This is a very big question to try to answer here, but I'll try to get you started.
Since you are only doing a one-port (reflection) measurement, you are quite correct that you don't need a full vector network analyzer. You could do this with either a vector voltmeter or an even simpler VSWR meter and a swept-frequency source (as your question suggests you intend).
The source should not require a DDS. A basic VCO should do, although of course the lower you can keep the harmonic distortion, the more accurate your measurements will be. VCOs covering more than an octave (as you need) are somewhat more difficult to find than narrowband ones covering frequency bands with important commercial uses. But you barely need more than an octave and you should be able to find one. Normally you would operate the VCO in a phase-locked loop, locked to a suitable reference frequency (say 10 MHz), to tune the the desired frequency in your 2 - 5 GHz band. This is the essence of an rf synthesizer.
On the receiver side, what you really need is a directional coupler to pick off the reflected signal coming back from your sample, and an rf detector to measure the reflection amplitude. A pair of matched couplers and detectors would let you measure the ratio between the transmitted signal and reflection for more accurate measurements.
Finally, the probe. I am not familiar with best practice for designing probes for liquid dielectric constant measurements, but I can give you some ideas about it. Details probably depend on physical factors like the viscosity of your sample, whether its a corrosive material, how easily residue from one sample could contaminate the next sample, the volume of the available samples, etc.
If you think the dielectric constant is basically constant over the 2-5 GHz range, then I'd suggest to build a resonant chamber and flood it with the sample, for example a roughly 1/2-wavelength segment of air-filled coaxial line, where you replace the air with your sample. If the dielectric constant changes, so will the resonant frequency of the chamber. Sweep the source frequency and find the reflection minimum or maximum to determine the sample dielectric constant. This type measurement could be done with just a VSWR meter as the sensor.
If the dielectric "constant" is changing in the 2-5 GHz range (due to material resonances, this is reasonably likely), then you will need a vector-voltmeter type detector. Again, though, I'd consider making a near-resonant chamber to hold the sample. With some math, the rf reflection from the chamber (amplitude and phase) will tell you the sample dielectric constant at each frequency.