If the design is new then testing components can make sense.
If it's a faulty PCB / unit from a proven / working device then I test with the components in circuit. Study of the circuit diagram tells you how the circuit should behave.
Unless the device has been abused in some way the components are rarely faulty if your on the end of a production line. Visual inspection, as the others have said, is your friend here.
Checks I make to start with are:
Check the rails
Any voltage references.(Zenner diodes and IC's)
Is the circuit receiving what is expected?
If op-amp based circuits are used I check the virtual earth's of the inverting circuits. They should be close to 0V or the mid rail if it's a mid rail based circuit. Any that are not I check the outputs are against the rails (If the virtual earth is not 0V / mid rail then the output should be against a rail unless limited by something else). Then I check the input is where it's expected.
Then it's feed in changing signals and see what in the signal path is not working.
The most common faults are...
Bad solder joints.
Solder splash.
Connectors not making contact. (Include connects wired wrong here and not just broken wires)
I once had some jumper pins that were open circuit because the flux had got on the pins and was stopping the jumper from making a physical connection. Both the Test Engineer and I were absolutely stunned. It took a coffee break and ruling out other faults to realise this. Just because it's never happened before doesn't mean it can't happen.