I have this board with three DIP ICs already mounted in their sockets.
The decoupling schematic is this:
So we have a sum of 8 ceramic caps and 2 electrolytic ones. So how do i solder them? Should i try soldering the tip of the capacitors tot he IC pin?
Generally when adding rework, bottom side is best for through hole boards. For SMT it can get more difficult and you end up having to use both sides.
You need to insulate all bare wires when doing rework, either by using insulated wiring or magnet wire, which can be tricky to clean. When adding components, and the legs are any length, insulate them by removing some insulation from rework wire and feeding the legs of the component through the insulation. The point of all this is you do not want something to move and short to other pins or traces.
All components added should be as flat to the board as possible, and wiring should be tidy. Finally, where wires are any length, tack them down with a suitable glue so they do not flop around or catch on anything. A touch of a cyanoacrylate (superglue) to tack things down is fine, but a more robust polymer based glue is more durable.
Note the image above, is NOT a great example of well done rework. The cap is flying in the air, held on only by the dabs of solder. For something like that you want to glue down the cap to make a solid mechanical bond.
ADDITION
However, as a comment on your actual problem. Having a high frequency digital board sandwidged onto the back of an audio board without shielding is a really bad idea. Decouplers on audio frequency analog chips do not do much, and is just an attempt to squash a problem rather than fixing it at the source.
Investigate a shield layer between the boards and make sure your grounding and power system is properly arranged.
Like Trevor pointed out, it should be no problem to mount the caps on the bottom side. If some of the pins are next to each other, you could consider to use 0805 SMD ceramic caps to keep trace/wire length at a minimum.
Furthermore, it would be best to remove the ICs from the sockets before soldering to prevent them from heating up too much.
For best bypassing, solder 0.1uF right across the top of the DIPs. Providing the needed charge surges right at the pins will best supply any high frequency charge demands of the opamps. Not pretty, but best, particularly if you were using digital ICs with 5nanosecond/50miiliAmp current demands.
From a quick analysis of the audio spectrum and front end design, I would have to say your problem is due to lack of ground plane in proximity to the Arduino noise to "shield" (misnomer) from EM coupling. (stray L and C ).
The input impedance is too high at 10M and unbalanced. You could try 50k, but even with battery operation, you are getting dirty white noise from all the logic noise radiated from the uC card. Even a shorted input is noisy as below.
This means you will need to create a shunt path for noise and divert it to analog ground in between the two cards or use a better Amp with a ground plane. in addition to this the impedance of 0V may be floating making everything high CM impedance again aggravating the poor unbalanced input design.
This "Lo-Fi" audio board may not be improved by adding caps. Your V- battery test may have proved this.
I suggest scrapping this design and separating the audio card from the Arduino with a reputable Guitar-preamp box with low noise balanced front end and possibly a big CM choke rated for audio at the guitar impedance.
You can try to make the noise worse or better with your fingers to verify the sensitive circuit nodes while holding analog ground and then a series Cap to Earth ground ( to block 60Hz ground faults if they occur but reduce RF white noise aliasing effects on ADC.)
simulate this circuit – Schematic created using CircuitLab
