Impedance is a large factor in this situation, as is sensitivity.
In the case of a speaker wire, your driving it with a very low output impedance amplifier (<< 1ohm) and driving a load with a fairly low impedance, a typical speaker may be from 3-50 ohms across the audio spectrum.
The voltage noise you see on a cable is very dependent on these impedances as the noise is really a coupled current, the voltage that is measured on the cable is a product of this current and its impedance path to ground.
In the case of a speaker wire this is a very low impedance so it takes a lot of coupled noise current to induce meaningful voltage changes in the signal. Shielding a speaker wire is really pointless unless its in a very high noise environment, like if you set a coil on top of a building mains transformer.
Another way to say this is that speakers are really current mode devices, the driving force is large currents and EMI coupled currents are really small in normal applications for speaker wire.
Line level signals are more sensitive for multiple reasons, one being that the signal they carry is generally later amplified, another being that the input impedance and source impedance for single ended line level audio are much higher. Input impedance is generally around 10k for single ended line level inputs, it takes far less current to create a large voltage noise on that input than it does with a speaker wire. This is why almost all line level audio cabling is shielded, be it RCA type running on coax or XLR running on STP.
Another way to say this is that line level transfer is almost always voltage mode, meaning that the receiving end is looking for voltage levels and drawing minimal current, as a result, small currents such as coupled noise become larger voltages and have a much bigger impact on the signal.
This same concept applies to many other issues. For instance with high impedance op amp inputs care has to be taken to minimize any coupled noise to that input signal for the same reasons that line level audio signals are more sensitive. Often guard rings or similar approaches are used.