As far as I understand, return current flows the "negative" channel for a differential signal connection. Only if it is not balanced well, there might be unwanted return current in reference GND or power plane.
But some say the return path for both positive and negative are always the reference plane, which get me confused...
It depends on the physical geometry of the transmission line.
I can route two tracks on a PCB, not too close together but matched in length, and use them as a differential pair. In this case, each of the individual lines will have its own return current on a nearby ground plane.
Or I can make a closely coupled pair of tracks on the PCB, with the line-to-line spacing less than the line-to-ground spacing. In this case most of the return current will be in the complementary line, but there will likely be some return current in the ground plane near each line.
Or I can make a purely differential line, like unshielded twisted pair, and pretty much the only return current for one line will be in the other and vice versa.
I'll take a slightly different approach to the other answers and use a differential transformer as an example. These are common on balanced microphone circuits.
simulate this circuit – Schematic created using CircuitLab
Figure 1. A balanced microphone circuit showing differential (green) and common mode (red) signals.
Note that the differential signals are in-phase on the transformer primary and so generate an output on the transformer secondary. Meanwhile the common-mode signals are out of phase and cancel out giving no output on the secondary. The transformer rejects the common mode signal.
As far as I understand, return current flows the "negative" channel for a differential signal connection.
This depends purely on which you take as the reference. If you start with the negative or inverting input then the "return current" is on the "positive" or non-inverting input.
Only if it is not balanced well, there might be unwanted return current in reference GND or power plane.
That would be true. In Figure 1 there is little possibility of this as the microphone capsule is floating with respect to ground. In practice there will be some capacitance to ground and in other applications it may be using a differential amplifier output with a direct ground reference. If, say, there are an uneven number of turns on the two halves of the transformer primary then the common mode rejection will be reduced and the CMRR will decrease.
But some say the return path for both positive and negative are always the reference plane, which get me confused...
If you think about this a bit, what happens if two out of phase signals are transmitted through the same path? They cancel out. Only the difference between the two signals will go through the ground path.
A properly balanced differential line has equal amounts of current flowing in both directions. A properly balanced differential transmission line does not need a reference plane (there are many transmission lines that don't have reference planes,anything that uses UTP or STP being some examples).
Of course if the lines are not balanced, current will flow along the reference plane (or sheild) but this will be minimal, because an unbalanced differential pair not longer functions like a differential pair.
The reference plane does change the characteristic impedance of a differential pair so reference planes must be taken into account when routing.
Any imbalance in parasitic capacitance will limit the CMRR for AC signals.
The return currents will flow thru whatever bits of metal form the 2nd plate of the parasitic capacitor.
Its your job to identify all the parasitic Efields, and bring those fields under your control.
