I suppose resonant stubs are an extension of your gimmick capacitor example, so I won't go further than merely mentioning the example.
Leaving the shield open on one side, negates its effect as far as magnetic shielding goes.
The shield carries two currents at high frequency: inside (signal) and outside (shield or CM).
At low frequencies (where the skin depth in the shield is comparable to its thickness), these currents bleed together over distance, and you get the expected DC result: the shield is just a resistor.
At high frequencies, the currents stay separated (shielded), by some degree depending on frequency and length. Like 10s of dB.
So we restrict analysis to high frequencies, where things are interesting.
When we have a signal going into coax, the return current flows inside the shield.
If the far end of the coax is grounded, then the return current flows onward to the destination. Signal quality is good, everyone is happy. Shield currents may be annoying, but to the extent they don't couple to the signal (so: depends on attenuation vs. frequency), they can be ignored. Which we know is worse at low frequencies, so this can introduce mains-frequency ground loop for example. The shield current can also act as an antenna (the inductive link between two boxes, at least one of which is isolated from ground plane / earth), or a end-shorted dipole (slot antenna) when the ends are joined by ground plane. Which can exacerbate EMC problems by providing a gain peak at harmonic frequencies, making the shield less effective there. (This is where ferrite beads shine the most: they introduce resistance to the CM mode, dampening these peaks.)
If the far end of the coax is open, then the return current has to flow back over the shield, up to the source, and back again along whatever ground plane lies between the two ends. The loop is completely unshielded to incident magnetic fields, signal quality is trash because the shield acts as a stub length in series with the signal, and any ground-loop voltage that might be present between connections, is superimposed directly on the signal (at all frequencies, not just low frequencies).
By most measures, the latter is strictly worse.
The one advantage that you get, is electrostatic shielding, which might be of value at frequencies below the stub resonance length. And, electric fields are the dominant concern at high impedances. So, all together: I think the advice arises from the bad old days of vacuum tube equipment, especially audio amplifiers where the bandwidth isn't enough to worry about stub lengths*, and stub lengths are small (within chassis).
*But the tubes might not know that. Typical receiving tubes have significant gain over 100MHz (if still rather little gain-bandwidth). "Grid stopper" resistors are a common sight, to dampen possible tuned-grid oscillation modes in these circuits.
So, it is best to ignore such advice, and ground responsibly.
A reminder that coax can be used differentially. The shield-to-circuit-GND connection can be "lifted" on some impedance (also serving to dampen shield resonances, as a ferrite bead does), as long as the signal receiver has adequate CMR and CMRR (range, rejection ratio) to handle expected noise levels. This completely solves GND loop problems up to the CMR. And RFI (and CMRR) can still be maintained by bypassing shield to circuit GND; the crossover frequency should pull in somewhere before CMRR gets too poor (since op-amps are limited in CMRR vs. F).