If we have a self-oscillator with multiple resonant frequencies, what are the criteria for the preferred frequency of oscillation?
For example, a push-pull oscillator with parallel-compensated two coupled inductors will have two resonance frequencies at the high coupling i.e \$f_{\rm{resonance}}\approx f_0/\sqrt{1\pm k}\$.
How do we determine the frequency of oscillation in such cases?
I agree with @TimWescott answer, however he missed one aspect of it which I expanded upon in this answer to a related question.
All linear oscillators are special cases of dynamical systems in which some constraints (namely Barkhausen Criteria) have been put in place to simplify the differential equations, analysis, and design. One assumption that is sometimes taught alongside those criteria, is that for every other possible oscillation frequency the loop gain must be strictly less than one. But in general these are just a set of behaviors that lie in a much wider parameter space in which chaos reigns.
Any dynamical system that is put together as an oscillator and can have more than one oscillation frequency (thus third-order or more) would have a parameter space that would follow a well-known path to chaos which includes period-doubling and complex oscillation patterns (e.g., squeging). As one of the defining characteristics of chaos is the exponential sensitivity to initial conditions and parameters, any minor parameter variation (e.g., temperature) or noise will alter the behavior making it completely unpredictable.
First, oscillator design is a book-length subject. In fact, general oscillator design is a book-length subject, and then you can branch off and write a book just about high-precision crystal oscillators, another one about LC oscillators, yet another one about power oscillators, and all the while the folks at NIST have an entire library of journal articles on precision time keeping. So this answer is necessarily short.
The short, practical answer to your question about what frequency an oscillator will operate when it has multiple possible modes of oscillation is -- whichever one you don't want. Unless having it stay steady at one or the other would be OK, in which case it'll shift from one to the other, unpredictably.
It is even possible to an oscillator oscillate at two widely different frequencies simultaneously. It's called squegging. This is generally easier to do when you're trying to optimize the oscillator for some parameter or another, rather than when you actually want it to squeg.
The way that you determine what frequency an oscillator will oscillate at is to use Bode-plot analysis (or root-locus, if you're crazy). Model the oscillator as an amplifier by breaking the signal chain at some point and finding the gain and phase as a function of frequency from the input to the output. For most normal circuits, oscillation will happen if there's a point with zero phase shift and greater than unity gain. Again for most circuits, the action of oscillating will change the operating point of the oscillator to bring the gain at zero phase shift to unity.
If you do this analysis and you find two points of zero phase shift and greater-than-unity gain, then it is going to be very difficult to predict which frequency the thing will oscillate at. Usually, the oscillator will start oscillating at whichever zero phase shift point has the most gain and will continue there. However, if the higher-gain frequency loses gain faster than the lower-gain frequency as oscillations build up, the oscillator may switch modes. Sometimes the oscillator will start oscillating at the lower gain frequency. Sometimes each turn-on event of the oscillator will be a roll of the dice, and you'll end up with whatever frequency it was in the mood to deliver that time.
For these reasons (unless you're purposely designing an oscillator to squeg), you really want to avoid this situation.
