Can you use a battery with the same chemistry, same rated voltage, but highter rated capacity?
Yes.
Higher capacity (measured in mAh) means that for the same use, the battery will last longer. It should not have any other effect (assuming the circuit does not rely on the battery's internal resistance).
Can you use a battery with a higher rated voltage?
In general, it is not a good idea. However, circuits might (or might not) have tolerances. Without knowing the circuit, it is probably a bad idea to try.
Also, the batteries themselves do not have a voltage always strictly equal to the rated voltage. The rated voltage is often a mean voltage or an approximate voltage the battery will keep most of the time. Batteries usually have a higher voltage when fully charged and a lower voltage when discharged enough.
For example, for an NiMH cell discharged with a current C/10 (where C is the rated capacity in mAH), the discharge curve is the following and the voltage starts at about 1.5 V:

The bottom line is: the real voltage of your battery pack when fully charged is 8 x 1.5 V = 12 V. So you should be fine applying maximum 12 V to the device. BUT...
Can you use an 11.1 V 2200 mAh Li-Po battery instead of a 9.6 V 1500 mAh NiMH battery?
Maybe, maybe not. It is best not to try.
As discussed above, the rated battery voltage is not the real voltage applied. The typical Li-Po battery has a rated voltage of 3.7 V and the following discharge curve:

The voltage goes from around 4.2 V or 4.3 V down to 3 V or 2.7 V (depending on the protection circuit).
It means that your 11.1 V battery (composed of 3 cells in series) has a real voltage of around 3 x 4.3 V = 12.9 V when fully charged, which is higher that the maximum voltage of your NiMH battery.
Without knowing the circuit and the maximum input voltage it can tolerate, it is better not to try it.