Probably the batteries are all connected in series (- to +). Check that.
If they are (which is almost certain) then you want to wire the + side of your power supply to where the + side of the +est battery usually goes: the one that's not connected to another battery's - side.
and, likewise, the - side of the power supply to where the - side of the -est battery usually goes: the one that's not connected to another battery's + side.
On an AA/AAA/C/D battery the - side is the flat side and the + side is the one that sticks out.
To try it out, you can probably use alligator clips (aka crocodile clips).
If you want to make a more permanent connection, there's probably no great option that doesn't involve cutting and joining wires; one mediocre idea is to insert only the 2 batteries at the ends, but jam the power supply wires between the batteries and the metal bits they touch. (Those 2 batteries will only be connected on one end, so they won't affect the circuit. You'd just use them to hold the wires in place)
There are a few ways to properly connect wires without soldering; such as screw terminal blocks (the kind with two rows of screws) or Wago connectors for example (not sure if they make Wago connectors small enough for the tiny wires that are probably in the remote. They're designed for house wiring). In this case you'd cut the wires that go from the remote control's circuit board to the battery box (cut near to the battery box), strip the insulation from the ends so the copper sticks out, insert that into your favourite connector of choice and clamp it down, then do the same with the power supply wires.