The AMS1117 is a linear regulator, designed to take in the DC battery voltage (say 12V) and output a regulated 3.3V, meaning that the 3.3V does not change (much) if the 12V changes (a lot.) Therefore, it is OK if Vin has an AC component (dips and spikes from motors starting and stopping, this is likely) but Vout should have almost no AC (since it is regulated and should prevent this.)

The amount of this regulation-ability is called Line Regulation
and Load Regulation
in the above datasheet. This says that for normal operation, you should only see 0.025VAC or 25mVAC. Since failure to regulate a steady DC voltage creates an AC voltage "riding on top of" the DC voltage, this is why I asked to check AC volts. Seeing 1.5-3VAC on Vout seems to indicate that the AMS1117 is losing it's regulation-ability.
All linear regulators have limits as to how much current (I) and power (Vout x I) they can output, as well as how much power they themselves can dissipate ((Vin-Vout) x I). They also have limits on how much external input and output capacitance exist, and these values are not mentioned in this datasheet. This can be important, as too much capacitance can cause the regulator to lose regulation, essentially trying forever to correct Vout which is not changing fast enough due to too much capacitance. In other words, a regulator can become an oscillator if too much capacitance is added to it.

So the AMS1117 could be oscillating due to too much output capacitance. I'd suggest removing C10 and measure Vout AC volts again. If lower AC volts, then this was the problem.
Another thing to consider is the Dropout Voltage rating for this regulator. The datasheet says that up to 1.3V can be dropped across it before it starts to lose regulation. What this means is, when outputting 3.3VDC, if Vin drops below 3.3+1.3 = 4.6VDC, Vout will drop as well. If this is happening repeatedly (hundreds or thousands of times a second from motors turning on and off), it could stress the regulator, manifesting as extra heat.
The next step in troubleshooting such issues (oscillation, momentary dips in Vin, etc.) is to use an oscilloscope to determine exactly what these voltages are doing over time.