What you are describing is called a "Self-powered function" in the USB 2.0 specification section 7.2.1.
The key points in the spec are...
No device shall supply (source) current on VBUS at its upstream facing
port at any time. From VBUS on its upstream facing port a device may
only draw (sink) current. They may not provide power to the pull-up
resistor on D+/D- unless VBUS is present (see Section 7.1.5). When
VBUS is removed, the device must remove power from the D+/D- pull-up
resistor within 10 seconds. On power-up, a device needs to ensure that
its upstream facing port is not driving the bus, so that the device is
able to receive the reset signaling. Devices must also ensure that the
maximum operating current drawn by a device is one unit load, until
configured.
The typical setup to switch between USB power or another power source is to tie the grounds together and use a pair of diodes (or ideal diodes) to switch between the power sources.
The USB specification wants the upstream facing interface on the device to not back-power the host in any way. This includes providing power to pullup resistors when VBUS is removed. For that reason, whatever chip is providing the USB interface (FTDI chip, MCU, etc) is usually powered directly from usb VBUS in order to meet the requirement of the USB specification. The rest of the system can be powered as you please.