Largely "connect away".
As long as the voltage is as required and the current supply is at last capable of meeting the maximum load current then it should work.
BUT
Be aware that some devices that appear to be just voltage sources may be more complex. This is unusual but not unknown.
eg If the output voltage measures a steady 4.1 to 4.2V open circuit and does not change much when loaded with eg a 20 ohm resistor then it is quite likely that it is a Lithium Ion charger. Using a simple higher current capable 4.2V source in its place would probably be utterly disastrous. I once needed to charge a LiIon battery when I did not have the correct charger while in China. I took the first available no-name power pack and measured its voltage, intending to make my own lashup charger. Eureka! 4.2V :-). As long as current max at 4.2V <= 1C for battery concerned it will probably charge OK.
Some few pieces of equipment need several voltages simultaneously and may care about applied and removed order or voltage ramp up rates etc. This is rare but happens. Examples may be printers which may have a 20 to 30 V supply (inkjet printhead?) and a 5V supply.
Some supplies have a complex current / voltage / time map that enables communication between supply and powered device re supply capability and device needs. An example is older Ericsson cell phones which accept 5V to 9V in but make decisions about subsequent charging which depend on how the charger responds to initial loading. Use of a much higher current supply than needed would probably be OK. Probably.
Some manufacturers make it hard in some or all cases to connect supplies other than their own. Unrecognized supplies will usually just not work rather than producing magic smoke. But caveat experimentator.
Apple is well known for this, Dell in some cases, Motorola in some cases, others perhaps.