Voltage is a measure of electromotive force (emf) or potential energy per unit charge. It's not really a force, not is it energy - but you can think of it as the 'force' or 'pressure' of the electricity.
What is it exactly? Nobody knows. What we do know is that opposite charges attract and like charges repel. Inside an atom the electrons (which each have 1 electron unit of negative charge) are attracted to the protons (which have 1 unit of positive charge). Chemical reactions in the battery force electrons towards the negative terminal. The electrons are still attracted to the protons, but they can't get back to them unless an external path is provided to the positive terminal.
The closer the electrons are to the protons in the atom, the more energy is required to remove them. This is the potential energy per unit charge. Current is the number of electrons that flow past a particular point in an electric circuit. Power is voltage multiplied by current, which is the rate of energy transfer from one place (or form) to another.
Voltage by itself is not power, but words are often used ambiguously. 120V AC is called mains 'power' because it provides power when current is drawn from it. AC is short for Alternating Current, even though no current has to flow for an AC voltage to be present.
The power you can get out of a battery is determined by both its voltage and the maximum current that it can deliver. But why only mark the battery with its voltage? Higher voltage can force more current through a circuit, and most devices are designed to work at a particular voltage. A battery with higher capacity (Ampere-hours or Watt-hours) can run a device for longer, but higher voltage could blow it up.