Summary:
You can probably double brightness by using a few more port pins.
Use of 4 transistors as column drivers will probably make a significant difference.
Use of a low side segment buffer IC plus high side transistors will give you all the drive you want.
Ports are presently being used "illegally" but damage is not occurring due to low drive of processor relative to LEd current ratings. Once proper drive is available series resistors will be needed to prevent damage.
Here is the data sheet for the 74HC244 buffer mentioned by arminb.
It provides 8 x inverting buffers.
The non inverting version is the 74HC240 - datasheet here .
Both these are OK in principal but notionally only provide 8 mA drive per circuit.
You can get various higher current versions such as these SN74LVC244a in stock at Digikey at 26 cents in ones - a bargain! :-).
Driving LEDS without resistors from a processor is bad practice whether it "works" or not. It loads the IC into "illegal" areas if it doesn't work and risks damaging the display if it does work (and still loads the IC illegally).
Your display currently drives 4 segments at a time x 4 columns.
IF you have 4 more port pins available you can (probably) double the brightness of your display by driving 8 segments at a time x 2 columns x 2 port pins per column. This allows you to multiplex your display at twice the rate so twie as bright IF current is maintained - which is why I doubled up the column drives.
At present you are driving a column of 4 LEDs via a portA pin but sinking it via 4 x port B pins. If you have 4 segments on then the brightness is limited by the ability of the single Port A pin to supply 4 x LEDs. If there is only one segment per column on then PortyA supplies one LED and port B sinks 1 LED BUT sink capacity is usually greater than source capacity on many processors, so PORT A drive will still probably be the limit.
SO by providing 4 x port A high side drivers you should increase brightness. These can be as simple as 4 x transistor emitter followers.
4 x NPN transistor (BC337 or similar)
Collectors to V+
Bases to Port A pins
Emitters to LED Anodes.
Adding a series resistor per LED would be "a good idea" [tm].
Knowing what your port pin source and sink capabilities are is essential if you wish to design the result.
Telling us what your processor is and providing links is always an extremely valuable thing to do.
If using transistors as above does not work well enough you can add per segment drive as well by using an octal buffer, as above, or by using segment drive transistors.
For a low side driver an eg ULN2803 octal driver would do an extremely good job. Using one of these plus high side emitter followers without a series resistor would produce a very very very bright display, but only for one cycle :-(. Yu can get other members of the ULN280x family which have different input drive requirements but the '2803 is generally the most useful.