Jonas,
Swing generalises your underlying architecture to provide you with a platform neutral user experience. The only heavyweight component (provided by the OS) is the JFrame container and the rest is pretty much handled by the Swing tookit. AWT on the other hard, asks the OS to draw all of it's UI components which means it's faster in a lot of ways as your using the native UI components specific to the OS. SWT tries to achieve a middle ground, for various standard components like buttons and labels (which are available on most OS), it lets the OS deal with those and for other specialised components, SWT will handle the creation for you.
That been said, I can outline the drawbacks.
(1) Since the toolkit creates and renders the components for you rather than asking the OS, you don't get to take advantage of the speed of the built in components provided by the OS.
(2) The UI isn't particuarly attrictive as it looks alien to most OS platforms regards of what look and feel you use.
(3) Some of the layout managers i.e GridBadLayout etc could be better simplified. I've lost count of the number of projects I've worked on where people have wrapped the GridBagLayout in some bespoke code to get a simpler way to use it.
I'd advise you to write a simple app in AWT, Swing and SWT and compare the development approaches an the final product between them all, then review the various comments made from other developers and decide which one works best. I've worked with Swing for many years and I used dislike SWT, but I'd come to realise that Swing is a lot more complicated than it needs to be when compared to other frameworks out there.