The layout of your board could use some work. First, if you're not familiar with bypassing, take a look at this question:
Decoupling caps, PCB layout
After that, lets look at your bypass capacitor placement. I picked two random bypass capacitors that were labeled on the board and drew the current loops:
Those loops are enormous, which means the capacitor won't be very effective at suppressing noise and at the same time you may have high frequency signals induced into those big loops. You want to make them as small as possible, if you do maybe that will improve your noise pick up.
The datasheet has a nice cartoon style layout on page 27, which you can probably use more or less as is. But to help, I took the foot print and placed capacitors as I would put them on this board:
The designers of this chip were very kind, leaving exactly enough space for each bypass capacitor right where it would most logically fit. As you can see, I've started with the capacitors and made sure they're in tight before routing any signals, connecting planes, etc. I've used 0402, but if you're hand assembling this and don't have a good magnifying lens, 0603 would probably fit more or less the same.
You can put the electrolytics and resistors further out. They're not as critical. I'm also not sure about that ground plane cut in your board or what it is doing. Is there another ground plane you're not showing? If not, you're routing over the cut plane, which is a bad idea, especially with the USB lines. I think you could get rid of it.
