That's the way smartcards work. If you want a USB stick, you know where to find it.
Smartcards are designed for security. They're designed to be tamper resistant. They're designed to store secrets and to commit suicide rather than give up their secrets. You can't (with reasonable effort, barring implementation flaws) extract the data from it, or clone it.
Smartcards are (supposed to be) designed so that if you try to bypass the software interface and read their memory directly, you can't. The only external interfaces are power and serial lines. Everything else is inside a Faraday cage (to counter emissions measurements) which is protected by a layer of corrosive chemicals (acids, I think) so that if you try to make a hole in the protection layer, the chip (especially the memory) will be damaged beyond recovery.
The chip is engineered to draw constant power, to avoid leaking information through power consumption. It's also engineered to allow operations to take constant time. So smartcard chips don't have hardware optimizations such as instruction pipelines or power management to reduce heat dissipation. Smartcard chips often contain parts that don't actually serve any purpose, to make emissions and power draw more constant or harder to analyze, and to make reverse engineering difficult.
A number of things make smartcards slower than non-security-sensitive chips. Since smartcards don't have their own power supply, they need to write to flash memory fairly often — they can't afford to cache things in RAM. Otherwise the adversary could cut the power at advantageous (for him) times. Software has to use constant-time algorithms (to avoid timing attacks), not the fastest algorithm possible. Software has to be written with redundancies, in case the adversary shines a laser at the card to flip some bits in memory and put the software in a state that shouldn't have been reachable (yes, that's a real attack).
Security measures such as these have a cost. Furthermore, since attacks keep improving, the design of the chips has to keep up. Most types of chips become more and more powerful following Moore's law, but with smartcards, the engineering improvements are mostly absorbed by the need to counter new attacks.
There are cards that support 4k RSA keys. But they're uncommon. A few years ago, Java Card 3 was introduced to support almost all the features of the Java language; I think a card capable of doing that would be capable of signing with 4k RSA. But they haven't really taken, partly because engineering improvements are taken up by increasing security protections, and partly because a lot of engineering improvements have gone into contactless cards, which require less power draw and faster response times.
But there isn't much of a market for 4k RSA. As smartcards were reaching that point, ECC became widely available. You can get the same security level as 4k RSA with much smaller ECC keys. The only drawback of switching from RSA to ECC is support for the algorithm throughout your infrastructure, but embedded devices such as smartcards are the hardest part: pretty much anything server-side supports ECC these days.
If you really need something more powerful, there exist USB dongles with “real” computing power, that can store cryptographic keys and perform cryptographic operations with a PKCS#11 interface. These dongles don't have nearly the same physical tamper resistance as a typical PKCS#11. It's a compromise between security and performance.
Further reading:
