What backs up the claim that C++ can be faster than a JVM or CLR with JIT?

Question

A reoccurring theme on SE I've noticed in many questions is the ongoing argument that C++ is faster and/or more efficient than higher level languages like Java. The counter-argument is that modern JVM or CLR can be just as efficient thanks to JIT and so on for a growing number of tasks and that C++ is only ever more efficient if you know what you're doing and why doing things a certain way will merit performance increases. That's obvious and makes perfect sense.

I'd like to know a basic explanation (if there is such a thing...) as to why and how certain tasks are faster in C++ than the JVM or CLR? Is it simply because C++ is compiled into machine code whereas the JVM or CLR still have the processing overhead of JIT compilation at run time?

When I try to research the topic, all I find is the same arguments I've outlined above without any detailed information as to understanding exactly how C++ can be utilized for high-performance computing.

Answer 1

It's all about the memory (not the JIT). The JIT 'advantage over C' is mostly limited to optimizing out virtual or non-virtual calls through inlining, something that the CPU BTB is already working hard to do.

In modern machines, accessing RAM is really slow (compared to anything the CPU does), which means applications that use the caches as much as possible (which is easier when less memory is used) can be up to a hundred times faster than those that don't. And there are many ways in which Java uses more memory than C++ and makes it harder to write applications that fully exploit the cache:

• There is a memory overhead of at least 8 bytes for each object, and the use of objects instead of primitives is required or preferred in many places (namely the standard collections).
• Strings consist of two objects and have an overhead of 38 bytes
• UTF-16 is used internally, which means that each ASCII character requires two bytes instead of one (the Oracle JVM recently introduced an optimizaion to avoid this for pure ASCII strings).
• There is no aggregate reference type (i.e. structs), and in turn, there are no arrays of aggregate reference types. A Java object, or array of Java objects, has very poor L1/L2 cache locality compared to C-structs and arrays.
• Java generics use type-erasure, which has poor cache locality compared to type-instantiation.
• Object allocation is opaque and has to be done separately for each object, so it is impossible for an application to deliberately lay out its data in a cache-friendly way and still treat it as structured data.

Some other memory- but not cache-related factors:

• There is no stack allocation, so all non-primitive data you work with has to be on the heap and go through garbage collection (some recent JITs do stack allocation behind the scenes in certain cases).
• Because there are no aggregate reference types, there is no stack passing of aggregate reference types. (Think efficient passing of Vector arguments)
• Garbage collection can hurt L1/L2 cache contents, and GC stop-the-world pauses hurt interactivity.
• Converting between data types always requires copying; you cannot take a pointer to a bunch of bytes you got from a socket and interpret them as a float.

Some of these things are tradeoffs (not having to do manual memory management is worth giving up a lot of performance for most people), some are probably the result of trying to keep Java simple, and some are design mistakes (though possibly only in hindsight, namely UTF-16 was a fixed length encoding when Java was created, which makes the decision to choose it a lot more understandable).

It's worth noting that many of these tradeoffs are very different for Java/JVM than they are for C#/CIL. The .NET CIL has reference-type structs, stack allocation/passing, packed arrays of structs, and type-instantiated generics.

Answer 2

Is it simply because C++ is compiled into assembly/machine code whereas Java/C# still have the processing overhead of JIT compilation at runtime?

Partially, but in general, assuming an absolutely fantastic state-of-the-art JIT compiler, proper C++ code still tends to perform better than Java code for TWO main reasons:

1) C++ templates provide better facilities for writing code that is both generic AND efficient. Templates provide the C++ programmer with a very useful abstraction that has ZERO runtime overhead. (Templates are basically compile-time duck-typing.) In contrast, the best you get with Java generics is basically virtual functions. Virtual functions always have a runtime overhead, and generally can't be inlined.

In general, most languages, including Java, C# and even C, make you choose between efficiency and generality/abstraction. C++ templates give you both (at the cost of longer compile times.)

2) The fact that the C++ standard doesn't have much to say about the binary layout of a compiled C++ program gives C++ compilers much more leeway than a Java compiler, allowing for better optimizations (at the cost of more difficulty in debugging sometimes.) In fact, the very nature of the Java language specification enforces a performance penalty in certain areas. For example, you can't have a contiguous array of Objects in Java. You can only have a contiguous array of Object pointers (references), meaning that iterating over an array in Java always incurs the cost of indirection. The value semantics of C++ however, enable contiguous arrays. Another difference is the fact that C++ allows objects to be allocated on the stack, whereas Java does not, meaning that, in practice, since most C++ programs tend to allocate objects on the stack, the cost of allocation is often close to zero.

One area where C++ might lag behind Java is any situation where many small objects need to be allocated on the heap. In this case, Java's garbage collection system will probably result in better performance than standard new and delete in C++ because the Java GC enables bulk deallocation. But again, a C++ programmer can compensate for this by using a memory pool or slab allocator, whereas a Java programmer has no recourse when faced with a memory-allocation pattern that the Java runtime is not optimized for.

Also, see this excellent answer for more information about this topic.

Answer 3

What the other answers (6 so far) seem to have forgotten to mention, but what I consider very important for answering this, is one of C++' very basic design philosophies, which was formulated and employed by Stroustrup from day #1:

You don't pay for what you don't use.

There are some other important underlying design principles that greatly shaped C++ (like that you shouldn't be forced into a specific paradigm), but You don't pay for what you don't use is right there among the most important ones.

---

In his book The Design and Evolution of C++ (usually referred to as [D&E]), Stroustrup describes what need he had that made him come up with C++ in the first place. In my own words: For his PhD thesis (something to do with network simulations, IIRC), he implemented a system in SIMULA, which he liked a lot, because the language was very good in allowing him to express his thoughts directly in code. However, the resulting program ran way too slow, and in order to get a degree, he rewrote the thing in BCPL, a predecessor of C. Writing the code in BCPL he describes as a pain, but the resulting program was fast enough to deliver results, which allowed him to finish his PhD.

After that, he wanted a language that would allow to translate real-world problems into code as directly as possible, but also allow the code to be very efficient.
In pursue of that, he created what would later become C++.

---

So the goal quoted above isn't merely one of several fundamental underlying design principle, it's very close to the raison d'etre for C++. And it can be found just about everywhere in the language: Functions are only virtual when you want them to (because calling virtual functions comes with a slight overhead) PODs are only initialized automatically when you explicitly request this, exceptions only cost you performance when you actually throw them (whereas it was an explicit design goal to allow the setup/cleanup of stackframes to be very cheap), no GC running whenever it feels like it, etc.

C++ explicitly chose to not to give you some conveniences ("do I have to make this method virtual here?") in exchange for performance ("no, I don't, and now the compiler can inline it and optimize the heck out of the whole thing!"), and, not surprisingly, this indeed resulted in performance gains compared to languages that are more convenient.

Answer 4

Do you know the Google research paper about that topic?

From the conclusion:

We find that in regards to performance, C++ wins out by a large margin. However, it also required the most extensive tuning efforts, many of which were done at a level of sophistication that would not be available to the average programmer.

This is at least a partially explanation, in the sense of "because real world C++ compilers produce faster code than Java compilers by empirical measures".

Answer 5

This is not a duplicate of your questions, but the accepted answer answers most of your question: A modern review of Java

To sum-up:

Fundamentally, the semantics of Java dictate that it is a slower language than C++.

So, depending on with which other language you compare C++, you might get or not the same answer.

In C++ you have:

• Capacity to do smart inlining,
• generic code generation that have strong locality (templates)
• as small and compact as possible data
• opportunities to avoid indirections
• predictable memory behaviour
• compiler optimizations possible only because of the usage of high level abstractions (templates)

These are the features or side-effects of the language definition that makes it theorically more efficient on memory and speed than any language that:

• use indirection massively ("everything is a managed reference/pointer" languages): indirection mean that the CPU have to jump in memory to get the necessary data, increasing the CPU cache failures, which means slowing down the processing - C uses also indirections a lot even if it can have small data as C++;
• generate big size objects which members are accessed indirectly: this is a consequence of having references by default, members are pointers so when you get a member you might not get data close to the core of the parent object, again triggering cache misses.
• use a garbarge collector: it just makes predictability of performance impossible (by design).

C++ agressive inlining of the compiler reduce or eliminate a lot of indirections. The capacity to generate small set of compact data makes it cache friendly if you don't spread these data all over the memory instead of packed together (both are possible, C++ just let you choose). RAII makes C++ memory behaviour predictable, eliminating a lot of problems in case of real-time or semi-real-time simulations, which require high speed. Locality problems, in general can be summed-up by this: the smaller the program/data, the faster the execution. C++ provide diverse ways to make sure your data are where you want it to be (in a pool, an array, or whatever) and that it is compact.

Obviously, there are other languages that can do the same, but they are just less popular because they don't provide as much abstraction tools as C++, so they are less useful in a lot of cases.

Answer 6

It is mainly about memory (as Michael Borgwardt said) with a bit of JIT inefficiency added in.

One thing not mentioned is the cache - to use the cache fully, you need your data to be laid out contiguously (ie all together). Now with a GC system, memory is allocated on the GC heap, which is quick, but as memory gets used the GC will kick in regularly and remove blocks that are no longer used and then compact the remaining together. Now apart from the obvious slowness of moving those used blocks together, this means that data you're using may not be stuck together. If you have an array of 1000 elements, unless you allocated them all at once (and then updated their contents rather than deleting and creating new ones- that will be created at the end of the heap) these will become scattered all over the heap, thus requiring several memory hits to read them all into the CPU cache. A C/C++ app will most likely allocate the memory for these elements and then you update the blocks with the data. (ok, there are data structures like a list that behave more like the GC memory allocations, but people know these are slower than vectors).

You can see this in operation simply by replacing any StringBuilder objects with String... Stringbuilders work by pre-allocating memory and filling it, and is a known performance trick for java/.NET systems.

Don't forget that the 'delete old and allocate new copies' paradigm is very heavily used in Java/C#, simply because people are told that memory allocations are really fast due to the GC, and so the scattered memory model gets used everywhere (except for stringbuilders, of course) so all your libraries tend to be wasteful of memory and use a lot of it, none of which gets the benefit of contiguity. Blame the hype around GC for this - they told you memory was free, lol.

The GC itself is obviously another perf hit - when it runs, it not only has to sweep through the heap, but it also has to free all unused blocks, and then it has to run any finalisers (though this used to be done separately the next time round with the app halted)(I don't know if it still is such a perf hit, but all docs I read say only use finalisers if really necessary) and then it has to move those blocks into position so the heap is compacted, and update the reference to the new location of the block. As you can see, its a lot of work!

Perf hits for C++ memory comes down to memory allocations - when you need a new block, you have to walk the heap looking for the next free space that is big enough, with a heavily fragmented heap, this is not nearly as fast as a GC's 'just allocate another block on the end' but I think it is not as slow as all the work the GC compaction does, and can be mitigated by using multiple fixed-sized block heaps (otherwise known as memory pools).

There's more... like loading assemblies out of the GAC that requires security checking, probe paths (turn on sxstrace and just look at what it's getting up to!) and general other overengineering that seems to be much more popular with java/.net than C/C++.

Answer 7

"Is it simply because C++ is compiled into assembly/machine code whereas Java/C# still have the processing overhead of JIT compilation at runtime?" Basically, yes!

Quick note though, Java has more overheads than just JIT compilation. For example, it does much more checking for you (which is how it does things like ArrayIndexOutOfBoundsExceptions and NullPointerExceptions). The garbage collector is another significant overhead.

There's a pretty detailed comparison here.

Answer 8

Bear in mind that the following is only comparing the difference between native and JIT compilation, and doesn't cover the specifics of any particular language or frameworks. There might be legitimate reasons to choose a particular platform beyond this.

When we claim that native code is quicker, we are talking about the typical use case of natively compiled code versus JIT compiled code, where the typical use of a JIT compiled application is to be run by the user, with immediate results (eg, no waiting on compiler first). In that case, I don't think anyone can claim with a straight face, that JIT compiled code can match or beat native code.

Let's assume we have a program written in some language X, and we can compile it with a native compiler, and again with a JIT compiler. Each work flow has the same stages involved, which can be generalized as (Code -> Intermediate Representation -> Machine Code -> Execution). The big difference between to two is which stages are seen by the user and which are seen by the programmer. With native compilation, the programmer sees all but the execution stage, but with the JIT solution, the compilation to machine code is seen by the user, in addition to execution.

The claim that A is faster than B is referring to the time taken for the program to run, as seen by the user. If we assume that both pieces of code perform identically in the Execution stage, we must assume that the JIT work flow is slower to the user, as he must also see the time T of the compilation to machine code, where T > 0. So, for any possibility of the JIT work flow to perform the same as the native work flow, to the user, we must decrease the time of Execution of the code, such that Execution + Compilation to machine code, are lower than only the Execution stage of the native work flow. This means we must optimize the code better in the JIT compilation than in the native compilation.

This however, is rather infeasible, since to perform the necessary optimizations to speed up Execution, we must spend more time in the compiling to machine code stage, and thus, any time we save as a result of the optimized code is actually lost, as we add it to the compilation. In other words, the "slowness" of a JIT based solution is not merely because of added time for the JIT compilation, but the code produced by that compilation performs slower than a native solution.

I'll use an example: Register allocation. Since memory access is some thousands of times slower than register access, we ideally want to use registers wherever possible and have as few memory accesses as we can, but we have a limited number of registers, and we must spill state into memory when we need a register. If we use a register allocation algorithm which takes 200ms to compute, and as a result we save 2ms of execution time - we're not making the best use of time for a JIT compiler. Solutions like Chaitin's algorithm, which can produce highly optimized code are unsuitable.

The role of the JIT compiler is to strike the best balance between compilation time and quality of produced code, however, with a large bias on fast compilation time, since you don't want to leave the user waiting. The performance of the code being executed is slower in the JIT case, as the native compiler is not bound (much) by time in optimising code, so is free to use the best algorithms. The possibility that overall compilation+execution for a JIT compiler can beat only execution time for natively compiled code is effectively 0.

But our VMs are not merely limited to JIT compilation. They employ Ahead-of-time compilation techniques, caching, hot swapping, and adaptive optimizations. So let's modify our claim that the performance is what the user sees, and limit it to the time taken for execution of the program (assume we've AOT compiled). We can effectively make the executing code equivalent to the native compiler (or perhaps better?). A big claim for VMs is that they may be able to produce better quality code then a native compiler, because it has access to more information - that of the running process, such as how often a certain function may be executed. The VM can then apply adaptive optimizations to the most essential code via hot swapping.

There's a problem with this argument though - it assumes that profile-guided optimization and the like is something unique to VMs, which is not true. We can apply it to native compilation too - by compiling our application with profiling enabled, recording the information, and then recompile the application with that profile. It's probably also worth pointing out that code hot swapping is not something that only a JIT compiler can do either, we can do it for native code - although the JIT based solutions for doing this are more readily available, and much easier on the developer. So the big question is: Can a VM offer us some information that native compilation cannot, which can boost the performance of our code?

I can't see it myself. We can apply most of the techniques of a typical VM to native code too - although the process is more involved. Similarly, we can apply any optimizations of a native compiler back to a VM which uses AOT compilation or adaptive optimizations. The reality is that the difference between natively run code, and that run in a VM is not as big as we've been made to believe. They ultimately lead to the same result, but they take a different approach to get there. The VM uses an iterative approach to produce optimized code, where the native compiler expects it from the start (and can be improved with an iterative approach).

A C++ programmer might argue that he needs the optimizations from the get-go, and shouldn't be waiting for a VM to figure out how to do them, if at all. This is probably a valid point with our current technology though, as the current level of optimizations in our VMs is inferior to what native compilers can offer - but that may not always be the case if the AOT solutions in our VMs improve, etc.

Answer 9

This article is a summary of a set of blog posts trying to compare speed of c++ vs c# and the issues you have to overcome in both languages to get high performance code. The summary is 'your library matter way more than anything, but if you are in c++ you can overcome that.' or 'modern languages have better libraries and are thus get faster results with lower effort' depending on your philosophical slant.

Answer 10

I think that the real question here is not "which is faster?" but "which has the best potential for higher performance?". Viewed on those terms, C++ clearly wins out - it's compiled to native code, there's no JITting, it's a lower level of abstraction, etc.

That's far from the full story.

Because C++ is compiled, any compiler optimizations must be done at compile time, and compiler optimizations that are appropriate for one machine may be completely wrong for another. It's also the case that any global compiler optimizations can and will favour certain algorithms or code patterns over others.

On the other hand, a JITted program will optimize at JIT time, so it can pull some tricks that a precompiled program cannot and can make very specific optimizations for the machine it's actually running on and the code that it's actually running. Once you get past the initial overhead of the JIT it has potential in some cases to be faster.

In both cases a sensible implementation of the algorithm and other instances of the programmer not being stupid will likely be far more significant factors, however - for example, it's perfectly possible to write completely brain-dead string code in C++ that will be walloped by even an interpreted scripting language.

Answer 11

JIT compilation actually has a negative impact on performance. If you design a "perfect" compiler and a "perfect" JIT compiler, the first option will always win in performance.

Both Java and C# are interpreted into intermediate languages, and then compiled to native code at runtime, which reduces the performance.

But now the difference is not that obvious for C#: Microsoft CLR produces different native code for different CPUs, thus making the code more efficient for the machine its running on, which isn't always done by C++ compilers.

P.S. C# is written very efficently and it hasn't many abstraction layers. This isn't true for Java, which isn't that efficient. So, in this case, with its greate CLR, C# programs often show better performance than C++ programs. For more about .Net and CLR take a look at Jeffrey Richter's "CLR via C#".