I understand that memory is a big bottleneck in modern computer systems, but can't a system with many simple cores be more efficient than one with single digit number of advanced cores for some tasks?
From what I understand a GPU is an extreme version of this, but isn't there some middle ground for certain tasks that needs a density/complexity which is somewhere between the two extremes?
Programming for parallelism is difficult, so most things are largely done sequentially which requires more complex processors. The clock limit prevents the processors from getting much more complex so instead we have several complex processors which mainly allow independent tasks to be run simultaneously or allow a single task to be split up into a few simultaneously threads when it is really obvious and simple to do so.
Because programming for parallelism is difficult, you can fairly accurately predict when the extra for effort massive parallelism will be invested on the programming side (i.e. where it is actually needed). This makes it easy to target exactly who actually will bother doing parallel work with many simple cores which is why you end up with general-purpose CPUs with fewer complex cores and special purpose GPUs with a large number of simpler cores.
Development is expensive so you need a sufficiently large market to support the development of a mixed or mid-complexity, multi-core processor.
From what I understand a GPU is an extreme version of this, but isn't there some middle ground for certain tasks that needs a density/complexity which is somewhere between the two extremes?
Modern GPUs are that middle ground. Where earlier GPUs were really simple "one instruction, different data, everyone waits until the slowest is done, next instruction" with only very limited instruction sets, modern GPUs compute units are far more general and independent.
There were multiple (if not many) attempts to do this before – and aside from niche usage, all of them failed due to neither falling into the sweet spot of independent high-performance CPUs nor cheap, low-power massively parallel simplistic shader units. Basically, you need to be efficient with both your CPU time and your memory bandwidth, and that means you either need few, but performant cores sharing main memory but having extensive local caches, or you need many, but centrally orchestrated simpler cores. It has shown you really can't have both easily – which is (my interpretation of) why it took the world so long to arrive at modern GPUs, and why only two companies are dominating the GPGPU market very clearly, with one clear leader.
Examples of these commercial failures include:
¹ Larrabee used very imperformant x86 cores in a manycore processor, and thus was middle ground and useless, taken over by GPUs on one side and classical workstation/server CPUs on the other, though one could argue that its successor, the Xeon Phi, has a better fate, but these really are more like lots of mighty x86-64 including AVX-512 on a single chip, so it's not the middle ground you hope for. The series got discontinued last year, mostly because lack of demand – GPUs on one side, classical x86-64 on the other side simply are more useful and better compute/Watt.
I'd like to take the frame challenge further than the other answers here. Until the end of the 20th Century, parallel computing was the domain of specialist servers and supercomputers; general-purpose consumer computers were based on a single processor core running at faster and faster clock speeds. Since then, multiple core processors have become the norm, and the number of cores included has slowly increased.
Modern architectures may include (even on a single chip):
Nonetheless, a lot of software is only able to make use of a single core, so general-purpose CPU cores are not likely to go away any time soon. That's because you can only make use of multiple cores in certain select cases:
Some tasks lend themselves to parallelism, some don’t. Broadly, tasks that tend to be repetitive with lots of time spent in loops can be partitioned into smaller sub tasks, marshaled by the host. Tasks that tend to be ‘thready’ with lots of branches aren’t so easy to break up.
Graphics and AI are examples that can be broken up into parallel tasks. TCP/IP, not so much.
Point being, there’s a place for both.
Umm, GPUs are not “an extreme version of a system with many simple cores”. Eg in an Nvidia GPU each SM (streaming multiprocessor) is a highly multithreaded processor. The data path is IIRC 16 lanes, each capable of 32-bit floating point calculations. The multi threading scheduling of a GPU is much more sophisticated than the scheduling for an MT CPU, snd is arguably comparable to that of an OOO ILP scheduler.
The big difference is that an OOO CPU needs to make many different scheduling decisions every cycle that affect the next 1-3 cycles. Largely different scheduling decisions for each instruction. Whereas the GPU’s scheduling decision applies to that set of 16 ALUs wide in space, also typically 2 to 4 cycles deep in time. Moreover, instead of scheduling a dependent instruction 1 or 2 cycles later, in a GPU the next dependent instruction from the same thread can only arrive many cycles later - in some GPUs no earlier than 40 cycles!!!! In between the GPU schedules operations group other threads. However, more and more even the massively parallel workloads of a GPU do not have enough parallelism, so more and more GPUs are exploiting ILP within the same “thread” - Nvidia’s terminology is “warp” - the same set of 16-space-wide X 2-time-deep operations executing in lockstep.
Ie the biggest difference between the CPU and GPU SM are the the SM amortizes the cost of control logic over many lockstep elements per thread/warp, and many independent control flow threads/warps within the same processor.
Plus, of course, a GPU has multiple if these processing elements. Many more in a single chip than CPUs do.
Overall, GPUs have a much lower ratio if control logic to compute/ALU logic than CPUs do. GPU control logic is pretty sophisticated, but it’s cost is spread out, amortized, over more ALUs - because in a GPU many of the ALUs are doing almost exactly the same thing at the same time.
—
But I am not addressing your question, am I? You asked “ why not many simple processors, rather than the big complicated single processors…”?
The name for “many independent processors” is MIMD. Multiple Instruction Mujtiple Data”. Now, a GPU is MIMD - but each of its independent processors is SIMD lockstep and MT multithreaded.
I think you are asking about “many independent simple processors”. A MIMD whose independent processors are, say, inky 32 or 64-bits wide, not-multithreaded, not suoerscakar, and not OOO. I like calling this a MIMD(1), whereas a GPU is a MIMD(SIMD(16x2)*MT(…))
So: here’s where a GPU beats a MIMD-1 —- the MIMD-1 has at least 16x more control logic, proportionally, than the simple GPU MIMd(SIMD(32)), even assuming that the GPZu cores are not MT or ILP. Ie the GPU wastes only 1/16 as much area and power on control logic as the MIMD-1.
—-
But again you are not asking about why a GPU beats a massively parallel MIMD-1. You are asking about why a massively parallel machine with simple MIMD-1 processing elements doesn’t beat a machine whose processing elements are much more complicated. Perhaps 16 or 1024x more complicated, so that you can only have 1/16 - 1/1024 if the processors. Say, “only” 8 CPUs per chip, versus perhaps 128 or even 8K MIMD-1s in the same chip.
Well, the MPP-MIMD-1 would beat the machine with much more complicated cores - for exactly the same workloads that the GPU beats the MIMD-1 machine. So the MIMD-1 gets squeezed out.
And for other workloads… well, if there is not enough parallelism to use all if the simple MIMD-1 processors, it is beaten by the big CPUs. And even if there us enough parallellsm, but if the irregular kind that GPUs cannot do well - well then the MIMD-1 cores are probably spending lots of time waiting for memory.
Also, for that matter, if you have 16x more MIMD-1 processors than you have big-CPUs, you have 16x more wires going to memory. Which eats away at that supposed 16x advantage.
Eg a MIMD-1 MPP would beat a big-CPU less-parallel machine, in a workload that had enough parallelism, if memory were free, 1-cycle latency, and if memory wires cost nothing. Or, equivalently, if each of the MIMD-1’s accessed private memory only. If logic gates were much bigger than wires… but when wiring area and power dominates the actual logic, the MIMD-1s lose more and more ground to fewer parallel bigger CPUs.
—-
There may still be a place for MIMD-1 MPPs - but it is squeezed between GPUs and big non-GPU CPUs.
—-
So much for the bad news. Now for the good news:
Programmers prefer thinking about completely independent MIMD CPUs to thinking about complex trade offs if MIMD and SIMD and MT.
In fact, the historical GPU programming model was to treat each of the lockstep lane-threads within a control-flow warp as an independent MIMD-1.
Conversely, another programming model than humans seem comfortable with is a single thread of control - but with data parallel operations, like operating in arbitrary arrays. Basically a vector processor.But again, humans like arbitrary vectors and arrays, snd don’t like to have to worry about tuning to different numbers of vector elements, etc.
In many ways a GPU is just a way if taking MIMD-1 code and running it in a less expensive micro architecture that shares control logic between what would otherwise be independent processing elements.
Ie the good news is that your “lots of simple processors” model wins in some domains -but as a software or programming model concept - one that us implemented more efficiently by either a GPU or fewer parallel but more ILP/MLP micro architectures.
Algorithms, proofs, thought processes tend to be a list of things to do in sequence. The human brain may be a massively parallel processor but the things it does in parallel are not as much conscious thought processes but acquired skills. As a consequence, the descriptions for what a computer should be doing tends to be structured in a comparatively linear manner, and computer instruction sets are structured in sequential steps rather than a set of tasks to be done with some interdependence. Superscalar architectures actually parallelise a few things while figuring out interdependence, and the original MIPS architecture ("Microprocessor without Interlocking Pipeline Stages") had the compiler already figure out those interdependencies and schedule operations accordingly.
But that is at a very minor and very low-level scale. It turns out that sequentiality very much pervades computing, thinking about computing, and programming and attempts at massive parallelism expressed reasonably naturally in architecture ("dataflow architecture") and/or programming ("Occam") did not really take off and some massively parallelisable systems like artificial neural networks get their programming rather via handwaving and "learning" rather than explicit instructions.
TL:DR: Yes, but not for most tasks. That's why the current iteration of this idea is hybrid CPUs with some performance, some efficiency cores, now that we have transistor budgets to throw that many cores onto a consumer laptop/desktop CPU.
but can't a system with many simple cores be more efficient than one with single digit number of advanced cores for some tasks?
"For some tasks" is the rub. They're a lot worse for many other tasks, ones that haven't been or can't easily be parallelized. For looping over a medium-sized array, for example, it's often not worth talking to other CPU cores about doing some of the work, because the latency involved is comparable to the time it would take just doing the work in a single thread.
And a "many simple-ish CPU" processor is worse than GPUs for tasks that are highly parallel, and don't have much data-dependent branching. I.e. where high latency can be tolerated to achieve the high throughput per-power and per-die-area GPUs can provide. So as other answers have pointed out, the middle ground between throughput-optimized GPUs and latency-optimized CPUs isn't very big in terms of commercial demand. (Stuff like CPU SIMD does it well enough for most things, although with power-efficiency becoming ever more important, there is room for hybrid CPUs with some efficiency cores.)
Per-thread performance is very important for things that aren't embarrassingly parallel. (And also because memory bandwidth / cache footprint scale some with number of threads for many workloads, so there's a lower limit on how simple/small you'd want to make each core without a totally different architecture such as a GPU.)
A system with fewer big cores can use SMT (Simultaneous Multi-Threading, for example hyperthreading) to make those big cores look like 2x as many smaller cores. (Or 4x or 8x, for example in IBM POWER CPUs.) This is not as power-efficient as actually having more smaller cores, but is in the same ballpark. And of course simple OS context-switching makes it possible to run as many software threads as you want on a core, while the reverse is not possible: there's no simple way to use lots of simple cores to run one thread fast.
There are diminishing returns The opposite of this question, Why not make one big CPU core? has
Related: Modern Microprocessors A 90-Minute Guide! has a section on SMT and multi-core, and is excellent background reading on CPU design constraints like power.
Making large cache-coherent systems is hard so it's hard to scale to huge numbers of CPU cores. The biggest Xeon and Epyc chips are packing 56 or 64 physical cores on one die.
Compare this with Xeon Phi compute cards which is pretty much what you wondered about: AVX-512 bolted on to low-power Silvermont cores, going up to 72 cores per card with some high-bandwidth memory. (And 4-way SMT to hide ALU and memory latency, so it actually supported 4x that many threads.)
They discontinued that line in 2018 due to lack of demand. This article says it's "never seen any commercial success in the market". You couldn't get big speedups from running existing binaries on it; code generally needed to be compile to take advantage of AVX-512. (I think Intel's toolchain was supposed to be able to auto-parallelize some loops so source changes might have been less necessary or smaller than for using GPUs). And it omitted AVX-512BW so it wasn't good for high-quality video encode (x264/x265 as opposed to fixed-function hardware); I think primarily good for FP work, which means it was competing with GPUs. (Some of the reason may be due to working on a new from-the-ground-up architecture for "exascale" computing, after seeing how the computing landscape evolved since the start of the Larrabee project in the mid 2000's; it was originally designed to be a more-programmable GPU based on x86 cores(!), but evolved into Xeon Phi once GPGPU did that better and they found a more appropriate nice.)
The latest iteration of your idea is to have a mix of cores, so you can still have a few fast cores for serial / latency-sensitive stuff.
Some code is only somewhat parallel, or has a few different threads doing separate tasks that are individually serial. (Not really distributing one task across many threads.)
ARM has been doing that for a while (calling it big.LITTLE), and Intel's new Alder Lake design with a mix of Performance (Golden Cove) cores and Efficiency (Gracemont) cores is exactly this: add some relatively-small throughput-optimized cores that don't push so far into the diminishing returns for spending more power to increase per-thread throughput.
So when doing "light weight" work where an E-core is enough to keep up with something that isn't useful to do faster (like playing a video or typing / clicking around on a web page), only that small core needs to be powered up.
Or when doing some number crunching / video encoding / whatever with lots of thread-level parallelism, 4 E-cores for the area of one P-core gives you more total throughput. But you still have some P-cores for tasks that don't (or simply weren't) parallelized. (I wrote in more detail about Alder Lake on superuser.com.)
Even the E cores on Alder Lake do fairly wide superscalar out-of-order exec, and can have quite good throughput on code where the instruction-level parallelism is easy for the CPU to find. (On ARM big.LITTLE, the little cores are often in-order, but still 3-wide superscalar with stuff like hit-under-miss caches to find some memory-level parallelism. e.g. Cortex-A53)
For most systems for general workloads, it's not commercially viable to not have any latency-optimized cores that have high single-threaded performance. Many tasks don't easily parallelize, or simply haven't been because that's much more programming effort. (Although low-end smartphones sometimes only use low-end cores; people would rather have cheap and slow than no phone at all, and power efficiency matters even more than for laptops.)
I already mentioned Xeon Phi, but years before that another interesting example was Sun UltraSPARC T1, aka Niagara, released in 2005.
It was an 8-core CPU (or 4 or 6 core for other models), at a time when x86 CPUs were only just starting1 to introduce dual-core such as Athlon X2. They weren't even trying to aim for high per-thread performance, which was essential for most interactive use back then. Instead they were aiming at server / database workloads with many connections so there was already plenty of thread-level parallelism for software, even back then.
Each core had a fairly simple pipeline, and was a "barrel" processor, rotating between up to 4 non-stalled logical cores aka hardware threads. (Basically like 4-way SMT, but in-order so instructions from separate threads never mix in an execution unit.) Keeping cores small and simple made locking overhead lower, I think.
32 logical cores was a huge amount back in 2005. (And later generations were multi-socket capable, allowing 2x or 4x that in a whole system). The wiki article mentions that without special compiler options (to auto-parallelize I assume), it left a lot of performance on the table for workloads that weren't already parallelized, like gzip or MySQL (back in 2005, and IDK if they were talking about a single big query or what). So that's the downside of having many weak cores, especially if you really lean into it like Sun did by making it with not much cache, and depending on the barrel processor to hide latency.
Footnote 1: Part of that was that most of the x86 market was for machines that would run Windows, and until Windows XP the mainstream versions weren't SMP-capable IIRC. But servers had been multi-socket for a long time, achieving SMP by having multiple physical CPU packages in separate sockets. But primarily it was that transistor budgets were still at a point where more cache and wider/deeper OoO exec were still providing significant gains.
Communication between cores is always going to be pretty expensive (at least high-latency), because distances are inherently far, and because of out-of-order exec to hide memory and L3-cache latency. A core can speculatively see its own actions, but stores can't be made visible to other cores speculatively, otherwise you'd have to roll them back on detecting a branch mispredict. (Which is completely impractical in terms of maintaining a consistent point to roll back to, and would defeat the purpose of having it be a separate core.)
So inter-core communication takes at least the latency of the interconnect and cache-coherency protocol, plus the latency of the store buffer and out-of-order exec window. (And the memory-ordering barriers normally involved limit out-of-order execution somewhat.)