22

10 gigabit Ethernet means that 10 billion bits are transmitted every second, but I don't understand how this is physically possible (let alone 100G Ethernet). The fastest CPUs today only run at ~8GHz, but even if transmission doesn't require a CPU, it still seems problematic.

At 10G, each bit lasts only 100 picoseconds, and in that time span, I would think that gate delays become an issue. It's not as simple as setting a line high or low for each bit, surely hundreds of transistors are required in order to output the complicated Ethernet waveform.

This seems like even more of a problem on the receiving end, since the waveform must be sampled at a very high rate, and if this uses ADCs, that introduces even more delay.

HaLailah HaZeh
  • 658
  • 5
  • 19
  • 6
    I guess the best bet would be to go and look through technical papers? – Eugene Sh. Jun 21 '17 at 21:27
  • 6
    Have a look at how the data is encoded, hint: Using PAM16 "magically" makes you need much less transitions. – PlasmaHH Jun 21 '17 at 21:28
  • 10
    8GHz * 64bit = 512Gbps. So really 10Gbps is nowhere near as fast as the CPU. – Tom Carpenter Jun 21 '17 at 21:30
  • 6
    Most of your assumptions about how it works are quite wrong - if it worked as you imagine, it probably wouldn't be possible. But it doesn't, and it is. – Chris Stratton Jun 21 '17 at 21:30
  • 1
    @TomCarpenter that's a meaningless calculation. – Chris Stratton Jun 21 '17 at 21:31
  • @ChrisStratton Okay, could you clarify these misconceptions? – HaLailah HaZeh Jun 21 '17 at 21:31
  • 4
    No. Stack Exchange sites are for answers to *specific problems*, they are not a substitute for reference materials or "how things work" type sites. – Chris Stratton Jun 21 '17 at 21:32
  • 1
    @ChrisStratton not pointless in the slightest. The point is one of the fact that an 8GHz 64bit CPU would have to process data with 64 parallel lines which is far far far more difficult than a measly 10G Ethernet line due to skew. – Tom Carpenter Jun 21 '17 at 21:33
  • @TomCarpenter most "8 GHz" CPUs can't actually do that at that rate. And even if they could in the core of the CPU, they couldn't get the data in and out. – Chris Stratton Jun 21 '17 at 21:34
  • 1
    Remember that CPU speed is determined by the *worst* path from one set of flipflops to another. Whereas ethernet is pipelined: you can have one end working on later bits at the same time as earlier bits are futher through the process. You can quite easily have sub-10ps gate delays for a single gate. – pjc50 Jun 21 '17 at 21:36
  • 8
    @ChrisStratton I disagree that this is off-topic. We have plenty of "How does X work?" questions in the archives, and this one is reasonably narrow. – Adam Haun Jun 21 '17 at 21:52
  • 2
    In the OP's defense, my usual minimum threshold to check Wikipedia before asking a question. Wikipedia's entry on 10G doesn't mention symbol rates, so it passes my bar. – Cort Ammon Jun 22 '17 at 01:12
  • 1
    Ever heard of a serdes? – Voltage Spike Jun 22 '17 at 06:58
  • CPUs are also very general purpose. You usually have hardware dedicated to handle the traffic - network cards to package and pack up and PCI busses to send it on. You can clock those much faster if they don't need to be nearly as big and as general purpose as a CPU. – mathreadler Jun 22 '17 at 09:51

3 Answers3

25

Chasing this answer down took a few different links, but it appears to boil down to this:
1. 4 differential pairs (8 wires total, but only 4 lanes).
2. 800 Mega Symbols a second.
3. Using PAM16, 16 symbols are used which translates into 4 bits per baud per lane.

Given that information you come up with 4 bits*800 Mhz*4 lanes which results in 12800 Mb/s or 12.8 Gb/s. Due to encoding of error correction and other overhead, they only expect you to get 10 Gb/s from it.

Notice that the wires themselves are only changing symbols or amplitude at a frequency of 800 MHz. That's pretty meh, in terms of switching speeds for transistors.

Now that's all for 10Gb Ethernet. How they do it for 100Gb Ethernet is a little more mind boggling. For that, it appears that they actually do pump up the frequency to 10.3 GHz or 25GHz. WHAT THE HECK? See here for that table. The difference in frequency is due to how many copper pair data lanes you choose to have. Whether or not anyone has actually made this 25GHz ethernet in copper would be interesting to know. It's possible that they've only spec'ed it out. When you start getting to those frequencies, your cables either need to be really short or you just switch over to fiber optics where you can send hundreds of beams of light down a single fiber. In that way, you don't have to go at crazy speeds, you just parallelize your data at the source and deparallelize it at the destination.

References if you want to look into it more: https://en.wikipedia.org/wiki/10_Gigabit_Ethernet#Copper
https://en.wikipedia.org/wiki/Pulse-amplitude_modulation
http://www.cablinginstall.com/articles/print/volume-15/issue-7/features/technology/twisted-pair-options-for-10-gigabit-ethernet.html
https://en.wikipedia.org/wiki/Differential_signaling

horta
  • 12,738
  • 22
  • 45
  • AFAIK, there's no Ethernet standard using PAM16. Some 40G & 100G standards might use PAM4. I don't think any 10G standard does. – The Photon Jun 21 '17 at 22:09
  • It appears that PAM16 requires 16 separate voltages to be shown sequentially per symbol. That means the effective frequency is actually 800MHz * 16 = 12.8GHz. Thus, the voltage level of each wire changes every 78.13 picoseconds. – HaLailah HaZeh Jun 21 '17 at 22:10
  • 1
    @HaLailahHaZeh Even if they haven't found a way to encode away that effective frequency. 12.8 GHz isn't unheard of on communication channels between CPUs in a system. – horta Jun 21 '17 at 22:17
  • @ThePhoton 802.3an is the standard that specifies PAM16. Maybe you're hinting at the fact that maybe no one has ever used it because it doesn't work for the reasons HaLailah brought up? Trying to shop for 10Gb ethernet connections tend to show only copper to fiber adapters/wires. – horta Jun 21 '17 at 23:11
  • I think this document would be very helpful: http://www.ikn.no/download/Whitepaper-10G-Ethernet-10-08.pdf – HaLailah HaZeh Jun 21 '17 at 23:18
  • 2
    @HaLailahHaZeh BTW, your comment above goes against that document you linked: the modulation rate is 800Mbaud, therefore the Nyquist frequency is _400 MHz_. You don't multiply the frequency by the number of voltage levels. This means that cabling capable of carrying frequencies up to ~500 MHz will suffice. – uint128_t Jun 21 '17 at 23:26
  • @uint128_t Sorry, I don't have much experience in this area. – HaLailah HaZeh Jun 21 '17 at 23:37
  • 1
    @HaLailahHaZeh 16 separate voltages is 4 bits, not 16 bits. – user253751 Jun 21 '17 at 23:42
  • 2
    Dammit. I checked all the -CX4 & -KX4 types before posting. Didn't realize there was a 10GBase-T. – The Photon Jun 21 '17 at 23:43
  • @immibis When did I say it was 16 bits? If I did, that's not what I meant. – HaLailah HaZeh Jun 21 '17 at 23:43
  • @HaLailahHaZeh: you said 800 * 16. It's 800 * 4(bits) * 4(wires) – slebetman Jun 22 '17 at 06:59
  • 1
    @HaLailahHaZeh: [PAM-16](https://en.wikipedia.org/wiki/pulse-amplitude_modulation) does use 16 separate voltages, but only one voltage is transmitted during each symbol-time. With 10GBASE-T, the 800 MSymbols/sec means the voltage of each wire changes every 1.25 nanoseconds. Thanks for the whitepaper; it is helpful. – davidcary Jun 22 '17 at 12:48
15

10G ethernet (as described by other answers) does not do signal transitions at 10 GHz, it uses multiple level encoding spread across 4 pairs to achieve 10 Gb/s.

However, 10+ gigabit serial transceivers are quite common on high speed chips. For instance PCIe, USB3.1, thunderbolt, and similar protocols all use 10 gbit/s serial rate on individual pairs.

You are correct that "bulk" logic can't keep up with that data rate. Certainly CPU cores don't operate at that frequency, but even the logic that implements things like PCIe interfaces can't operate at that speed. Instead they use dedicated high speed SERDESs.

Data is routed with the IC in wide parallel buses. A dedicated piece of hardware does the serial-to-parallel or parallel-to-serial conversion right near the input/output. The SERDES does an absolute minimum of actual logic. Transmitters are very simple. It will have a PLL to generate the high speed serial data clock and the parallel to serial logic. Receivers are more complicated, they need to do clock recovery on the incoming data, and also framing detection to make sure that the bits are grouped properly. All in all, only a tiny bit of logic has to operate at the ultra-high speed. Yes, the propagation delays through transitions are extremely important, and the circuit has to be carefully designed so that all the signals line up properly.

Evan
  • 2,449
  • 10
  • 17
  • And Thunderbolt only works 20-40 gbit/s on thin copper wires because the cables are active to amplify the weak signals and as such, quite expensive. – chx Jun 21 '17 at 23:06
  • 10G transmitters are actually quite a bit more complicated than what you describe. They do pre-emphasis/de-emphasis. That is they actually intentionally distort the signal in a way which will make the signal look cleaner on the receiving end. – Timmy Brolin Jan 12 '22 at 00:40
3

Lots of 10 GBit/s Ethernet links are actually optical (e.g. 10GBASE-SR or 10GBASE-LR, see https://en.wikipedia.org/wiki/10_Gigabit_Ethernet) although there is also 10GBASE-T over twisted pair cables with 8P8C ('RJ45') connectors as described by @horta. As far as I know is this quite power hungry compared to the optical variants.

Transfer of data from the CPU (or rather memory) to the Ethernet card happens usually through the PCIe bus in an x86 based computer. PCIe Gen 1 lanes have a useable data transfer rate of 2 Gbit/s second (after the 8/10 bit encoding). With 8 lanes the theoretical maximum is 16 GBit/s (per direction), sufficient to drive a single port of 10 GBit/s Ethernet.

The CPU deposits the data to be transmitted in the RAM and then instructs the network card where to pick up (DMA) and similarly for reception the CPU allocates buffers and informs the network card about it when then typically generates an interrupt when the buffer(s) were filled. Note that the bandwidth to RAM is typically much larger than the one of the PCIe bus.

Today we have PCIe Gen 3 widely available which has useable data rate of roughly 8 GBit/s per lane and direction. A 16 lane slot can theoretically handle 128 GBit/s, sufficient for 100 GBit/s Ethernet (PCIe Gen 4 has been announced officially recently).

So the 'trick' to achieve high throughput inside the PC (without having to go to exorbitant signalling speeds) is to use parallel buses (RAM) or multiple serial lanes (PCIe).

For 100 Gbit/s Ethernet one typically has four links with 25 GBaud signalling speed (100GBASE-SR4, 100GBASE-LR4, 100GBASE-CR4), there are also a standards for cables with ten links (e.g. fibre pairs) of 10 Gbit/s (100GBASE-CR10, 100GBASE-SR10, 100GBASE-CR10). For longer distance links, there are also standards using only a single fibre, either using four wavelengths (100GBASE-CWDM4) or using two polarization modes and QPSK (100GBASE-ZR).

For extremely high link speeds over long haul links (such as the Marea transatlantic cable with 20 Terabit/s per fibre pair) one packs as many transmitters at different wavelengths as possible into the useable wavelength band of the fibres and amplifiers, also known as Dense Wavelength Division Multiplexing (DWDM). Note that such a multiplexer/demultiplexer is typically an optical only device at its core and is fed by multiple lower bandwidth streams which can be processed electronically in parallel.

To achieve 20 TBit/s one then also has use advanced modulation techniques where at each clock cycle multiple amplitudes and phases can be transmitted (I have seen 64QAM in a whitepaper) therefore transmitting multiple bits per clock cycle, similar to the 10GBASE-T standard described by @horta.

Andre Holzner
  • 596
  • 3
  • 6