8

For Cisco Hardware dCEF, based on some documents published on Cisco website, at the ingress line card/interface, conceptually it looks up the FIB with dst IP address, and gets a pointer to an adjacency table entry, where L2 rewrite information is stored, e.g. nexthop mac, etc.

But what confuses me is, doesn't the L2 rewrite happen on egress line card/interface? If so, then why this adjacency table is stored on ingress? Or where is the adjacency table look up happening? ingress or egress? If this is on ingress, are the L2 rewrite information carried over from ingress card to egress line card? Wouldn't that be a waste of fabric bandwidth?

wei
  • 203
  • 1
  • 3
  • 1
    Think about: how can a packet be forwarded to the egress interface if the ingress interface doesn't have any adjacency? – Ricky Jun 11 '14 at 16:59
  • Well, you dont have to store the who adjacency table, you can store some sort of ajacency id, then on egress you can use this adjacency id to look up the adjacency table and get the L2 rewrite information. In fact, if this adjacency table is stored on ingress asic, you will have to store every egress line card's adjacency info to ingress line card, wouldn't that be a huge mem waste? – wei Jun 12 '14 at 01:54

2 Answers2

9

But what confuses me is, doesn't the L2 rewrite happen on egress line card/interface?

Not really, the forward / drop decision, L2 adjacency lookup, decrement TTL, IP Checksum calculation, etc... all happen on the ingress linecard.

Conceptually, you can break the information flow into a control-plane and data plane, even within the router chassis. It seems that most of your confusion revolves around how the control plane works... this is a quick diagram I hacked up to illustrate...

IPC and CEF

  • The Route Processor builds the CEF table from layer-2 adjacency information (including ethernet, ppp, sonet, etc...) as well as all preferred routes
  • The entire CEF and adjacency table information is packed into IPC messages, which are sent between the Route Processor and all linecards. Individual CEF entries are rendered as XDRs within the IPC message. An XDR is just a Cisco-specific way of writing CEF entries into an IPC message.
  • Individual linecards unpack the XDRs from the IPC messages and build (what should be) an exact copy of the Route Processor's CEF and adjacency table on the linecard.
  • After the linecard completes revisions to the CEF and adjacency tables, a platform-specific process runs on the linecard to compute the data structures required for the platform's hardware to forward and rewrite packets on the linecard itself.

Synchronized IPC is pretty critical to dCEF's operation; if you don't keep messages synchronized between all linecards, you can wind up with prefix inconsistencies.

The mechanics of how the router does this is platform-specific, so I'll reference the platform I know best which is Catalyst 6500 with Supervisor720 / Supervisor2T. The forwarding & rewrite engine on a Catalyst 6500 dCEF linecard is actually a miniature copy of the Supervisor itself; so the whole IP forwarding and switching process executes just like it does as if the packet was centrally forwarded on the supervisor. The ingress dCEF linecard looks up the required information in the CAM / CEF table, and then builds a header which it attaches to the packet.

The egress linecard looks at the header and uses the adjacency information inside it to write the packet onto the wire.

Why this adjacency table is stored on ingress?

So you can make the entire forwarding decision on ingress.

If this is on ingress, are the L2 rewrite information carried over from ingress card to egress line card?

Yes

Wouldn't that be a waste of fabric bandwidth?

I don't think so, but then again I could be biased :-)

Mike Pennington
  • 29,876
  • 11
  • 78
  • 152
  • Thanks a lot for the detailed answer! I am basically assuming linecard fib is synced with rp. My confusion is actually around data plane, because i know some non-cisco products are doing adjacency table lookup on egress, so i am trying to find out if cisco really chooses to do it in ingress and why, to me, this wastes memory and fabric bandwidth. – wei Jun 13 '14 at 03:33
  • Waste is subjective. You have to do the lookup somewhere; one could argue that doing a lookup on egress wastes resources and makes the product more expensive. The debate could go on for a long time throwing rocks at the various angles of the object of your dislike. Bottom-line, Cisco chooses to spend a modest amount of memory on the ingress linecard, and the adjacency information is not sent through the fabric on the Catalyst6500 anyway; adjacency results are sent to the egress linecard via a dedicated RBUS. – Mike Pennington Jun 13 '14 at 09:43
  • For more reading regarding the RBUS: [Sup720 Architecture paper](http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-6500-series-switches/prod_white_paper0900aecd80673385.html) – Mike Pennington Jun 13 '14 at 09:46
3

When distributed Cisco Express Forwarding is enabled, line cards, such as the VIP line cards or the Cisco 12000 Series Internet Router line cards, maintain an identical copy of the FIB and adjacency tables. The line cards perform express forwarding between port adapters, thus relieving the RP of involvement in the switching operation. distributed Cisco Express Forwarding uses an interprocess communication (IPC) mechanism to ensure synchronization of FIB tables and adjacency tables on the RP and line cards. -- Cisco

The RP (running various routing protocol processes) builds the FIB and publishes it to all linecards. There's one FIB, but it's replicated on every linecard. (yes, sometimes they get out of sync.)

Ricky
  • 31,438
  • 2
  • 43
  • 84
  • Yeah, that part i understand. The part I am confused about is that all cisco published documents seem to imply that L2 rewrite information are looked up on ingress, which doesnt make much sense to me. – wei Jun 12 '14 at 02:01