But what confuses me is, doesn't the L2 rewrite happen on egress line card/interface?
Not really, the forward / drop decision, L2 adjacency lookup, decrement TTL, IP Checksum calculation, etc... all happen on the ingress linecard.
Conceptually, you can break the information flow into a control-plane and data plane, even within the router chassis. It seems that most of your confusion revolves around how the control plane works... this is a quick diagram I hacked up to illustrate...
- The Route Processor builds the CEF table from layer-2 adjacency information (including ethernet, ppp, sonet, etc...) as well as all preferred routes
- The entire CEF and adjacency table information is packed into IPC messages, which are sent between the Route Processor and all linecards. Individual CEF entries are rendered as XDRs within the IPC message. An XDR is just a Cisco-specific way of writing CEF entries into an IPC message.
- Individual linecards unpack the XDRs from the IPC messages and build (what should be) an exact copy of the Route Processor's CEF and adjacency table on the linecard.
- After the linecard completes revisions to the CEF and adjacency tables, a platform-specific process runs on the linecard to compute the data structures required for the platform's hardware to forward and rewrite packets on the linecard itself.
Synchronized IPC is pretty critical to dCEF's operation; if you don't keep messages synchronized between all linecards, you can wind up with prefix inconsistencies.
The mechanics of how the router does this is platform-specific, so I'll reference the platform I know best which is Catalyst 6500 with Supervisor720 / Supervisor2T. The forwarding & rewrite engine on a Catalyst 6500 dCEF linecard is actually a miniature copy of the Supervisor itself; so the whole IP forwarding and switching process executes just like it does as if the packet was centrally forwarded on the supervisor. The ingress dCEF linecard looks up the required information in the CAM / CEF table, and then builds a header which it attaches to the packet.
The egress linecard looks at the header and uses the adjacency information inside it to write the packet onto the wire.
Why this adjacency table is stored on ingress?
So you can make the entire forwarding decision on ingress.
If this is on ingress, are the L2 rewrite information carried over from ingress card to egress line card?
Wouldn't that be a waste of fabric bandwidth?
I don't think so, but then again I could be biased :-)