Prefixes with aggregate labels not fully tracerouting across MPLS core

Question

I have two routers, A (Cat6500 w/SUP720-3BXL, IOS 12.2(33)SXH4) and B (Nexus 7K w/SUP1, NX-OS 5.2(4)), separated by several hops across an MPLS core, each with VRF ABC. Router A has two directly connected routes and four static routes within this VRF.

RouterA# show ip bgp vpnv4 vrf ABC labels
   Network          Next Hop      In label/Out label
Route Distinguisher: 65000:123 (ABC)
   10.30.10.0/24    10.30.200.1     154/nolabel
   10.30.20.0/24    10.30.200.1     88/nolabel
   10.30.30.0/24    10.30.200.1     38/nolabel
   10.30.40.0/24    10.30.200.1     147/nolabel
   10.30.200.0/24   0.0.0.0         IPv4 VRF Aggr:95/nolabel(ABC)
   10.90.90.0/24    0.0.0.0         IPv4 VRF Aggr:95/nolabel(ABC)
   10.133.242.0/25  192.168.255.3   nolabel/17
   10.133.242.128/26
                    192.168.255.3   nolabel/18
   10.255.255.224/29
                    192.168.255.3   nolabel/492474

Per-prefix labeling is used for this VRF on both routers. Notice that the two directly connected routes receive a shared aggregate label (95) whereas the four static routes each receive a unique label.

Router B agrees on the VPN labels to use:

RouterB# show bgp vpnv4 unicast labels vrf ABC
BGP routing table information for VRF default, address family VPNv4 Unicast
BGP table version is 17042469, local router ID is 192.168.255.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath

   Network            Next Hop            In label/Out label
Route Distinguisher: 65000:123     (VRF ABC)
*>i10.30.10.0/24      172.26.64.1         nolabel/154
*>i10.30.20.0/24      172.26.64.1         nolabel/88
*>i10.30.30.0/24      172.26.64.1         nolabel/38
*>i10.30.40.0/24      172.26.64.1         nolabel/147
*>i10.30.200.0/24     172.26.64.1         nolabel/95
*>i10.90.90.0/24      172.26.64.1         nolabel/95
*>l10.255.255.224/29  0.0.0.0             492474/nolabel (ABC)

From router B, I can traceroute to both of the directly connected networks on router A with no problem:

RouterB# traceroute 10.30.200.10 vrf ABC
traceroute to 10.30.200.10 (10.30.200.10), 30 hops max, 40 byte packets
 1  192.168.254.97 (192.168.254.97) (AS 65000)  19.226 ms  19.369 ms  19.079 ms
      [Label=63 E=0 TTL=1 S=0, Label=95 E=0 TTL=1 S=1]
 2  192.0.2.151 (192.0.2.151) (AS 65000)  23.309 ms  28.027 ms  18.977 ms
      [Label=39 E=0 TTL=1 S=0, Label=95 E=0 TTL=2 S=1]
 3  192.168.251.62 (192.168.251.62) (AS 65000)  21.576 ms  24.265 ms  21.503 ms
      [Label=59 E=0 TTL=1 S=0, Label=95 E=0 TTL=1 S=1]
 4  10.30.200.10 (10.30.200.10) (AS 65000)  19.155 ms *  19.414 ms

However, traceroutes to all of the statically-learned routes timeout across the MPLS path and pick back up only at their last hops:

RouterB# traceroute 10.30.10.10 vrf ABC
traceroute to 10.30.10.10 (10.30.10.10), 30 hops max, 40 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  10.30.200.10 (10.30.200.10) (AS 65000)  19.065 ms  19.281 ms  18.68 ms
      [Label=154 E=0 TTL=1 S=1]
 5  10.30.10.10 (10.30.10.10) (AS 65000)  19.420 ms  19.377 ms  19.73 ms

Both traceroutes above should be following the exact same path, and there are no filtering mechanisms in place along it. The same thing happens in the reverse direction as well. What am I missing? What's the difference between BGP routes learned by direct connection versus static configuration with regard to MPLS/label forwarding?

Answer 1

The difference between aggregate labels and normal labels is such that normal labels directly point to L2 rewrite details (an interface and L2 address). This means a normal label will be label switched by the egress PE node directly out, without doing an IP lookup.

Adversely, aggregate labels can potentially represent many different egress options, so L2 rewrite information is not associated with the label itself. This means that an egress PE node must perform an IP lookup for the packet, to determine appropriate L2 rewrite information.

Typical reasons why you might have an aggregate label instead of normal label are:

1. Need to perform neighbor discovery (IPv4 ARP, IPv6 ND)
2. Need to perform ACL lookup (egress ACL in customer interface)
3. Running whole VRF under single label (table-label)

Some of these restrictions (particularly 2) are not valid to all platforms.

How traceroute is affected in MPLS VPN environment is by the transit P, when generating the TTL exceeded message, will not know how to return it (it does not have routing table entry to the sender). So a transit P node will send the TTL exceeded message with original label stack all the way to the egress PE node, in hope that the egress PE note has an idea of how to return the TTL exceeded message to the sender.
This feature is automatically on in Cisco IOS but needs 'icmp-tunneling' configured in Juniper JunOS.

Based on this, I would suspect that perhaps your CE devices are not accepting packets when source address is a P node link network, and as they are not accepting the ICMP message, they are not able to return it to the sender.
A Possible way test to this theory would be to enable per-vrf label:

• IOS: mpls label mode all-vrfs protocol bgp-vpnv4 per-vrf
• JunOS: set routing-instances FOO vrf-table-label

Generally speaking I do not recommend propagating TTL, especially on VPN environment, at least in our case customers get confused and anxious about it. They worry why their VPN has foreign addresses showing.

Another thing which confuses people causing them to open a support ticket, is when they are running a traceroute from say the UK to the US, because they see >100ms latency between two core routers in UK, not realizing that the whole path has same latency all the way to the west coast of the US, because all the packets take a detour from there.

This issue is mostly unfixable by design, however in IOS you can determine how many labels at most to pop (mpls ip ttl-expiration pop N) when you are generating TTL exceeded. This gives you a somewhat decent approximation if INET == 1 label, VPN == >1 label, so you can configure it so that VPN traffic is tunnelled and INET traffic gets directly returned without egress PE node detour. But as I said, this is just an approximation of desired functionality, as features like in-transit repairs may cause your label stack not to be always same size for the same service.