9

I have two routers, A (Cat6500 w/SUP720-3BXL, IOS 12.2(33)SXH4) and B (Nexus 7K w/SUP1, NX-OS 5.2(4)), separated by several hops across an MPLS core, each with VRF ABC. Router A has two directly connected routes and four static routes within this VRF.

RouterA# show ip bgp vpnv4 vrf ABC labels
   Network          Next Hop      In label/Out label
Route Distinguisher: 65000:123 (ABC)
   10.30.10.0/24    10.30.200.1     154/nolabel
   10.30.20.0/24    10.30.200.1     88/nolabel
   10.30.30.0/24    10.30.200.1     38/nolabel
   10.30.40.0/24    10.30.200.1     147/nolabel
   10.30.200.0/24   0.0.0.0         IPv4 VRF Aggr:95/nolabel(ABC)
   10.90.90.0/24    0.0.0.0         IPv4 VRF Aggr:95/nolabel(ABC)
   10.133.242.0/25  192.168.255.3   nolabel/17
   10.133.242.128/26
                    192.168.255.3   nolabel/18
   10.255.255.224/29
                    192.168.255.3   nolabel/492474

Per-prefix labeling is used for this VRF on both routers. Notice that the two directly connected routes receive a shared aggregate label (95) whereas the four static routes each receive a unique label.

Router B agrees on the VPN labels to use:

RouterB# show bgp vpnv4 unicast labels vrf ABC
BGP routing table information for VRF default, address family VPNv4 Unicast
BGP table version is 17042469, local router ID is 192.168.255.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath

   Network            Next Hop            In label/Out label
Route Distinguisher: 65000:123     (VRF ABC)
*>i10.30.10.0/24      172.26.64.1         nolabel/154
*>i10.30.20.0/24      172.26.64.1         nolabel/88
*>i10.30.30.0/24      172.26.64.1         nolabel/38
*>i10.30.40.0/24      172.26.64.1         nolabel/147
*>i10.30.200.0/24     172.26.64.1         nolabel/95
*>i10.90.90.0/24      172.26.64.1         nolabel/95
*>l10.255.255.224/29  0.0.0.0             492474/nolabel (ABC)

From router B, I can traceroute to both of the directly connected networks on router A with no problem:

RouterB# traceroute 10.30.200.10 vrf ABC
traceroute to 10.30.200.10 (10.30.200.10), 30 hops max, 40 byte packets
 1  192.168.254.97 (192.168.254.97) (AS 65000)  19.226 ms  19.369 ms  19.079 ms
      [Label=63 E=0 TTL=1 S=0, Label=95 E=0 TTL=1 S=1]
 2  192.0.2.151 (192.0.2.151) (AS 65000)  23.309 ms  28.027 ms  18.977 ms
      [Label=39 E=0 TTL=1 S=0, Label=95 E=0 TTL=2 S=1]
 3  192.168.251.62 (192.168.251.62) (AS 65000)  21.576 ms  24.265 ms  21.503 ms
      [Label=59 E=0 TTL=1 S=0, Label=95 E=0 TTL=1 S=1]
 4  10.30.200.10 (10.30.200.10) (AS 65000)  19.155 ms *  19.414 ms

However, traceroutes to all of the statically-learned routes timeout across the MPLS path and pick back up only at their last hops:

RouterB# traceroute 10.30.10.10 vrf ABC
traceroute to 10.30.10.10 (10.30.10.10), 30 hops max, 40 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  10.30.200.10 (10.30.200.10) (AS 65000)  19.065 ms  19.281 ms  18.68 ms
      [Label=154 E=0 TTL=1 S=1]
 5  10.30.10.10 (10.30.10.10) (AS 65000)  19.420 ms  19.377 ms  19.73 ms

Both traceroutes above should be following the exact same path, and there are no filtering mechanisms in place along it. The same thing happens in the reverse direction as well. What am I missing? What's the difference between BGP routes learned by direct connection versus static configuration with regard to MPLS/label forwarding?

10
  • Is the topic wrong? Looks like aggregate labels traceroute fine, normal labels don't? What platform is this? Are there anything configured with regards TTL hiding or any other specific commands? In VPN the traceroute always goes to the egress PE before TTL exceeded is generated, so for some reason for non-aggregate label you're not actually generating TTL exceeded.
    – ytti
    Jun 13, 2013 at 18:29
  • Updated question to reflect platforms (IOS and NX-OS). Jun 13, 2013 at 18:37
  • HW would be appreciated as well, sup720-3bxl has HW limitations when dealing with TTL decrement in MPLS environment. Do you experience the problem to both directions or only one direction?
    – ytti
    Jun 13, 2013 at 18:38
  • The same thing happens with statically-learned routes the reverse direction as well. Router A is running a SUP720-3BXL; could elaborate on the limitations you mention? Jun 13, 2013 at 18:48
  • Unfortunately thinking bit more the sup720-3blx (or PFC3B to be exact, PFC3C is fixed) issue does not explain this. Since then you'd only miss egress PE in traceroute completely (no stars). And it wouldn't have same issue to both directions, this is most curious to me how the problem happens from nexus7k to 7600 and 7600 to nexus7k.
    – ytti
    Jun 13, 2013 at 19:06

1 Answer 1

9

The difference between aggregate labels and normal labels is such that normal labels directly point to L2 rewrite details (an interface and L2 address). This means a normal label will be label switched by the egress PE node directly out, without doing an IP lookup.

Adversely, aggregate labels can potentially represent many different egress options, so L2 rewrite information is not associated with the label itself. This means that an egress PE node must perform an IP lookup for the packet, to determine appropriate L2 rewrite information.

Typical reasons why you might have an aggregate label instead of normal label are:

  1. Need to perform neighbor discovery (IPv4 ARP, IPv6 ND)
  2. Need to perform ACL lookup (egress ACL in customer interface)
  3. Running whole VRF under single label (table-label)

Some of these restrictions (particularly 2) are not valid to all platforms.

How traceroute is affected in MPLS VPN environment is by the transit P, when generating the TTL exceeded message, will not know how to return it (it does not have routing table entry to the sender). So a transit P node will send the TTL exceeded message with original label stack all the way to the egress PE node, in hope that the egress PE note has an idea of how to return the TTL exceeded message to the sender.
This feature is automatically on in Cisco IOS but needs 'icmp-tunneling' configured in Juniper JunOS.

Based on this, I would suspect that perhaps your CE devices are not accepting packets when source address is a P node link network, and as they are not accepting the ICMP message, they are not able to return it to the sender.
A Possible way test to this theory would be to enable per-vrf label:

  • IOS: mpls label mode all-vrfs protocol bgp-vpnv4 per-vrf
  • JunOS: set routing-instances FOO vrf-table-label

Generally speaking I do not recommend propagating TTL, especially on VPN environment, at least in our case customers get confused and anxious about it. They worry why their VPN has foreign addresses showing.

Another thing which confuses people causing them to open a support ticket, is when they are running a traceroute from say the UK to the US, because they see >100ms latency between two core routers in UK, not realizing that the whole path has same latency all the way to the west coast of the US, because all the packets take a detour from there.

This issue is mostly unfixable by design, however in IOS you can determine how many labels at most to pop (mpls ip ttl-expiration pop N) when you are generating TTL exceeded. This gives you a somewhat decent approximation if INET == 1 label, VPN == >1 label, so you can configure it so that VPN traffic is tunnelled and INET traffic gets directly returned without egress PE node detour. But as I said, this is just an approximation of desired functionality, as features like in-transit repairs may cause your label stack not to be always same size for the same service.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.