0

I've been stuck on this one in my lab for a while. Any help is appreciated.

Here's my CPE edge topology:

https://i.imgur.com/UZ4u0XG.png

Take the below scenario:

I have 2 circuits coming in that I am doing per-packet load balancing across for my WAN links (Fa0/0 & Fa0/1). I get my default route installed with IPCP after PPP negotiation for both links has completed. Looks normal as below:

CPE-4# sh ip route
S*    0.0.0.0/0 [1/0] via 10.161.3.1

CEF shows I have both paths there:

CPE-4# sh ip cef
Prefix               Next Hop             Interface
0.0.0.0/0            10.161.3.1           Dialer1
                     10.161.3.1           Dialer2

Both of my sessions are up:

CPE-4#sh caller
                                                  Active    Idle
  Line           User               Service       Time      Time
  con 0          -                  TTY           01:12:07  00:00:00
  Vi2            LNS1               PPPoE         00:58:27  00:00:01
  Vi3            LNS1               PPPoE         01:03:54  00:00:08

I shut down one of the wan links (Fa0/0) and I get the expected PPP debug logs:

*Oct 16 18:23:19.279: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down
*Oct 16 18:23:20.279: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
*Oct 16 18:24:16.279: Vi2 PPP: Missed 5 keepalives, taking LCP down
*Oct 16 18:24:16.279: Vi2 PPP DISC: Missed too many keepalives
*Oct 16 18:24:16.283: PPP: NET STOP send to AAA.
*Oct 16 18:24:16.299: Vi2 IPCP: Event[DOWN] State[Open to Starting]
*Oct 16 18:24:16.299: Vi2 IPCP: Event[CLOSE] State[Starting to Initial]
*Oct 16 18:24:16.303: Vi2 LCP: O TERMREQ [Open] id 3 len 4
*Oct 16 18:24:16.307: Vi2 LCP: Event[CLOSE] State[Open to Closing]
*Oct 16 18:24:16.307: Vi2 PPP: Phase is TERMINATING
*Oct 16 18:24:16.359: Di1 Deleted neighbor route from AVL tree: topoid 0, address 10.161.3.1
*Oct 16 18:24:16.359: Di1 IPCP: Remove route to 10.161.3.1
*Oct 16 18:24:16.359: Di1 IPCP: Remove default route thru 10.161.3.1
*Oct 16 18:24:16.395: Vi2 LCP: Event[DOWN] State[Closing to Initial]
*Oct 16 18:24:16.399: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to down
*Oct 16 18:24:16.415: Vi2 PPP: Phase is DOWN
*Oct 16 18:24:16.443: %DIALER-6-UNBIND: Interface Vi2 unbound from profile Di1
*Oct 16 18:24:16.483: %LINK-3-UPDOWN: Interface Virtual-Access2, changed state to down

My secondary WAN link is still up, we can confirm that:

CPE-4#sh caller
                                                  Active    Idle
  Line           User               Service       Time      Time
  con 0          -                  TTY           01:12:07  00:00:00
  Vi3            LNS1               PPPoE         01:03:54  00:00:08

But now my default route is gone

CPE-4# sh ip route
Gateway of last resort is not set

It's definitely not in there with CEF:

CPE-4# sh ip cef
Prefix               Next Hop             Interface
0.0.0.0/0            no route

I can still ping my next hop and everybody knows the way to get there:

CPE-4#ping 10.161.3.1
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/54/68 ms

CPE-4#sh ip route

Gateway of last resort is not set

      10.0.0.0/32 is subnetted, 1 subnets
C        10.161.3.1 is directly connected, Dialer1

CPE-4(config-if)#do sh ip cef
Prefix               Next Hop             Interface
0.0.0.0/0            no route
0.0.0.0/8            drop
0.0.0.0/32           receive
10.161.3.1/32        attached             Dialer1

To get the default route back I need to bounce Vi3 (2nd WAN Link) and re-negoatie PPP and get the IPCP route that way.

Here are the dialer configs. Pretty standard stuff:

interface Dialer1
 mtu 1492
 ip address negotiated
 ip load-sharing per-packet
 encapsulation ppp
 dialer pool 1
 dialer idle-timeout 0
 dialer persistent
 dialer-group 1
 ppp chap hostname testuser@gns3.homelab
 ppp chap password 0 password
 ppp ipcp route default
 no cdp enable

interface Dialer2
 mtu 1492
 ip address negotiated
 ip load-sharing per-packet
 encapsulation ppp
 dialer pool 2
 dialer idle-timeout 0
 dialer persistent
 dialer-group 2
 ppp chap hostname testuser2@gns3.homelab
 ppp chap password 0 password
 ppp ipcp route default
 no cdp enable

Followed by the WAN links:

interface FastEthernet0/0
 description *** WAN 1 ***
 no ip address
 duplex full
 pppoe enable group global
 pppoe-client dial-pool-number 1

interface FastEthernet0/1
 description *** WAN 2 ***
 no ip address
 duplex full
 pppoe enable group global
 pppoe-client dial-pool-number 2

For reference, the debug logs from PPP and CEF when shutting down one interface: https://hastebin.com/wolobujeyo.yaml - in this instance Dialer2 is the interface that is currently active. Dialer1 was shutdown.

Most notable logs are below, which state that Dialer2 has taken over the default route:

*Oct 16 20:48:10.019: FIBpath: {mod} [v4-ah-10.161.3.1-Di2 67561A40(1)] Linked path to oce IP adj out of Dialer2 675AFA60
*Oct 16 20:48:10.023: FIBpathlist_ifnums: [1/0:v4-ah-10.161.3.1-Di2 67561474(1)]inserted Dialer2(10) 0.0.0.0
*Oct 16 20:48:10.115: FIBpathlist_ifnums: [1/0:v4-rcrsv-10.161.3.1 675613D4(2)] updated Dialer2(10) 0.0.0.0

Same thing happens if I clear the VAI too:

CPE-4#clear int vi3

*Oct 17 11:31:22.839: %DIALER-6-UNBIND: Interface Vi3 unbound from profile Di1
*Oct 17 11:31:22.923: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to down
*Oct 17 11:31:22.935: %LINK-3-UPDOWN: Interface Virtual-Access3, changed state to down

.

CPE-4#sh ip cef
Prefix               Next Hop             Interface
0.0.0.0/0            no route
0.0.0.0/8            drop
0.0.0.0/32           receive
10.161.3.1/32        attached             Dialer2

.

CPE-4#sh ip route
Gateway of last resort is not set

VAI comes back up:

*Oct 17 11:31:45.063: %DIALER-6-BIND: Interface Vi3 bound to profile Di1
*Oct 17 11:31:45.079: %LINK-3-UPDOWN: Interface Virtual-Access3, changed state to up
*Oct 17 11:31:46.259: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to up

Route is back

CPE-4#sh ip cef
Prefix               Next Hop             Interface
0.0.0.0/0            10.161.3.1           Dialer1
                     10.161.3.1           Dialer2
0.0.0.0/8            drop
0.0.0.0/32           receive
10.161.3.1/32        attached             Dialer1
                     attached             Dialer2

LNS also gives me two IP addresses as it's two separate user accounts logging in via RADIUS:

CPE-4#sh ip int br
Interface              IP-Address      OK? Method Status                Protocol
Dialer1                172.16.100.1    YES IPCP   up                    up
Dialer2                172.16.100.2    YES IPCP   up                    up

I can confirm further that it's PPP/IPCP doing something as I can add static routes in manually as well and they do not clear when PPP goes down:

CPE-4#sh ip cef
Prefix               Next Hop             Interface
0.0.0.0/0            attached             Dialer1
                     attached             Dialer2
                     10.161.3.1           Dialer1
                     10.161.3.1           Dialer2
0.0.0.0/8            drop
0.0.0.0/32           receive
10.161.3.1/32        attached             Dialer1
                     attached             Dialer2

After clearing one VAI:

CPE-4#sh ip cef
Prefix               Next Hop             Interface
0.0.0.0/0            attached             Dialer1
                     attached             Dialer2
0.0.0.0/8            drop
0.0.0.0/32           receive
10.161.3.1/32        attached             Dialer1

So IPCP or PPP is definitely the issue here.

I have tried this across multiple firmware versions and also very different hardware. This replication is done on a Cisco 7206VXR NPE-400 15.2(4)S3. I have also replicated the issue on Cisco 877, 887, 891, 927-4P, 1921, 1911, 2911, ISR 4331, ISR 1111-4P/8P. Firmware versions from 15.2 - 15.7.

Due to the range that I'm able to replicate this on I do not think it is a bug, more than I'm missing something in how IPCP works.

Is anybody able to explain why my default route is getting removed when one WAN link goes down, despite there being two - both with valid routes and CEF entries?

3
  • "I have 2 circuits coming in that I am doing per-packet load balancing across for my WAN links" Per-packet load balancing is not really what you want. It causes all types of problems, from slowing TCP to killing real-time protocols due to out-of-order packet delivery. TCP can recover, but it does slow it down. Real-time protocols cannot tolerate much, and it can simply render them useless. – Ron Maupin Oct 17 '20 at 16:25
  • In practice, it rarely creates any serious problems when both links are mostly identical. (same speed, latency, etc. If there's a 10ms difference between links, then you'll have a mess, but MLPPP would as well.) – Ricky Oct 17 '20 at 21:37
  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you can post and accept your own answer. – Ron Maupin Dec 17 '20 at 22:35
1

In Cisco's mind, this should be handled by multi-link PPP, but having had serious problems with their quarter-baked MLPPP code, we ("I") switched the problem customers to CEF per-packet.

In this mode, you can't rely on IPCP to setup the pseudo-ECMP. The route IPCP creates does not include the interface, so when duplicate entries are added -- 0/0 via 10.161.3.1 -- there's no way to know which to remove, thus both are deleted. If you don't rely on IPCP, this will work just fine. HOWEVER, you have to always connect to the same RAS. (or route to interfaces, which is never a good practice.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.