I've been stuck on this one in my lab for a while. Any help is appreciated.
Here's my CPE edge topology:
https://i.sstatic.net/a1R6U.png
Take the below scenario:
I have 2 circuits coming in that I am doing per-packet load balancing across for my WAN links (Fa0/0 & Fa0/1). I get my default route installed with IPCP after PPP negotiation for both links has completed. Looks normal as below:
CPE-4# sh ip route
S* 0.0.0.0/0 [1/0] via 10.161.3.1
CEF shows I have both paths there:
CPE-4# sh ip cef
Prefix Next Hop Interface
0.0.0.0/0 10.161.3.1 Dialer1
10.161.3.1 Dialer2
Both of my sessions are up:
CPE-4#sh caller
Active Idle
Line User Service Time Time
con 0 - TTY 01:12:07 00:00:00
Vi2 LNS1 PPPoE 00:58:27 00:00:01
Vi3 LNS1 PPPoE 01:03:54 00:00:08
I shut down one of the wan links (Fa0/0) and I get the expected PPP debug logs:
*Oct 16 18:23:19.279: %LINK-5-CHANGED: Interface FastEthernet0/0, changed state to administratively down
*Oct 16 18:23:20.279: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to down
*Oct 16 18:24:16.279: Vi2 PPP: Missed 5 keepalives, taking LCP down
*Oct 16 18:24:16.279: Vi2 PPP DISC: Missed too many keepalives
*Oct 16 18:24:16.283: PPP: NET STOP send to AAA.
*Oct 16 18:24:16.299: Vi2 IPCP: Event[DOWN] State[Open to Starting]
*Oct 16 18:24:16.299: Vi2 IPCP: Event[CLOSE] State[Starting to Initial]
*Oct 16 18:24:16.303: Vi2 LCP: O TERMREQ [Open] id 3 len 4
*Oct 16 18:24:16.307: Vi2 LCP: Event[CLOSE] State[Open to Closing]
*Oct 16 18:24:16.307: Vi2 PPP: Phase is TERMINATING
*Oct 16 18:24:16.359: Di1 Deleted neighbor route from AVL tree: topoid 0, address 10.161.3.1
*Oct 16 18:24:16.359: Di1 IPCP: Remove route to 10.161.3.1
*Oct 16 18:24:16.359: Di1 IPCP: Remove default route thru 10.161.3.1
*Oct 16 18:24:16.395: Vi2 LCP: Event[DOWN] State[Closing to Initial]
*Oct 16 18:24:16.399: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access2, changed state to down
*Oct 16 18:24:16.415: Vi2 PPP: Phase is DOWN
*Oct 16 18:24:16.443: %DIALER-6-UNBIND: Interface Vi2 unbound from profile Di1
*Oct 16 18:24:16.483: %LINK-3-UPDOWN: Interface Virtual-Access2, changed state to down
My secondary WAN link is still up, we can confirm that:
CPE-4#sh caller
Active Idle
Line User Service Time Time
con 0 - TTY 01:12:07 00:00:00
Vi3 LNS1 PPPoE 01:03:54 00:00:08
But now my default route is gone
CPE-4# sh ip route
Gateway of last resort is not set
It's definitely not in there with CEF:
CPE-4# sh ip cef
Prefix Next Hop Interface
0.0.0.0/0 no route
I can still ping my next hop and everybody knows the way to get there:
CPE-4#ping 10.161.3.1
Success rate is 100 percent (5/5), round-trip min/avg/max = 44/54/68 ms
CPE-4#sh ip route
Gateway of last resort is not set
10.0.0.0/32 is subnetted, 1 subnets
C 10.161.3.1 is directly connected, Dialer1
CPE-4(config-if)#do sh ip cef
Prefix Next Hop Interface
0.0.0.0/0 no route
0.0.0.0/8 drop
0.0.0.0/32 receive
10.161.3.1/32 attached Dialer1
To get the default route back I need to bounce Vi3 (2nd WAN Link) and re-negoatie PPP and get the IPCP route that way.
Here are the dialer configs. Pretty standard stuff:
interface Dialer1
mtu 1492
ip address negotiated
ip load-sharing per-packet
encapsulation ppp
dialer pool 1
dialer idle-timeout 0
dialer persistent
dialer-group 1
ppp chap hostname [email protected]
ppp chap password 0 password
ppp ipcp route default
no cdp enable
interface Dialer2
mtu 1492
ip address negotiated
ip load-sharing per-packet
encapsulation ppp
dialer pool 2
dialer idle-timeout 0
dialer persistent
dialer-group 2
ppp chap hostname [email protected]
ppp chap password 0 password
ppp ipcp route default
no cdp enable
Followed by the WAN links:
interface FastEthernet0/0
description *** WAN 1 ***
no ip address
duplex full
pppoe enable group global
pppoe-client dial-pool-number 1
interface FastEthernet0/1
description *** WAN 2 ***
no ip address
duplex full
pppoe enable group global
pppoe-client dial-pool-number 2
For reference, the debug logs from PPP and CEF when shutting down one interface: https://hastebin.com/wolobujeyo.yaml - in this instance Dialer2 is the interface that is currently active. Dialer1 was shutdown.
Most notable logs are below, which state that Dialer2 has taken over the default route:
*Oct 16 20:48:10.019: FIBpath: {mod} [v4-ah-10.161.3.1-Di2 67561A40(1)] Linked path to oce IP adj out of Dialer2 675AFA60
*Oct 16 20:48:10.023: FIBpathlist_ifnums: [1/0:v4-ah-10.161.3.1-Di2 67561474(1)]inserted Dialer2(10) 0.0.0.0
*Oct 16 20:48:10.115: FIBpathlist_ifnums: [1/0:v4-rcrsv-10.161.3.1 675613D4(2)] updated Dialer2(10) 0.0.0.0
Same thing happens if I clear the VAI too:
CPE-4#clear int vi3
*Oct 17 11:31:22.839: %DIALER-6-UNBIND: Interface Vi3 unbound from profile Di1
*Oct 17 11:31:22.923: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to down
*Oct 17 11:31:22.935: %LINK-3-UPDOWN: Interface Virtual-Access3, changed state to down
.
CPE-4#sh ip cef
Prefix Next Hop Interface
0.0.0.0/0 no route
0.0.0.0/8 drop
0.0.0.0/32 receive
10.161.3.1/32 attached Dialer2
.
CPE-4#sh ip route
Gateway of last resort is not set
VAI comes back up:
*Oct 17 11:31:45.063: %DIALER-6-BIND: Interface Vi3 bound to profile Di1
*Oct 17 11:31:45.079: %LINK-3-UPDOWN: Interface Virtual-Access3, changed state to up
*Oct 17 11:31:46.259: %LINEPROTO-5-UPDOWN: Line protocol on Interface Virtual-Access3, changed state to up
Route is back
CPE-4#sh ip cef
Prefix Next Hop Interface
0.0.0.0/0 10.161.3.1 Dialer1
10.161.3.1 Dialer2
0.0.0.0/8 drop
0.0.0.0/32 receive
10.161.3.1/32 attached Dialer1
attached Dialer2
LNS also gives me two IP addresses as it's two separate user accounts logging in via RADIUS:
CPE-4#sh ip int br
Interface IP-Address OK? Method Status Protocol
Dialer1 172.16.100.1 YES IPCP up up
Dialer2 172.16.100.2 YES IPCP up up
I can confirm further that it's PPP/IPCP doing something as I can add static routes in manually as well and they do not clear when PPP goes down:
CPE-4#sh ip cef
Prefix Next Hop Interface
0.0.0.0/0 attached Dialer1
attached Dialer2
10.161.3.1 Dialer1
10.161.3.1 Dialer2
0.0.0.0/8 drop
0.0.0.0/32 receive
10.161.3.1/32 attached Dialer1
attached Dialer2
After clearing one VAI:
CPE-4#sh ip cef
Prefix Next Hop Interface
0.0.0.0/0 attached Dialer1
attached Dialer2
0.0.0.0/8 drop
0.0.0.0/32 receive
10.161.3.1/32 attached Dialer1
So IPCP or PPP is definitely the issue here.
I have tried this across multiple firmware versions and also very different hardware. This replication is done on a Cisco 7206VXR NPE-400 15.2(4)S3. I have also replicated the issue on Cisco 877, 887, 891, 927-4P, 1921, 1911, 2911, ISR 4331, ISR 1111-4P/8P. Firmware versions from 15.2 - 15.7.
Due to the range that I'm able to replicate this on I do not think it is a bug, more than I'm missing something in how IPCP works.
Is anybody able to explain why my default route is getting removed when one WAN link goes down, despite there being two - both with valid routes and CEF entries?
