1

TLDR; routed port on vPC switch 1 needs to be accessible from vPC switch 2.

Our data center has a pair of Nexus 5596T with L3 card being used as the core. These are vPC pair, HSRP members, all vlan SVIs live here. All vlans are in a VRF. All routes are static at this location, since it's basically a quad 0 to one side and an RFC-1918 to another side. I have the vPC heartbeat on mgmt0 in its own VRF:

vpc domain 1
  system-priority 5000
  peer-keepalive destination 172.31.255.1
  delay restore 150
  peer-gateway

There is an etherchannel group between the two for the peer link:

interface port-channel1
  description N5K Peer Link
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

I had a need to add a routed port to one of the 5k's, which I placed in the VRF as so:

int eth1/6
vrf member VRF-name
no switchport
ip addr 1.1.1.1/30
no shut

I am capable of pinging the remote device (1.1.1.2) from the VRF on the switch hosting the routed port (5k-1), however from 5k-2 I am not able to ping and 1.1.1.1 doesn't appear in the routing table due to the static routing config. I don't see a clear way to add the route to 5k-2 other than simply pointing it to a local IP on 5k-1 such as:

ip route 1.1.1.0/30 [IP on 5k-1]

What am I missing here? Because we are vPC to L2 switches and servers, I might end up with traffic to 1.1.1.2 getting black-holed depending on which switch receives that traffic.

config:

!Command: show running-config
!Time: Sat Nov 12 13:34:07 2016

version 7.1(3)N1(2)
hostname Sac_N5596-1

no feature telnet
cfs eth distribute
feature pim
feature eigrp
feature pbr
feature udld
feature interface-vlan
feature hsrp
feature lacp
feature dhcp
feature vpc
feature lldp
feature vtp
feature fex

logging level feature-mgr 0

ssh key rsa 2048
no ip domain-lookup
ip access-list SDWAN
  10 permit ip any 10.9.0.0/16
ip access-list SNMP-ACL
  1 permit ip 10.254.225.140/32 any
class-map type qos match-all class-iscsi
  match cos 4
class-map type queuing class-iscsi
  match qos-group 3
policy-map type qos policy-qos
  class class-iscsi
    set qos-group 3
policy-map type queuing policy-queuing
  class type queuing class-fcoe
    bandwidth percent 0
  class type queuing class-iscsi
    bandwidth percent 95
  class type queuing class-default
    bandwidth percent 5
class-map type network-qos class-iscsi
  match qos-group 3
policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216
    multicast-optimize
policy-map type network-qos policy-nq
  class type network-qos class-iscsi
    mtu 9216
    pause no-drop
system qos
  service-policy type network-qos jumbo
vtp mode transparent
vtp domain SAC

ntp peer 10.254.1.254 use-vrf internal
ntp server 63.145.169.3 use-vrf internal
ntp server 69.36.224.15 use-vrf internal
ntp source-interface Vlan1

vlan 1
vlan 30
  name UC-DEVICES
vlan 100
  name TBD_Removed_100
vlan 105
  name iDRAC_MGMT
vlan 110
  name User_10
vlan 160
  name SERVER
vlan 170
  name Database
vlan 171
  name Application
vlan 172
  name Web
vlan 200
  name iSCSI
vlan 205
  name TBD_Removed_205
vlan 225
  name BACKUPS
vlan 250
  name DMZ
vlan 350
  name LB-DMZ
vlan 360
  name LB-SERVER
vlan 370
  name LB-DataBase
vlan 371
  name LB-Application
vlan 372
  name LB-Web
vlan 800
  name Network_Endpoints
vlan 900
  name MPLS
vlan 998
  name Dummy-VLAN
spanning-tree vlan 1-998 priority 24576
route-map SDWAN permit 10
  match ip address SDWAN
  set ip next-hop verify-availability 1.1.1.1
service dhcp
ip dhcp relay
vrf context internal
  ip route 0.0.0.0/0 10.254.1.1
  ip route 10.0.0.0/12 10.99.99.1
  ip route 10.5.0.0/16 10.254.1.1
  ip route 10.16.0.0/16 10.254.1.1
  ip route 10.17.0.0/16 10.254.1.1
  ip route 10.249.0.0/16 10.99.99.1
  ip route 10.254.254.0/24 10.254.1.1
  ip route 172.16.60.0/24 10.254.1.20
  ip route 192.4.1.0/24 10.99.99.1
  ip route 192.168.0.0/16 10.99.99.1
  ip route 192.168.5.0/24 10.254.1.1
  ip route 192.168.13.0/24 10.254.1.1
  ip route 192.168.195.0/24 10.99.99.1
  ip route 192.168.250.0/24 10.99.99.1
vrf context management
vpc domain 1
  system-priority 5000
  peer-keepalive destination 172.31.255.2
  delay restore 150


interface Vlan1
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.1.253/24
  hsrp version 2
  hsrp 1
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.254.1.2

interface Vlan30
  description UC Devices VLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.30.253/24
  hsrp version 2
  hsrp 30
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.254.30.1

interface Vlan105
  description iDRAC/MGMT
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.5.253/24
  hsrp version 2
  hsrp 105
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.254.5.1
  ip dhcp relay address 10.254.60.50

interface Vlan110
  description User Vlan
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.10.253/24
  hsrp version 2
  hsrp 110
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.254.10.1
  ip dhcp relay address 10.254.60.50

interface Vlan160
  description Server VLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.60.253/24
  hsrp version 2
  hsrp 160
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.254.60.1

interface Vlan170
  description DataBase
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.170.253/24
  hsrp version 2
  hsrp 170
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.254.170.1

interface Vlan171
  description Application
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.171.253/24
  hsrp version 2
  hsrp 171
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.254.171.1

interface Vlan172
  description Web
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.172.253/24
  hsrp version 2
  hsrp 172
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.254.172.1

interface Vlan200
  description iSCSI Traffic
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.100.253/24
  hsrp version 2
  hsrp 200
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.254.100.1

interface Vlan225
  description Backup Solution vLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.225.253/24
  hsrp version 2
  hsrp 225
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.254.225.1

interface Vlan360
  description Load Balancer Servers VLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 172.16.60.253/24
  hsrp version 2
  hsrp 360
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 172.16.60.1

interface Vlan370
  description Load Balancer DataBase VLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 172.16.170.253/24
  hsrp version 2
  hsrp 370
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 172.16.170.1

interface Vlan371
  description Load Balancer Application VLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 172.16.171.253/24
  hsrp version 2
  hsrp 371
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 172.16.171.1

interface Vlan372
  description Load Balancer Web VLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 172.16.172.253/24
  hsrp version 2
  hsrp 372
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 172.16.172.1

interface Vlan800
  description Network_Endpoints
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.0.253/24
  hsrp version 2
  hsrp 800
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.254.0.1

interface Vlan900
  description MPLS
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.99.99.253/24
  hsrp version 2
  hsrp 900
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.99.99.2

interface port-channel1
  description Nexus to Nexus
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface port-channel5
  description Trunk to DevStation_3750X
  switchport mode trunk
  speed 10000

interface port-channel20
  description To 3850 Stack
  switchport mode trunk
  vpc 20

interface Ethernet1/1
  description Firewall LAN Handoff - Primary

interface Ethernet1/2
  description Uplink to MPLS router - Primary
  switchport access vlan 900

interface Ethernet1/3
  description Firewall DMZ
  switchport access vlan 250

interface Ethernet1/4
  description Uplink to SilverPeak
  switchport access vlan 800
  spanning-tree port type edge

interface Ethernet1/5
  description SilverPeak Web Mgmt
  switchport access vlan 160
  spanning-tree port type edge

interface Ethernet1/6
  description SilverPeak LAN0
  no switchport
  vrf member internal
  ip address 1.1.1.1/30


interface Ethernet1/29
  description N5K Peer Link
  switchport mode trunk
  channel-group 1 mode active

interface Ethernet1/30
  description N5K Peer Link
  switchport mode trunk
  channel-group 1 mode active

interface Ethernet1/31
  description N5K Peer Link
  switchport mode trunk
  channel-group 1 mode active

interface Ethernet1/32
  description N5K Peer Link
  switchport mode trunk
  channel-group 1 mode active


interface mgmt0
  vrf member management
  ip address 172.31.255.1/30
clock timezone PST -8 0
clock summer-time PDT 2 Sun Mar 02:00 1 Sun Nov 02:00 60
line console
line vty
boot kickstart bootflash:/n5000-uk9-kickstart.7.1.3.N1.2.bin
boot system bootflash:/n5000-uk9.7.1.3.N1.2.bin

Nexus 2:

!Command: show running-config
!Time: Sat Nov 12 13:34:32 2016

version 7.1(3)N1(2)
hostname Sac_N5596-2

no feature telnet
cfs eth distribute
feature pim
feature eigrp
feature pbr
feature udld
feature interface-vlan
feature hsrp
feature lacp
feature dhcp
feature vpc
feature lldp
feature vtp
feature fex


no ip domain-lookup
ip access-list SDWAN
  10 permit ip any 10.9.0.0/16
ip access-list SNMP-ACL
  1 permit ip 10.254.225.140/32 any
class-map type qos match-all class-iscsi
  match cos 4
class-map type queuing class-iscsi
  match qos-group 3
policy-map type qos policy-qos
  class class-iscsi
    set qos-group 3
policy-map type queuing policy-queuing
  class type queuing class-fcoe
    bandwidth percent 0
  class type queuing class-iscsi
    bandwidth percent 95
  class type queuing class-default
    bandwidth percent 5
class-map type network-qos class-iscsi
  match qos-group 3
policy-map type network-qos jumbo
  class type network-qos class-default
    mtu 9216
    multicast-optimize
policy-map type network-qos policy-nq
  class type network-qos class-iscsi
    mtu 9216
    pause no-drop
system qos
  service-policy type network-qos jumbo
vtp mode transparent
vtp domain SAC

ntp peer 10.254.2.253
ntp server 63.145.169.3
ntp server 69.36.224.15
ntp source-interface Vlan1

vlan 1
vlan 30
  name UC-DEVICES
vlan 100
  name TBD_Removed_100
vlan 105
  name iDRAC_MGMT
vlan 110
  name User_10
vlan 160
  name SERVER
vlan 170
  name DATABASE
vlan 171
  name Application
vlan 172
  name Web
vlan 200
  name iSCSI
vlan 205
  name TBD_Removed_205
vlan 225
  name BACKUPS
vlan 250
  name DMZ
vlan 350
  name LB-DMZ
vlan 360
  name LB-SERVER
vlan 370
  name LB-DataBase
vlan 371
  name LB-Application
vlan 372
  name LB-Web
vlan 800
  name Network_Endpoints
vlan 900
  name MPLS
vlan 998
  name Dummy-VLAN
spanning-tree vlan 1-998 priority 28672
route-map SDWAN permit 10
  match ip address SDWAN

service dhcp
ip dhcp relay
vrf context internal
  ip route 0.0.0.0/0 10.254.1.1
  ip route 10.0.0.0/12 10.99.99.1
  ip route 10.5.0.0/16 10.254.1.1
  ip route 10.16.0.0/16 10.254.1.1
  ip route 10.17.0.0/16 10.254.1.1
  ip route 10.249.0.0/16 10.99.99.1
  ip route 10.254.254.0/24 10.254.1.1
  ip route 172.16.60.0/24 10.254.1.20
  ip route 192.168.0.0/16 10.99.99.1
  ip route 192.168.5.0/24 10.254.1.1
  ip route 192.168.13.0/24 10.254.1.1
  ip route 192.168.195.0/24 10.99.99.1
  ip route 192.168.250.0/24 10.99.99.1
  ip route 192.254.1.0/24 10.99.99.1
vrf context management
vpc domain 1
  system-priority 5000
  peer-keepalive destination 172.31.255.1
  delay restore 150
  peer-gateway


interface Vlan1
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.1.254/24
  hsrp version 2
  hsrp 1
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 10.254.1.2

interface Vlan30
  description UC Devices VLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.30.254/24
  hsrp version 2
  hsrp 30
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 10.254.30.1

interface Vlan105
  description iDRAC/MGMT
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.5.254/24
  hsrp version 2
  hsrp 105
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 10.254.5.1
  ip dhcp relay address 10.254.60.50

interface Vlan110
  description User Vlan
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.10.254/24
  hsrp version 2
  hsrp 110
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 10.254.10.1
  ip dhcp relay address 10.254.60.50

interface Vlan160
  description Server VLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.60.254/24
  hsrp version 2
  hsrp 160
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 10.254.60.1

interface Vlan170
  description DataBase
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.170.254/24
  hsrp version 2
  hsrp 170
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 10.254.170.1

interface Vlan171
  description Application
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.171.254/24
  hsrp version 2
  hsrp 171
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 10.254.171.1

interface Vlan172
  description Web
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.172.254/24
  hsrp version 2
  hsrp 172
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 10.254.172.1

interface Vlan200
  description iSCSI Traffic
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.100.254/24
  hsrp version 2
  hsrp 200
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 10.254.100.1

interface Vlan225
  description Backup Solution vLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.225.254/24
  hsrp version 2
  hsrp 225
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 10.254.225.1

interface Vlan360
  description Load Balancer Servers VLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 172.16.60.254/24
  hsrp version 2
  hsrp 360
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 172.16.60.1

interface Vlan370
  description Load Balancer DataBase VLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 172.16.170.254/24
  hsrp version 2
  hsrp 370
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 172.16.170.1

interface Vlan371
  description Load Balancer Application VLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 172.16.171.254/24
  hsrp version 2
  hsrp 371
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 172.16.171.1

interface Vlan372
  description Load Balancer Web VLAN
  no shutdown
  vrf member internal
  no ip redirects
  ip address 172.16.172.254/24
  hsrp version 2
  hsrp 372
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 172.16.172.1

interface Vlan800
  description Network_Endpoints
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.254.0.254/24
  hsrp version 2
  hsrp 800
    preempt delay minimum 240
    priority 150
    timers 1 3
    ip 10.254.0.1

interface Vlan900
  description MPLS
  no shutdown
  vrf member internal
  no ip redirects
  ip address 10.99.99.254/24
  hsrp version 2
  hsrp 900
    preempt delay minimum 240
    priority 130
    timers 1 3
    ip 10.99.99.2

interface port-channel1
  description N5K Peer Link
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link


interface Ethernet1/29
  description N5K Peer Link
  switchport mode trunk
  channel-group 1 mode active

interface Ethernet1/30
  description N5K Peer Link
  switchport mode trunk
  channel-group 1 mode active

interface Ethernet1/31
  description N5K Peer Link
  switchport mode trunk
  channel-group 1 mode active

interface Ethernet1/32
  description N5K Peer Link
  switchport mode trunk
  channel-group 1 mode active


interface mgmt0
  vrf member management
  ip address 172.31.255.2/30
clock timezone PST -8 0
clock summer-time PDT 2 Sun Mar 02:00 1 Sun Nov 02:00 60
line console
line vty
boot kickstart bootflash:/n5000-uk9-kickstart.7.1.3.N1.2.bin
boot system bootflash:/n5000-uk9.7.1.3.N1.2.bin
Improve this question
6
  • Can you include the complete config of both 5k please?
    – hertitu
    Nov 12, 2016 at 20:20
  • edited post to include config.
    – Rex Sheffield
    Nov 12, 2016 at 21:55
  • To make it more readable you can use the 'code formatting'. Anyway I think I understand the setup now, but not sure what you are asking... as you say ip route 1.1.1.0/30 [IP on 5k-1] seems to be all that is needed. Optionally you could create a "routing vlan" in which you only put an SVI of each switch, and use that vlan to route the traffic over (in other words use the SVI of switch1 in that vlan, as next hop for your route)
    – hertitu
    Nov 12, 2016 at 22:11
  • OK, cool. I wanted to make sure that I wasn't missing something here.
    – Rex Sheffield
    Nov 12, 2016 at 22:18
  • Don't use the hsrp addresses (.1) because they could be owned by switch2 at some point. Use one of the real addresses (.253). I didn't check the entire config tbh but if all vlans are configure similarly then yes it shouldn't matter which one you use. FWIW Personally I would consider it more 'clean' to create a separate routing vlan.
    – hertitu
    Nov 12, 2016 at 22:20

1 Answer 1

Reset to default
1

resolution was to ip route 1.1.1.0/30 [IP on 5k-1]

Improve this answer
1
  • You should accept your answer so that the question doesn't keep popping up forever, looking for an answer.
    – Ron Maupin ♦
    Dec 17, 2016 at 0:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.