2

I am trying to replace our existing Virtual Chassis way of doing mutli-chassis LAG with EVPN multi-homing using VXLAN. My understanding is we should be able to replace any LAG we have with evpn-multihoming.

The reason is I want to get rid of the single point of failure of virtual-chass' shared route engine (we've had some issues) and this is part of a stepping stone to moving to a spine-leaf network, using VXLAN. So it makes sense to leverage this now.

I am trying to achieve what should be a fairly basic setup - two QFX 5110 switches with an ESI LAG, and EVPN to share MACs between the two independent switches.

The Problem

Fundamentally, my problem is I can't get remote MAC learning to work over EVPN.

  • As soon as I add any VXLAN VNI config to my existing VLAN100, my ethernet-switching table goes completely empty, with no MACs showing. And no local bridging of packets. Connectivity is completely lost.
{master:0}
root@core-01> show ethernet-switching table

{master:0}
root@core-01>
  • If I remove the VXLAN config from VLAN100, MACs show in the ethernet-switching table and I get local bridging to work, but not remote.
    • I get some routes shared over BGP EVPN, but I only get type 1, 3 or 4 routes, never type 2.
    • My understanding is I need type 2 routes to learn the actual MACs.
    • My vtep interface shows as a member of vlan 100, but never listed as a forwarding option for any of the MACs.

Setup Details

  • Using OSPF to learn loopbacks
  • Using iBGP as my underlay.
  • Only using a single EVPN instance.
  • Not using any routing-instances/VRFs - just the default global table for now.
  • Using Junos 17.3R1.10

EVPN Multihoming setup

Troubleshooting Output

Not sure exactly what is most relevant here, but I've included some basic info below

root@core-01> show route receive-protocol bgp 10.10.216.170

inet.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)

:vxlan.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)

bgp.evpn.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
  Prefix                  Nexthop              MED     Lclpref    AS path
  3:10.10.216.170:100::100::10.10.216.170/248 IM
*                         10.10.216.170               100        I
  4:10.10.216.170:0::650000100113044501:10.10.216.170/296 ES
*                         10.10.216.170               100        I

default-switch.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
  Prefix                  Nexthop              MED     Lclpref    AS path
  3:10.10.216.170:100::100::10.10.216.170/248 IM
*                         10.10.216.170               100        I

__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
  Prefix                  Nexthop              MED     Lclpref    AS path
  4:10.10.216.170:0::650000100113044501:10.10.216.170/296 ES
*                         10.10.216.170               100        I



root@core-01> show evpn instance
                            Intfs       IRB intfs         MH      MAC addresses
Instance                    Total   Up  Total   Up  Nbrs  ESIs    Local  Remote
__default_evpn__                                       1
default-switch                114    2      0    0     1     1        0       0

{master:0}

{master:0}
root@core-01> show vlans

Routing instance        VLAN name             Tag          Interfaces
default-switch          VLAN100               100
                                                           ae1.0*
                                                           ge-0/0/2.0*
                                                           vtep.32769*

EDIT: A reboot has helped slightly

After rebooting both switches, I can now see my ethernet switching table populate and I am seeing EVPN type 2 routes, so MACs are being learned it seems. In the process of rebooting I've lost access to one of the two switches so I need to sort that out and continue troubleshooting.

Unfortunately even though things are looking better, I still don't have network connectivity.

{master:0}
root@core-02> show ethernet-switching table

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)


Ethernet switching table : 5 entries, 5 learned
Routing instance : default-switch
   Vlan                MAC                 MAC      Logical                Active
   name                address             flags    interface              source
   VLAN100             00:15:fa:b6:76:41   D        vtep.32769             10.10.216.169
   VLAN100             00:15:fa:bc:18:01   D        vtep.32769             10.10.216.169
   VLAN100             00:1d:aa:56:4b:b8   D        vtep.32769             10.10.216.169
   VLAN100             c0:67:af:0a:9d:c0   D        vtep.32769             10.10.216.169
   VLAN100             e8:65:49:d5:43:40   D        vtep.32769             10.10.216.169

And I'm seeing EVPN type 2 routes coming from the other switch:

root@core-02> show route receive-protocol bgp 10.10.216.169

inet.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)

:vxlan.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)

bgp.evpn.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
  Prefix                  Nexthop              MED     Lclpref    AS path
  2:10.10.216.169:100::100::00:15:fa:b6:76:41/304 MAC/IP
*                         10.10.216.169               100        I
  2:10.10.216.169:100::100::00:15:fa:bc:18:01/304 MAC/IP
*                         10.10.216.169               100        I
  2:10.10.216.169:100::100::00:1d:aa:56:4b:b8/304 MAC/IP
*                         10.10.216.169               100        I
  2:10.10.216.169:100::100::c0:42:d0:03:b1:01/304 MAC/IP
*                         10.10.216.169               100        I
  2:10.10.216.169:100::100::c0:67:af:0a:9d:c0/304 MAC/IP
*                         10.10.216.169               100        I
  2:10.10.216.169:100::100::e8:65:49:d5:43:40/304 MAC/IP
*                         10.10.216.169               100        I
  3:10.10.216.169:100::100::10.10.216.169/248 IM
*                         10.10.216.169               100        I

default-switch.evpn.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
  Prefix                  Nexthop              MED     Lclpref    AS path
  2:10.10.216.169:100::100::00:15:fa:b6:76:41/304 MAC/IP
*                         10.10.216.169               100        I
  2:10.10.216.169:100::100::00:15:fa:bc:18:01/304 MAC/IP
*                         10.10.216.169               100        I
  2:10.10.216.169:100::100::00:1d:aa:56:4b:b8/304 MAC/IP
*                         10.10.216.169               100        I
  2:10.10.216.169:100::100::c0:42:d0:03:b1:01/304 MAC/IP
*                         10.10.216.169               100        I
  2:10.10.216.169:100::100::c0:67:af:0a:9d:c0/304 MAC/IP
*                         10.10.216.169               100        I
  2:10.10.216.169:100::100::e8:65:49:d5:43:40/304 MAC/IP
*                         10.10.216.169               100        I
  3:10.10.216.169:100::100::10.10.216.169/248 IM
*                         10.10.216.169               100        I

UPDATE 2: JunOS update fixed LACP Bug

  • As above, after rebooting both switches, I started seeing type 2 routes and started seeing MACs learned over the VTEP interface in my ethernet-switching table.

  • The next issue was no MAC learning on either LAG interface to the server. This was because despite ge-0/0/1 being up on both switches, traffic refused to pass and as a consequence LACP would not bring the ae1 interface up on either switch, or on the server bond0. The LACP stats showed Port Disabled which is due to no begin event occurring within LACP: https://www.juniper.net/documentation/en_US/junos/topics/reference/command-summary/show-lacp-interfaces.html

  • I found that if I removed the LACP+LAG config (I didn't try just turning off LACP active mode) my configuring a separate VLAN 100 access port to use instead, I got connectivity to the server right away.

  • I know my LACP config on ae1 was working fine prior to enabling VXLAN+EVPN, so I put this down to another possible bug in JunOS, in the same way that I had to reboot the switches previously.

The Fix

Based on this I decided just to upgrade to the current JTAC recommended version.

This has worked. I upgraded:

  • From: JunOS 17.3R1
  • To: JunOS 20.2R2.11

Immediately after doing this on each switch, ae1 came up on that switch, and MAC learning on ae1 began working, and I have full reachability.

I can't confirm that a JunOS upgrade would have avoided the need to restart after enabling VXLAN VNI mapping on my VLAN100, but given I haven't read anything about having to restart, I'd say an upgrade would have fixed this too.

Here's my working EVPN routes:

{master:0}
root@core-02> show route protocol evpn

inet.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)

:vxlan.inet.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)

inet6.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)

bgp.evpn.0: 15 destinations, 15 routes (15 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1:10.10.216.170:0::650000100113044501::FFFF:FFFF/192 AD/ESI
                   *[EVPN/170] 02:38:08
                       Indirect
1:10.10.216.170:100::650000100113044501::0/192 AD/EVI
                   *[EVPN/170] 02:38:09
                       Indirect
3:10.10.216.170:100::100::10.10.216.170/248 IM
                   *[EVPN/170] 02:38:07
                       Indirect
4:10.10.216.170:0::650000100113044501:10.10.216.170/296 ES
                   *[EVPN/170] 02:38:09
                       Indirect

default-switch.evpn.0: 12 destinations, 12 routes (12 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1:10.10.216.170:100::650000100113044501::0/192 AD/EVI
                   *[EVPN/170] 02:38:09
                       Indirect
3:10.10.216.170:100::100::10.10.216.170/248 IM
                   *[EVPN/170] 02:38:07
                       Indirect

__default_evpn__.evpn.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1:10.10.216.170:0::650000100113044501::FFFF:FFFF/192 AD/ESI
                   *[EVPN/170] 02:38:08
                       Indirect
4:10.10.216.170:0::650000100113044501:10.10.216.170/296 ES
                   *[EVPN/170] 02:38:09
                       Indirect

{master:0}
root@core-02>

Configuration

The two main sources I've used to perform the configuration are:

From core-01:

root@core-01# show vlans
VLAN100 {
    vlan-id 100;
    vxlan {
        vni 100;
        ingress-node-replication;
    }
}
default {
    vlan-id 1;
    l3-interface irb.0;
}

root@core-01# show protocols
bgp {
    group internal-peers {
        type internal;
        description "iBGP Peers";
        local-address 10.10.216.169;
        family evpn {
            signaling;
        }
        authentication-key  ## SECRET-DATA
        peer-as 65000;
        neighbor 10.10.216.170;
    }
}
ospf {
    reference-bandwidth 100g;
    area 0.0.0.0 {
        interface et-0/0/49.0 {
            authentication {
                md5 66 key  ## SECRET-DATA
            }
        }
        interface et-0/0/51.0 {
            authentication {
                md5 66 key  ## SECRET-DATA
            }
        }
        interface lo0.0 {
            passive;
        }
    }
}
evpn {
    vni-options {
        vni 100 {
            vrf-target export target:1:100; ## Warning: 'export' is deprecated
        }
    }
    encapsulation vxlan;
    multicast-mode ingress-replication;
    extended-vni-list all;
}

policy-options {
    policy-statement underlay-import {
        term VXLAN100 {
            from community VXLAN100;
            then accept;
        }
    }
    community VXLAN100 members target:1:100;
}
switch-options {
    vtep-source-interface lo0.0;
    route-distinguisher 10.10.216.169:100;
    vrf-import underlay-import;
    vrf-target target:65000:100;
}


root@core-01# show interfaces ge-0/0/1
description "Test - Server NIC#1";
ether-options {
    802.3ad ae1;
}

root@core-01# show interfaces ae1
description "Test - Server";
mtu 9216;
esi {
    00:65:00:00:10:01:13:04:45:01;
    all-active;
}
aggregated-ether-options {
    lacp {
        active;
        periodic fast;
        system-id 65:00:00:10:01:00;
    }
}
unit 0 {
    family ethernet-switching {
        interface-mode access;
        vlan {
            members VLAN100;
        }
    }
}
Improve this question
15
  • 2
    Are you sure the ## Warning: requires 'bgp' license warning isn't your problem here? Without that, you'll never be able to distribute mac addresses via EVPN.
    – Teun Vink ♦
    Nov 23, 2020 at 8:38
  • @TeunVink You might be right - I was wondering about that. But A) In my experience BGP generally still works despite the warning, and B) If you look at some CLI output I just added, I am receiving some EVPN route-types. I just think think I am missing some things. I will probably figure out how to re-add the licenses while I see what others have to say!
    – Geekman
    Nov 23, 2020 at 8:41
  • @TeunVink FWIW I have just added the licenses to both, no more warnings, but still see nothing in ethernet switching table, either locally or remotely while I have VXLAN configured.
    – Geekman
    Nov 23, 2020 at 9:19
  • @Geekman Correct, it's an honor based license so it'll work but you wouldn't get support if you called JTAC. I'll post an answer shortly.
    – Jordan Head
    Nov 23, 2020 at 13:43
  • 2
    @JordanHead Thanks for confirming. FYI I rebooted both switches and after doing this, I got my ethernet-switching table populating. Doh! Still not working though, and I locked myself out of one device, but I'll continue troubleshooting in 8 hours or so.
    – Geekman
    Nov 23, 2020 at 14:12

1 Answer 1

Reset to default
3

TLDR: Updated JunOS to the latest recommended version for QFX 5110 as per this JTAC document: https://kb.juniper.net/InfoCenter/index?page=content&id=KB21476&actp=METADATA#qfx_series

Junos 18.4R2-S5 and subsequent service releases / 20.2R2

This is how I fixed the initial empty MAC table bug:

  • After rebooting both switches, I started seeing type 2 routes and started seeing MACs learned over the VTEP interface in my ethernet-switching table. Only had to reboot once for this bug to disappear.

This is how I fixed the LAG/LACP bug:

  • I found that if I removed the LACP+LAG config (I didn't try just turning off LACP active mode) my configuring a separate VLAN 100 access port to use instead, I got connectivity to the server right away.

  • To fix this even with LACP enabled, I upgraded from: JunOS 17.3R1 to: JunOS 20.2R2.11

Immediately after doing this on each switch, ae1 came up on that switch, and MAC learning on ae1 began working, and I have full reachability.

See original post for more details on working EVPN routes+MAC learning after applying this fix.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.