1

I'm trying to setup Out Of Band management on a MLNX-OS based Mellanox switch

It seems by default the mgmt0-1 interfaces are in the same VRF as all other interfaces, and I assume I have to create a new VRF and put the management interface in it.

However this MLNX-OS documentation I found has only a small paragraph on VRF and I can't figure out how to set the mgmt interface into a specific VRF. I suppose is it through a statement like:

   vrf definition management rd 10.12.15.73:0

but it doesn't work and I don't understand how the "rd" part works.

The switch is a MSN2700 with MLNX-OS version 3.6.3508. The switch current configuration is:

##
## Running database "initial"
## Generated at 2009/10/25 02:22:29 +0000
## Hostname: SW2
##

##
## Running-config temporary prefix mode setting
##
no cli default prefix-modes enable

##
## Interface Split configuration
##
   interface ethernet 1/1 module-type qsfp-split-4 force
   interface ethernet 1/3 module-type qsfp-split-4 force
   interface ethernet 1/5 module-type qsfp-split-4 force
   interface ethernet 1/25 module-type qsfp-split-2 force
   interface ethernet 1/26 module-type qsfp-split-2 force

##
## Interface Ethernet configuration
##
   interface ethernet 1/1/1-1/1/4 speed 10000 force
   interface ethernet 1/3/1-1/3/4 speed 10000 force
   interface ethernet 1/5/1-1/5/4 speed 10000 force
   interface ethernet 1/17-1/18 speed 40000 force
   interface ethernet 1/32 speed 10000 force
   interface ethernet 1/1/1-1/1/4 mtu 9000 force
   interface ethernet 1/3/1-1/3/4 mtu 9000 force
   interface ethernet 1/5/1-1/5/4 mtu 9000 force
   interface ethernet 1/17-1/18 mtu 9000 force
   interface ethernet 1/25/1-1/25/2 mtu 9000 force
   interface ethernet 1/26/1-1/26/2 mtu 9000 force
   interface ethernet 1/17 switchport mode trunk
   interface ethernet 1/18 switchport mode trunk
   interface ethernet 1/25/1 switchport mode trunk
   interface ethernet 1/25/2 switchport mode trunk
   interface ethernet 1/26/1 switchport mode trunk
   interface ethernet 1/26/2 switchport mode trunk
   interface ethernet 1/32 description Uplink

##
## LAG configuration
##
   lacp

##
## VLAN configuration
##
   vlan 1047
   vlan 2014-2015
   vlan 3012
   vlan 1047 name "VDI-Client"
   vlan 2014 name "STOR1"
   vlan 2015 name "STOR2"
   vlan 3012 name "VMNet"
   interface ethernet 1/25/1 switchport trunk allowed-vlan none
   interface ethernet 1/25/2 switchport trunk allowed-vlan none
   interface ethernet 1/26/1 switchport trunk allowed-vlan none
   interface ethernet 1/26/2 switchport trunk allowed-vlan none
   interface ethernet 1/1/1 switchport access vlan 2014
   interface ethernet 1/1/2 switchport access vlan 2014
   interface ethernet 1/1/3 switchport access vlan 3012
   interface ethernet 1/1/4 switchport access vlan 3012
   interface ethernet 1/3/1 switchport access vlan 2015
   interface ethernet 1/3/2 switchport access vlan 2015
   interface ethernet 1/3/3 switchport access vlan 2015
   interface ethernet 1/3/4 switchport access vlan 2015
   interface ethernet 1/5/1 switchport access vlan 2014
   interface ethernet 1/5/2 switchport access vlan 2015
   interface ethernet 1/5/3 switchport access vlan 2014
   interface ethernet 1/5/4 switchport access vlan 2015
   interface ethernet 1/25/1 switchport trunk allowed-vlan add 1047
   interface ethernet 1/25/1 switchport trunk allowed-vlan add 2014-2015
   interface ethernet 1/25/1 switchport trunk allowed-vlan add 3012
   interface ethernet 1/25/1 switchport trunk allowed-vlan remove 1
   interface ethernet 1/25/2 switchport trunk allowed-vlan add 1047
   interface ethernet 1/25/2 switchport trunk allowed-vlan add 2014-2015
   interface ethernet 1/25/2 switchport trunk allowed-vlan add 3012
   interface ethernet 1/25/2 switchport trunk allowed-vlan remove 1
   interface ethernet 1/26/1 switchport trunk allowed-vlan add 1047
   interface ethernet 1/26/1 switchport trunk allowed-vlan add 2014-2015
   interface ethernet 1/26/1 switchport trunk allowed-vlan add 3012
   interface ethernet 1/26/1 switchport trunk allowed-vlan remove 1
   interface ethernet 1/26/2 switchport trunk allowed-vlan add 1047
   interface ethernet 1/26/2 switchport trunk allowed-vlan add 2014-2015
   interface ethernet 1/26/2 switchport trunk allowed-vlan add 3012
   interface ethernet 1/26/2 switchport trunk allowed-vlan remove 1

##
## L3 configuration
##
   vrf definition management
   vrf definition management rd 10.12.15.73:0
   ip routing vrf default
   interface vlan 1
   interface vlan 10
   interface vlan 1047
   interface vlan 3012
   interface vlan 1 ip address 10.12.99.43 255.255.255.248
   interface vlan 1047 ip address 10.11.47.254 255.255.255.0
   interface vlan 3012 ip address 10.10.47.254 255.255.255.0
   ip route 10.0.0.0 /8 10.12.99.41

##
## DHCP relay configuration
##
   ip dhcp relay instance 1 vrf default
   ip dhcp relay instance 1 address 10.10.47.1
   ip dhcp relay instance 1 address 10.10.47.2
   interface vlan 1047 ip dhcp relay instance 1
   interface vlan 3012 ip dhcp relay instance 1

##
## Network interface configuration
##
no interface mgmt0 dhcp
   interface mgmt0 ip address 10.12.15.73 /24

##
## Other IP configuration
##
   ip route 0.0.0.0 0.0.0.0 10.12.15.254
   hostname SW2
   ip domain-list mydomain.tld
   ip name-server xxx.xxx.xxx.xxx

##
## Other IPv6 configuration
##
no ipv6 enable

##
## Local user account configuration
##
   <stripped>

##
## AAA remote server configuration
##
<stripped>

##
## SNMP configuration
##
 <stripped>

##
## Network management configuration
##
# web proxy auth basic password ********

##
## X.509 certificates configuration
##
#
# Certificate name system-self-signed, ID *****************
# (public-cert config omitted since private-key config is hidden)

##
## Persistent prefix mode setting
##
cli default prefix-modes enable

So what is the proper way to configure OOB management on MLNX-OS?
Is it trough a VRF?
If so how can I assign mgmt0 to a specific VRF instance?

EDIT: opened a case with Mellanox, they confirm that by default the management interfaces are in default VRF (thus not OOB) and that I have to put them in a separate VRF. Still have to sort out with them how to do it.

Improve this question
4
  • Have you checked the command vrf forwarding vrf_name under interface configuration mode? Not sure if it works for Mgmt interface or not, but it is mentioned to work under routed ethernet, VLAN and Loopback interfaces (Page 980 in the documentation).
    – Hung Tran
    Commented Nov 13, 2017 at 18:42
  • Yes I checked it, but there's no VRF command available on mgmt interface. My guess is that I must use the default VRF for management and put all other traffic in a new VRF. Still waiting for an update on the case; but will definitely try this approach.
    – JFL
    Commented Nov 13, 2017 at 19:45
  • Yes, it is worth a try.
    – Hung Tran
    Commented Nov 13, 2017 at 19:47
  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you can provide and accept your own answer.
    – Ron Maupin ♦
    Commented Feb 19, 2018 at 20:03

5 Answers 5

Reset to default
2

I contacted Mellanox customer support with the same problem. There is a command to create a dedicated VRF for the management interface. This is easy to do - from the CLI, just run the command

vrf definition mgmt

See here for the Mellanox documentation

Improve this answer
0

It seems by default the mgmt0-1 interfaces are in the same VRF as all other interfaces...

I do not see it mentioned in the provided document. And according to it, the setup for OOB management on Mellanox seems straightforward.

We need to configure an IP address on mgmt0 (or mgmt1) and put it in OOB Management network. There is no reason to put it in a VRF instance. This document would give you more information.

Updated answer:

Looks like you misunderstood the point here.

OBB Management traffic should hit Mgmt Interface over L2 network/segment, not by routing through the device itself.

enter image description here

Improve this answer
6
  • yes it shoudl be, but the problem is that it's not OOB at all, but inband. It is in the same instance as the other switch configuration, since the switch act as layer 3 with an IP route for traffic routed for the switch, the management interface use this route and is not reachable
    – JFL
    Commented Nov 10, 2017 at 10:44
  • @JFL This is written in their document Mellanox switches support out-of-band (OOB) dedicated interfaces (e.g. mgmt0, mgmt1) and in-band dedicated interfaces and In-band management is a management path passing through the data ports. In-band management can be created over one of the VLANs in the systems. In this case, if you access this box via Mgmt interfaces, it is OOB.
    – Hung Tran
    Commented Nov 10, 2017 at 11:18
  • @JFL some cases I know, you have a Terminal host/server resides in OOB Management network (probably L2 only segment). If you want to access devices in your network through OOB, you have to jump into this host/server first then access those devices.
    – Hung Tran
    Commented Nov 10, 2017 at 11:32
  • The mgmt interface was reachable from a remote network when there was no other route installed. As soon as I added a route for normal switch operation the interface was no more reachable.This cleary show that it is in no way isolated. I perfectly know what is written in the documentation, but it is not how the switch behave in practice.
    – JFL
    Commented Nov 10, 2017 at 13:51
  • I got your point, but this is not what I wan to achieve. Requiring a pure layer 2 management network for an enterprise class switch is nonsense. And the fact I can actually reach the mgtm interface through routing when there's a route for it in the main routing table prove that is it not isolated from the main forwarding plane. That's NOT OOB.
    – JFL
    Commented Nov 10, 2017 at 14:37
0

I have just rolled out 2 of them, the way I have deployed is utilising the OOB physical interface, this is attached to the management plane which according to many manufacturer this interface is isolated on the board level from the data plane, which gives you total isolation. I have setup a management network with a dedicated management switch where all the production switch are connected to it via their own management interface. I found this solution optimal for our DC.

Comments and suggestion are welcome to review the design.

Improve this answer
0

After a troubleshooting session for another issue with Mellanox support engineers, they confirmed that the only way to isolate the Management interface from the routing plane is to leave it alone in the default VRF and put all other config in a separate VRF.

The management interface cannot bu put in another VRF. Current version of Onyx OS (3.9.0612) allows 6 VRF in addition to the default one.

Mellanox is aware of the issue, they have many complaints about it and it should be addressed in a future release.

Improve this answer
0

I can confirm that with Onyx 3.9.1014, mgmt0 is always in the default vrf and there's no way to move it. Thus, you must follow Stuggi's comment to move everything else to a different VRF if you want separation.

With Onyx 3.9.3220, mgmt0 can be placed in a mgmt vrf

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.