Leaf-Spine network without routing

Question

If I have a leaf-spine network with all the servers on the same subnet, say 10.10.0.0/16, do I need a router (or l3) since everything is on the same network and the network is flat?

Answer 1

do I need a router (or l3) since everything is on the same network and the network is flat

Technically, no - if all nodes reside in that flat network.

However, such a large subnet is not good practice due to limited scaling and the potential propagation of any L2 problems.

The core-distribution links should always be routed (L3) instead of switched. In current practice, the distribution-access links are increasingly becoming L3 as well which provides even better scalability. Most often, L3 switches are used between core and distribution, and between distribution and access.

With a smaller network (and subnet) you would use a collapsed core topology where the access switches connect to the core directly. Again, good practice is to route those links.

Using routed instead of bridged links can improve total scalability (when it's not practical to propagate each MAC address throughout the whole network), total throughput (in contrast to STP blocking redundant links you can use equal-cost multi-path routing) and resilience (a complex network can fail over quicker on a link-state routing protocol than by R/MSTP).

Shortest Path Bridging heavily borrows from ECMP and scales significantly better in an L2 scenario. However, the industry hasn't quite caught on yet with standard switches, so L3 with ECMP is currently a better and more future-proof way.

All in all, L3 vs L2 depends on the level of scalability you have in mind. A network with a few hundred nodes that is not growing (are you sure?) usually works well in a flat L2 design.

Answer 2

Thechnically, you do not need L3 capable switches or routing to build a network that has spine-and-leaf characteristics, but you'll need multichassis link aggregation capability.

With a MLAG capable platform (such as the Cisco Nexus series with VPC), you can build a leaf and spine topology that has the main features commonly expected, using nothing but classic switching of packets based on L2 headers (well, Ethernet, usually), all without running an ecapsulation overlay (like VXLAN) and without the need for a control plane (like BGP EVPN):

• all leaves connected to all spines
• all links forwarding (no STP blocked links)
• growth in overall capacity by adding more leaves
• growth in overall bandwidth by adding more (parallel) leaf-to-spine links
• consistently low number of forwarding hops from any leaf to any other

There's a few catches:

• growth in overall bandwidth by adding more spines can be somewhat of a challenge, als MLAG setups are generally restricted to pairs of devices.
• spines are an MLAG pair, and not quite as independent as with a L3 spine and leaf
• there's some vendor-lock-in with the given MLAG implementation.
• the fundamental behaviour and operation of a (large) switching domain is still the same: You should keep a form of spanning-tree running, you should set your STP root correctly, you'll see MAC address learning, flushing of MAC address tables upon topology changes (adding a new leaf), unknown unicast flooding, and you'll have to deal with multicast (IGMP Snooping).

[Addon 1]

A Layer3/routed spine-and-leaf design usually comes with immense stability and resilience, and can tolerate (with some performance degradations) faults on a spines (can survive with just one single spine), multiple faults on leaf/spine links etc.

With an MLAG based L2-design, all of this is depends on the capabilites and resilience features that the vendor built into their MLAG feature. Cisco alone hast (at least) three technologies that allow MLAG (StackWise, VSS, vPC). Be sure to understand in depth how your vendor and platform handle separation of data and control planes in their MLAG products, and how failures impact the given platform.

[/Addon 1]

Answer 3

A true, 'Leaf Spine' topology is a routed design and each leaf should be its own L2 domain (or multiples of). You can run a leaf spine 'looking' architecture as a single L2, however it will only look like leaf spine, not behave like it.