How to design a good (small) OSPF network

Question

Ok, lets try again. It seems Im having problem asking the right questions so lets take it from the beginning. I work at a HPC center at a university. We have a number of small (<300 nodes) clusters and a few storage systems. To our site we get a statically routed /22 network.

Today we have the whole /22 in a single broadcast domain and we have decided to redesign and renumber the network for better scalability and performance. To do this we have bought two new L3 switches (Dell s6000-on) to use as our backbone. Connected to these we will have the aggregation switches of the individual clusters (which are HP 5400) and it will look something like this:

Now we want to renumber the hosts with private IP-adresses (one net per cluster) and use OSPF internally, to reach the internet we still use a default route to some central university router.

One important thing is that we still need to have some hosts with public IPs reachable from the internet (a few login nodes per cluster and some other services).

1. My first question has to do with the links between Core and aggregation. I could use MC-LAG (Dell VLT), Stacking or just OSPF everywhere.

   • With MC-LAG I can have all links on the same vlan and its very easy to have other vlans span the whole network
   • But it feels to me that ptp-links and OSPF eclb is more robust and scalable.
   • Or I can use switch stacking which gives me a virtual chassi and get rid of the need for MC-LAG. But I don't really trust the stability of such a technique.

2. The machines on private IP-adresses still need internet, so I want to NAT them out. But our new core switches doesn't support NAT (as far as I know) so we are setting up a NAT-router on the side. Is there some smart way in OSPF to redirect traffic from privates IPs to the nat-router? Or should I use PBR?

3. Do you have any solution for the public IPs? They are sort of scattered throughout the datacenter so there is not a single access point for these machines. It would be very helpful to have a large vlan with a /24 or so that spanned the whole network, see question 1. But maybe the best way is to just split up the network into smaller ones and route them with OSPF as needed.

4. Some of you say its a bad idea to have both public and private IPs in the same OSPF area. If this is so, what do you think we should do instead?

I hope this makes it more clear what Im after. Please feel free to ask for clarifications

Answer 1

1. My first question has to do with the links between Core and aggregation. I could use MC-LAG or pure OSPF. With MC-LAG I can have all links on the same vlan and its very easy to have other vlans span the whole network, but on the other hand, it feels to me that ptp-links and OSPF eclb is more robust and scalable.

Go for ospf L3 links, better stability, and scales better, easier to monitor and trouble shoot.

2. The machines on private IP-adresses still need internet, so I want to NAT them out. But our new core switches doesn't support NAT (as far as I know) so we are setting up a NAT-router on the side. Is there some smart way in OSPF to redirect traffic from privates IPs to the nat-router? Or should I use PBR?

Opt1: PBR - is messy, and hard to maintain, single nat device is single point of failure.

Opt2: central NAT on outgoing university router

Opt3: get two routers/firewalls/devices that can sit between your core switch and the University network that can do the NAT for you

3. Do you have any solution for the public IPs? They are sort of scattered throughout the datacenter so there is not a single access point for these machines. It would be very helpful to have a large vlan with a /24 or so that spanned the whole network, see question

With the L3 recommended option, there can be smaller public IP vlan/subnets in each area.

take your public /22 and subnet with /28(64net x 14hosts) or /26(16Net x 64hosts) each, then have a vlan in each cluster for public IP machines. Also easier to filter/protect in future

But maybe the best way is to just split up the network into smaller ones and route them with OSPF as needed.

Agreed

4. Some of you say its a bad idea to have both public and private IPs in the same OSPF area. If this is so, what do you think we should do instead?

This comes from a corporate mindset where security is high on the priority list if not on top.

In corporates NO external access is allowed to internal machines, all public facing/accessible machines are in a DMZ behind FW controlling inside and outside access.

Access to internal machines would be provided to staff through a 2 factor authenticated VPN link in most cases, no need for public IP's on internal machines.

If you opt for the recommended NAT capable devices between core and rest of Campus network, you can do one-to-one nat there as required and all internal hosts can have private ip's. This also provides the possibility for better security as only specific services can be allowed inbound to hosts.