Network Design Improvement Advice

Question

We are thinking about redesigning our network. The topology is a star and is comprised of several similar POPs. The entire network is based on Cisco devices. The core router is a Cisco 7600 and two 4900 distribution switches. On each POP, we have an access switch, which is a ME3400 (L3 capable) and CMTS 7200 with a NPE-G2 Processor Card.

The 7200s are connected by trunks to the core router (don't like it) and Load-balance is done in L3 by OSPF (one area). I can't implement multiple area OSPF because the CPE public subnets are too difficult to summarize and the public range is split over several POPs without hierarchy, so this would be inefficient. The actual design was done several years ago when EtherChannel was uncommon.

So far, I've been thinking:

• Implementation of EtherChannel between the access switches and CMTS.
• Implementation of a L3 boundary between core network and POP.

I've not found a solution for the redundancy on the core beside L3 OSPF. Suggestion are welcome.

I've also been thinking about changing the public interconnection IP to private ones, due to security concern.

---

Edit 1 - Additional Info

• There are 8 PoPs and we don't plan to expand any further than 15.
• We have split a public subnet range of /19 over the PoPs without caring about summarization or hierarchy. Our criteria to split and provide subnet is the number of end-CPE. For example, if a PoP is estimated with 1500 CPE connected we allocate 2048 public address. Due to the growth, we didn't make contiguous distribution so we have several small ranges over the PoP. Some PoPs have a 700 CPE and we provide 4 range non-contiguous subnets. Smooth migration is a hot-topic because we are on production so the change of one PoP shouldn't affect the other.
• No firewall filtering is planned on the feed - we reach 3Gb of traffic but some ACL to restrict access to our internal infrastructure is on analysis.
• We are connected through BGP to our ISP and we have a fail-over link with a partner. Here the change/modification are only intend to our core network, I'm not focused to perform any change on this part.

Edit 2 - IOS on device

• Core Router : Cisco CISCO7604 - c7600rsp72043_rp Software (c7600rsp72043_rp-IPSERVICESK9-M), Version 12.2(33)SRE3
• Distribuition Switch : cisco WS-C4900M - Catalyst 4500 L3 Switch Software (cat4500e-IPBASEK9-M), Version 15.0(2)SG2
• Access Switch : cisco ME-3400G-12CS-A - ME340x Software (ME340x-METROIPACCESSK9-M), Version 12.2(60)EZ3
• CMTS 7200 NPE-G2 : cisco uBR7225VXR - 7200 Software (UBR7200P-JK9SU2-M), Version 12.2(33)SCG5 (soon to be upgraded to SCG6)

Edit 3 - Improve Topology Picture

I change the picture to improve the understanding of our network. We have a joint venture with a company where we provide each other Core Redundancy as showed in the image. In normal operation, we have only 1 ISP but if our core router go down we have the redundant feed through our partner. I add on picture the type of subnet and distribution, we can find on CPE side.

Thanks for your suggestions and advice.

Answer 1

You have an elegant physical setup, but logically, it’s deficient in a lot of ways. You’re relying on layer 2 for all of your connections, which doesn’t provide any real failover your setup should provide. Luckily, you aren’t wasting any bandwidth with STP because you don’t have any redundant links; VLAN 100 and VLAN 200 only have a single way out, if I’m reading this correctly.

I can't implement multiple area OSPF because the CPE public subnets are too difficult to summarize and the public range is split over several POPs without hierarchy, so this would be inefficient.

I don’t even think you need to go as far as setting up multiple areas, the segment your concerned with is fairly small. Cisco recommends you stay under 50 routers per area, though, most will agree that you can exceed that amount by quite a significant margin and still be considered healthy. I have about 80 routers per area with no issues; Ron has even more than that without any hiccups.

• Implementation of EtherChannel between the access switches and CMTS.

This all hinges off what happens higher up in your network. If you’re significantly oversubscribing your distribution to core links, then you may not see any real benefit in increasing your bandwidth at the access layer. The same holds true if you’re doing this with your access to distribution.

Again, as most have mentioned in the comments, implementing OSPF would be pretty easy. If you were utilizing a layer 3 protocol, you would share that load across equal/unequal cost links, too. Perhaps a redundant link between your distribution switches would allow each access switch a redundant path out. As it stands, if one of your access links goes down, you lose an entire subnet (VLAN 100 or VLAN 200).