Cannot Load Internal Website over MPLS

Question

We have an MPLS set up between two subnets. Everything (that is, I have RDP both directions as well as file-sharing) seems to be working correctly except for two things, which may be related.

• Windows computers do not populate the "Network" with devices from the other subnet.
• From the remote computer, we cannot load our internal website located on the host network.

WINS is enabled and working, and both computers know who is the DNS server and who is the WINS server.

The webserver is at 192.168.1.20 (Windows Server 2003) and the computer (Windows 7) in the remote subnet is at 192.168.2.249. File-sharing and RDP work both ways.

So, the host subnet is 192.168.0.0/23 and the remote subnet is 192.168.2.0/23. Each of the Windstream routers have two ports - one for MPLS and one for internet. Right now, the internet port at the remote is not connected. The internet port on the host router runs through our firewall router before entering the host network, but the MPLS port at the host plugs directly into the switch-trunk.

The two Windstream Routers each have an MPLS port:

• 192.168.1.2 = Host MPLS
• 192.168.2.2 = Remote MPLS

The Windstream routers also each have an open internet port to which I have attached a Firewall Router for filtering the internet, making the internet gateways:

• 192.168.1.1 = Host Gateway
• 192.168.2.1 = Remote Gateway

I read here that I needed to list the subnets in Active Directory Sites and Services, so I've created the two subnets as members of the one site.

For my testing, the firewalls are turned off on both computers plus the webserver.

The ISP (Windstream) confirms that MTU is set to 1500, and in their testing, their pings are always sent with a size of 1500.

So what can I try to get these two issues solved?

Here is a map:

[update] When I run Wireshark on the webserver and watch the request for the webpage, I see alot of retransmissions on the http calls. Here is the report:

It looks like http traffic from the remote to the host is what is getting retransmissions. But I've got no equipment that can block it. The firewalls are all off.

So, I can telnet to the webserver from a machine on the same subnet, but I can't telnet to it from a machine on the remote subnet - it reports:

c:\>telnet 192.168.1.20 80
Connecting To 192.168.1.20...Could not open connection to the host, on port 80: Connect failed

Trace-Route from the webserver to the remote computer:

Trace-Route from the remote computer to the webserver:

[update] I enabled IIS on the remote laptop, and I am able to pull that webpage from a computer at the host location. This has always been the case, however, in my tests. The http traffic, therefore, is only one way.

Answer 1

The two Windstream Routers each have an MPLS port:

• 192.168.1.2 = Host MPLS
• 192.168.2.2 = Remote MPLS

The Windstream routers also each have an open internet port to which I have attached a Firewall Router for filtering the internet, making the internet gateways:

• 192.168.1.1 = Host Gateway
• 192.168.2.1 = Remote Gateway

Problem Summary

Your problem is that you've got multiple routers on the same subnet:

• The Internet gateway at 192.168.1.1
• The Windstream MPLS router at 192.168.1.2

This is a faulty design; I completely understand that it "seems like" there is nothing wrong with this, but as you're discovering, this is a hard way to build the network.

Following up on our chat discussion, the exact problem is that SAINTJOSEPH's TCP SYN packets are delivered correctly; however, stateful inspection on your PaloAlto firewall drops SAINTSERVIUS' TCP SYN-ACK response, due to asymmetric routing in your environment. This is illustrated in the two following diagrams.

TCP SYN from SAINTJOSEPH to SAINTSERVIUS:

Your PaloAlto firewall drops the TCP SYN-ACK from SAINTSERVIUS to SAINTJOSEPH:

This tracert makes it very clear that SAINTSERVIUS defaults through 192.168.1.1 (the PaloAlto firewall):

Long Term Solutions

You should only have a single router next-hop for every subnet; however, you currently have two. This results in the drops shown in your wireshark screen capture (which indicates that TCP SYN-ACK packets never reach Windstream's MPLS network).

These are two long-term solutions which we discussed... I'm also including the quick hack we did to validate that the problem is stateful packet drops on the PaloAltos. I am assuming you are using multiple interfaces on the PaloAlto.

• Option A: Keep the PaloAlto firewalls inline with your interoffice connections (more maintenance)
• Option B: Move the PaloAlto firewalls in front of the internet connections (less maintenance, but also less comfortable)
• Option C: Quick Windows static routing hack

Better Design, Option A (MPLS re-addressing required)

This is a another way to lay out the subnet for SAINTSERVIUS (retaining your numbering scheme with /23 subnets, although it's not required...). This option keeps interoffice routing on the PaloAlto firewalls, which feels more familiar to you right now.

This is a long-term solution. You would have to work with Windstream to readdress your infrastructure. This is a lot of work. In my opinion, keeping the firewalls in the middle of your interoffice traffic is less preferable.

Better Design, Option B (MPLS re-addressing required)

This is a another way to lay out the subnet for SAINTSERVIUS (retaining your numbering scheme with /23 subnets, although it's not required...). This option offloads interoffice LAN routing to your PowerConnect switches, and relies on the PaloAlto firewalls to protect against internet threats.

This is a long-term solution. You would have to work with Windstream to readdress your infrastructure. This is a lot of work, but it offers the advantage of not having to deal with firewalls for your interoffice communication.

Suboptimal Workaround, Option C (what you used after our chat)

Open a cmd.exe window and add this route on SAINTSERVIUS as the Administrator:

route ADD 192.168.2.0 MASK 255.255.254.0 192.168.1.2

Closing remarks

I should mention that you are one of the few people who followed through with sufficient details about your question When you started, we didn't have enough details to answer the question. Now, the issues are very clear; thank you for putting the time / effort into documenting the problem well.

Personal note: If you would like an electronic copies of the diagrams as SVG / Inkscape format, please feel free to contact me at my personal email (listed in my user profile).