2

I am having an issue with an offsite location accessing a website that is hosted on 3 servers behind a Coyotoepoint Load-Balancer, at our Corporate Office. We currently have well over 500 locations accessing this site with no issues, but this one location is causing problems.

All our locations terminate at a firewall at our Corporate Office over an IPsec VPN. The website is served up by our Load-Balancer using a cluster IP. The problem location can ping the cluster IP with no issue, but when they try to access the site it fails.

From packet captures I'm seeing a lot of TCP Out-Of-Order errors and a TCP Dup ACK. On the working sites I don't see any of those.

** Update ** Bellow is a ASCII Diagram of how my remote sites access my load-balancers


+---------------------CORPORATE OFFICE----------------------------------------------------+
|                                                                                         |
|                            192.168.15.5/16 +--------------+ 192.168.11.1/30             |
|                            Inside Interface|  Cisco ASA   | DMZ Interface               |
|                                           ++--------------+<----------+                 |
|   +--------------------+<-----------------+                           +---------------+ |
|   |                    |                                              |  Meraki MX90  | |
|   |  Coyotepoint       |                                              |192.168.11.2/30| |
|   | 192.168.10.140/16  |                                              |Gateway: .1    | |
|   | Gateway: 40.2      |                                              +---------------+ |
|   |                    |    +------------------------------------+               ^      |
|   +--------------------+    |          Cisco Router              |               |      |
|                        |    |         192.168.40.2/16            |               |      |
|                        |    |Default Route to Remote Sites: .15.5|               |      |
|  +------------------+  |    +------------------------------------+               |      |
|  |      Web1        |  |                                                         |      |
|  |192.168.10.145/16 |  |                                                         |      |
|  |Gateway: 40.2     |  |                                                         |      |
|  +------------------+<-+   +-----------------------------------------------------|------+
|  +------------------+  |   |                                                     |
|  |      Web2        |  |   |                                                     |
|  |192.168.10.146/16 |  |   |                                                     |
|  |Gateway: 40.2     |  |   |                                                 IPsec VPN
|  +------------------+<-+   |                                                     |
|  +------------------+  |   |                                     +               |
|  |      Web3        |  |   |                                     +---------------+
|  |192.168.10.147/16 |  |   |                                     | Remote Sites  |
|  |Gateway: 40.2     |  |   |                                     +---------------+
|  +------------------+<-+   |
+----------------------------+
3
  • Has the load balanced site ever worked across the problematic tunnel? When did the problem start? Have you pinged using DF and full MTU sized packets? – This Nov 23 '13 at 12:54
  • The load balancer has never worked with the problem site. I can bypass the load balancer and the site will load directly from one of the servers. I did a "ping -f -l <IP of Cluster>. At working sites I'm able to ping at 1332 Bytes, but at the site I'm having issues with, I can't go over 1310 bytes. That isn't just limited to the Cluster IP either. If I ping anything at my corporate office 1310 bytes is the max I can do across the tunnel. All working sites can ping max 1404 across the tunnel and 1472 outside, problem site is 1310/1374. Could a MTU setting on the modem cause this issue? – Luke Nov 24 '13 at 20:51
  • We are using Meraki MX60 firewalls to build dynamic IPsec tunnels to our head end Meraki MX90 at corporate. These devices don't really have configs in the traditional sense. They are cloud managed from a web based dashboard. There are no configurable MTU settings on a MX60 appliance. Since the problem site also has a lower maximum accessing servers outside the tunnel it makes me think the cable modem may not be setup to 1500 MTU. I might also want to open a ticket with Meraki. They can see much deeper into the device than I can through the dashboard. – Luke Nov 24 '13 at 22:03
2

Updated Answer

Originally I thought it was upgrading the firmware of my load-balancers that fixed my issues. That turns out to be coincidence. The actual problem involved a setting on the load-balancer itself.

The Web servers in our cluster have to know the internal IP of the remote site in order to know what location is trying to access the page, then display information that pertains only to that site. In order to do that I have to enable a setting call Spoofing on the Coyotepoint.

By default Web servers in a load-balanced cluster only get requests from the load-balancer's cluster IP. When they are operating like that all traffic from the server going to the hosts and vice versa is going through the load-balancer. When Spoofing is turned on, the Web servers are seeing the Source Address as the remote site's internal IP. Instead of routing their response back through the load-balancer they just send the responses out their default gateway.

Coyotepoint's best practices is to set the Web servers default-gateway to the cluster IP when spoofing is turned on. I have changed all the default gateway of my web servers to the cluster ip, and everything is working. Coyotepoint is convinced that is what was causing my TCP Out-Of-Order errors. Also Coytoepoint stated there has been significant networking changes on their 10.2.0a code, but didn't give me anything specific.

Updated network diagram:


+---------------------CORPORATE OFFICE----------------------------------------------------+
|                                                                                         |
|                            192.168.15.5/16 +--------------+ 192.168.11.1/30             |
|                            Inside Interface|  Cisco ASA   | DMZ Interface               |
|                                           ++--------------+<----------+                 |
|   +--------------------+<-----------------+                           +---------------+ |
|   |                    |                                              |  Meraki MX90  | |
|   |  Coyotepoint       |                                              |192.168.11.2/30| |
|   | 192.168.10.140/16  |                                              |Gateway: .1    | |
|   | Gateway: 40.2      |                                              +---------------+ |
|   |                    |    +------------------------------------+               ^      |
|   +--------------------+    |          Cisco Router              |               |      |
|                        |    |         192.168.40.2/16            |               |      |
|                        |    |Default Route to Remote Sites: .15.5|               |      |
|  +------------------+  |    +------------------------------------+               |      |
|  |      Web1        |  |                                                         |      |
|  |192.168.10.145/16 |  |                                                         |      |
|  |Gateway: 10.140   |  |                                                         |      |
|  +------------------+<-+   +-----------------------------------------------------|------+
|  +------------------+  |   |                                                     |
|  |      Web2        |  |   |                                                     |
|  |192.168.10.146/16 |  |   |                                                     |
|  |Gateway: 10.140   |  |   |                                                 IPsec VPN
|  +------------------+<-+   |                                                     |
|  +------------------+  |   |                                     +               |
|  |      Web3        |  |   |                                     +---------------+
|  |192.168.10.147/16 |  |   |                                     | Remote Sites  |
|  |Gateway: 10.140   |  |   |                                     +---------------+
|  +------------------+<-+   |
+----------------------------+

Original Answer

This issue was fixed by upgrading my Coytoepoint firmware version from 8.6.0i-patch1 to 10.2.0a. The version I was running was the most up to date I could go to without reconfigurring my load-balancer from scratch. The configs don't transfer when you update from 8.6.0i-patch1 to v10.

This site seems to have some other issues that I will be looking into, but the original problem has been solved.

Thanks everyone for your help and suggestions.

3
  • Luke, two things... First, if possible could you find out what bug was fixed by that Coyotepoint upgrade? Also please don't forget to accept this answer in a couple of days – This Nov 25 '13 at 14:39
  • I still have my open ticket with Coyote point and already emailed the saved states from the new firmware. Their engineers are looking into what is different. Once I have their response, would I update it to the answer, or the original question? – Luke Nov 25 '13 at 14:50
  • Hi Luke, just update your answer with the bug info – This Nov 25 '13 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.