Adtran NetVanta router on MPLS network can only reach other MPLS sites if source address set

Question

I have an MPLS network with 5 remote locations with 1544 routers, our central office, and our data center. All of the traffic from the remote locations routes out the central office unless it goes down, in which case everything routes out the data center connection.

If I want to ping anything on the public internet from the router cli, I can. If I want to ping/traceroute anything on the MPLS network that is not local, I have to give the command with "source", i.e. ping 10.0.1.11 source 10.5.0.1 or I get nothing. Netflow is working fine because I could define the source as being eth 0/1.

The default route is going out the ppp MPLS interface, and no one inside the 10.5.0.0/23 network has any issues getting everywhere they need to, including servers at the data center that I cannot reach from the router itself without the "source 10.5.0.1" being added.

And there doesn't appear to be any way to define "source" for the syslog stuff. So the adtrans at the remote sites cannot get their syslog data to the syslog server.

These routers were primarily configured by the telco before we took ownership, so I don't know why they did certain things the way they did. e.g. should proxy-arp actually be enabled for this to work? Or could that break something else in this config? Do you see something else that would fix this problem?

Routing table:

#sh ip route
Codes: C - connected, S - static, R - RIP, O - OSPF, B - BGP
       E1 - OSPF external type 1, E2 - OSPF external type 2
       IA - OSPF inter area

Gateway of last resort is 208.46.133.129 ppp 1

S    0.0.0.0/0 [1/1/0] via 208.46.133.129, ppp 1
C    10.5.0.0/23 is directly connected, eth 0/1
S    67.14.0.0/17 [1/1/0] via 208.46.133.129, ppp 1
S    67.14.128.0/19 [1/1/0] via 208.46.133.129, ppp 1
C    208.46.133.128/30 is directly connected, ppp 1
C    208.46.133.129/32 is directly connected, ppp 1

Here is what should be the relevant parts of the config from one of the five remote routers (they are all functionally identical):

ip firewall
no ip firewall alg msn
no ip firewall alg mszone
no ip firewall alg h323
!
ip flow export destination 10.0.1.11 7003 source eth 0/1
ip flow export destination 10.0.1.2 9996 source eth 0/1
ip flow cache timeout active 5
ip flow top-talkers
!
no ethernet cfm
!
interface eth 0/1
  description desc LAN Block 10.5.0.0/23
  ip address  10.5.0.1  255.255.254.0
  no ip proxy-arp
  ip access-policy LAN
  ip flow ingress
  ip flow egress
  no shutdown
  no lldp send-and-receive
!
interface ppp 1
  description DS1NT
  ip address  y.y.133.130  255.255.255.252
  no rtp quality-monitoring
  ppp multilink interleave
  ppp multilink fragmentation
  ppp multilink
  no shutdown
  cross-connect 1 t1 1/1 1 ppp 1
  cross-connect 2 t1 1/2 1 ppp 1
!
ip access-list extended DATA-ACL
  permit ip 10.5.0.0 0.0.1.255  10.0.0.0 0.0.1.255
  permit ip 10.5.0.0 0.0.1.255  10.1.0.0 0.0.0.255
  permit ip 10.5.0.0 0.0.1.255  5.5.0.0 0.0.255.255
  permit ip 10.5.0.0 0.0.1.255  5.6.0.0 0.0.255.255
  deny   ip 10.5.0.0 0.0.1.255  10.0.0.0 0.255.255.255
  deny   ip 10.5.0.0 0.0.1.255  172.16.0.0 0.15.255.255
  deny   ip 10.5.0.0 0.0.1.255  192.168.0.0 0.0.255.255
  permit ip 10.5.0.0 0.0.1.255  any
!
ip policy-class LAN
  allow list DATA-ACL
!
ip route 0.0.0.0 0.0.0.0 208.46.133.129
ip route 5.6.0.0 255.255.224.0 10.1.0.1
ip route x.x.0.0 255.255.128.0 y.y.133.129
ip route x.x.128.0 255.255.224.0 y.y.133.129

Answer 1

If You don't configure the source address and send the packet to 10.0.1.11, it routed by the default route, and will get the source address from interface attached to the default GW (I suspect 208.46.133.129) automatically, but not 10.5.0.1. Something in the configuration of network (routes, ACLs, IPSec policy, MPLS filters ...) prevents to communication from this address.

Answer 2

This is normal and expected behavior.

It would be a real problem if you could ping from any router, anywhere, without sourcing, every time - this is generally a "no no".

Imagine you are an IP packet header. If you do not specify a SOURCE IP address, what will be the "Source IP" entry in the header?

Whatever is filled in there by default (non-routed loopback, physical non-routed interface, etc) is what is sent out of the default gateway. When you source, you are giving your IP header a valid address & subnet to return to. It is a best practice to source accordingly. Also, there are security implications of not explicitly defining these things.

I hope that helps.

Source: Network Engineer for 6+ years @ Fortune 50 company.

Answer 3

This is really just an expanded explanation of things that have been touched on in the other two answer, but with a little more depth. My apologies for the length, but I was trying not to gloss over any obvious details/assumptions, as that is usually where the problem lies.

First, let's start with the selection of the source IP for data that is originated from the router, such as your pings, traceroutes, syslog, etc. So most router software will do a routing lookup on the destination address, and then use the local interface that is closest to the destination as the source. So in this case, other than for traffic onto the local LAN (10.5.0.1/23), the traffic will exit the router on the PPP interface so the source IP will be 208.46.133.130.

Now, without looking at all of the devices in the path, I can't say this for sure, but just based on the fact that these routers only have static and connected routes, I'm assuming that a device in the traffic path does not have a 'complete' routing table. So I'll give you one scenario that I've seen that will result in this sort of behavior. So let's say that this 10.0.1.11 IP is your syslog server, and it has as a default gateway a seperate layer 3 switch or router in the hub site that is at 10.0.1.1, and then the firewall to the Internet is at 10.0.1.2, and let's put the MPLS router at 10.0.1.3. So if you only have static routes throughout the network, one possibility would be that the firewall at 10.0.1.2 does have a static route for the 208.46.133.128/30 network, or some larger aggregate of it that points back to the MPLS router. But if the server's gateway only the static routes for the LAN addresses (10.5.0.0/23, etc.) that point back to the MPLS and a default route to the firewall, what happens then? The return traffic from the server will go to it's default gateway, and then the default route from there will go to the firewall, and it doesn't know to go back to the MPLS cloud for the ISP/uplink side of the remote MPLS router, but it does when traffic is sourced from the LAN side of the router or the LAN itself.

So long description short, based on using only static routes, there are probably 'holes' in the routing tables of various devices, and because only an admin has access to the router, missing routes for these 'infrastructure' IP addresses are usually not noticed. The way to track this down is to go to every single hop in the path and look at the routing table for the source and destination addresses. At some point you'll find a device that has a route back to the 10 address, and doesn't have a route for the 208 address, or just a default route that points in the wrong direction. My longer term suggestion would be to implement some form of dynamic routing. With statics keeping track of whether or not all the needed routes are added to every new device, or that each new subnet has a route added to all the existing devices is tedious and error prone, and you very quickly lose any advantage/simplicity that was the supposed reason to avoid dynamic routing in the first place.