I've been having some strange issues with some site-to-site IPsec tunnels using ESP and pre-shared keys that I've been working on for the past two weeks with no success. The tunnel has no problem coming up, but certain traffic just doesn't want to pass over the link and I cannot for the life of me figure out why. We have 12 tunnels to remote locations, all configured identically and just about all with identical hardware (all Cisco 867VAE or 861 routers except for two of the sites), but these problems are only affecting 4 of the tunnels.
I have a mountain of details and a list of all the different types of things that do and don't work over the tunnel, but to keep it simple let's just stick with this particular one: I'm unable to send pings from the main office (where our Cisco ASA 5512-X is) to the remote location (where the router is; two of these locations are Cisco 867VAE routers, one is an old Cisco 2811 router, and one is a TP-Link TD-W8960N) over the VPN tunnel. However, pings from the remote location to the main office over the VPN tunnel work just fine. Any idea what might cause this behavior?
For reference there are four different ISPs involved, so I'm assuming it's not an issue with one particular ISP, but just in case it's relevant: We have three bonded T1's at our main office through Time Warner, while at the remote locations we have two with T1's (one through AT&T and one through Level 3) and two with DSL through CenturyLink. In case it matters none of our tunnels are very heavily used at all (usually just supporting two or three computers and a couple IP phones), so it definitely shouldn't be network congestion causing this somehow.
I've already triple-checked my configuration, rebooted everything, wiped out the configuration and re-added it, tried a different model Cisco router at the location with the 2811, and just for the heck of it I played with a variety of MTU and MSS values on both the ASA and remote router, but no difference. I can provide additional details on traffic that works and doesn't work as well as specific configuration details or show command output if needed.