This got pretty long so I'm just going to put the answer at the top instead of the bottom, read the whole thing if you want more background and to read a rant.
Fix and summary of the issue
Essentially the problem is that your pointing your default gateway to another device that is still local to the clients as it's default gateway which is causing asymmetric routing and making connection inconsistencies that the applications/hosts probably don't like much. You can turn on redirects on the router interface or the switch but that's still not going to change that you're sending your packets to one device and receiving them from another. To remidiate this you have a couple options:
- Segment off those two devices. You generally do not need two separate routing devices in an end network. Keep your transit upstream and keep your LANs clean as this will also help clear chatter that your servers don't need to hear (as much as they love receiving RIP updates)
- An exception to this would be to configure the switch and the router to have some sort of first hop redundancy protocol (VRRP,GLBP,HSRP,etc.) the caveat with this is that you would need an identical next hop (or equal cost way to reach the destination) and be sure to specify the master and why it's priority would decrement to ensure traffic paths.
The just of it is that you need to create boundaries in order to segment traffic and control where the decisions to send traffic is being made. How it is now with two possible routing devices, it's leading straight down the road to a lack of security controls and irregular traffic patterns. I would really suggest setting a layer 3 transit up behind the scene to reach between eachother (possibly just take the IP addresses off your switch if the router handle it). That way devices have one exit, and that one exit leads to how everything is going to reach everything else, giving you central control and simplifying troubleshooting.
ANALYSIS AND RCA OF ISSUE FROM GIVEN OUTPUTS
Since both the Gateway and the Gateway's next hop are both on the same LAN as your end devices, the redirect is there to let them know they no longer need to forward traffic to them since there is a more direct route available for them to use (even if you don't want them to be using it, a network device will still see it as a better option as it's one less hop the end device would have to take)
This essentially creates local asymmetric routing:
Path out w/o ICMP-REDIR:(SER->RTR->SW->NEWNET)
Return Path:(NEWNET->RTR->SER)
This is the path that is taken every time.
Where as the ones that do accept the redirects get a temporary route statement in their table, once that times out they forward to the default gateway again, who proceeds to give them another redirect. How everything is taking an expected path so it is happy:
1st Attempt out: (SER->RTR|| Use this route instead)
Path out ICMP-REDIR: (SER->SW->NEWNET)
Return Path:(NEWNET->RTR->SER)
When you are dealing with things that have connection tables/states, applications that are expecting certain consistencies and possibly even source based routing on your hosts and you locally have asymmetric routing, thing will either kinda of limp along somehow or they will completely break. Neither one is what you want.
A good analogy would be if you invited your SO over and told her to pick up some love tubes on the way, you to the door and make sure the mood is right in the lovepad you open the door with eyes closed, lips puckered and you're handed your condoms, it's not your girlfriend handing them to you though, it's some weird stranger who just happened to get them for you after hearing that you wanted them, you'd probably be a bit weirded out.
The redirect would be the equivalent of you SO telling you that they wouldn't be able to make it there tonight but they have a friend in the area to help us out and get what we need.
SECURITY CONSIDERATIONS OF REDIRECTS AND MULTIPLE L3 DEVICES IN ACCESS LAN
If you decide you're going to put an access list on the switch because you don't want it to access XYZ server, well that's cool, I can just change my gateway to the other device that routes making that a completely null point (assuming they both know how to get there). The reason I'm bringing this up is because generally when I run across this happening it's because someone decided they were going to put their firewall on the network as the exit point, but then they put in a layer 3 switch with an IP in every network and that's the default gateway. It completely voids the fact that you have a firewall, because the switch is local to everything, I don't need to have my access to internal resources inspected and passed, I just go through the switch. This may not be the case for what your doing but it gets the "you should have rigid boundaries" point across. And you NEED to turn off redirects everywhere unless you have some reason that I have yet to hear. If you need to get to a specific host/subnet, add a static route on your host instead of dynamically learning it from an unsecured protocol, you're opening yourself up to DOS from route poisoning and MITM attacks. Anyone can spoof an ICMP redirect, and like when those redirects are sent it's just that device guessing what it should be based on what's in it's table, which might be completely wrong if it was from a reconfigured router advertisement.
If you have three user LANs put that on one switch have the default gateway for that switch be your firewall/router in a separate transit network defined as a separate zone. Do the same for each border you have and make sure they all go through one place that controls access to resources between them.
