I'm giving you a high-level overview of my problem, no tech details.
I'd like to know how to solve this problem.
Problem Effect #1
We got a Netapp replacing another NAS. The Netapp is connected to our Enterasys/Extreme network gear.
Since we have the Netapp we have managed to create trouble in the network several times.
Meaning: When performing the following kind of transfers to or from the Netapp:
- lots of data
- transfer not necessarily fast (sometimes only 10MB/s = 100Mb/s)
- NFS or Networker (backup/restore) traffic
- potentially lots of small files (NFS) but also large binary streams (NDMP)
it seems to have had an effect on our firewalls (VRRP cluster) such that they start failing over because VRRP HELLO packets are getting lost, potentially on every VLAN, not only on the ones where the Netapp is connected to.
Even when keeping the Netapp <-> backup/whatever traffic to the same L2 segment this happens. What could still be involved at that layer? It does not happen always or often, but often enough to indicate a problemm.
I was thinking about the buffering of the switches maybe posing problems? Can that even be? What would happen in those cases, would packets really get lost or would everything just be slower?
How do I solve this problem?
Problem Effect #2
The FDB on the switches gets messed up: After a quick MASTER/BACKUP flap of the firewalls, the VRRP MAC is seen as indicated by the red arrows on the attached diagram (for some VLANs). So it's sending packets for that address in a loop I guess.
When I then issue a manual spanning tree flap by disconnecting one of the redundant links (the dotted lines leaving the datacenter switches) everything goes back to normal.
Note that in the meantime the setup is even simpler, the Netapp and test server connected to the same switch, still same effect.
In general
I said there are no routing protocols because we don't use much routing, it's reasonable to do so here. We also don't use gvrp and such. Putting wrong vlans on one end of the link and then have it automagically appear on the other end is no automatism I desire. But that is another discussion. We may chat about it, sure.
If I had errors to show you I would. But then I would probably be able to fix it myself. Since I have nothing so far, I am asking. I can merely notice the effect of something and am looking for potential causes. Switch CPU is already nice and next time I will take a live look. What about the buffering inside the switches? Can that be a factor?
- Have you done some research before asking the question?
Yes, to no avail, that's why am here. I sniffed, I analysed the config, I re-read the operating manuals. Nothing.
- Have you explained what you've already tried to solve your problem?
As I said we found that there has never been a problem as long as the new NAS wasn't there. Sure, this invites speculation. But good system administrators often operate by their guts in order to localize problems. I don't think I'm a bad admin, but my guts don't lead me anywhere here.
Currently I have set up a lab with one of our core switches isolated with only the NAS and a server on one VLAN and two PC pinging each other on another VLAN while the server and the NAS happily copy files to each other. Tomorrow I will see if there was packet loss on the PCs. However this is by far no simulation of a larger network.
- Have you explained why this question is important for you to resolve?
Well, having the network firewalls down is not nice.
- Have you provided some relevant context about the needs of the business environment and/or software applications that you're supporting?
When someone performs a major transfer to or from the NAS, that should have no impact on the rest of the network. Those transfers need not be "major" by the way to cause problem, as I explained.
- Have you included a diagram of the problem (especially if it involves more than two devices)?
If you insist I can do that, but the whole network is quite complicated. But it's a standard enterprise network, with core, distribution and edge. Some spanning tree, some routers, some firewalls. No routing protocols whatsoever. We prefer to remain in control. ;-)
- Are you sure you've provided enough details such as operating systems, addresses, protocols, link types, link speeds, or interface names?
Probably not. The NAS has a 10G card. However the problem also arises when it is connected with the 1GbE interface only.
- Have you included information about what manuals or reference material you consulted when trying to resolve your question?
No, because aside from the install and config manuals of the items involved I currently wouldn't know where to look, as I pointed out.
- Have you specified which hardware you're using, including firmware version numbers?
It's a Netapp FAS2240 with Release 8.2P3 7-Mode. The switches are Enterasys G-Series.
- If your question includes a configuration, have you checked that it's correctly formatted?
No config for now.
- Have you included relevant details about timing, syslog, or application log entries?
When the firewalls flap we can't rely on anything pretty much anymore. But as I said the local firewall logs as well as local tcpdump clearly show VRRP HELLOs missing.
- If your configuration results in an error, have you included the exact error, as well as any tracebacks?
Yes, the VRRP HELLOs get missing on VLANs independent from the NAS' VLAN therefore causing trouble with the firewalls.
- If your question doesn't include a configuration, are you sure the configurations are not required? Hint: most people will want to see configurations even if you think they aren't necessary
What part would you like to see?
- If your configuration produces different results to what you expected, have you stated what you expected, why you expected it, and the actual results?
What configuration would be required? The NAS, the firewalls, the whole network?
- Have you checked that your question looks reasonable in terms of formatting?
I have and it is.
- Have you checked the spelling and grammar to the best of your ability?
Yes.
- Have you read the whole question to yourself carefully, to make sure it makes sense and contains enough information for someone coming to it without any of the context that you already know?
I'm sorry if it seems to broad but I need a few shots in the dark here as I said.
