Trying to get a handle on an apparent network problem that only affects unix systems for a specific time period each day. On a network with about 400 nodes I have 3 servers that exhibit network up/down (flapping) behavior a few times between 19:00 and 20:30 each night. Two of the servers are OLD HPUX systems and one is Solaris 10. The apparent network problem presents itself with client disconnections and alarms from my monitoring server (ping test). Servers don't show anything useful in their logs, and other Windows servers exhibit no problems at all.
I should also mention that these servers have been online for several years without issue. Problem seemingly popped up overnight and we don't have any known new devices or configuration changes to correlate with the timeframe.
In an attempt to better see and manage traffic, we moved one of the HPUX servers behind a transparent firewall. The firewall has no deny rules, it's just monitoring. But strangely the HPUX server behind the firewall stopped having trouble and became stable in all subsequent nights (the other, outside the firewall remains troublesome). My only guess is that the firewall is implicitly blocking some non-TCP traffic that isn't apparent in its logging.
Regarding the network itself, there are only two switches between the clients and server. Neither show any errors within their respective logs.
Has anyone seen a situation like this or does anyone have a suggestion? I think I can fix the server network flapping using the firewall, but this doesn't address the root cause that I'd really like to identify.
It's been suggested that the problem could be a result of multicast/broadcast traffic overloading the old 10mbit NIC on the HPUX server, but since I can see this in the firewall and I don't see any increase/decrease during the problem period, I'm not convinced.
All advice is appreciated! Thank you!