Diagnostics ARP Table 504 Nginx Gateway Timeout
-
@johnpoz Sounds good.
I just checked the arpwatch 'database' entries, and none of the MACs on the WANs are in there - only local ones.
-
Yeah arpwatch shouldn't do it - but it was just off the cuff guess to something that "could" maybe do it..
Drawing a blank - maybe after some more coffee something will come to me.. But atleast we have a piece of the puzzle to work with.. What would cause pfsense to arp for every IP in a range that is on the wan.. Not thinking of any packages you could add that could be setup to scan??
The nmap package "could" do it for sure - but you would have to kick that off..
-
@johnpoz Hmm....good point - I have 'arping' installed, and that could have been at the same time I added the 2nd WAN (can't really recall).
Maybe it's the culprit as that's kind of what it's job is. I might uninstall it and see if that helps.
-
Yeah arping could prob do it too - but was not aware it had any sort of scanner option or that you could do it on a schedule sort of thing.. I currently don't have that installed.
-
@johnpoz Nope that's not it.
Uninstalled 'arping', then deleted all MACs on both WANs, leaving only the standard 2 on each.
My failover WAN is already back up to 127 again.
My primary WAN is up to 1021.
-
Nope, I'm not sure what could cause that either. Weird.
I would check the system processes. See if you have anything running that is obviously and arp process.
-
checking what processes are running would be good start for sure - but also you could just do process of elimination - how many packages exactly do you have installed? Nmap could do it for sure - but then that would have, or should have to be triggered.
You could look to see if you have any crons scheduled.
-
I already checked for 'arp' processes and there's nothing over and above what 'arpwatch' is doing:
/usr/local/sbin/arpwatch -z -Z -f /usr/local/arpwatch/arp_igb2.5.dat -i igb2.5 /usr/local/sbin/arpwatch -z -Z -f /usr/local/arpwatch/arp_igb2.99.dat -i igb2.99 /usr/local/sbin/arpwatch -z -Z -f /usr/local/arpwatch/arp_igb2.999.dat -i igb2.999 /usr/local/sbin/arpwatch -z -Z -f /usr/local/arpwatch/arp_igb2.10.dat -i igb2.10 /usr/local/sbin/arpwatch -z -Z -f /usr/local/arpwatch/arp_igb2.20.dat -i igb2.20 /usr/local/sbin/arpwatch -z -Z -f /usr/local/arpwatch/arp_igb2.50.dat -i igb2.50My WANs are on diff. physical NICs, so different 'igb' numbers to this one - so presumably arpwatch is behaving itself.
As for cron, it all appears kosher to me:
1,31 0-5 * * * root /usr/bin/nice -n20 adjkerntz -a 1 3 1 * * root /usr/bin/nice -n20 /etc/rc.update_bogons.sh 1 1 * * * root /usr/bin/nice -n20 /etc/rc.dyndns.update */60 * * * * root /usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot 30 12 * * * root /usr/bin/nice -n20 /etc/rc.update_urltables 1 0 * * * root /usr/bin/nice -n20 /etc/rc.update_pkg_metadata 0,15,30,45 * * * * root /etc/rc.filter_configure_sync 0 11 4-10 * * root /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc >> /var/log/pfblockerng/extras.log 2>&1 */1 * * * * root /usr/sbin/newsyslog 1 3 * * * root /etc/rc.periodic daily 15 4 * * 6 root /etc/rc.periodic weekly 30 5 1 * * root /etc/rc.periodic monthly */1 * * * * root /usr/local/pkg/servicewatchdog_cron.php */5 * * * * root /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc */2 * * * * root /usr/bin/nice -n20 /sbin/pfctl -q -t snort2c -T expire 900 5 1,7,13,19 * * * root /usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_for_rule_updates.php 15 * * * * root /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron >> /var/log/pfblockerng/pfblockerng.log 2>&1 16 3 * * * root /usr/local/pkg/acme/acme_command.sh "renewall" | /usr/bin/logger -t ACME 2>&1"/etc/cron.d/at" only has 'atrun' in it.
I do have nmap installed, but I've had that for as long as I remember without issues - and like you said - it does nothing unless triggered/called explicitely.
Tried restarting 'arpwatch' and deleting all MAC entries but they soon returned.
Uninstalled 'arpwatch' and nmap - no difference
Had HAproxy installed (not currently being used) so uninstalled that. No change.
Rebooted - no difference.The only 'arp' process I've seen pop up periodially is:
/usr/sbin/arp --libxo json -anI'm not sure what they are related to.
At this point I'm a bit stumped.
-
Think I've "solved" it.
ntopng seems to be the culprit - I disabled it and all the ARPs have stopped. I haven't modified ntopng in ages, so maybe the config has become corrupted and it's doing something it's not supposed to be. Odd one.
I'll wait awhile before re-enabling it, but I will delete the data/config and re-install it fresh to ensure it's happier than it was.
Barring another re-occurance, thanks for the help folks!
-
@manicmoose said in Diagnostics ARP Table 504 Nginx Gateway Timeout:
ntopng seems to be the culprit
While I don't profess to being a ntop guru by any means.. I don't get why it would be arping like that. I get what your saying that it stopped what your seeing - but not sure why it do that.. Ntop out of the box sure shouldn't be arping every IP for networks its attached too..
-
@johnpoz Agreed - and it never used to, which is why I think it's become a little confused...
Time to reset its brain when I re-enable it (later).
-
It will it you have enabled this:
Active Network Discovery Toggle the periodic discovery of network devices using multiple techniques that include ARP scan, MDNS and SSDP.That's not enabled by default.
-
Nor should it be enabled on your wan interface ;)
-
Yep, I get all that.
As mentioned, I hadn't altered the ntopng config for ages, so I don't think that was it - but I've blown away the config now so I can't check.
Nevertheless, it's solved so that's all I care about.
-
@manicmoose
Thanks for the information, unfortunately wasn't able to check in for a week.I have uninstalled ntopng, but still got a long waiting time to see the ARP entries. Will do a tcpdump to check further.
-
did flush the cache? The items would still be there for 20 minutes even if you uninstall what was arping for them.
-
@lddj9090
How can i clear the arp cache on pfsesne, all at once? -
2-5 minutes.
-
arp -d -a
or from the gui click the clear arp table.
-
Cleared the cache and tried again, still spinning for ages and timeout. Could "snort" be somehow involved?
