Diagnostics ARP Table 504 Nginx Gateway Timeout

[johnpoz]

checking what processes are running would be good start for sure - but also you could just do process of elimination - how many packages exactly do you have installed? Nmap could do it for sure - but then that would have, or should have to be triggered.

You could look to see if you have any crons scheduled.

[manicmoose]

I already checked for 'arp' processes and there's nothing over and above what 'arpwatch' is doing:

/usr/local/sbin/arpwatch -z -Z -f /usr/local/arpwatch/arp_igb2.5.dat -i igb2.5
/usr/local/sbin/arpwatch -z -Z -f /usr/local/arpwatch/arp_igb2.99.dat -i igb2.99
/usr/local/sbin/arpwatch -z -Z -f /usr/local/arpwatch/arp_igb2.999.dat -i igb2.999
/usr/local/sbin/arpwatch -z -Z -f /usr/local/arpwatch/arp_igb2.10.dat -i igb2.10
/usr/local/sbin/arpwatch -z -Z -f /usr/local/arpwatch/arp_igb2.20.dat -i igb2.20
/usr/local/sbin/arpwatch -z -Z -f /usr/local/arpwatch/arp_igb2.50.dat -i igb2.50

My WANs are on diff. physical NICs, so different 'igb' numbers to this one - so presumably arpwatch is behaving itself.

As for cron, it all appears kosher to me:

1,31	0-5	*	*	*	root	/usr/bin/nice -n20 adjkerntz -a
1	3	1	*	*	root	/usr/bin/nice -n20 /etc/rc.update_bogons.sh
1	1	*	*	*	root	/usr/bin/nice -n20 /etc/rc.dyndns.update
*/60	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot
30	12	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_urltables
1	0	*	*	*	root	/usr/bin/nice -n20 /etc/rc.update_pkg_metadata
0,15,30,45	*	*	*	*	root	/etc/rc.filter_configure_sync
0	11	4-10	*	*	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc >> /var/log/pfblockerng/extras.log 2>&1
*/1	*	*	*	*	root	/usr/sbin/newsyslog
1	3	*	*	*	root	/etc/rc.periodic daily
15	4	*	*	6	root	/etc/rc.periodic weekly
30	5	1	*	*	root	/etc/rc.periodic monthly
*/1	*	*	*	*	root	/usr/local/pkg/servicewatchdog_cron.php
*/5	*	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_cron_misc.inc
*/2	*	*	*	*	root	/usr/bin/nice -n20 /sbin/pfctl -q -t snort2c -T expire 900
5	1,7,13,19	*	*	*	root	/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/snort/snort_check_for_rule_updates.php
15	*	*	*	*	root	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron >> /var/log/pfblockerng/pfblockerng.log 2>&1
16	3	*	*	*	root	/usr/local/pkg/acme/acme_command.sh "renewall" | /usr/bin/logger -t ACME 2>&1

"/etc/cron.d/at" only has 'atrun' in it.

I do have nmap installed, but I've had that for as long as I remember without issues - and like you said - it does nothing unless triggered/called explicitely.

Tried restarting 'arpwatch' and deleting all MAC entries but they soon returned.

Uninstalled 'arpwatch' and nmap - no difference
Had HAproxy installed (not currently being used) so uninstalled that. No change.
Rebooted - no difference.

The only 'arp' process I've seen pop up periodially is:

/usr/sbin/arp --libxo json -an

I'm not sure what they are related to.

At this point I'm a bit stumped.

[manicmoose]

Think I've "solved" it.

ntopng seems to be the culprit - I disabled it and all the ARPs have stopped. I haven't modified ntopng in ages, so maybe the config has become corrupted and it's doing something it's not supposed to be. Odd one.

I'll wait awhile before re-enabling it, but I will delete the data/config and re-install it fresh to ensure it's happier than it was.

Barring another re-occurance, thanks for the help folks!

[johnpoz]

@manicmoose said in Diagnostics ARP Table 504 Nginx Gateway Timeout:

ntopng seems to be the culprit

While I don't profess to being a ntop guru by any means.. I don't get why it would be arping like that. I get what your saying that it stopped what your seeing - but not sure why it do that.. Ntop out of the box sure shouldn't be arping every IP for networks its attached too..

[manicmoose]

@johnpoz Agreed - and it never used to, which is why I think it's become a little confused...

Time to reset its brain when I re-enable it (later).

[stephenw10]

It will it you have enabled this:

Active Network Discovery

Toggle the periodic discovery of network devices using multiple techniques that include ARP scan, MDNS and SSDP.

That's not enabled by default.

[johnpoz]

Nor should it be enabled on your wan interface ;)

[manicmoose]

Yep, I get all that.

As mentioned, I hadn't altered the ntopng config for ages, so I don't think that was it - but I've blown away the config now so I can't check.

Nevertheless, it's solved so that's all I care about.

[lddj9090]

@manicmoose
Thanks for the information, unfortunately wasn't able to check in for a week.

I have uninstalled ntopng, but still got a long waiting time to see the ARP entries. Will do a tcpdump to check further.

[johnpoz]

did flush the cache? The items would still be there for 20 minutes even if you uninstall what was arping for them.

[lddj9090]

@lddj9090
How can i clear the arp cache on pfsesne, all at once?

[lddj9090]

@johnpoz

2-5 minutes.

[johnpoz]

arp -d -a

or from the gui click the clear arp table.

[lddj9090]

@johnpoz

Cleared the cache and tried again, still spinning for ages and timeout. Could "snort" be somehow involved?

[johnpoz]

You need to figure out why your wan has anything it.. Have gone over why there should be nothing in your wan network in your arp cache other than your gateway..

NOTHING should just be arping for every IP on a network.. especially our wan.. So there is something you set to arp, or some other reason.

You need to sniff on your wan and figure out what - are you arping, is it gratuitous arps? etc..

Out of the box pfsense with any package installed shouldn't be arping neither your wan network your lan side networks.. It would have to be some setting you did.. Like the ntop setting for network discovery.

[lddj9090]

@johnpoz
Alright, I will investigate further and talk back to you later. Thanks for the support so far!

[stephenw10]

@lddj9090 Do you actually have WAN side ARP entries with public IPs?

Are they eventually being resolved by reverse DNS?

Could a completely different problem than @manicmoose had.

Steve

[jimp]

Last time someone was seeing ARP for random hosts on WAN it was from having their subnet mask on the WAN interface set to something ridiculous like 0 instead of what it should have been (e.g. 24, 29, etc.)

If your WAN is set to a static address, double check your IP address and subnet mask settings.

Whatever is upstream may be answering ARP for anything with its own MAC. That's unusual but not unheard of.

[lddj9090]

@stephenw10

As far as I am aware, there aren't any public IP in my ARP output.

Here is a snippet of the ARP -a output: arp_snippet_pfsense.txt

We do have high availability sync enabled. As well as a bunch of Virtual IPs (CARP). Even though they all seem to be configured correctly, could that cause something unexpected?

[lddj9090]

@jimp

I double-checked the WAN Interfaces static address settings, they are all setup correctly.