Diagnostics ARP Table 504 Nginx Gateway Timeout

lddj9090 · Jun 28, 2021, 2:24 PM

@johnpoz
Alright, I will investigate further and talk back to you later. Thanks for the support so far!

stephenw10 · Jun 28, 2021, 2:58 PM

@lddj9090 Do you actually have WAN side ARP entries with public IPs?

Are they eventually being resolved by reverse DNS?

Could a completely different problem than @manicmoose had.

Steve

jimp · Jun 28, 2021, 3:29 PM

Last time someone was seeing ARP for random hosts on WAN it was from having their subnet mask on the WAN interface set to something ridiculous like 0 instead of what it should have been (e.g. 24, 29, etc.)

If your WAN is set to a static address, double check your IP address and subnet mask settings.

Whatever is upstream may be answering ARP for anything with its own MAC. That's unusual but not unheard of.

lddj9090 · Jun 29, 2021, 8:02 AM

@stephenw10

As far as I am aware, there aren't any public IP in my ARP output.

Here is a snippet of the ARP -a output: arp_snippet_pfsense.txt

We do have high availability sync enabled. As well as a bunch of Virtual IPs (CARP). Even though they all seem to be configured correctly, could that cause something unexpected?

lddj9090 · Jun 29, 2021, 8:06 AM

@jimp

I double-checked the WAN Interfaces static address settings, they are all setup correctly.

johnpoz · Jun 29, 2021, 10:49 AM

And how many entries in your arp table? 1000's ??

lddj9090 · Jun 29, 2021, 11:19 AM

@johnpoz
254

johnpoz · Jun 29, 2021, 11:42 AM

Then I doubt your problem is related to what the other poster was seeing in slow downs do to trying to resolve public IP names.

Is pfsense not dhcp or dns for this network - where resolving names could be hindered, for ptr?

lddj9090 · Jun 30, 2021, 12:46 PM

@johnpoz said in Diagnostics ARP Table 504 Nginx Gateway Timeout:

hindered

Pfsense is DHCP for certain networks, not for all though. Also pfsense handles DNS request from our internal DNS server, so we can use DNS blocklists.

Pfsense DNS Override has active settings for some domains. Could there be an issue?

johnpoz · Jun 30, 2021, 1:20 PM

how long does it take to display your arp -a table?

from a cmd line? Where @stephenw10 was saying there could be a delay in the table being displayed is lots and lots of IPs in the mac that could not resolve, or take a while to resolve via external dns because they were public, and would do a public PTR query for their names.

If you have directly attached network to pfsense, so it has devices in its mac table - but the dns for these is somewhere downstream or upstream that pfsense has a problem resolving - then guess this could cause a delay in the output of your mac address table. Which could cause a timeout in the gui, etc.

But with only a few hundred entries and local dns, etc.. I don't see how that could be a problem.. When from say ssh you do a arp -a for your arp table how fast does it display?

jimp · Jun 30, 2021, 1:08 PM

You could edit the source and disable DNS for the page. There should probably be an option on that page to disable it anyhow since it really does not scale well.

diff --git a/src/usr/local/pfSense/include/www/diag_arp.inc b/src/usr/local/pfSense/include/www/diag_arp.inc
index 568657f2ff..eed0b38eb2 100644
--- a/src/usr/local/pfSense/include/www/diag_arp.inc
+++ b/src/usr/local/pfSense/include/www/diag_arp.inc
@@ -88,7 +88,7 @@ function prepare_ARP_table($json = false) {
        // Resolve hostnames and replace Z_ with "".  The intention
        // is to sort the list by hostnames, alpha and then the non
        // resolvable addresses will appear last in the list.
-       $dnsavailable=1;
+       $dnsavailable=0;
        $dns = trim(_getHostName("", "8.8.8.8"));
        if ($dns == "") {
                $dns = trim(_getHostName("", "8.8.4.4"));

lddj9090 · Jun 30, 2021, 1:20 PM

@johnpoz said in Diagnostics ARP Table 504 Nginx Gateway Timeout:

mac that could not resolve

Is it possible to see which ARP requests could not be resolved or have to be resolved external?

And how to figure out if pfsense has a problem resolving devices in its mac table? As I think this has to be the case. Pfsense has direct attached network.

It takes about 3-5 minutes to display the ARP -a command output via ssh. Also when I check the output, I just see IP/Macs form our network (different VLANs).

lddj9090 · Jun 30, 2021, 1:23 PM

@jimp
What do you mean by "disable DNS for the page"?

lddj9090 · Jun 30, 2021, 2:15 PM

I am now on a holiday, will check on it when I am back. Thanks for the support so far.

jimp · Jul 1, 2021, 2:06 PM

@lddj9090 said in Diagnostics ARP Table 504 Nginx Gateway Timeout:

@jimp
What do you mean by "disable DNS for the page"?

It will stop the page from using DNS to resolve hostnames. DNS resolution is almost certainly what is slowing it down.

Instead you'll just get a list of the MAC addresses and IP addresses without the hostnames.

johnpoz · Jun 30, 2021, 5:29 PM

@lddj9090 said in Diagnostics ARP Table 504 Nginx Gateway Timeout:

figure out if pfsense has a problem resolving devices in its mac table?

By why which IP/mac combo's don't show a name.. And where that could/should resolve, etc.

example

Here is snip of mine. There are a few that don't resolve - but most of them do.

[21.05-RELEASE][admin@sg4860.local.lan]/root: arp -a
? (192.168.6.106) at 8a:75:de:b2:b7:bd on igb2.6 expires in 1166 seconds [vlan]
? (192.168.6.104) at 64:79:f0:d6:60:50 on igb2.6 expires in 1186 seconds [vlan]
? (192.168.6.105) at 38:80:df:f4:23:43 on igb2.6 expires in 1194 seconds [vlan]
sg4860.wguest.local.lan (192.168.6.253) at 00:08:a2:0c:e6:20 on igb2.6 permanent [vlan]
Alexa-Kitchen.local.lan (192.168.4.76) at a0:d0:dc:56:53:42 on igb2.4 expires in 1101 seconds [vlan]
Alexa-Bedroom.local.lan (192.168.4.77) at 40:a9:cf:c2:b0:99 on igb2.4 expires in 974 seconds [vlan]
alexa-show.local.lan (192.168.4.78) at b8:5f:98:0e:22:14 on igb2.4 expires in 653 seconds [vlan]
Alexa-cpu.local.lan (192.168.4.79) at b4:7c:9c:a8:d5:f0 on igb2.4 expires in 1198 seconds [vlan]
island2.local.lan (192.168.4.72) at 18:69:d8:39:e4:23 on igb2.4 expires in 944 seconds [vlan]
? (192.168.4.203) at 9a:4c:a7:b3:8f:31 on igb2.4 expires in 1000 seconds [vlan]
island1.local.lan (192.168.4.71) at 18:69:d8:3a:03:cd on igb2.4 expires in 1194 seconds [vlan]
nightstand.local.lan (192.168.4.65) at 68:57:2d:98:38:84 on igb2.4 expires in 981 seconds [vlan]
office1.local.lan (192.168.4.61) at 68:57:2d:98:39:14 on igb2.4 expires in 900 seconds [vlan]
sg4860.wpsk.local.lan (192.168.4.253) at 00:08:a2:0c:e6:20 on igb2.4 permanent [vlan]
office2.local.lan (192.168.4.62) at 68:57:2d:98:36:83 on igb2.4 expires in 1136 seconds [vlan]
office3.local.lan (192.168.4.63) at 68:57:2d:98:28:22 on igb2.4 expires in 925 seconds [vlan]
end-table.local.lan (192.168.4.56) at d4:a6:51:cf:c5:45 on igb2.4 expires in 928 seconds [vlan]
guestroom.local.lan (192.168.4.57) at 18:69:d8:39:f7:e9 on igb2.4 expires in 1001 seconds [vlan]
S30.local.lan (192.168.4.89) at 88:3f:4a:f0:cb:9c on igb2.4 expires in 1115 seconds [vlan]
hall1.local.lan (192.168.4.58) at d4:a6:51:d1:a2:8a on igb2.4 expires in 1055 seconds [vlan]
hall2.local.lan (192.168.4.59) at d4:a6:51:d2:0b:d4 on igb2.4 expires in 909 seconds [vlan]
D2.local.lan (192.168.4.52) at 18:69:d8:3a:08:d8 on igb2.4 expires in 931 seconds [vlan]
D3.local.lan (192.168.4.53) at 18:69:d8:39:d9:f4 on igb2.4 expires in 922 seconds [vlan]
D4.local.lan (192.168.4.54) at 18:69:d8:39:f2:ca on igb2.4 expires in 1198 seconds [vlan]
D5.local.lan (192.168.4.55) at 18:69:d8:39:ed:ba on igb2.4 expires in 1194 seconds [vlan]

I haven't gotten around to setting up some of them with reservations, and those do not resolve. I will fix that at some point, etc.

But if you had hundreds of these that do not resolve - and its trying then sure that could cause a delay in the output..

The simple solution is prob just do as shown and disable the dns part of arp table. See if I do with -n -a, none of mine resolve

[21.05-RELEASE][admin@sg4860.local.lan]/root: arp -n -a
? (192.168.6.106) at 8a:75:de:b2:b7:bd on igb2.6 expires in 1183 seconds [vlan]
? (192.168.6.104) at 64:79:f0:d6:60:50 on igb2.6 expires in 1169 seconds [vlan]
? (192.168.6.105) at 38:80:df:f4:23:43 on igb2.6 expires in 1140 seconds [vlan]
? (192.168.6.253) at 00:08:a2:0c:e6:20 on igb2.6 permanent [vlan]
? (192.168.4.76) at a0:d0:dc:56:53:42 on igb2.4 expires in 1049 seconds [vlan]
? (192.168.4.77) at 40:a9:cf:c2:b0:99 on igb2.4 expires in 1167 seconds [vlan]
? (192.168.4.78) at b8:5f:98:0e:22:14 on igb2.4 expires in 456 seconds [vlan]
? (192.168.4.79) at b4:7c:9c:a8:d5:f0 on igb2.4 expires in 1158 seconds [vlan]
? (192.168.4.72) at 18:69:d8:39:e4:23 on igb2.4 expires in 1048 seconds [vlan]
? (192.168.4.203) at 9a:4c:a7:b3:8f:31 on igb2.4 expires in 1006 seconds [vlan]
? (192.168.4.71) at 18:69:d8:3a:03:cd on igb2.4 expires in 997 seconds [vlan]
? (192.168.4.65) at 68:57:2d:98:38:84 on igb2.4 expires in 1084 seconds [vlan]
? (192.168.4.61) at 68:57:2d:98:39:14 on igb2.4 expires in 1004 seconds [vlan]
? (192.168.4.253) at 00:08:a2:0c:e6:20 on igb2.4 permanent [vlan]
? (192.168.4.62) at 68:57:2d:98:36:83 on igb2.4 expires in 939 seconds [vlan]
? (192.168.4.63) at 68:57:2d:98:28:22 on igb2.4 expires in 1028 seconds [vlan]
? (192.168.4.56) at d4:a6:51:cf:c5:45 on igb2.4 expires in 1032 seconds [vlan]
? (192.168.4.57) at 18:69:d8:39:f7:e9 on igb2.4 expires in 1106 seconds [vlan]
? (192.168.4.89) at 88:3f:4a:f0:cb:9c on igb2.4 expires in 918 seconds [vlan]
? (192.168.4.58) at d4:a6:51:d1:a2:8a on igb2.4 expires in 858 seconds [vlan]
? (192.168.4.59) at d4:a6:51:d2:0b:d4 on igb2.4 expires in 1013 seconds [vlan]
? (192.168.4.52) at 18:69:d8:3a:08:d8 on igb2.4 expires in 1035 seconds [vlan]
? (192.168.4.53) at 18:69:d8:39:d9:f4 on igb2.4 expires in 1026 seconds [vlan]
? (192.168.4.54) at 18:69:d8:39:f2:ca on igb2.4 expires in 1001 seconds [vlan]
? (192.168.4.55) at 18:69:d8:39:ed:ba on igb2.4 expires in 997 seconds [vlan]

lddj9090 · Jul 1, 2021, 2:06 PM

@jimp

After deleting some DNS Override entries, the GUI ARP table shows up without any problems. Seem to have fixed the issue!

Thanks a lot for the outstanding support.

johnpoz · Jul 1, 2021, 2:13 PM

Well your PTR zone there - yeah if you were asking some other NS to resolve the ptrs - and it didn't know them or you couldn't actually talk to that NS then yeah you could have issues when your trying to resolve hundreds of IPs to their names..

lddj9090 · Jul 1, 2021, 3:04 PM

@johnpoz
Exactly, I think not beeing able to resolve the IP's caused the problem in the end. Thanks for your help mate!

lddj9090 · Jul 14, 2021, 7:30 AM

Alright, to clarify for other users who may have problems with this:

Configuration under DNS Resolver -> General Settings -> Domain Overrides looked like:

Domain:
18.16.172-in.addr.arpa (our VLAN)

Lookup Server IP Address:
172.16.18.1 (Pfsense interface for this VLAN)

The reason I configured it like this was, I wanted that clients from that VLAN do not use an outside lookup server. Instead, using our Pfsense itself.

After deleting this entry, the ARP table shows up nicely.