Diagnostics ARP Table 504 Nginx Gateway Timeout

lddj9090 · Jun 22, 2021, 12:41 PM

Hi

Problem: 504 Gateway Timeout whenever opening Diagnostic -> ARP Table

Branch: 2.5.1

I know the nginx is configured to send a timeout after 60 seconds. Is it therefore safe to extend the 60 seconds to 120 seconds? Or does it affect different parts of pfsense somehow?

Or is there a better solution to fix the 504 Gateway Timeout and to see the ARP Table again?

I checked out https://mybroadband.co.za/forum/threads/pfsense-2-5-0-504-gateway-timeout.1126720/ and some other Google hits, but did not really find an answer to my question.

Any help is appreciated, thanks.

johnpoz · Jun 22, 2021, 2:50 PM

So your saying it takes over 60 seconds to show your arp table? How many entries are in it?

lddj9090 · Jun 22, 2021, 2:50 PM

@johnpoz it doesn't load the ARP Table due to the timeout, which happens after 60 seconds yes. There are roughly 285 entries.

Funny though, just now when I tried to open the ARP Table it loaded fairly quickly. The other 15 times I tried opening it, it gave me a Timeout. How come?

johnpoz · Jun 22, 2021, 3:18 PM

@lddj9090 said in Diagnostics ARP Table 504 Nginx Gateway Timeout:

How come?

No idea.. But yeah it should load pretty much instant - its output of the arp table.. Not like it should have to be doing anything fancy with the formatting, etc. Or filtering, etc..

285 items isn't all that many, now if you said 28000 or something ;)

stephenw10 · Jun 22, 2021, 7:35 PM

Yeah, that seems unreasonable slow. How long does it take at the CLI to return arp -a?

The first time you run that it can take longer as it appears to reverse DNS lookup anything public.

Steve

johnpoz · Jun 22, 2021, 7:47 PM

It would also return anything local I would think, but those should already be in the dns if pfsense was doing local dns..

Not sure why anything non local would be in arp cache ;) Unless there were you were running public space, on your wan side sure.. But there really shouldn't be more than say your gateway in that network.

Only thing I see in my arp table on the public side, other than my own interface

The gateway.

stephenw10 · Jun 22, 2021, 8:23 PM

Yeah, I only noticed because I have a bunch of fake subnets on this test box and it resolved them:

ec2-100-23-58-10.us-west-2.compute.amazonaws.com (100.23.58.10) at 90:ec:77:0f:74:49 on lagg0 permanent [ethernet]
ec2-100-23-58-1.us-west-2.compute.amazonaws.com (100.23.58.1) at 02:51:3d:7f:87:00 on lagg0 expires in 226 seconds [ethernet]

And took noticeable time to do so.

If you were using a public subnet and had 300 entries I could imagine that introducing significant delay.

Steve

manicmoose · Jun 27, 2021, 12:02 PM

I'm seeing the same thing. I have 1074 entries currently in my arp table.

I don't recall there being so many arp entries on my WAN interface before - this is a new development.

The only change I've made recently is I added a second WAN and have them in a failover group....maybe that makes a difference??

johnpoz · Jun 27, 2021, 12:31 PM

@manicmoose said in Diagnostics ARP Table 504 Nginx Gateway Timeout:

maybe that makes a difference??

No, you really shouldn't be seeing arp entries in your public wan.. You should have no need to talk to these other ips. And no reason for them to arp for you, etc.

You could be on a network with 1000s of other IPs - but doesn't mean everyone has every other IP in their arp table. Only reason is they arp for your IP, or you arp for theirs for some reason. Or they are sending out gratuitous. Or some other sort of traffic where you would then need to arp..

Might want to sniff on your wan and see where the arps are coming from, some sort of traffic that has your machine then arp for the IP, or devices arping for yours?

Since they should expire the traffic would have to be something that is quite regular if your seeing that many entries.. Freebsd/Pfsense does use a long arp cache time of 20 minutes. But still to have that many listed has to be some sort of regular traffic causing it.

Your not monitoring say a broadcast IP for your wan network are you, vs just the gateway IP.. That could cause it as you would get bunch of devices reply to the arp for a broadcast address.

edit

I have 1074 entries currently in my arp table.

Do they all point to the same mac? Here I just pinged a couple of IPs on my wan network.. And they show up in arp table now since there was an answer - but they point to same mac (my cable modem)

manicmoose · Jun 27, 2021, 12:31 PM

@johnpoz Thanks John.

For both of the WANs, the 'gateway' and 'monitor' IP's are the same, so that should be okay...

This is what I'm seeing via a tcpdump on the WAN interface:

22:16:06.191949 ARP, Request who-has xxx.yyy.85.245 tell xxx.yyy.85.21, length 28
22:16:06.192019 IP xxx.yyy.85.21.41070 > xxx.yyy.85.42.5353: 54784 PTR (QM)? _services._dns-sd._udp.local. (46)
22:16:06.193097 ARP, Request who-has xxx.yyy.85.246 tell xxx.yyy.85.21, length 28
22:16:06.194178 ARP, Request who-has xxx.yyy.85.247 tell xxx.yyy.85.21, length 28
22:16:06.195259 ARP, Request who-has xxx.yyy.85.248 tell xxx.yyy.85.21, length 28
22:16:06.196338 ARP, Request who-has xxx.yyy.85.249 tell xxx.yyy.85.21, length 28
22:16:06.197414 ARP, Request who-has xxx.yyy.85.250 tell xxx.yyy.85.21, length 28
22:16:06.198492 ARP, Request who-has xxx.yyy.85.251 tell xxx.yyy.85.21, length 28
22:16:06.199571 ARP, Request who-has xxx.yyy.85.252 tell xxx.yyy.85.21, length 28
22:16:06.200648 ARP, Request who-has xxx.yyy.85.253 tell xxx.yyy.85.21, length 28

Is this the kind of thing you were talking about/looking for?

NB. My WAN IP is the: xxx.yyy.85.21

johnpoz · Jun 27, 2021, 12:38 PM

yeah for some reason your arping for the the whole range? That makes no sense..

But yeah if you are arping for them - and you get an answer, then yeah they would be in your arp table..

I have no idea what in pfsense would be doing that.. Mine sure isn't

@stephenw10 do you know of something in pfsense that could do that, or a package maybe? Arpwatch enabled on the wan interface?? Would be a guess, but I thought it just listens for arps - doesn't actual arp for stuff on its own.

manicmoose · Jun 27, 2021, 12:39 PM

@johnpoz Yep, I've never seen it before either.

Short of a reboot - is there some subsystem I could restart to try and 'reset' it's brain, or are there multiple involved making it difficult to tell which one(s) would need to be done?

Like the OP, I'm on 2.5.1-release as well.

Edit: I have arpwatch installed/enabled, but neither of the WANs is selected to be monitored.

johnpoz · Jun 27, 2021, 12:39 PM

Lets see if @stephenw10 has some idea - off the top of my head I have no idea what would cause that.. But does for sure explain why so many entries.

manicmoose · Jun 27, 2021, 12:41 PM

@johnpoz Sounds good.

I just checked the arpwatch 'database' entries, and none of the MACs on the WANs are in there - only local ones.

johnpoz · Jun 27, 2021, 12:53 PM

Yeah arpwatch shouldn't do it - but it was just off the cuff guess to something that "could" maybe do it..

Drawing a blank - maybe after some more coffee something will come to me.. But atleast we have a piece of the puzzle to work with.. What would cause pfsense to arp for every IP in a range that is on the wan.. Not thinking of any packages you could add that could be setup to scan??

The nmap package "could" do it for sure - but you would have to kick that off..

manicmoose · Jun 27, 2021, 12:53 PM

@johnpoz Hmm....good point - I have 'arping' installed, and that could have been at the same time I added the 2nd WAN (can't really recall).

Maybe it's the culprit as that's kind of what it's job is. I might uninstall it and see if that helps.

johnpoz · Jun 27, 2021, 1:03 PM

Yeah arping could prob do it too - but was not aware it had any sort of scanner option or that you could do it on a schedule sort of thing.. I currently don't have that installed.

manicmoose · Jun 27, 2021, 1:10 PM

@johnpoz Nope that's not it.

Uninstalled 'arping', then deleted all MACs on both WANs, leaving only the standard 2 on each.

My failover WAN is already back up to 127 again.
My primary WAN is up to 1021.

stephenw10 · Jun 27, 2021, 4:31 PM

Nope, I'm not sure what could cause that either. Weird.

I would check the system processes. See if you have anything running that is obviously and arp process.

johnpoz · Jun 27, 2021, 5:42 PM

checking what processes are running would be good start for sure - but also you could just do process of elimination - how many packages exactly do you have installed? Nmap could do it for sure - but then that would have, or should have to be triggered.

You could look to see if you have any crons scheduled.