WAN is up, but gateway is down

Lanes

I went from a stable and working pfSense install to some really weird issues overnight.

I can access my pfSense box via VPN
I can ping external hosts from pfSense web gui
I cannot access the internet from the LAN

Apinger reports the gateway being down, and the logs show some message I do not understand. This message appears 18 times in a 10 min window, but now it just logs WAN being down after a few reboot attempts by me.
"Could not bind socket on address() for monitoring address (WAN_DHCP) with error Can't assign requested address"

I have left out the IPs in the below message.

What's going on? Am I looking at the correct logs? Is my ISP blocking my pfSense box?

I should also mention that squid is crashing when starting squidGuard, but I had that issue at some point before, and don't think it is related.

Lanes

I just realised from seeing the logged ip in the above message that I can access the Internet when connecting via VPN, but not directly from the LAN.

Posting from my iPhone via 4G now, but had forgotten to d/c the VPN connection.

Also, I am running 2.2.

Mr. Jingles

Can you try an alternative monitoring IP for that gateway? I've always had to do that, otherwise the gateway appears down. Put in a 8.8.8.8 for testing, for example.

Lanes

@Mr.:

Can you try an alternative monitoring IP for that gateway? I've always had to do that, otherwise the gateway appears down. Put in a 8.8.8.8 for testing, for example.

I already changed it to 8.8.8.8 with the same results. The gateway is actually reported as 'up' for a brief moment, but it drops within seconds. Not sure if it ever is up, or if the system just defaults to report it as up until it is confirmed down.

stephenw10

@Lanes:

I cannot access the internet from the LAN

How are you testing this? Since you have Squid and Squidguard installed testing general web access may be affected. Can you ping external sites from the LAN subnet?

What sort of IPs are those that you've left out, real public addresses or private addresses? Do they look correct from what you were seeing before?

Did this behaviour start spontaneously or when you upgraded to 2.2?

You could try disabling gateway monitoring completely or enabling debug logging to get more info in System > Advanced, Miscellaneous tab.

Steve

Visseroth

in System -> Routing edit your WAN interface and see if you can set your gateway static and try again and if you are using DHCP make sure it says "dynamic" or just delete what is set in the gateway.

It's worth a shot

Lanes

I had my ISP start logging my connection over the weekend, but they have yet to find anything.

In the meantime Snort started to crash and a web server behind the firewall was given an IP in the 10.50.1.0/24 range, it is normally configured to get a static IP and I have no idea where it got the IP from…

I completely removed Squid and did a ipconfig /refresh on all devices, and now everything is working again (apart from Snort). Squid was disabled from the gui, but appeared to have some remnants giving errors.

I have not yet figured out what caused those logs from the gateway, and I can't even be bothered to start investigating what causes Snort to shut down. I'll wipe the disk and do a complete reinstallation.

stephenw10

If you don't have a dhcp server running on pfSense then you might have some rogue dhcp server on your network. That can cause all sorts of odd symptoms.

Steve

q54e3w

I hope you don't mind me gate crashing your thread, I'm having a similar issue and perhaps have some more info that might help someone help us. I'm using Virgin in the UK with their (not very)SuperHub in modem mode to avoid any double NAT etc. I've noticed this issue and suspect it may have started as I migrated through the 2.2 builds.

It usually starts with my modem dropping the line for some reason, modem crashes(?) or possibly DHCP expiry etc.

From my limited knowledge of networking it links like the events that take place look like this….

VPN and Default Gateway go down and show no connection
This process appears to repeat until something is kicked/rebooted
WAN polls DHCP server, no DHCPOFFERS received
WAN assigns itself recorded lease 213.81.85.148 & Router 213.81.85.1
Deletes existing routes and tries for a DHCP from ISP servers
Fails due to no route...and repeat....

The DHCP logs look like this....(igb4=WAN)

Feb 16 16:00:04 pfsense dhclient: PREINIT
Feb 16 16:00:04 pfsense dhclient: EXPIRE
Feb 16 16:00:04 pfsense dhclient: Deleting old routes
Feb 16 16:00:04 pfsense dhclient: PREINIT
Feb 16 16:00:04 pfsense dhclient[36312]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 1
Feb 16 16:00:05 pfsense dhclient[36312]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 2
Feb 16 16:00:07 pfsense dhclient[36312]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 3
Feb 16 16:00:11 pfsense dhclient[36312]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 7
Feb 16 16:00:18 pfsense dhclient[36312]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 14
Feb 16 16:00:32 pfsense dhclient[36312]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 16
Feb 16 16:00:48 pfsense dhclient[36312]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 8
Feb 16 16:00:56 pfsense dhclient[36312]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 9
Feb 16 16:01:05 pfsense dhclient[36312]: No DHCPOFFERS received.
Feb 16 16:01:05 pfsense dhclient[36312]: Trying recorded lease 213.81.85.148
Feb 16 16:01:05 pfsense dhclient: TIMEOUT
Feb 16 16:01:05 pfsense dhclient: Starting add_new_address()
Feb 16 16:01:05 pfsense dhclient: ifconfig igb4 inet 213.81.85.148 netmask 255.255.255.0 broadcast 255.255.255.255 
Feb 16 16:01:05 pfsense dhclient: New IP Address (igb4): 213.81.85.148
Feb 16 16:01:05 pfsense dhclient: New Subnet Mask (igb4): 255.255.255.0
Feb 16 16:01:05 pfsense dhclient: New Broadcast Address (igb4): 255.255.255.255
Feb 16 16:01:05 pfsense dhclient: New Routers (igb4): 213.81.85.1
Feb 16 16:01:06 pfsense dhclient: New Routers (igb4): 213.81.85.1
Feb 16 16:01:07 pfsense dhclient: Deleting old routes
Feb 16 16:01:07 pfsense dhclient[36312]: bound: immediate renewal.
Feb 16 16:01:07 pfsense dhclient[36312]: DHCPREQUEST on igb4 to 62.253.131.54 port 67
Feb 16 16:01:07 pfsense dhclient[37920]: send_packet: No route to host
Feb 16 16:01:07 pfsense dhclient[80138]: DHCPREQUEST on igb4 to 62.253.131.54 port 67
Feb 16 16:01:07 pfsense dhclient[37920]: send_packet: No route to host
Feb 16 16:01:09 pfsense dhclient[80138]: DHCPREQUEST on igb4 to 62.253.131.54 port 67
Feb 16 16:01:09 pfsense dhclient[37920]: send_packet: No route to host

N.B 62.253.131.54 is Virgins hardware.

My gateway.log is full of this error…

Feb 16 16:08:33 pfsense apinger: Could not bind socket on address(213.81.85.148) for monitoring address 213.81.85.1(WAN_DHCP) with error Can't assign requested address

The only way to resolve this appears to be to either reset the modem and my pfSense box.

Rebooting the modem alone starts of a process which looks something like
Modem assigns itself 192.168.100.1, assign WAN 192.168.100.3, wait 30 seconds for renewal, error 'cant bind to dhcp address'…repeat forever until pfSense rebooted too.

Feb 16 16:11:44 pfsense dhclient[19523]: igb4 link state up -> down
Feb 16 16:11:47 pfsense dhclient[19523]: igb4 link state down -> up
Feb 16 16:11:47 pfsense dhclient: EXPIRE
Feb 16 16:11:47 pfsense dhclient: Deleting old routes
Feb 16 16:11:47 pfsense dhclient: PREINIT
Feb 16 16:11:47 pfsense dhclient[19523]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 1
Feb 16 16:11:48 pfsense dhclient[19523]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 1
Feb 16 16:11:49 pfsense dhclient[19523]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 1
Feb 16 16:11:50 pfsense dhclient[19523]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 2
Feb 16 16:11:52 pfsense dhclient[19523]: DHCPDISCOVER on igb4 to 255.255.255.255 port 67 interval 2
Feb 16 16:11:52 pfsense dhclient[19523]: DHCPOFFER from 192.168.100.1
Feb 16 16:11:52 pfsense dhclient: ARPSEND
Feb 16 16:11:54 pfsense dhclient: ARPCHECK
Feb 16 16:11:54 pfsense dhclient[19523]: DHCPREQUEST on igb4 to 255.255.255.255 port 67
Feb 16 16:11:55 pfsense dhclient[19523]: DHCPACK from 192.168.100.1
Feb 16 16:11:55 pfsense dhclient: BOUND
Feb 16 16:11:55 pfsense dhclient: Starting add_new_address()
Feb 16 16:11:55 pfsense dhclient: ifconfig igb4 inet 192.168.100.3 netmask 255.255.255.0 broadcast 192.168.100.255 
Feb 16 16:11:55 pfsense dhclient: New IP Address (igb4): 192.168.100.3
Feb 16 16:11:55 pfsense dhclient: New Subnet Mask (igb4): 255.255.255.0
Feb 16 16:11:55 pfsense dhclient: New Broadcast Address (igb4): 192.168.100.255
Feb 16 16:11:55 pfsense dhclient: New Routers (igb4): 192.168.100.1
Feb 16 16:11:55 pfsense dhclient: Adding new routes to interface: igb4
Feb 16 16:11:55 pfsense dhclient: /sbin/route add default 192.168.100.1
Feb 16 16:11:55 pfsense dhclient: Creating resolv.conf
Feb 16 16:11:55 pfsense dhclient[19523]: bound to 192.168.100.3 -- renewal in 30 seconds.
Feb 16 16:11:59 pfsense dhcpd: Internet Systems Consortium DHCP Server 4.2.6
Feb 16 16:11:59 pfsense dhcpd: Copyright 2004-2014 Internet Systems Consortium.
Feb 16 16:11:59 pfsense dhcpd: All rights reserved.
Feb 16 16:11:59 pfsense dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb 16 16:11:59 pfsense dhcpd: Internet Systems Consortium DHCP Server 4.2.6
Feb 16 16:11:59 pfsense dhcpd: Copyright 2004-2014 Internet Systems Consortium.
Feb 16 16:11:59 pfsense dhcpd: All rights reserved.
Feb 16 16:11:59 pfsense dhcpd: For info, please visit https://www.isc.org/software/dhcp/
Feb 16 16:11:59 pfsense dhcpd: Wrote 0 deleted host decls to leases file.
Feb 16 16:11:59 pfsense dhcpd: Wrote 0 new dynamic host decls to leases file.
Feb 16 16:11:59 pfsense dhcpd: Wrote 47 leases to leases file.
Feb 16 16:11:59 pfsense dhcpd: Listening on BPF/igb0/a0:36:9f:2d:eb:a8/192.168.10.0/24
Feb 16 16:11:59 pfsense dhcpd: Sending on   BPF/igb0/a0:36:9f:2d:eb:a8/192.168.10.0/24
Feb 16 16:11:59 pfsense dhcpd: Listening on BPF/ix1/90:e2:ba:37:fb:a5/192.168.110.0/24
Feb 16 16:11:59 pfsense dhcpd: Sending on   BPF/ix1/90:e2:ba:37:fb:a5/192.168.110.0/24
Feb 16 16:11:59 pfsense dhcpd: Listening on BPF/ix0/90:e2:ba:37:fb:a4/192.168.101.0/24
Feb 16 16:11:59 pfsense dhcpd: Sending on   BPF/ix0/90:e2:ba:37:fb:a4/192.168.101.0/24
Feb 16 16:11:59 pfsense dhcpd: Listening on BPF/igb6/0c:c4:7a:07:d5:52/192.168.60.0/24
Feb 16 16:11:59 pfsense dhcpd: Sending on   BPF/igb6/0c:c4:7a:07:d5:52/192.168.60.0/24
Feb 16 16:11:59 pfsense dhcpd: Listening on BPF/igb7/0c:c4:7a:07:d5:53/192.168.70.0/24
Feb 16 16:11:59 pfsense dhcpd: Sending on   BPF/igb7/0c:c4:7a:07:d5:53/192.168.70.0/24
Feb 16 16:11:59 pfsense dhcpd: Listening on BPF/igb5/0c:c4:7a:07:d5:51/192.168.50.0/24
Feb 16 16:11:59 pfsense dhcpd: Sending on   BPF/igb5/0c:c4:7a:07:d5:51/192.168.50.0/24
Feb 16 16:11:59 pfsense dhcpd: Can't bind to dhcp address: Address already in use
Feb 16 16:11:59 pfsense dhcpd: Please make sure there is no other dhcp server
Feb 16 16:11:59 pfsense dhcpd: running and that there's no entry for dhcp or
Feb 16 16:11:59 pfsense dhcpd: bootp in /etc/inetd.conf.   Also make sure you
Feb 16 16:11:59 pfsense dhcpd: are not running HP JetAdmin software, which
Feb 16 16:11:59 pfsense dhcpd: includes a bootp server.
Feb 16 16:11:59 pfsense dhcpd: 
Feb 16 16:11:59 pfsense dhcpd: If you did not get this software from ftp.isc.org, please
Feb 16 16:11:59 pfsense dhcpd: get the latest from ftp.isc.org and install that before
Feb 16 16:11:59 pfsense dhcpd: requesting help.
Feb 16 16:11:59 pfsense dhcpd: 
Feb 16 16:11:59 pfsense dhcpd: If you did get this software from ftp.isc.org and have not
Feb 16 16:11:59 pfsense dhcpd: yet read the README, please read it before requesting help.
Feb 16 16:11:59 pfsense dhcpd: If you intend to request help from the dhcp-bugs at isc.org
Feb 16 16:11:59 pfsense dhcpd: mailing list, please read the section on the README about
Feb 16 16:11:59 pfsense dhcpd: submitting bug reports and requests for help.
Feb 16 16:11:59 pfsense dhcpd: 
Feb 16 16:11:59 pfsense dhcpd: Please do not under any circumstances send requests for
Feb 16 16:11:59 pfsense dhcpd: help directly to the authors of this software - please
Feb 16 16:11:59 pfsense dhcpd: send them to the appropriate mailing list as described in
Feb 16 16:11:59 pfsense dhcpd: the README file.
Feb 16 16:11:59 pfsense dhcpd: 
Feb 16 16:11:59 pfsense dhcpd: exiting.
Feb 16 16:11:59 pfsense dhclient[30299]: connection closed
Feb 16 16:11:59 pfsense dhclient[30299]: exiting.
Feb 16 16:12:00 pfsense dhclient: PREINIT
Feb 16 16:12:00 pfsense dhclient[16209]: DHCPREQUEST on igb4 to 255.255.255.255 port 67
Feb 16 16:12:01 pfsense dhclient[16209]: DHCPACK from 192.168.100.1
Feb 16 16:12:01 pfsense dhclient: REBOOT

When everything works properly, the Gateway is assigned 213.81.85.1 and my WAN 213.81.85.148.

Would love to hear if there's a way to coerce the pfsense box to return to normal duties after one of these hiccups.

xtofh

Just wanted to let you know that I have the same issue with my cablre provider (Telenet in Belgium).

It also occured since upgrading to 2.2. (coming from 2.1.5)

Similar issues in the post here: https://forum.pfsense.org/index.php?topic=88068.15

stephenw10

Are you seeing identical issues to those described there?
You can trigger this by simply pulling and replacing the WAN connection? And only rebooting fixes it?
Have you tried a switch in between the WAN interface and the modem?

Steve

xtofh

Ouch, the issue I'm having is slightly different, sorry about that..
(however, pulling wan does not change anything, reboot does)

I don't have a switch between wan/modem, never had. (no problems with 2.1.5)

stephenw10

Try putting a switch in between. If the issue is triggered by the link going down momentarily that can prove it.
Interesting that pulling WAN does nothing. Also that 2.1.5 worked fine. :-\

Steve

xtofh

Ok, good advice.

I've put a switch in between, 2.2 active again, let's see how it goes. I'll report back. (in a few hours or days, depending on the working)

iorx

Hi!

Reusing topic because it same-same here. I've received my ISP dhcp logfiles and it looks like pfsense just stops asking for an IP-address. As result I loose my WAN.
Can't really find in my logs what is causing this.

Need help where to look and how to resolve this.

pfSense 2.2.2

pfSense searching for IP. Get a offer but doesn't request.


Jun  7 19:26:55 dhcpd: DHCPDISCOVER from 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:26:56 dhcpd: DHCPOFFER on 10.100.100.170 to 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:27:15 dhcpd: DHCPDISCOVER from 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:27:15 dhcpd: DHCPOFFER on 10.100.100.170 to 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:27:17 dhcpd: DHCPDISCOVER from 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:27:17 dhcpd: DHCPOFFER on 10.100.100.170 to 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:27:20 dhcpd: DHCPDISCOVER from 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:27:20 dhcpd: DHCPOFFER on 10.100.100.170 to 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:27:24 dhcpd: DHCPDISCOVER from 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:27:24 dhcpd: DHCPOFFER on 10.100.100.170 to 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:27:29 dhcpd: DHCPDISCOVER from 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:27:29 dhcpd: DHCPOFFER on 10.100.100.170 to 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:27:52 dhcpd: DHCPDISCOVER from 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:28:09 dhcpd: DHCPDISCOVER from 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:28:09 dhcpd: DHCPOFFER on 10.100.100.170 to 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:28:56 dhcpd: DHCPDISCOVER from 00:15:5e:02:81:17 (fw) via 10.100.100.161
Jun  7 19:28:56 dhcpd: DHCPOFFER on 10.100.100.170 to 00:15:5e:02:81:17 (fw) via 10.100.100.161

xtofh

iorx, if you're using intel e1000 physical nic's, try the solution I implemented (thanks to cmb) last friday:

https://forum.pfsense.org/index.php?topic=96325.0

Until now (5 days and counting) it's going good, so I'm hopeful.