Since a few day's losing the wan (dhcp) ip address

TravisD

I'll try to scare up more details, but I was having this same problem on 2.0.2, so I went ahead and moved to a 2.1 BETA to see if it helped. No real improvement. For a while I could get the WAN IP back by release/renew in the Status->Interfaces page.

Since it just started happening all of a sudden I've been assuming it was something external to pfSense. I went ahead and swapped out my cablemodem since Comcast has been warning me that it needed to be done anyway since my old one was not DOCSIS 3.0.

No change there, so swapped in a new USB Ethernet adaptor (SiiG) and still no change. New ethernet cable too.

It's been stable for a bit now, but I still don't trust it. Only thing that seems to be at all different now is that I killed by Bittorrent client (only had a single torrent running, nothing excessive). I'll try bringing it back up later if things seem stable and see what happens. Also just updated to the latest snapshot.

The is running on a 1.6GHz Ion, booting from USB (NanoBSD image).

If it comes back, I will return with logs and such.

Zorac

I've tested with two different modems as well with no improvement. 2.0.1 had no issues, and early versions of the 2.1 beta were reliable for me as well.

TravisD

No luck, died again. Couldn't even get it back with release/renew in the GUI.

Feb 26 23:19:36 gw kernel: ue0: link state changed to UP
Feb 26 23:19:37 gw php: : DEVD Ethernet detached event for wan
Feb 26 23:19:37 gw dhclient[39419]: connection closed
Feb 26 23:19:37 gw dhclient[39419]: exiting.
Feb 26 23:19:37 gw php: : The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf ue0 > /tmp/ue0_output 2> /tmp/ue0_error_output' returned exit code '15', the output was '' 
Feb 26 23:19:39 gw php: : DEVD Ethernet attached event for wan
Feb 26 23:19:39 gw php: : HOTPLUG: Configuring interface wan
Feb 26 23:19:39 gw dhclient: PREINIT
Feb 26 23:19:39 gw kernel: ue0: link state changed to DOWN
Feb 26 23:19:39 gw check_reload_status: Linkup starting ue0
Feb 26 23:19:39 gw kernel: ue0: link state changed to DOWN
Feb 26 23:19:39 gw dhclient: Starting delete_old_states()
Feb 26 23:19:39 gw dhclient: Comparing IPs: Old: New: 
Feb 26 23:19:39 gw dhclient: Comparing Routers: Old: New: 
Feb 26 23:19:39 gw dhclient[56796]: DHCPREQUEST on ue0 to 255.255.255.255 port 67
Feb 26 23:19:40 gw dhclient[56796]: DHCPREQUEST on ue0 to 255.255.255.255 port 67
Feb 26 23:19:41 gw dhclient[56796]: DHCPREQUEST on ue0 to 255.255.255.255 port 67
Feb 26 23:19:41 gw check_reload_status: Linkup starting ue0
Feb 26 23:19:41 gw kernel: ue0: link state changed to UP
Feb 26 23:19:42 gw php: : DEVD Ethernet detached event for wan
Feb 26 23:19:42 gw dhclient[56945]: connection closed
Feb 26 23:19:42 gw dhclient[56945]: exiting.
Feb 26 23:19:42 gw php: : The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf ue0 > /tmp/ue0_output 2> /tmp/ue0_error_output' returned exit code '15', the output was '' 
Feb 26 23:19:44 gw php: : DEVD Ethernet attached event for wan
Feb 26 23:19:44 gw php: : HOTPLUG: Configuring interface wan
Feb 26 23:19:44 gw dhclient: PREINIT
Feb 26 23:19:44 gw kernel: ue0: link state changed to DOWN
Feb 26 23:19:44 gw check_reload_status: Linkup starting ue0
Feb 26 23:19:44 gw kernel: ue0: link state changed to DOWN
Feb 26 23:19:44 gw dhclient: Starting delete_old_states()
Feb 26 23:19:44 gw dhclient: Comparing IPs: Old: New: 
Feb 26 23:19:44 gw dhclient: Comparing Routers: Old: New: 
Feb 26 23:19:44 gw dhclient[68249]: DHCPREQUEST on ue0 to 255.255.255.255 port 67
Feb 26 23:19:46 gw dhclient[68249]: DHCPREQUEST on ue0 to 255.255.255.255 port 67
Feb 26 23:19:46 gw check_reload_status: Linkup starting ue0
Feb 26 23:19:46 gw kernel: ue0: link state changed to UP
Feb 26 23:19:47 gw php: : DEVD Ethernet detached event for wan
Feb 26 23:19:47 gw dhclient[68398]: connection closed
Feb 26 23:19:47 gw dhclient[68398]: exiting.
Feb 26 23:19:47 gw php: : The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf ue0 > /tmp/ue0_output 2> /tmp/ue0_error_output' returned exit code '15', the output was '' 
Feb 26 23:19:49 gw php: : DEVD Ethernet attached event for wan
Feb 26 23:19:49 gw php: : HOTPLUG: Configuring interface wan
Feb 26 23:19:49 gw dhclient: PREINIT
Feb 26 23:19:49 gw kernel: ue0: link state changed to DOWN
Feb 26 23:19:49 gw dhclient: Starting delete_old_states()
Feb 26 23:19:49 gw kernel: ue0: link state changed to DOWN
Feb 26 23:19:49 gw dhclient: Comparing IPs: Old: New: 
Feb 26 23:19:49 gw check_reload_status: Linkup starting ue0
Feb 26 23:19:49 gw dhclient: Comparing Routers: Old: New: 
Feb 26 23:19:49 gw dhclient[72692]: DHCPREQUEST on ue0 to 255.255.255.255 port 67
Feb 26 23:19:50 gw dhclient[72692]: DHCPREQUEST on ue0 to 255.255.255.255 port 67
Feb 26 23:19:51 gw check_reload_status: Linkup starting ue0
Feb 26 23:19:51 gw kernel: ue0: link state changed to UP

Going off now to try and downgrade to 2.0.1. Any pointers to where to find it for download still?

Zorac

some of the mirrors keep the old versions….

http://pfsense.mirrors.ovh.net/pfsense.org/downloads/old/

let me know how it works out for you.

TravisD

Thanks! That had it. Wasn't able to downgrade via the GUI, so off to yank the USB stick and re-image it.

TravisD

Update: Doesn't look like this is a 2.1 issue, unless it's something in the 2.1 config file.

I did a clean re-install of 2.0.1 and restored my config file (taken from the 2.1 instance). Same problem still occurring. Guess I'm in the wrong sub forum at this point :(

Back to the drawing board…

abeowitz

I've been having this problem since January. I found that if I pull out the ethernet cable and replug it back in, it'll start working again for a while.

It's been getting worse recently. Thought it was my DSL or Clearwire, but bypassing the FW proved they were fine.

I thought it might be my USB ethernet adapters. (Since my box only has one ethernet port, I have to use USB for WAN and OPT1.) But other brands of USB ethernet also failed.

After reading this thread, I went back to 2.0.2, but forgot that my ethernet card isn't supported in FreeBSD 8.1, so I tried the 2.1 beta disc I made back on Nov 29th, and things seem very stable at the moment. – I started fresh with minimal configuration/tweaking and have not upgraded.

I did have an AMD proc and powernow settings enabled. In retrospect, I should've tested that, or some of the other advanced ethernet settings...

TravisD

Update: Stable overnite running on 2.0.1. Will continue to monitor.

tworush

woutervb
I have the same problem my work around is creating a script which pings google and if packet lost my network card restarts

copy and save this file in the root home directory called wan_check.sh

#!/usr/bin/env bash
#/usr/local/bin/network_check

# Script to monitor and restart network card

maxPloss=10 #Maximum percent packet loss before a restart

restart_networking() {
        # Add any commands need to get network back up and running
        sh /root/wan.sh

}

# First make sure we can resolve google, otherwise 'ping -w' would hang
if ! $(host -W5 www.google.com > /dev/null 2>&1); then
        #Make a note in syslog
        logger "network_check: Network connection is down, restarting network ..."
        restart_networking
        exit
fi

# Initialize to a value that would force a restart
# (just in case ping gives an error and ploss doesn't get set)
ploss=101
# now ping google for 10 seconds and count packet loss
ploss=$(ping -q -w10 www.google.com | grep -o "[0-9]*%" | tr -d %) > /dev/null 2>&1

if [ "$ploss" -gt "$maxPloss" ]; then
        logger "Packet loss ($ploss%) exceeded $maxPloss, restarting network ..."
        restart_networking
fi

This code will restart Wan if ping loses packets. change de1 to your wan interface
name this file wan.sh

ifconfig de1 down
sleep 5
ifconfig de1 up
dhclient de1

change the attriputes (Chmod) to 777 on both files
and setup cron job to check the wan interface every 30 min
30 0 * * * root /root/wan_check.sh

Note I had to make the folowing change because running under Hyper-v. Change Speed and duplex to "100 base tx full duplex" this located under interface–> WAN

abeowitz

What do you all have powerd set to?

Mine seems to fail when powerd is checked. But that could be just coincidence… (Looking for patterns.)

Zorac

mine is unchecked (off)

abeowitz

So, I updated from 2012-11-29 to today and the problem returned.

However, reconfiguring from scratch, I made the WAN & OPT IP's static, and it seems stable. (Though static means I have to NAT again through the modems.)

TravisD

Do you guys have OpenVPN configured (but perhaps not used) by any chance? I'm starting to think that this may have started for me about the time I started messing with OpenVPN (I got interested when the IOS OpenVPN App came out). I came across a post here somewhere that seemed to correlate one of the various error messages with OpenVPN.

I just whacked the OpenVPN config on my system to see if it stabilizes at all. We'll see what happens.

Zorac

I do have openvpn configured. I use it for my tablet when I'm away from home.

TravisD

I just did a 100% clean install of 2.1. Didn't bring in any old config, just configured a handfull of NATs that I need. No VPN, etc. We'll see how this one goes…

TravisD

At least I seem to be moving the ball around… (Oh, my apologies for spreading this around to multiple areas of the support forum. Since I'm now on 2.1, I'll try and put things here and redirect the others to here. Bad forum behavior, I know - sorry!)

Fresh, clean, no-legacy install of 2.1 -- still fails, but failed differently this time.


Feb 28 23:26:00 gw check_reload_status: Linkup starting ue0
Feb 28 23:26:00 gw kernel: ue0: link state changed to DOWN
Feb 28 23:26:00 gw kernel: ue0: link state changed to UP
Feb 28 23:26:01 gw check_reload_status: Linkup starting ue0
Feb 28 23:26:03 gw php: : DEVD Ethernet detached event for wan
Feb 28 23:26:03 gw dhclient[52833]: connection closed
Feb 28 23:26:03 gw dhclient[52833]: exiting.
Feb 28 23:26:03 gw php: : DEVD Ethernet attached event for wan
Feb 28 23:26:03 gw php: : HOTPLUG: Configuring interface wan
Feb 28 23:26:03 gw dhclient: PREINIT
Feb 28 23:26:03 gw kernel: arpresolve: can't allocate llinfo for 68.48.154.1
Feb 28 23:26:03 gw kernel: arpresolve: can't allocate llinfo for 68.48.154.1
Feb 28 23:26:03 gw dhclient: Starting delete_old_states()
Feb 28 23:26:03 gw dhclient: Comparing IPs: Old: New: 
Feb 28 23:26:03 gw dhclient: Comparing Routers: Old: New: 
Feb 28 23:26:03 gw dhclient[68325]: DHCPREQUEST on ue0 to 255.255.255.255 port 67
Feb 28 23:26:03 gw dhclient[68325]: DHCPACK from 73.129.106.1
Feb 28 23:26:03 gw dhclient: REBOOT
Feb 28 23:26:03 gw dhclient: Starting delete_old_states()
Feb 28 23:26:03 gw dhclient: Comparing IPs: Old: New: 68.48.155.52
Feb 28 23:26:03 gw dhclient: Comparing Routers: Old: New: 68.48.154.1
Feb 28 23:26:03 gw dhclient: Removing states through old gateway '' (new gateway '68.48.154.1')
Feb 28 23:26:03 gw dhclient: Starting add_new_address()
Feb 28 23:26:03 gw dhclient: ifconfig ue0 inet 68.48.155.52 netmask 255.255.254.0 broadcast 255.255.255.255 
Feb 28 23:26:03 gw dhclient: New IP Address (ue0): 68.48.155.52
Feb 28 23:26:03 gw dhclient: New Subnet Mask (ue0): 255.255.254.0
Feb 28 23:26:03 gw dhclient: New Broadcast Address (ue0): 255.255.255.255
Feb 28 23:26:03 gw dhclient: New Routers (ue0): 68.48.154.1
Feb 28 23:26:03 gw dhclient: Adding new routes to interface: ue0
Feb 28 23:26:03 gw dhclient: /sbin/route add default 68.48.154.1
Feb 28 23:26:03 gw dhclient: Creating resolv.conf
Feb 28 23:26:03 gw check_reload_status: rc.newwanip starting ue0
Feb 28 23:26:03 gw dhclient[68325]: bound to 68.48.155.52 -- renewal in 163608 seconds.
Feb 28 23:26:04 gw php: : Clearing states to old gateway 68.48.154.1.
Feb 28 23:26:06 gw php: : rc.newwanip: Informational is starting ue0.
Feb 28 23:26:06 gw php: : rc.newwanip: on (IP address: ) (interface: wan) (real interface: ue0).
Feb 28 23:26:06 gw php: : rc.newwanip: Failed to update wan IP, restarting…
Feb 28 23:26:06 gw check_reload_status: Configuring interface wan
Feb 28 23:26:08 gw dhclient[78874]: dhclient already running, pid: 76098.
Feb 28 23:26:08 gw dhclient[78874]: exiting.
Feb 28 23:26:08 gw php: : The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf ue0 > /tmp/ue0_output 2> /tmp/ue0_error_output' returned exit code '1', the output was '' 
Feb 28 23:26:22 gw check_reload_status: Updating all dyndns
Feb 28 23:26:22 gw check_reload_status: Restarting ipsec tunnels
Feb 28 23:26:22 gw check_reload_status: Restarting OpenVPN tunnels/interfaces
Feb 28 23:26:22 gw check_reload_status: Reloading filter
Feb 28 23:26:25 gw php: : DynDns: updatedns() starting
Feb 28 23:26:25 gw php: : DynDNS (travisd.dyndns.org): running get_failover_interface for wan. found ue0
Feb 28 23:26:25 gw php: : DynDNS (travisd.dyndns.org) There was an error trying to determine the public IP for interface - wan(ue0). Probably interface is not a WAN interface.
Feb 28 23:26:25 gw php: : Could not find IPv4 gateway for interface (wan).
Feb 28 23:26:25 gw php: : Could not find IPv4 gateway for interface (wan).
Feb 28 23:26:25 gw php: : Could not find IPv4 gateway for interface (wan).
Feb 28 23:26:25 gw php: : Could not find IPv4 gateway for interface (wan).
Feb 28 23:26:25 gw php: : Could not find IPv4 gateway for interface (wan).
Feb 28 23:26:26 gw php: : DynDns: updatedns() starting
Feb 28 23:26:26 gw php: : DynDNS (travisd.dyndns.org): running get_failover_interface for wan. found ue0
Feb 28 23:26:26 gw php: : DynDNS (travisd.dyndns.org) There was an error trying to determine the public IP for interface - wan(ue0). Probably interface is not a WAN interface.

Files as they looked before release/renew:


[2.1-BETA1][root@gw.tubas.net]/root(7): cat /tmp/ue0*
fe80::201:5cff:fe22:1281
dhclient already running, pid: 76098.
exiting.

[2.1-BETA1][root@gw.tubas.net]/var/etc(13): cat dhclient_wan.conf 
interface "ue0" {
timeout 60;
retry 15;
select-timeout 0;
initial-interval 1;

 script "/sbin/dhclient-script";
}

No significant DHCP traffic on tcpdump at this time on the WAN interface (filtering on port 67). Removing the filter, I saw normal looking arp traffic.

tcpdump while doing release/renew (at least that works again to restore service!) produced minimal output, but it looks reasonable to me:


23:32:43.335315 IP (tos 0x10, ttl 16, id 0, offset 0, flags [none], proto UDP (17), length 328)
    c-68-48-155-52.hsd1.md.comcast.net.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 00:50:b6:59:70:f5 (oui Unknown), length 300, xid 0xba01a458, Flags [none] (0x0000)
   Client-Ethernet-Address 00:50:b6:59:70:f5 (oui Unknown) [|bootp]
23:32:43.354619 IP (tos 0x0, ttl 64, id 60020, offset 0, flags [none], proto UDP (17), length 338)
    73.129.106.1.bootps > c-68-48-155-52.hsd1.md.comcast.net.bootpc: BOOTP/DHCP, Reply, length 310, hops 1, xid 0xba01a458, Flags [none] (0x0000)
   Your-IP c-68-48-155-52.hsd1.md.comcast.net
   Gateway-IP 73.129.106.1
   Client-Ethernet-Address 00:50:b6:59:70:f5 (oui Unknown) [|bootp]

And again the post-renew output from the various dhcp files:


[2.1-BETA1][root@gw.tubas.net]/var/etc(18): cat /tmp/ue0*
fe80::201:5cff:fe22:1281
dhclient: PREINIT
dhclient: Starting delete_old_states()
OLD_IP: not found
dhclient: Comparing IPs: Old: New: 
dhclient: Comparing Routers: Old: New: 
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPACK from 73.129.106.1
bound to 68.48.155.52 -- renewal in 163409 seconds.
68.48.154.1
[2.1-BETA1][root@gw.tubas.net]/var/etc(19): cat dhc
dhclient_wan.conf dhcp6c_wan_script.sh* 
[2.1-BETA1][root@gw.tubas.net]/var/etc(19): cat dhclient_wan.conf 
interface "ue0" {
timeout 60;
retry 15;
select-timeout 0;
initial-interval 1;

 script "/sbin/dhclient-script";
}

So, something changed by doing a fresh install of the 2.1 snapshot, but things still seme broken.

abeowitz

Even with static IPs on WAN and OPT1, I still get lots of these in my dmesg logs:

ue1: link state changed to DOWN
ue1: link state changed to UP
ue0: link state changed to DOWN
ue0: link state changed to UP
pfsync0: promiscuous mode disabled
ue0: link state changed to DOWN
ue0: link state changed to UP
ue0: link state changed to DOWN
ue0: link state changed to UP
pfsync0: promiscuous mode enabled

I'm getting small % packet loss, which seems to correlate to the link states. But there's no interruption in web surfing.

I don't have OpenVPN enabled/configured. I do have the export keys module installed, but haven't done anything with that yet.

TravisD

So I had configured a couple of dynamic DNS updaters, but decided to pull them out for one reason or another.

Upon deleting them, this appears in the system.log:


Mar  1 01:05:29 gw php: /services_dyndns.php: The command '/bin/rm /conf/dyndns_*.cache' returned exit code '1', the output was 'rm: /conf/dyndns_wandyndns'travisd.dyndns.org'0.cache: Read-only file system rm: /conf/dyndns_wanopendns'travisd.dyndns.org'1.cache: Read-only file system'

…and the WAN link dies again (loses it's IP address)

Manually trying to remove these replicates the issue:


[2.1-BETA1][root@gw.tubas.net]/conf(12): rm -f dyndns_*
rm: dyndns_wandyndns'travisd.dyndns.org'0.cache: Read-only file system
rm: dyndns_wanopendns'travisd.dyndns.org'1.cache: Read-only file system

Edit: More logs, this is what spits out now when it dies (and before manual recovery via release/renew)


Mar  1 01:34:58 gw check_reload_status: Linkup starting ue0
Mar  1 01:34:58 gw kernel: ue0: link state changed to DOWN
Mar  1 01:34:58 gw kernel: ue0: link state changed to UP
Mar  1 01:34:58 gw check_reload_status: Linkup starting ue0
Mar  1 01:34:58 gw kernel: ue0: link state changed to DOWN
Mar  1 01:34:58 gw kernel: ue0: link state changed to UP
Mar  1 01:34:58 gw check_reload_status: Linkup starting ue0
Mar  1 01:34:58 gw check_reload_status: Linkup starting ue0
Mar  1 01:35:01 gw php: : DEVD Ethernet detached event for wan
Mar  1 01:35:01 gw php: : DEVD Ethernet attached event for wan
Mar  1 01:35:01 gw dhclient[23035]: connection closed
Mar  1 01:35:01 gw dhclient[23035]: exiting.
Mar  1 01:35:01 gw php: : DEVD Ethernet detached event for wan
Mar  1 01:35:01 gw php: : HOTPLUG: Configuring interface wan
Mar  1 01:35:01 gw php: : DEVD Ethernet attached event for wan
Mar  1 01:35:01 gw php: : HOTPLUG: Configuring interface wan
Mar  1 01:35:01 gw dhclient: PREINIT
Mar  1 01:35:01 gw dhclient[44035]: dhclient already running, pid: 43420.
Mar  1 01:35:01 gw dhclient[44035]: exiting.
Mar  1 01:35:01 gw php: : The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf ue0 > /tmp/ue0_output 2> /tmp/ue0_error_output' returned exit code '1', the output was '' 
Mar  1 01:35:01 gw dhclient: Starting delete_old_states()
Mar  1 01:35:01 gw dhclient: Comparing IPs: Old:  New: 
Mar  1 01:35:01 gw dhclient: Comparing Routers: Old:  New: 
Mar  1 01:35:01 gw dhclient[43420]: DHCPREQUEST on ue0 to 255.255.255.255 port 67
Mar  1 01:35:01 gw dhclient[43420]: DHCPACK from 73.129.106.1
Mar  1 01:35:01 gw dhclient: REBOOT
Mar  1 01:35:01 gw dhclient: Starting delete_old_states()
Mar  1 01:35:01 gw dhclient: Comparing IPs: Old:  New: 68.48.155.52
Mar  1 01:35:01 gw dhclient: Comparing Routers: Old:  New: 68.48.154.1
Mar  1 01:35:01 gw dhclient: Removing states through old gateway '' (new gateway '68.48.154.1')
Mar  1 01:35:01 gw dhclient: Starting add_new_address()
Mar  1 01:35:01 gw dhclient: ifconfig ue0 inet 68.48.155.52 netmask 255.255.254.0 broadcast 255.255.255.255 
Mar  1 01:35:01 gw dhclient: New IP Address (ue0): 68.48.155.52
Mar  1 01:35:01 gw dhclient: New Subnet Mask (ue0): 255.255.254.0
Mar  1 01:35:01 gw dhclient: New Broadcast Address (ue0): 255.255.255.255
Mar  1 01:35:01 gw dhclient: New Routers (ue0): 68.48.154.1
Mar  1 01:35:01 gw dhclient: Adding new routes to interface: ue0
Mar  1 01:35:01 gw dhclient: /sbin/route add default 68.48.154.1
Mar  1 01:35:01 gw dhclient: Creating resolv.conf
Mar  1 01:35:01 gw check_reload_status: rc.newwanip starting ue0
Mar  1 01:35:01 gw dhclient[43420]: bound to 68.48.155.52 -- renewal in 159739 seconds.
Mar  1 01:35:02 gw php: : Clearing states to old gateway 68.48.154.1.
Mar  1 01:35:04 gw php: : rc.newwanip: Informational is starting ue0.
Mar  1 01:35:04 gw php: : rc.newwanip: on (IP address: ) (interface: wan) (real interface: ue0).
Mar  1 01:35:04 gw php: : rc.newwanip: Failed to update wan IP, restarting...
Mar  1 01:35:04 gw check_reload_status: Configuring interface wan
Mar  1 01:35:06 gw dhclient[54110]: dhclient already running, pid: 51797.
Mar  1 01:35:06 gw dhclient[54110]: exiting.
Mar  1 01:35:06 gw php: : The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf ue0 > /tmp/ue0_output 2> /tmp/ue0_error_output' returned exit code '1', the output was '' 
Mar  1 01:35:22 gw check_reload_status: Updating all dyndns
Mar  1 01:35:22 gw check_reload_status: Restarting ipsec tunnels
Mar  1 01:35:22 gw check_reload_status: Restarting OpenVPN tunnels/interfaces
Mar  1 01:35:22 gw check_reload_status: Reloading filter
Mar  1 01:35:25 gw php: : Could not find IPv4 gateway for interface (wan).
Mar  1 01:35:25 gw php: : Could not find IPv4 gateway for interface (wan).
Mar  1 01:35:25 gw php: : Could not find IPv4 gateway for interface (wan).
Mar  1 01:35:25 gw php: : Could not find IPv4 gateway for interface (wan).
Mar  1 01:35:25 gw php: : Could not find IPv4 gateway for interface (wan).

Edit 2: From the next time it died, here's the dhclient output:


[2.1-BETA1][root@gw.tubas.net]/tmp(62): cat ue*
dhclient already running, pid: 69584.
exiting.

And yes, it is indeed running:

_dhcp 69584 0.0 0.1 3388 1464 ?? SNs 1:52AM 0:00.01 dhclient: ue0 (dhclient)

Edit 3: The next time it died, I tried running dhclient manually:


[2.1-BETA1][root@gw.tubas.net]/sbin(74):  /sbin/dhclient -c /var/etc/dhclient_wan.conf ue0
dhclient already running, pid: 54736.
exiting.

[2.1-BETA1][root@gw.tubas.net]/sbin(75): ps auxw | grep dhcl
root   46928  0.0  0.1  3388  1332  ??  INs   1:58AM   0:00.00 dhclient: ue0 [priv] (dhclient)
_dhcp  54736  0.0  0.1  3388  1464  ??  SNs   1:58AM   0:00.06 dhclient: ue0 (dhclient)
root   94481  0.0  0.1  3536  1248   1  S+    2:00AM   0:00.00 grep dhcl

[2.1-BETA1][root@gw.tubas.net]/sbin(76): kill -KILL 54736 46928

[2.1-BETA1][root@gw.tubas.net]/sbin(77): /sbin/dhclient -c /var/etc/dhclient_wan.conf ue0
dhclient: PREINIT
ifconfig: ioctl (SIOCAIFADDR): File exists
dhclient: Starting delete_old_states()
OLD_IP: not found
dhclient: Comparing IPs: Old:  New: 
dhclient: Comparing Routers: Old:  New: 
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPACK from 73.129.106.1
bound to 68.48.155.52 -- renewal in 158952 seconds.
[2.1-BETA1][root@gw.tubas.net]/sbin(78):

…and this brought it back up.

wallabybob

@TravisD:

Since I'm now on 2.1,

But what particular build of 2.1?

@TravisD:


Feb 28 23:26:00 gw check_reload_status: Linkup starting ue0
Feb 28 23:26:00 gw kernel: ue0: link state changed to DOWN
Feb 28 23:26:00 gw kernel: ue0: link state changed to UP
Feb 28 23:26:01 gw check_reload_status: Linkup starting ue0
Feb 28 23:26:03 gw php: : DEVD Ethernet detached event for wan
Feb 28 23:26:03 gw dhclient[52833]: connection closed
Feb 28 23:26:03 gw dhclient[52833]: exiting.
Feb 28 23:26:03 gw php: : DEVD Ethernet attached event for wan
Feb 28 23:26:03 gw php: : HOTPLUG: Configuring interface wan
Feb 28 23:26:03 gw dhclient: PREINIT
Feb 28 23:26:03 gw kernel: arpresolve: can't allocate llinfo for 68.48.154.1
Feb 28 23:26:03 gw kernel: arpresolve: can't allocate llinfo for 68.48.154.1
Feb 28 23:26:03 gw dhclient: Starting delete_old_states()
Feb 28 23:26:03 gw dhclient: Comparing IPs: Old: New: 
Feb 28 23:26:03 gw dhclient: Comparing Routers: Old: New: 
Feb 28 23:26:03 gw dhclient[68325]: DHCPREQUEST on ue0 to 255.255.255.255 port 67
Feb 28 23:26:03 gw dhclient[68325]: DHCPACK from 73.129.106.1
Feb 28 23:26:03 gw dhclient: REBOOT
Feb 28 23:26:03 gw dhclient: Starting delete_old_states()
Feb 28 23:26:03 gw dhclient: Comparing IPs: Old: New: 68.48.155.52
Feb 28 23:26:03 gw dhclient: Comparing Routers: Old: New: 68.48.154.1
Feb 28 23:26:03 gw dhclient: Removing states through old gateway '' (new gateway '68.48.154.1')
Feb 28 23:26:03 gw dhclient: Starting add_new_address()
Feb 28 23:26:03 gw dhclient: ifconfig ue0 inet 68.48.155.52 netmask 255.255.254.0 broadcast 255.255.255.255 
Feb 28 23:26:03 gw dhclient: New IP Address (ue0): 68.48.155.52
Feb 28 23:26:03 gw dhclient: New Subnet Mask (ue0): 255.255.254.0
Feb 28 23:26:03 gw dhclient: New Broadcast Address (ue0): 255.255.255.255
Feb 28 23:26:03 gw dhclient: New Routers (ue0): 68.48.154.1
Feb 28 23:26:03 gw dhclient: Adding new routes to interface: ue0
Feb 28 23:26:03 gw dhclient: /sbin/route add default 68.48.154.1
Feb 28 23:26:03 gw dhclient: Creating resolv.conf
Feb 28 23:26:03 gw check_reload_status: rc.newwanip starting ue0
Feb 28 23:26:03 gw dhclient[68325]: bound to 68.48.155.52 -- renewal in 163608 seconds.
Feb 28 23:26:04 gw php: : Clearing states to old gateway 68.48.154.1.
[b]Feb 28 23:26:06 gw php: : rc.newwanip: Informational is starting ue0.[/b]
Feb 28 23:26:06 gw php: : rc.newwanip: on (IP address: ) (interface: wan) (real interface: ue0).
Feb 28 23:26:06 gw php: : rc.newwanip: Failed to update wan IP, restarting…

The log extract you posted shows multiple dhclient copies running. This seems like a bad idea - they could easily interfere with each other in unpredictable ways. I think I have seen recent posts addressing this issue. The multiple reports: Feb 28 23:26:06 gw php: : rc.newwanip: Informational is starting ue0. in the original log you posted seem strange in that ue0 shouldn't need to be started (not sure exactly what that started means) immediately after the interface IP address has been set and routes added etc. PERHAPS there is a synchronisation issue between multiple processes and it generally works OK in some systems (slower systems like mine) but doesn't generally work OK on faster systems.

@abeowitz:

Even with static IPs on WAN and OPT1, I still get lots of these in my dmesg logs:

ue1: link state changed to DOWN
ue1: link state changed to UP
ue0: link state changed to DOWN
ue0: link state changed to UP
pfsync0: promiscuous mode disabled
ue0: link state changed to DOWN
ue0: link state changed to UP
ue0: link state changed to DOWN
ue0: link state changed to UP
pfsync0: promiscuous mode enabled

Its likely that for a time the NICs didn't detect carrier from the devices they are connected to.

@TravisD:

Manually trying to remove these replicates the issue:


[2.1-BETA1][root@gw.tubas.net]/conf(12): rm -f dyndns_*
rm: dyndns_wandyndns'travisd.dyndns.org'0.cache: Read-only file system
rm: dyndns_wanopendns'travisd.dyndns.org'1.cache: Read-only file system

Are you running nanoBSD based variant of pfSense? Please post the version string from the home page of your pfSense box.

TravisD

Yes, I'm running a nanoBSD variant. Currently:

2.1-BETA1 (i386)
built on Thu Feb 28 04:26:51 EST 2013

(but over the last several days I've been trying 2.0.1, 2.0.2, 2.0.3-SNAPSHOT, and a few 2.1 snapshots.

This is running on: Intel(R) Atom(TM) CPU 330 @ 1.60GHz

It definitely appears to be a dhclient thing at this point. If I manually kill the running dhclient process and re-start it, things come right back up.


[2.1-BETA1][root@gw.tubas.net]/sbin(86): /sbin/dhclient -c /var/etc/dhclient_wan.conf ue0
dhclient already running, pid: 72467.
exiting.
[2.1-BETA1][root@gw.tubas.net]/sbin(87): kill -KILL 72467
[2.1-BETA1][root@gw.tubas.net]/sbin(88): /sbin/dhclient -c /var/etc/dhclient_wan.conf ue0
dhclient: PREINIT
ifconfig: ioctl (SIOCAIFADDR): File exists
dhclient: Starting delete_old_states()
OLD_IP: not found
dhclient: Comparing IPs: Old:  New: 
dhclient: Comparing Routers: Old:  New: 
DHCPREQUEST on ue0 to 255.255.255.255 port 67
DHCPACK from 73.129.106.1
bound to 68.48.155.52 -- renewal in 158108 seconds.