WAN link goes down every 12 hours (DHCP related?)

voxadam

My WAN link to Comcast goes down about every twelve hours. It has me totally confused. I can't figure out if it's related to DHCP or not. I can say that nothing out of the ordinary was going on on my network when this happens. It happens whether or not anyone is actively using any of the systems on the network. It can happen when I'm working, browsing, and streaming Netflix or it can happen everyone in the house is asleep and most everything is down or idle. If memory serves this has been happening for the past five days, ever since I replaced my again WRT54GS running DD-WRT with a proper firewall running pfSense.

Below you'll find the my system, dhcp, and gateway logs for the time that the link goes down. If you need any other logs or other information I'll be more than happy to provide it.

system.log: http://pastebin.com/mxuXd55w
dhcp.log: http://pastebin.com/tvAihELQ
gateways.log: http://pastebin.com/2cZqjJBa

Thanks for your help. I can't wait to get this thing up and running properly so I can really screw with it. :D

–adam

wallabybob

@voxadam:

My WAN link to Comcast goes down about every twelve hours.

If presume you mean it goes down and remains unusable. The dhclient log entries show dhclient acquiring a new IP address at 12:54:17. The system log shows:```

Jun 19 12:54:14 check_reload_status: Linkup starting ue0
Jun 19 12:54:14 kernel: ue0: link state changed to DOWN
Jun 19 12:54:14 kernel: ue0: link state changed to UP
Jun 19 12:54:14 check_reload_status: Linkup starting ue0
Jun 19 12:54:16 php: : DEVD Ethernet attached event for wan
Jun 19 12:54:16 php: : HOTPLUG: Configuring interface wan
Jun 19 12:54:17 php: : DEVD Ethernet detached event for wan
Jun 19 12:54:17 check_reload_status: rc.newwanip starting ue0

Notice the two link state change events reported at 12:54:14 and their order. Notice the attach event (presumably triggered by the link state change to UP) is reported BEFORE the detach event (presumably triggered by the link state change to down). This leads me to wonder if the  "link up" processing (adding routes etc) is overtaking the "link down" processing (removing routes etc) with the result that the tidying up on "link down" is actually undoing some of work of "link up" leaving your WAN interface "unusable".

I notice you are using a USB ethernet interface. A number of users have reported them unreliable with pfSense/FreeBSD.

There is some resemblance between what you reported and the topic at [http://forum.pfsense.org/index.php/topic,63411.msg342805.html#msg342805](http://forum.pfsense.org/index.php/topic,63411.msg342805.html#msg342805) - perhaps the same solution will work for you.

voxadam

Thanks for the quick replay wallabybob. I know the USB ethernet interface isn't optimal but it's what I had laying around the the Dell OptiPlex GX620 "ultra small form factor" box I'm using doesn't have any PCI/PCIe expansion so it's what I used. I might take the box down tonight and try connecting the Comcast DOCSIS bridge to the on board ethernet and connect my LAN to the USB adapter. Another option might be to find a VLAN capable switch and use it to connect both my WAN and LAN to the OptiPlex or junk the Dell and get something with real expansion.

Thanks for the link. The symptoms sure look similar. It's just hard to tell if the issue lies in the DOCSIS bridge, the USB ethernet adapter, Comcast's DHCP, or some combination.

I suppose another issue might be the USB reenumerating or something causing the ethernet link to drop sporadically. I just SSHed in and checked usbconfig, the ethernet adapter was sharing a EHCI root hub with another device that wasn't being used. I removed the unused device so the D-Link DUB-E100 ethernet interface is the only device on the hub. Who knows, stranger things have caused me problems.

Again, thanks for the help. I'll keep plugging away at this. I'm still very new to pfSense and FreeBSD in general, I'm more familiar with Linux, Windows (ugh…), and some OS X; I'm excited to add another OS to my list.

ZPrime

I'm having similar issues, although they aren't clockwork ("every 12 hours"), and I'm using different hardware (no USB). I noticed the same sort of "down/up" weirdness and arpresolve errors in the logs though.

I'm on a Soekris net6501-70 and using one of the em Intel interfaces. When it starts acting up, I can watch the link light on the Soekris and cable modem flap on and off in time with the logs. It is not a hardware problem, because if I connect the modem to an unconfigured port (same hardware and driver, just em3 instead of em0), it does not flap… until I change that port to be my WAN port in pfSense and save it, at which point the flapping starts again.

If I reboot pfSense it goes away for some period of time, but I haven't been able to nail it down to anything as short as every 12 hours. It's more like "a week or three" for me.

Smells like something software-related somehow, since the flapping only happens when the port is configured. If it was layer2/hardware-specific, I would expect it to flap regardless of pfSense's configuration. Also, I can check the modem logs (by connecting a different device straight to the modem, since the WAN port on the router is flapping) or observe the front of the modem and the signal levels are fine during the problem timeframe.

I don't have the actual log files, but my pfSense dumps syslogs to my Synology NAS so I can pull up the relevant timeframe there if anyone wants specific info. I can actually see dhclient get an IP from the ISP, but then the interface goes down and the process repeats ad nauseam.

voxadam

I was pretty sure about the clockwork timing until this morning when it went down at nearly midnight and then again at 0130 local time. I've swapped interfaces so that the USB NIC is connected to my LAN and the onboard Broadcom interface is connected to my DOCSIS bridge.

I have a sneaking suspicion that there is something going on between the DOCSIS bridge (or Comcast's DHCP server) and the USB NIC. I have six or seven hours of uptime since swapping the ethernet interfaces and my logs are clean. It's 2300 here right now so I should know in the next six hours if changing the interface "fixed" the problem.

ZPrime

I wouldn't hold your breath. Again, mine seems to go fine for a week or two or three, and then it suddenly goes weird, and the only fix is a reboot of pfSense. After screwing around quite a bit with it when it was acting up, I'm fairly convinced it is a software problem at this point - why else would connecting it to an unconfigured (but otherwise identical) interface be fine?

voxadam

I've never gone more than a day without my WAN flapping. It may all be moot as I'm changing to a new ISP (or at least trying it out for a month) in less than a week.

What version of pfSense are you running? I just realized that I probably posted in the wrong subforum initially as I've been running 2.1-rc0 since day one. This is my first experience with pfSense and I've only had a bit of experience with FreeBSD (years ago), though I am familiar with Linux, OS X, and a few other UNIX-like operating systems.

ZPrime

I'm on 2.0.3 and seeing this behavior… You should probably be posting unreleased stuff in that forum, but it scares me even more if the same "bug" (?) is still present in 2.1 (I was considering trying RC-whatever if this persists in 2.0.x).

voxadam

Well, my WAN has been up for "1 Day 00 Hour 27 Minutes 31 Seconds". I guess this means (knock on wood) it's not a pfSense issue but rather either a bug in the USB NIC, Comcast, or a combination of the two. I suppose it could be a USB subsystem bug as well but I doubt that. I might try freebsd-net@freebsd.org, or Bill Paul who, according to the man page, authored the axe driver to see if I can pin down exactly what's happening before I change ISPs.