LAN dropping it's Assigned IP address.
Hello.. I am currently testing a new machine I am sending out to an outside terminal and noticed some odd behavior this morning.
Although everything was working fine through the VPN tunnel to this remote box, the LAN interface completely dropped it's IP assignment. This of course meant that even though the tunnel was up and established and I could contact the box using it's VPN ip address, it made no difference because the LAN wasn't functional.
After restarting the IP has returned and things are functioning like normal (it's only been about 4 hours since then), but I was wondering if anyone had any thoughts on this. This is PFsense version 2.1.3-amd64 in installed on a ASRock AD2550R/U3S3 Mini ITX Server motherboard with Intel 82574L chipsets. It's functions are as a general Firewall and OpenVPN client. (Site2Site) Very sorry, I didn't check to see if the interface was simply down or if the IP address was just cleared. I'll make sure to do that if this happens again.
Alright, some new information about this. This is something that tends to happen after about 24-30 hours every time and is very odd. I did check to make sure that ifconfig showed the lan IP and that it at least had a knowledge of what the ip was suppose to be. It still showed the IP address, but the pfsense cli menu didn't show anything assigned for em0(lan). I copied the logs off and am going to try to see what I can find. I did see in dhcpd logs that it said randomly about 4am that it stopped listening because there wasn't an ipv4 subnet assigned to the interface.
I'll sanitize the logs of anything identifying and attach them. Also, I'm not on the latest PFsense, so I'll upgrade it and see if that helps.
Is the LAN IP address dynamic? Maybe it's not able to reach it's DHCP server? (I don't recall if the DHCP client logs to System or DHCP.)
First off, thanks for the response. :)
Ah, no, the LAN ip is static and pfsense's dhcp server is listening on that interface. Also, a note on how much it is actually dropping the interface IP address.. I can access this pfsense firewall through the vpn tunnel using it's vpn address (220.127.116.11) when the lan ip address(192.168.1.1) drops, but I can't access the web interface using 192.168.1.1. Likewise, the dhcp server doesn't assign any ips after that and setting a nic static to try to communicate with pfsense's lan interface does nothing. It really does just look like it is completely dropping that ip address all together..
I haven't seen that happen unless the interface physically goes down. (loose or bad cable, failing interface, etc.) The next time it happens check to see if ifconfig shows "status: no carrier".
AH, good to know. Honestly, I haven't run into this situation period with pfsense, so I'll make sure to double check that. But that does make sense.. I haven't run into any problems with em1, the wan interface. What makes me cringe though is that the I don't think it will be the cable, since it handles flawlessly until the problem arises and I'm not dropping any packets/there aren't any rx/tx errors in ifconfig. :(
I think that, regardless of whether or not I see the no-carrier or similar errors in ifconfig, I will try swapping the wan and lan nic assignments. If I start dropping the wan connection I can assume it has something to do with either the nic itself of the way the pfsense is handling the driver modules. If I do see that this is a compatibility issue of some kind, the sooner the better. I will have to order in a replacement and test before the go-live date of the new terminal. D:
This is a long shot but:
Server motherboard with Intel 82574L chipsets
Packet of death? http://blog.krisk.org/2013/02/packets-of-death.html
Many other things to try before that.
Yes, check what is still working, ifconfig, link leds etc.
Hmm… this time doesn't sound like the issue but also does.. Link lights are always still on and blinking away and I don't end up with real logs showing PCI scan errors..
Still, I won't just brush this off. This behavior is very odd and I've run into plenty of issues where the interface goes down in the OS, only showing with ifconfig -a, but never an issue where it still appears up but actually isn't.
Some more info for Packets of death and, in particular, ways of testing packets of death on your intel nic.
I'll definitely give this concept a shot and see if that kills it. If it does, it sounds like I'm going to just have to go with another board. I wouldn't be able to risk the down time if a fix wasn't adequate. Basically, the trucking industry = 24x7x365 and cheap owners = finding the most reliable and inexpensive equipment possible given a time frame. :-/ The thanks I have towards the developers on this project can't be expressed enough. What a solid and amazing project. :) Seriously, you may need to work out a kink or two, but it always works perfect after that it would seem. I've used it at least 20 different companies I've done IT for and it's been amazing.
Update on this:
I'm fairly certain I pinned down the issue. I did some more poking around and noticed THIS issue. I only noticed it after I started to ping and left it going for a long time though. Once it hit this point the IP stopped showing up in the cli config manager.
So now, I need to add the most up to date module for the nic I think.. I'll give you guys a heads up to see if that worked..
You're seeing the error message "ping: sendto: No buffer space available"?
Still might be a faulty card or cable: https://doc.pfsense.org/index.php/No_buffer_space_available
I would still recommend swapping the LAN and WAN NIC assignments as you planned to do earlier since that would help diagnose a hardware issue.
Blah, sorry, I forgot to mention that I tried swapping them and had the same situation. I also read on some freebsd forums about this nic showing these symptoms when both the onbord nics are in use.. And as for it being a faulty cable, I tried 4 different straight through cables (going through a switch) and then 2 different cross over cables going straight into the laptop. I also used two laptops. (I've had situations like this where I ended up going through hell only to find out I used several bad cables or had a bad laptop nic.) Also, taking the interface down and back up made things function again like normal.
It sounds like updating the module normally fixes this issue, but I've just decided that I would rather just pick a board that I know will function with pfsense well and go with it instead. This being an off site location and so many states away I think it would just be safer to go that route.
That being said, I love this motherboard to death and might just purchase one for myself. It has 8SATA ports, USB 3.0, a decent server processor and has been pretty much rated perfect by everyone. I almost just said "Screw it" and used freebsd 9 or 10 and set up the packages manually, using the newest modules. Anyways.. If I do end up purchasing one, I think I'll pursue a solution to this problem and revive the thread again later. But yes, it does sound like making the newest modules for the PRO/1000 chipest packs from intel would have worked.
OH, thank you guys so much for all your help on this. Sorry, I couldn't stick it out so that we could all see what was happening, but I just was running out of time..
Sounds like a timer is being involved…
Do you loose your LAN IP after a long period of inactivity ?
We experienced this kind of problem with site2site IPSEC VPN with PF & Cisco old PIC : the Tunnel (phase 1) gets up as soon as you connect the VPN BUT if no traffic is going through, the Phase 2 won't never goes up. It goes up as soon as you perform a simple ping or whatever else trafic on that subnet.
The side effect of that was that after a long period of inactivity (few hours), the phase 2 went down automatically...
This was a keep alive problem that had to be tuned Cisco side...
You'd better choose INTEL nics.
Yeah, those are intel nics.. And the problem wasn't with the VPN connections because I could ping anything at the main office from this vpn server.. Also, I could access the remote vpn server using it's vpn ip address (18.104.22.168) and, after looking through the vpn logs there weren't any errors with either phase I or II setting and the link established successfully. Bringing the nic up and down fixed the issue for about 26 hours.
Sounds like the NIC is getting itself into an unrecoverable situation somehow. Assuming it's not PoD (which it probably isn't) try turning off any hardware offloading options or disabling MSI/MSI-X.
Ah, I decided to go with a different board actually guys.. (I posted that earlier.) But I do intend on getting this board for myself. It's just such a beautiful board and would make such an amazing pfsense box that I want to give it a shot in my free time. :D But I will be broke for a while..