Overwrote Configuration Now I Have Issues
-
In order to get rid of a VLAN tag on my LAN I took a backup of my config and edited it to remove the VLAN then restored that config. Two things then began to happen, upon reboot I would end up with no internet unless I did one of two things. Go to my WAN interface and hit save or change the speed and duplex to 100baseTX full-duplex. Which isn't horrible but why would this even be a problem all of a sudden? Also on 2.6 the dashboard gateway widget would show my WAN gateway as offline while on 2.7 it shows as online but is unable to ping anything on the WAN although the same fix applies to both versions.
The second thing that happens is when the system is under load like when I download a torrent I will get around 30% packet loss and lose connection to the internet (while the torrent is still downloading fine no less). At first I thought this might be buffer bloat but I'm not uploading anything and I've never had to mess with traffic shaping in the past. I've managed to work around this by disabling gateway monitoring action on the three OpenVPN clients I'm running (under System - Routing).
Would anyone happen to have any insight as to why overwriting my configuration with the same config would have caused any of these problems?
-
@jaskerx What brand network card?
I’d guess your description of lowering speed and packet loss point to a connection or hardware problem. Try different patch cables, or a different WAN NIC if possible.
-
@SteveITS It is an Intel I350 4-port although I'm only using ports igb0 and igb1. I would find it suspicious that a hardware problem would wait until I overwrote my config as there where no problems previous.
-
Unfortunately I don't have any extra patch cables or NIC's to try as replacements.
-
It sounds like you might have a bad or missing default route after boot.
Go to System > Routing > Gateways ad make sure the default IPv4 gateway is set as the WAN.
Steve
-
@stephenw10 Default gateway IPv4 has always been set to WAN_DHCP, so nothing has changed there. Although after I had restored the config I found out that WAN_DHCP seems to be dynamically created when the WAN is initialized as when the WAN was offline (after boot) there was no WAN_DHCP entry under System - Routing. At one point I manually created the WAN_DHCP entry but going back and diff'ing old config files where that entry didn't exist I decided to just leave it default and just let pfSense create it automatically.
-
Yes, by default WAN_DHCP is dynamically created. If you edit that gateway it becomes permanent in the config.
Resaving the WAN recreates the routes on it and reapplies the interface settings. So it's likely something is being lost or set incorrectly that is restored when doing that.
-
@stephenw10 When I was trying to figure this out after it first happened I was all through the logs and the only thing that stood out was
PHP rc.bootup: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf igb0 > /tmp/igb0_output 2> /tmp/igb0_error_output' returned exit code '1', the output was ''I'm pretty sure at one point I even tried to set speed and duplex to autoselect and that didn't work either. Still nothing in the logs that related to the WAN failing to initialize or the port failing to set its speed. Had to ssh into the box and check /tmp/igb0_error_output and was something like no link.
Everything under the WAN interface was default aside from now having to change the speed and duplex, here is a screenshot. Haven't edited anything under WAN_DHCP either so that is all default as well. If something is being lost or set incorrectly would that make this a bug?
-
That dhclient error is common at boot and doesn't usually cause a problem.
What exactly did you remove from the config?
I would check the interface status at the command line after booting. Something must be missing that gets set when you re-save the WAN.
-
@stephenw10 When I saved the config and edited it all I did was change the LAN interface from VLAN2 back to igb1 removing the VLAN tag and that part went according to plan.
How do I check the interfaces from command line? Something like
ip address? Might be a while before I get around to rebooting the box again as everything is working fine. It's more of a general curiosity at this point. -
I guess ifconfig is what I would use?
-
Yup. For example:
[23.09-DEVELOPMENT][admin@4100.stevew.lan]/root: ifconfig ix3 ix3: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: WAN options=49138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,NETMAP,HWSTATS,MEXTPG> ether 90:ec:77:1f:8a:5f inet 172.21.16.232 netmask 0xffffff00 broadcast 172.21.16.255 inet 45.65.87.21 netmask 0xffffffc0 broadcast 45.65.87.63 vhid 1 inet6 fe80::92ec:77ff:fe1f:8a5f%ix3 prefixlen 64 scopeid 0x8 carp: MASTER vhid 1 advbase 1 advskew 0 peer 224.0.0.18 peer6 ff02::12 media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>And the routing with:
[23.09-DEVELOPMENT][admin@4100.stevew.lan]/root: netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 172.21.16.1 UGS ix3 10.1.8.0/24 link#24 U ovpns1 10.1.8.1 link#10 UHS lo0 10.1.9.0/24 link#21 U ovpnc2 10.1.9.2 link#10 UHS lo0 10.10.10.1 link#10 UH lo0 10.45.11.1 link#10 UHS lo0 10.45.11.2 link#19 UH gre0 ... -
@stephenw10 I changed the speed and duplex on the WAN back to default and rebooted and when the box comes back up the WAN gateway says its online but there is no globe icon (indicating there is internet?). Went to System - Routing and default IPv4 gateway is still WAN_DHCP so I hit save on the page and the globe icon immediately appears but I have to manually restart my three openvpn's to get those to come up. It seems I can fix this a number of ways I can A. Hit save on the WAN interface page which is still the best way as it automatically brings up all my VPN's, B. Hit save on the System - Routing page but then I still have to manually restart the VPN's, C. Just leave the speed and duplex of the WAN interface as 100baseTX full-duplex and when I reboot everything starts fine. Here is output of ifconfig igb0 and netstat -rn before reboot and after:
Before Reboot
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: WAN options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> ether HIDDEN inet6 HIDDEN%igb0 prefixlen 64 scopeid 0x1 inet HIDDEN netmask 0xfffffc00 broadcast HIDDEN media: Ethernet 100baseTX <full-duplex> status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Routing tables Internet: Destination Gateway Flags Netif Expire default pub IP UGS igb0 4.2.2.1 10.20.70.1 UGHS ovpnc1 4.2.2.2 10.35.102.1 UGHS ovpnc2 4.2.2.3 10.31.86.1 UGHS ovpnc3 10.10.10.1 link#6 UH lo0 10.20.70.0/24 link#12 U ovpnc1 10.20.70.101 link#6 UHS lo0 10.31.86.0/24 link#14 U ovpnc3 10.31.86.244 link#6 UHS lo0 10.35.102.0/24 link#13 U ovpnc2 10.35.102.67 link#6 UHS lo0 pub IP/22 link#1 U igb0 pub IP link#6 UHS lo0 127.0.0.1 link#6 UH lo0 192.168.0.0/24 link#2 U igb1 192.168.0.1 link#6 UHS lo0 192.168.3.0/24 link#10 U igb1.3 192.168.3.1 link#6 UHS lo0 192.168.4.0/24 link#11 U igb1.4 192.168.4.1 link#6 UHS lo0 192.168.10.0/24 link#9 U igb1.10 192.168.10.1 link#6 UHS lo0After Reboot
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: WAN options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> ether HIDDEN inet6 HIDDEN%igb0 prefixlen 64 scopeid 0x1 inet HIDDEN netmask 0xfffffc00 broadcast HIDDEN media: Ethernet autoselect (100baseTX <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Routing tables Internet: Destination Gateway Flags Netif Expire 10.10.10.1 link#6 UH lo0 pub IP/22 link#1 U igb0 pub IP link#6 UHS lo0 127.0.0.1 link#6 UH lo0 192.168.0.0/24 link#2 U igb1 192.168.0.1 link#6 UHS lo0 192.168.3.0/24 link#10 U igb1.3 192.168.3.1 link#6 UHS lo0 192.168.4.0/24 link#11 U igb1.4 192.168.4.1 link#6 UHS lo0 192.168.10.0/24 link#9 U igb1.10 192.168.10.1 link#6 UHS lo0When I leave the speed and duplex as default I'm not getting a default gateway for some reason, any idea why? Even backed up the config and looked and the default gateway is set as WAN_DHCP. Weird, it seems that restoring that config broke stuff relating to gateways and gateway monitoring.
-
Yes, so as I initially thought it's losing it's default gateway. Or rather it comes up at boot without a default selected.
That implies the gateway set is not valid at boot which is odd.
Try this: edit the WAN_DHCP gateway in System > Routing > Gateways and change the monitor IP to something remote and save it.
That will create a gateway entry for it in the config which means it will always be valid.
-
@stephenw10 I set 8.8.8.8 as the WAN_DHCP monitor IP and rebooted, the only thing that changed in the routing table was the top line:
8.8.8.8 pub IP UGHS igb0Still no default gateway on reboot, same behavior as before.
-
Hmm, check the config file directly. Make sure the gateway name is in upper case there where it's defined and where it's set as default.
There was a bug in an earlier version that created lower case names. -
@stephenw10 From the config file that I just took:
<gateway_item> <interface>wan</interface> <gateway>dynamic</gateway> <name>WAN_DHCP</name> <weight>1</weight> <ipprotocol>inet</ipprotocol> <descr><![CDATA[Interface WAN_DHCP Gateway]]></descr> <gw_down_kill_states></gw_down_kill_states> </gateway_item> <defaultgw4>WAN_DHCP</defaultgw4>Had to go back to the last six backups of the config I've taken and the gateway_item for WAN_DHCP is in the last two but does not exist past that, which is strange because I didn't edit WAN_DHCP until you told me to. The gw_down_kill_states line seems to be new for 2.7.
-
Mmm, dynamic gateways are not stored in the config unless you apply some custom setting to them like the monitor IP.
However that shouldn't be needed to set the default gateway.
Check the system and routing logs at boot. Something is failing to set the default route or removing it.
Do you have multiple gateways defined? The OpenVPN interfaces perhaps? If you have OpenVPN clients one of them could be pushing a new default route.
-
@stephenw10 Changed speed and duplex on WAN to default once again and rebooted, been checking the logs under System - General but I don't see anything unusual Pastebin for reference (hope I sanitized that good enough). There is nothing under the System - Routing logs save for one entry from April 15 of this year about starting radvd 2.19, which I removed hoping that new logs would be created but that has not been the case as the log is still empty.
As for defining multiple gateways, yes I have the WAN_DHCP and three OpenVPN clients. The OpenVPN's are combined into a gateway group and is configured as is detailed in this guide. I do have Don't add/remove routes checked in the config of all three so I am unsure of how they could be pushing a new default route. Also if they were pushing a new default route wouldn't that show up as the default when I use netstat -rn at boot?
-
Hmm, what I don't see there is:
Jul 4 20:17:26 php-cgi 447 rc.bootup: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP' Jul 4 20:17:26 php-cgi 447 rc.bootup: Default gateway setting Interface WAN_DHCP Gateway as default.Or similar to that.
But what I do see is that it appears that igb0 links some seconds after igb1 and the VLANs on that:
Jul 4 09:53:25 kernel igb1: link state changed to UP Jul 4 09:53:42 kernel igb0: link state changed to UPWhat are those NICs actually connected to?
I think you're seeing that dhclient error because at that point in the boot it's trying to start it on an interface that is down.
If you simply disconnect and reconnect the WAN cable after boot does that also restore the default route?
Steve
