Overwrote Configuration Now I Have Issues
-
@stephenw10 Default gateway IPv4 has always been set to WAN_DHCP, so nothing has changed there. Although after I had restored the config I found out that WAN_DHCP seems to be dynamically created when the WAN is initialized as when the WAN was offline (after boot) there was no WAN_DHCP entry under System - Routing. At one point I manually created the WAN_DHCP entry but going back and diff'ing old config files where that entry didn't exist I decided to just leave it default and just let pfSense create it automatically.
-
Yes, by default WAN_DHCP is dynamically created. If you edit that gateway it becomes permanent in the config.
Resaving the WAN recreates the routes on it and reapplies the interface settings. So it's likely something is being lost or set incorrectly that is restored when doing that.
-
@stephenw10 When I was trying to figure this out after it first happened I was all through the logs and the only thing that stood out was
PHP rc.bootup: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf igb0 > /tmp/igb0_output 2> /tmp/igb0_error_output' returned exit code '1', the output was ''I'm pretty sure at one point I even tried to set speed and duplex to autoselect and that didn't work either. Still nothing in the logs that related to the WAN failing to initialize or the port failing to set its speed. Had to ssh into the box and check /tmp/igb0_error_output and was something like no link.
Everything under the WAN interface was default aside from now having to change the speed and duplex, here is a screenshot. Haven't edited anything under WAN_DHCP either so that is all default as well. If something is being lost or set incorrectly would that make this a bug?
-
That dhclient error is common at boot and doesn't usually cause a problem.
What exactly did you remove from the config?
I would check the interface status at the command line after booting. Something must be missing that gets set when you re-save the WAN.
-
@stephenw10 When I saved the config and edited it all I did was change the LAN interface from VLAN2 back to igb1 removing the VLAN tag and that part went according to plan.
How do I check the interfaces from command line? Something like
ip address? Might be a while before I get around to rebooting the box again as everything is working fine. It's more of a general curiosity at this point. -
I guess ifconfig is what I would use?
-
Yup. For example:
[23.09-DEVELOPMENT][admin@4100.stevew.lan]/root: ifconfig ix3 ix3: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500 description: WAN options=49138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,NETMAP,HWSTATS,MEXTPG> ether 90:ec:77:1f:8a:5f inet 172.21.16.232 netmask 0xffffff00 broadcast 172.21.16.255 inet 45.65.87.21 netmask 0xffffffc0 broadcast 45.65.87.63 vhid 1 inet6 fe80::92ec:77ff:fe1f:8a5f%ix3 prefixlen 64 scopeid 0x8 carp: MASTER vhid 1 advbase 1 advskew 0 peer 224.0.0.18 peer6 ff02::12 media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>) status: active nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>And the routing with:
[23.09-DEVELOPMENT][admin@4100.stevew.lan]/root: netstat -rn Routing tables Internet: Destination Gateway Flags Netif Expire default 172.21.16.1 UGS ix3 10.1.8.0/24 link#24 U ovpns1 10.1.8.1 link#10 UHS lo0 10.1.9.0/24 link#21 U ovpnc2 10.1.9.2 link#10 UHS lo0 10.10.10.1 link#10 UH lo0 10.45.11.1 link#10 UHS lo0 10.45.11.2 link#19 UH gre0 ... -
@stephenw10 I changed the speed and duplex on the WAN back to default and rebooted and when the box comes back up the WAN gateway says its online but there is no globe icon (indicating there is internet?). Went to System - Routing and default IPv4 gateway is still WAN_DHCP so I hit save on the page and the globe icon immediately appears but I have to manually restart my three openvpn's to get those to come up. It seems I can fix this a number of ways I can A. Hit save on the WAN interface page which is still the best way as it automatically brings up all my VPN's, B. Hit save on the System - Routing page but then I still have to manually restart the VPN's, C. Just leave the speed and duplex of the WAN interface as 100baseTX full-duplex and when I reboot everything starts fine. Here is output of ifconfig igb0 and netstat -rn before reboot and after:
Before Reboot
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: WAN options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> ether HIDDEN inet6 HIDDEN%igb0 prefixlen 64 scopeid 0x1 inet HIDDEN netmask 0xfffffc00 broadcast HIDDEN media: Ethernet 100baseTX <full-duplex> status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Routing tables Internet: Destination Gateway Flags Netif Expire default pub IP UGS igb0 4.2.2.1 10.20.70.1 UGHS ovpnc1 4.2.2.2 10.35.102.1 UGHS ovpnc2 4.2.2.3 10.31.86.1 UGHS ovpnc3 10.10.10.1 link#6 UH lo0 10.20.70.0/24 link#12 U ovpnc1 10.20.70.101 link#6 UHS lo0 10.31.86.0/24 link#14 U ovpnc3 10.31.86.244 link#6 UHS lo0 10.35.102.0/24 link#13 U ovpnc2 10.35.102.67 link#6 UHS lo0 pub IP/22 link#1 U igb0 pub IP link#6 UHS lo0 127.0.0.1 link#6 UH lo0 192.168.0.0/24 link#2 U igb1 192.168.0.1 link#6 UHS lo0 192.168.3.0/24 link#10 U igb1.3 192.168.3.1 link#6 UHS lo0 192.168.4.0/24 link#11 U igb1.4 192.168.4.1 link#6 UHS lo0 192.168.10.0/24 link#9 U igb1.10 192.168.10.1 link#6 UHS lo0After Reboot
igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: WAN options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> ether HIDDEN inet6 HIDDEN%igb0 prefixlen 64 scopeid 0x1 inet HIDDEN netmask 0xfffffc00 broadcast HIDDEN media: Ethernet autoselect (100baseTX <full-duplex>) status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> Routing tables Internet: Destination Gateway Flags Netif Expire 10.10.10.1 link#6 UH lo0 pub IP/22 link#1 U igb0 pub IP link#6 UHS lo0 127.0.0.1 link#6 UH lo0 192.168.0.0/24 link#2 U igb1 192.168.0.1 link#6 UHS lo0 192.168.3.0/24 link#10 U igb1.3 192.168.3.1 link#6 UHS lo0 192.168.4.0/24 link#11 U igb1.4 192.168.4.1 link#6 UHS lo0 192.168.10.0/24 link#9 U igb1.10 192.168.10.1 link#6 UHS lo0When I leave the speed and duplex as default I'm not getting a default gateway for some reason, any idea why? Even backed up the config and looked and the default gateway is set as WAN_DHCP. Weird, it seems that restoring that config broke stuff relating to gateways and gateway monitoring.
-
Yes, so as I initially thought it's losing it's default gateway. Or rather it comes up at boot without a default selected.
That implies the gateway set is not valid at boot which is odd.
Try this: edit the WAN_DHCP gateway in System > Routing > Gateways and change the monitor IP to something remote and save it.
That will create a gateway entry for it in the config which means it will always be valid.
-
@stephenw10 I set 8.8.8.8 as the WAN_DHCP monitor IP and rebooted, the only thing that changed in the routing table was the top line:
8.8.8.8 pub IP UGHS igb0Still no default gateway on reboot, same behavior as before.
-
Hmm, check the config file directly. Make sure the gateway name is in upper case there where it's defined and where it's set as default.
There was a bug in an earlier version that created lower case names. -
@stephenw10 From the config file that I just took:
<gateway_item> <interface>wan</interface> <gateway>dynamic</gateway> <name>WAN_DHCP</name> <weight>1</weight> <ipprotocol>inet</ipprotocol> <descr><![CDATA[Interface WAN_DHCP Gateway]]></descr> <gw_down_kill_states></gw_down_kill_states> </gateway_item> <defaultgw4>WAN_DHCP</defaultgw4>Had to go back to the last six backups of the config I've taken and the gateway_item for WAN_DHCP is in the last two but does not exist past that, which is strange because I didn't edit WAN_DHCP until you told me to. The gw_down_kill_states line seems to be new for 2.7.
-
Mmm, dynamic gateways are not stored in the config unless you apply some custom setting to them like the monitor IP.
However that shouldn't be needed to set the default gateway.
Check the system and routing logs at boot. Something is failing to set the default route or removing it.
Do you have multiple gateways defined? The OpenVPN interfaces perhaps? If you have OpenVPN clients one of them could be pushing a new default route.
-
@stephenw10 Changed speed and duplex on WAN to default once again and rebooted, been checking the logs under System - General but I don't see anything unusual Pastebin for reference (hope I sanitized that good enough). There is nothing under the System - Routing logs save for one entry from April 15 of this year about starting radvd 2.19, which I removed hoping that new logs would be created but that has not been the case as the log is still empty.
As for defining multiple gateways, yes I have the WAN_DHCP and three OpenVPN clients. The OpenVPN's are combined into a gateway group and is configured as is detailed in this guide. I do have Don't add/remove routes checked in the config of all three so I am unsure of how they could be pushing a new default route. Also if they were pushing a new default route wouldn't that show up as the default when I use netstat -rn at boot?
-
Hmm, what I don't see there is:
Jul 4 20:17:26 php-cgi 447 rc.bootup: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP' Jul 4 20:17:26 php-cgi 447 rc.bootup: Default gateway setting Interface WAN_DHCP Gateway as default.Or similar to that.
But what I do see is that it appears that igb0 links some seconds after igb1 and the VLANs on that:
Jul 4 09:53:25 kernel igb1: link state changed to UP Jul 4 09:53:42 kernel igb0: link state changed to UPWhat are those NICs actually connected to?
I think you're seeing that dhclient error because at that point in the boot it's trying to start it on an interface that is down.
If you simply disconnect and reconnect the WAN cable after boot does that also restore the default route?
Steve
-
@stephenw10 NIC is a Intel I350-T4 port igb0 is connected to the DSL modem and port igb1 is connect to a 16-port TP-Link switch, ports igb2 and igb3 are not connected to anything.
When I first started troubleshooting this I believe I tried unplugging the modem and plugging it back in and that seemed to give igb0 an IP from my ISP. Keep in mind that this first started happening with 2.6 and on boot it would tell me my WAN_DHCP gateway was offline while on 2.7 the gateway is online, it has an IP from the ISP but for some reason has no globe icon by the default gateway and is unable to ping anything from the WAN. I don't think I ever tried just unplugging the cable and plugging it back in (may give that a try). On 2.6 I was also able to go to Status - Interfaces and renew the lease on the WAN, I have not tried this on 2.7 because it already has an IP.
-
The globe icon indicates the current default gateway/route. You don't see one because for some reason it's not setting a default route when the dhcp client pulls a lease. Even though it is pulling a public IP and a gateway.
Check the DHCP logs for the dhclient entries when it connects. Are there errors shown when it tries to add the gateway as a router?I think you may be hitting two issues here. Try renewing the dhcp lease in 2.7 and see if that also adds the default route.
-
@stephenw10 Here is a Pastebin Link for the DHCP log during the same time as the other log (July 4: 09:52-09:54) and I am not seeing any errors when it's adding new routers or any errors at all for that matter.
Next time I reboot I will try to release and renew the IP, I imagine it will work but that's just an assumption. Also not sure what good it will do.
-
Mmm, nothing in that log, looks fine.
When you reboot pfSense does the WAN actually link ~20s after the LAN, looking at the link LEDs?
That seems odd if the upstream modem is already up. I bet putting a switch between the WAN and modem would stop this happening. That would be a good test.If that does solve it we might be able to work around it with the dhcp client timing values.
-
@stephenw10 Rebooted again and tried to release and renew the WAN IP which I think it's safe to say that it doesn't work, makes the WAN come up as "Pending" then it just sits there and doesn't choose a default route.
As for watching the NIC activity lights I'm sure the LAN is coming up around 20 seconds or more before the WAN does, as for putting a switch between the modem and the router I'm having trouble finding my old one so that test might be a no go.
I'm also getting a post code error beep from the Asus motherboard now, which I didn't have before. The internet is being less than helpful in finding out it's meaning, and there is nothing in the manual. I hear the POST beep, then beep, one second pause, beep, beep, beep, one second pause, beep. It's either two errors in one about RAM not being seated correctly (all the RAM is showing up in the GUI), or it's complaining about the video card that the system doesn't have, or maybe because the monitor is off. Sigh... computers.
