Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Overwrote Configuration Now I Have Issues

    Scheduled Pinned Locked Moved General pfSense Questions
    37 Posts 3 Posters 3.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      jaskerx
      last edited by

      In order to get rid of a VLAN tag on my LAN I took a backup of my config and edited it to remove the VLAN then restored that config. Two things then began to happen, upon reboot I would end up with no internet unless I did one of two things. Go to my WAN interface and hit save or change the speed and duplex to 100baseTX full-duplex. Which isn't horrible but why would this even be a problem all of a sudden? Also on 2.6 the dashboard gateway widget would show my WAN gateway as offline while on 2.7 it shows as online but is unable to ping anything on the WAN although the same fix applies to both versions.

      The second thing that happens is when the system is under load like when I download a torrent I will get around 30% packet loss and lose connection to the internet (while the torrent is still downloading fine no less). At first I thought this might be buffer bloat but I'm not uploading anything and I've never had to mess with traffic shaping in the past. I've managed to work around this by disabling gateway monitoring action on the three OpenVPN clients I'm running (under System - Routing).

      Would anyone happen to have any insight as to why overwriting my configuration with the same config would have caused any of these problems?

      S 1 Reply Last reply Reply Quote 0
      • S
        SteveITS Galactic Empire @jaskerx
        last edited by

        @jaskerx What brand network card?

        I’d guess your description of lowering speed and packet loss point to a connection or hardware problem. Try different patch cables, or a different WAN NIC if possible.

        Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
        When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
        Upvote 👍 helpful posts!

        J 1 Reply Last reply Reply Quote 0
        • J
          jaskerx @SteveITS
          last edited by

          @SteveITS It is an Intel I350 4-port although I'm only using ports igb0 and igb1. I would find it suspicious that a hardware problem would wait until I overwrote my config as there where no problems previous.

          J 1 Reply Last reply Reply Quote 0
          • J
            jaskerx @jaskerx
            last edited by

            Unfortunately I don't have any extra patch cables or NIC's to try as replacements.

            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              It sounds like you might have a bad or missing default route after boot.

              Go to System > Routing > Gateways ad make sure the default IPv4 gateway is set as the WAN.

              Steve

              J 1 Reply Last reply Reply Quote 0
              • J
                jaskerx @stephenw10
                last edited by

                @stephenw10 Default gateway IPv4 has always been set to WAN_DHCP, so nothing has changed there. Although after I had restored the config I found out that WAN_DHCP seems to be dynamically created when the WAN is initialized as when the WAN was offline (after boot) there was no WAN_DHCP entry under System - Routing. At one point I manually created the WAN_DHCP entry but going back and diff'ing old config files where that entry didn't exist I decided to just leave it default and just let pfSense create it automatically.

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  Yes, by default WAN_DHCP is dynamically created. If you edit that gateway it becomes permanent in the config.

                  Resaving the WAN recreates the routes on it and reapplies the interface settings. So it's likely something is being lost or set incorrectly that is restored when doing that.

                  J 1 Reply Last reply Reply Quote 0
                  • J
                    jaskerx @stephenw10
                    last edited by

                    @stephenw10 When I was trying to figure this out after it first happened I was all through the logs and the only thing that stood out was

                    PHP rc.bootup: The command '/sbin/dhclient -c /var/etc/dhclient_wan.conf igb0 > /tmp/igb0_output 2> /tmp/igb0_error_output' returned exit code '1', the output was ''
                    

                    I'm pretty sure at one point I even tried to set speed and duplex to autoselect and that didn't work either. Still nothing in the logs that related to the WAN failing to initialize or the port failing to set its speed. Had to ssh into the box and check /tmp/igb0_error_output and was something like no link.

                    Everything under the WAN interface was default aside from now having to change the speed and duplex, here is a screenshot. Haven't edited anything under WAN_DHCP either so that is all default as well. If something is being lost or set incorrectly would that make this a bug?

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      That dhclient error is common at boot and doesn't usually cause a problem.

                      What exactly did you remove from the config?

                      I would check the interface status at the command line after booting. Something must be missing that gets set when you re-save the WAN.

                      J 1 Reply Last reply Reply Quote 0
                      • J
                        jaskerx @stephenw10
                        last edited by

                        @stephenw10 When I saved the config and edited it all I did was change the LAN interface from VLAN2 back to igb1 removing the VLAN tag and that part went according to plan.

                        How do I check the interfaces from command line? Something like ip address? Might be a while before I get around to rebooting the box again as everything is working fine. It's more of a general curiosity at this point.

                        J 1 Reply Last reply Reply Quote 0
                        • J
                          jaskerx @jaskerx
                          last edited by

                          I guess ifconfig is what I would use?

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            Yup. For example:

                            [23.09-DEVELOPMENT][admin@4100.stevew.lan]/root: ifconfig ix3
                            ix3: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
                            	description: WAN
                            	options=49138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,NETMAP,HWSTATS,MEXTPG>
                            	ether 90:ec:77:1f:8a:5f
                            	inet 172.21.16.232 netmask 0xffffff00 broadcast 172.21.16.255
                            	inet 45.65.87.21 netmask 0xffffffc0 broadcast 45.65.87.63 vhid 1
                            	inet6 fe80::92ec:77ff:fe1f:8a5f%ix3 prefixlen 64 scopeid 0x8
                            	carp: MASTER vhid 1 advbase 1 advskew 0
                            	      peer 224.0.0.18 peer6 ff02::12
                            	media: Ethernet autoselect (1000baseT <full-duplex,rxpause,txpause>)
                            	status: active
                            	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
                            

                            And the routing with:

                            [23.09-DEVELOPMENT][admin@4100.stevew.lan]/root: netstat -rn
                            Routing tables
                            
                            Internet:
                            Destination        Gateway            Flags     Netif Expire
                            default            172.21.16.1        UGS         ix3
                            10.1.8.0/24        link#24            U        ovpns1
                            10.1.8.1           link#10            UHS         lo0
                            10.1.9.0/24        link#21            U        ovpnc2
                            10.1.9.2           link#10            UHS         lo0
                            10.10.10.1         link#10            UH          lo0
                            10.45.11.1         link#10            UHS         lo0
                            10.45.11.2         link#19            UH         gre0
                            ...
                            
                            J 1 Reply Last reply Reply Quote 0
                            • J
                              jaskerx @stephenw10
                              last edited by

                              @stephenw10 I changed the speed and duplex on the WAN back to default and rebooted and when the box comes back up the WAN gateway says its online but there is no globe icon (indicating there is internet?). Went to System - Routing and default IPv4 gateway is still WAN_DHCP so I hit save on the page and the globe icon immediately appears but I have to manually restart my three openvpn's to get those to come up. It seems I can fix this a number of ways I can A. Hit save on the WAN interface page which is still the best way as it automatically brings up all my VPN's, B. Hit save on the System - Routing page but then I still have to manually restart the VPN's, C. Just leave the speed and duplex of the WAN interface as 100baseTX full-duplex and when I reboot everything starts fine. Here is output of ifconfig igb0 and netstat -rn before reboot and after:

                              Before Reboot

                              igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                              	description: WAN
                              
                              options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
                              	ether HIDDEN
                              	inet6 HIDDEN%igb0 prefixlen 64 scopeid 0x1
                              	inet HIDDEN netmask 0xfffffc00 broadcast HIDDEN
                              	media: Ethernet 100baseTX <full-duplex>
                              	status: active
                              	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                              
                              Routing tables
                              
                              Internet:
                              Destination        Gateway            Flags     Netif Expire
                              default             pub IP                UGS        igb0
                              4.2.2.1            10.20.70.1         UGHS     ovpnc1
                              4.2.2.2            10.35.102.1        UGHS     ovpnc2
                              4.2.2.3            10.31.86.1         UGHS     ovpnc3
                              10.10.10.1         link#6              UH          lo0
                              10.20.70.0/24      link#12            U        ovpnc1
                              10.20.70.101       link#6             UHS         lo0
                              10.31.86.0/24      link#14            U        ovpnc3
                              10.31.86.244       link#6             UHS         lo0
                              10.35.102.0/24     link#13            U        ovpnc2
                              10.35.102.67       link#6             UHS         lo0
                              pub IP/22            link#1             U          igb0
                              pub IP                 link#6             UHS         lo0
                              127.0.0.1            link#6             UH          lo0
                              192.168.0.0/24     link#2             U          igb1
                              192.168.0.1        link#6             UHS         lo0
                              192.168.3.0/24     link#10            U        igb1.3
                              192.168.3.1        link#6             UHS         lo0
                              192.168.4.0/24     link#11            U        igb1.4
                              192.168.4.1        link#6             UHS         lo0
                              192.168.10.0/24    link#9             U       igb1.10
                              192.168.10.1       link#6             UHS         lo0
                              

                              After Reboot

                              igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
                              	description: WAN
                              	options=4e100bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
                              	ether HIDDEN
                              	inet6 HIDDEN%igb0 prefixlen 64 scopeid 0x1
                              	inet HIDDEN netmask 0xfffffc00 broadcast HIDDEN
                              	media: Ethernet autoselect (100baseTX <full-duplex>)
                              	status: active
                              	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                              	
                              Routing tables
                              
                              Internet:
                              Destination        Gateway            Flags     Netif Expire
                              10.10.10.1         link#6             UH          lo0
                              pub IP/22           link#1             U          igb0
                              pub IP                link#6             UHS         lo0
                              127.0.0.1          link#6             UH          lo0
                              192.168.0.0/24     link#2             U          igb1
                              192.168.0.1        link#6             UHS         lo0
                              192.168.3.0/24     link#10            U        igb1.3
                              192.168.3.1        link#6             UHS         lo0
                              192.168.4.0/24     link#11            U        igb1.4
                              192.168.4.1        link#6             UHS         lo0
                              192.168.10.0/24    link#9             U       igb1.10
                              192.168.10.1       link#6             UHS         lo0
                              

                              When I leave the speed and duplex as default I'm not getting a default gateway for some reason, any idea why? Even backed up the config and looked and the default gateway is set as WAN_DHCP. Weird, it seems that restoring that config broke stuff relating to gateways and gateway monitoring.

                              1 Reply Last reply Reply Quote 0
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Yes, so as I initially thought it's losing it's default gateway. Or rather it comes up at boot without a default selected.

                                That implies the gateway set is not valid at boot which is odd.

                                Try this: edit the WAN_DHCP gateway in System > Routing > Gateways and change the monitor IP to something remote and save it.

                                That will create a gateway entry for it in the config which means it will always be valid.

                                J 1 Reply Last reply Reply Quote 0
                                • J
                                  jaskerx @stephenw10
                                  last edited by

                                  @stephenw10 I set 8.8.8.8 as the WAN_DHCP monitor IP and rebooted, the only thing that changed in the routing table was the top line:

                                  8.8.8.8            pub IP        UGHS       igb0
                                  

                                  Still no default gateway on reboot, same behavior as before.

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Hmm, check the config file directly. Make sure the gateway name is in upper case there where it's defined and where it's set as default.
                                    There was a bug in an earlier version that created lower case names.

                                    J 1 Reply Last reply Reply Quote 0
                                    • J
                                      jaskerx @stephenw10
                                      last edited by

                                      @stephenw10 From the config file that I just took:

                                                       <gateway_item>
                                      			<interface>wan</interface>
                                      			<gateway>dynamic</gateway>
                                      			<name>WAN_DHCP</name>
                                      			<weight>1</weight>
                                      			<ipprotocol>inet</ipprotocol>
                                      			<descr><![CDATA[Interface WAN_DHCP Gateway]]></descr>
                                      			<gw_down_kill_states></gw_down_kill_states>
                                      		</gateway_item>
                                      		<defaultgw4>WAN_DHCP</defaultgw4>
                                      

                                      Had to go back to the last six backups of the config I've taken and the gateway_item for WAN_DHCP is in the last two but does not exist past that, which is strange because I didn't edit WAN_DHCP until you told me to. The gw_down_kill_states line seems to be new for 2.7.

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10S
                                        stephenw10 Netgate Administrator
                                        last edited by

                                        Mmm, dynamic gateways are not stored in the config unless you apply some custom setting to them like the monitor IP.

                                        However that shouldn't be needed to set the default gateway.

                                        Check the system and routing logs at boot. Something is failing to set the default route or removing it.

                                        Do you have multiple gateways defined? The OpenVPN interfaces perhaps? If you have OpenVPN clients one of them could be pushing a new default route.

                                        J 1 Reply Last reply Reply Quote 0
                                        • J
                                          jaskerx @stephenw10
                                          last edited by

                                          @stephenw10 Changed speed and duplex on WAN to default once again and rebooted, been checking the logs under System - General but I don't see anything unusual Pastebin for reference (hope I sanitized that good enough). There is nothing under the System - Routing logs save for one entry from April 15 of this year about starting radvd 2.19, which I removed hoping that new logs would be created but that has not been the case as the log is still empty.

                                          As for defining multiple gateways, yes I have the WAN_DHCP and three OpenVPN clients. The OpenVPN's are combined into a gateway group and is configured as is detailed in this guide. I do have Don't add/remove routes checked in the config of all three so I am unsure of how they could be pushing a new default route. Also if they were pushing a new default route wouldn't that show up as the default when I use netstat -rn at boot?

                                          1 Reply Last reply Reply Quote 0
                                          • stephenw10S
                                            stephenw10 Netgate Administrator
                                            last edited by

                                            Hmm, what I don't see there is:

                                            Jul 4 20:17:26 	php-cgi 	447 	rc.bootup: Gateway, none 'available' for inet, use the first one configured. 'WAN_DHCP'
                                            Jul 4 20:17:26 	php-cgi 	447 	rc.bootup: Default gateway setting Interface WAN_DHCP Gateway as default. 
                                            

                                            Or similar to that.

                                            But what I do see is that it appears that igb0 links some seconds after igb1 and the VLANs on that:

                                            Jul 4 09:53:25 	kernel 		igb1: link state changed to UP
                                            Jul 4 09:53:42 	kernel 		igb0: link state changed to UP
                                            

                                            What are those NICs actually connected to?

                                            I think you're seeing that dhclient error because at that point in the boot it's trying to start it on an interface that is down.

                                            If you simply disconnect and reconnect the WAN cable after boot does that also restore the default route?

                                            Steve

                                            J 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.