Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    My pfSense keeps breaking (novel inside…)

    Scheduled Pinned Locked Moved General pfSense Questions
    46 Posts 5 Posters 19.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      soteriologist
      last edited by

      PLEASE HELP!!!  I'm about to go bald from this crap.  :-[

      This is the second time this same exact thing has happened to me and it is outrageously frustrating…

      As the story goes:  I'm making a bunch of different configuration changes to my Firewall (NAT/Rules) trying to get the right configuration setup to allow for a sort-of complex rule-set when all of a sudden traffic stops flowing out of pfSense.  I can't see a reason why.

      I enable logging for the default "allow LAN to any rule" so I can see what's going in and out of the firewall and if it's blocking anything.
      At first it shows states in the state table.  And it shows that the firewall is NOT blocking and allowing me to connect to 4.2.2.3 for dns queries... but nothing else.

      Eventually as I go about doing all of the below, states no longer show any longer.  NONE.  Not even me connecting into the firewall itself.  The entire States tab is empty.  Same for the firewall tab.  It won't even show me successfully connecting to 4.2.2.3.

      SOOooo I make a back up of my config (just in case I need to do anything drastic).
      I plug directly into my WAN devices and get a nice internet connection.
      I reboot pfSense... that does nothing.
      I reset the states...  that does nothing.
      I reload my config... that does nothing.
      I power cycle all WAN devices attached to pfSense... that does nothing.
      I go about disabling all of my changes... that does nothing.
      I disable all installed packages... that does nothing.

      Then I start with the drastic...
      I remove all installed packages... that does nothing.
      I DELETE all of my changes... that does nothing.
      I reset pfSense to factory defaults and reset just the basic connections (IP for the LAN and WAN along with the gateway for the WAN to be associated with)... that does nothing.

      I'm able to get into the pfSense perfectly find through the web configurator.  I'm able to SSH in.  I'm able to see the shell through VGA and everything seems to respond good.

      The ONLY other weirdness is that after ALL of this and I'm setting every back up again when I change the IP addresses for the interfaces through the VGA console it stops for what seems like an eternity on "Please wait while the changes are saved to WAN...  Reloading filter..."  Keyboard input shows up on the screen, BUT the system doesn't do anything other than display what I'm typing.  so I hit the power button on the front of the server and it'll show pfSense go through the shutdown process, and I'll boot it back up and it'll show my changed IP address (which I can browse to).

      It's done this EXACTLY the same twice now.

      The only thing I can do to get the system working again is to reinstall pfSense from the CD onto the hard drive and completely override all the files.  :-\

      NOT. FUN.

      Any idea on where to start on this?  Why this might be happening?  How I can prevent it?
      I haven't even had the chance to really use pfSense yet.  First time it happened was after I had it up and running and was using it for about 3 days.  And one night after work hours I was finishing up some configuration on it, it did this to me and I had to completely switch back to my old equipment.  This time it happened while still in the testing stage before I even had a chance to put it back as my primary firewall.  I'm getting a bit burned out on this monster of a project.  It's just plain frustrating.  :'(

      I am ussing an brand new SSD in the server.  Do you think that could be causing the problem?  It's corrupting States table or some other file and causing it to be unreadable/writable?  And even though I'm telling pfSense to reset to factory default it's not necessarily re-creating said corrupted table or file?

      1 Reply Last reply Reply Quote 0
      • W
        wallabybob
        last edited by

        @soteriologist:

        I'm making a bunch of different configuration changes to my Firewall (NAT/Rules) trying to get the right configuration setup to allow for a sort-of complex rule-set when all of a sudden traffic stops flowing out of pfSense.  I can't see a reason why.

        What traffic stops flowing? (Apparently you still have contact with the web GUI.) What reasons did you look for? Did you check interface status?

        @soteriologist:

        I enable logging for the default "allow LAN to any rule" so I can see what's going in and out of the firewall and if it's blocking anything.
        At first it shows states in the state table.  And it shows that the firewall is NOT blocking and allowing me to connect to 4.2.2.3 for dns queries… but nothing else.

        What other traffic did you try and what was reported?

        @soteriologist:

        Eventually as I go about doing all of the below, states no longer show any longer.  NONE.  Not even me connecting into the firewall itself.  The entire States tab is empty.  Same for the firewall tab.  It won't even show me successfully connecting to 4.2.2.3.

        Does the browser show signs of stalling before completing the States display?

        @soteriologist:

        SOOooo I make a back up of my config (just in case I need to do anything drastic).
        I plug directly into my WAN devices and get a nice internet connection.
        I reboot pfSense… that does nothing.
        I reset the states...  that does nothing.
        I reload my config... that does nothing.
        I power cycle all WAN devices attached to pfSense... that does nothing.
        I go about disabling all of my changes... that does nothing.
        I disable all installed packages... that does nothing.

        You are getting browser response so the box is at least doing that - that is the box is not doing nothing! So I suspect "does nothing" means there is something you are expecting it to do but it is not clear to me what the box is expected to do but isn't doing.

        @soteriologist:

        Then I start with the drastic…
        I remove all installed packages... that does nothing.
        I DELETE all of my changes... that does nothing.
        I reset pfSense to factory defaults and reset just the basic connections (IP for the LAN and WAN along with the gateway for the WAN to be associated with)... that does nothing.

        I'm able to get into the pfSense perfectly find through the web configurator.  I'm able to SSH in.  I'm able to see the shell through VGA and everything seems to respond good.

        Again, please elaborate on "does nothing".

        @soteriologist:

        The ONLY other weirdness is that after ALL of this and I'm setting every back up again when I change the IP addresses for the interfaces through the VGA console it stops for what seems like an eternity on "Please wait while the changes are saved to WAN…  Reloading filter..."  Keyboard input shows up on the screen, BUT the system doesn't do anything other than display what I'm typing.

        Please type Ctrl-T (hold down the  Ctrl key, press the "T" key, release the Ctrl key) on the console a few times at (say) 10 seconds apart and report what is displayed.

        @soteriologist:

        so I hit the power button on the front of the server and it'll show pfSense go through the shutdown process, and I'll boot it back up and it'll show my changed IP address (which I can browse to).

        It's done this EXACTLY the same twice now.

        The only thing I can do to get the system working again is to reinstall pfSense from the CD onto the hard drive and completely override all the files.  :-\

        That sort of shutdown risks file corruption.

        @soteriologist:

        Any idea on where to start on this?

        Provide answers to the above questions.

        @soteriologist:

        Why this might be happening?  How I can prevent it?

        I don't have enough evidence to answer these.

        @soteriologist:

        I am ussing an brand new SSD in the server.  Do you think that could be causing the problem?

        No evidence yet for that.

        @soteriologist:

        It's corrupting States table or some other file and causing it to be unreadable/writable?

        As best I know state tables are kept in RAM allocated to the kernel.

        @soteriologist:

        And even though I'm telling pfSense to reset to factory default it's not necessarily re-creating said corrupted table or file?

        At best, reset to factory default restores the initial configuration parameters (firewall rules, IP address, password etc). It does not recover corrupt system or package files.

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          Perhaps the novel is overheating your cpu?  :D

          But seriously…
          The fact that:

          I remove all installed packages… that does nothing.
          I DELETE all of my changes... that does nothing.
          I reset pfSense to factory defaults and reset just the basic connections (IP for the LAN and WAN along with the gateway for the WAN to be associated with)... that does nothing.

          By 'nothing' I'm assuming you mean no internet access from LAN side clients but please elaborate on that.

          This implies to me that something has altered the underlying FreeBSD config in a way that isn't controlled by pfSense. When you reset to factory defaults you are replacing the config.xml file with the default one but not resetting the entire OS or replacing binaries as you do when you re-install.
          This is likely to be caused by a package. What packages do are you using?

          Steve

          1 Reply Last reply Reply Quote 0
          • S
            soteriologist
            last edited by

            Interfaces all show that their connections are up.

            I can connect fine from the LAN side to the web gui, and through SSH.  But all traffic on the WAN side won't leave.  There are no states of any sort showing.  There's no active connections.

            When I check the states and firewall the browser has fully loaded the page whenever I look at it.

            By does nothing, I mean ALL of the following:
            There are not states listed AT ALL.
            There is no firewall traffic listed AT ALL.
            I'm able to get into the box… BUT no traffic is leaving it.

            1 Reply Last reply Reply Quote 0
            • S
              soteriologist
              last edited by

              As for what traffic was tried through the box,
              icmp 8
              tcp 53
              tcp 80
              tcp 443
              tcp 25
              tcp 143
              and a bunch of voip traffic in the 6K block of ports.

              1 Reply Last reply Reply Quote 0
              • S
                soteriologist
                last edited by

                The package I'm using is:
                pfSense-2.0.1-RELEASE-amd64.iso

                that I have burned to a CD and am installing from an internal DVD drive onto my 64GB SSD that's in the machine.

                As for resetting, I'm doing so by clicking on the reset to factory defaults options inside the WebGui.  So it's resetting with what options it has in place using that built in function.

                1 Reply Last reply Reply Quote 0
                • S
                  soteriologist
                  last edited by

                  I can't remember ALL of the packages that I had installed the first time this happened but some of them were:
                  pfBlocker
                  file manager
                  squid
                  squidguard
                  and maybe one or two more (all reporting ones)

                  This last time around I had just the following:
                  file manager
                  squid
                  squidguard
                  and I tried out the widescreen theme.

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Well none of those packages look like obvious suspects, never the less I would try without any packages to rule that out.  :-\

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • S
                      soteriologist
                      last edited by

                      I've already uninstalled all the packages.

                      The current state I'm in is:
                      No packages installed.
                      Reset to factory defaults.
                      Only the most basic settings have been applied in order to get an internet connection up on it.

                      And yet I still see no traffic.   :-\

                      It's as I were creating a super-massive star when all of a sudden it imploded into a supernova and warped into a blackhole.   :'(
                      I want my super-massive pfSense star back.

                      1 Reply Last reply Reply Quote 0
                      • S
                        soteriologist
                        last edited by

                        The only thing left for me to do is re-install and start from scratch… AGAIN.

                        But before I do that, I figured I'd post on here to see if someone had a suggestion to diagnose this shit and hopefully stop others from running into the same problem  AND hopefully prevent me from running through it all over again a third time.

                        I figured if there truly is a horribad bug somewhere in the code, someone would want to know about it and get it fixed.

                        1 Reply Last reply Reply Quote 0
                        • S
                          soteriologist
                          last edited by

                          ok,  so I turned on accessing the web configurator from the wan side.  One of my internet connections is DSL which uses a wired/wireless router/modem combo.  So I've plugged my laptop into one of the other wired ports on the little DSL router and can access my pfSense box through the WAN port there.  So at least SOME traffic is flowing through that connection.    But it's not showing up in the states/firewall logs.

                          Still no pings, webpages, email, etc are going through it though.    :'(
                          Can't get a connection to the internet through the pfSense.  :-\

                          1 Reply Last reply Reply Quote 0
                          • W
                            wallabybob
                            last edited by

                            @soteriologist:

                            The only thing left for me to do is re-install and start from scratch… AGAIN.

                            There are still a number of alternatives, including plugging your laptop into the DSL router and attempting to access the pfSense WAN port.

                            @soteriologist:

                            One of my internet connections is DSL

                            What are the others?

                            @soteriologist:

                            So I've plugged my laptop into one of the other wired ports on the little DSL router and can access my pfSense box through the WAN port there.

                            Can you also access the management interface on the DSL router? What does it tell you about the WAN interface of the DSL router?

                            What is the interface type of your pfSense WAN interface? (Static? DHCP? PPP?)

                            Please post the output of of the pfSense shell command```

                            netstat -rn -f inet;  traceroute -n 8.8.8.8

                            1 Reply Last reply Reply Quote 0
                            • S
                              soteriologist
                              last edited by

                              Tried pinging with pfSense's web configurator (under Diagnostics >> Ping) to both 4.2.2.2 and google.com (along with a handful of other sites) and get no response.  Tried  pinging the DSL router, and get a response.  Tried pinging my laptop that is also plugged into the same router and get a response form my laptop's ip address.

                              Still nothing shows in states/firewall logs though?

                              I'll try that traceroute command.

                              1 Reply Last reply Reply Quote 0
                              • S
                                soteriologist
                                last edited by

                                Here are the results:

                                netstat -rn -f inet ; traceroute -n 8.8.8.8
                                Routing tables

                                Internet:
                                Destination        Gateway            Flags    Refs      Use  Netif Expire
                                127.0.0.1          link#12            UH          0    3412    lo0
                                192.168.2.0/24    link#7            U          0    10355    em2
                                192.168.2.2        link#7            UHS        0        0    lo0
                                192.168.168.0/24  link#5            U          0      341    em0
                                192.168.168.1      link#5            UHS        0        0    lo0
                                traceroute: findsaddr: failed to connect to peer for src addr selection.

                                1 Reply Last reply Reply Quote 0
                                • W
                                  wallabybob
                                  last edited by

                                  You don't have a default route hence most of the traffic that would normally go out the WAN interface doesn't go out the WAN interface because there isn't a route saying that is where it should go.

                                  Your pfSense WAN interface type is? (Depending on that I might be able to give you a pfSense shell command to add a default route.) But that won't help if the upstream link from your DSL router is broken. Can you get status of the upstream (to the Internet) link on the DSL router?

                                  What version of pfSense are you running? Please post the version information from the home page of your pfSense box.

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Hmm, you have no default route and no route to anywhere outside your network. Problem!
                                    Is your WAN connection up? (or was it when you did this).
                                    @soteriologist:

                                    I've already uninstalled all the packages.

                                    The reason I suspected packages is that they sometimes either overwrite things they shouldn't or remove things they shouldn't when you uninstall them.

                                    Uninstalling all the packages is not necessarily the same thing as never having installed them!  ::)

                                    Something has messed up your routing table, either directly or by messing up something that controls the routing table.

                                    Steve

                                    Edit: Typed too slow.

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      soteriologist
                                      last edited by

                                      I'm able to get an internet connection fine through all of my WAN devices I've had in the past (and currently have) attached to pfSense.   Even when I plug in using the same cables/ports that pfSense would use to those devices.

                                      I ruled out any hardware problems at the get-go.

                                      As for my current version:
                                      Version 2.0.1-RELEASE (amd64)
                                      built on Mon Dec 12 18:16:13 EST 2011
                                      FreeBSD 8.1-RELEASE-p6

                                      As for the default route, I can check mark that box for the interface.  Right now it's unchechked because I had a loadbalancing group created and had "Allow default gateway switching" under "System >> Advanced >> Miscellaneous >> Load Balancing"  checked.

                                      I can recheck to have just that default DSL line checked as the "Default Gateway" and uncheck the other setting… brb.

                                      1 Reply Last reply Reply Quote 0
                                      • C
                                        cmb
                                        last edited by

                                        You need to set a default gateway even if you're policy routing your egress traffic. And uncheck the default gateway switching.

                                        What type of WANs?

                                        1 Reply Last reply Reply Quote 0
                                        • S
                                          soteriologist
                                          last edited by

                                          I just rechecked to have just that default DSL line checked as the "Default Gateway" and unchecked "Allow default gateway switching" under "System >> Advanced >> Miscellaneous >> Load Balancing".
                                          Still no ping response beyond the router/modem it's plugged into.
                                          Stil no internet connection.
                                          Still nothing in state/firewall logs.

                                          Re-ran traceroute and this is what I have now:

                                          netstat -rn -f inet;  traceroute -n 8.8.8.8

                                          Routing tables

                                          Internet:
                                          Destination        Gateway            Flags    Refs      Use  Netif Expire
                                          default            192.168.2.2        US          0       51    em2
                                          127.0.0.1          link#12            UH          0     3524    lo0
                                          192.168.2.0/24     link#7             U           0    19094    em2
                                          192.168.2.2        link#7             UHS         0        0    lo0
                                          192.168.168.0/24   link#5             U           0      341    em0
                                          192.168.168.1      link#5             UHS         0        0    lo0
                                          traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets

                                          1 Reply Last reply Reply Quote 0
                                          • S
                                            soteriologist
                                            last edited by

                                            The three WANs that I have are:
                                            One DSL connection through a Verizon router/modem
                                            One T1 through an AdTran DSU/CSU
                                            One T1 through a Cisco DSU/CSU

                                            AT THE MOMENT I'm ONLY using the DSL for testing.  Just to simplify things and because the entire company is actively using the two T1s at office.  But when I had everything plugged in during off hours, they were all working fine until… well... everything stopped working.  So I had to put everything back they way I had it in the very late hours of the night before everyone came back in the next day and BACK TO THE DRAWING BOARD!

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.