Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PfSense 2.3 LAN interface stops routing traffic - stops working after 2 or 3 day

    Scheduled Pinned Locked Moved General pfSense Questions
    88 Posts 31 Posters 44.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J Offline
      jeffvfren
      last edited by

      I'm having the same issue. Disable CPU cores, still under monitoring.
      Hopefully the fix release asap.

      Update:
      It does not work for me, the issue just happen again.

      1 Reply Last reply Reply Quote 0
      • H Offline
        h311m4n
        last edited by

        Add us to the list as well.

        We have a virtual pfsense cluster on our DRC site with and ipsec tunnel to our prod site. Since the update, we've had constant crashes from the master on the DRC site.

        It has been stable for the past 2-3 days now, but with a VCENTER session open, I see a constant CPU usage warning on the master on the other side. We have done the CPU workaround too but as soon as we launch our VEEAM replications, we can be pretty much sure that the ipsec tunnel will fall. Def looks like all the UDP traffic is basically doing a DOS…

        1 Reply Last reply Reply Quote 0
        • C Offline
          cmb
          last edited by

          We're still working on tracking this down. Have it narrowed down to something in our IPsec changes (which are just back-ports from FreeBSD -CURRENT), but that's still 80 change sets potentially related.

          1 Reply Last reply Reply Quote 0
          • H Offline
            h311m4n
            last edited by

            @cmb:

            We're still working on tracking this down. Have it narrowed down to something in our IPsec changes (which are just back-ports from FreeBSD -CURRENT), but that's still 80 change sets potentially related.

            Good to know you're working on it!  :)

            1 Reply Last reply Reply Quote 0
            • M Offline
              marcvb
              last edited by

              We also are having this problem with 2.3 , it happens each week.
              We use multiple ipsec connections.
              The pfsense is inside a vmware.
              2Gb ram
              4 cores

              I think the state table size is also bigger: 82% (165500/201000)
              When i want to open the state table it also crashes Allowed memory size of 268435456 bytes exhausted.
              I will put more ram in the system when i have a downtime window.

              –--
              Yes i got my down window upgraded the ram to 4Gb, can now view the states without any problem.

              1 Reply Last reply Reply Quote 0
              • J Offline
                j.koopmann
                last edited by

                Happened again today twice but with different details.

                1. LAN went down. For whatever reason my script doing ifconfig down and up did not help. I tried manually with no luck. Retried and introduced longer sleep between down and up and then it worked. This was the first crash in several days.

                2. appr. 60 minutes later LAN was down again. However this time I did not even manage to get any result on the serial console. I had to power down/up. No core dump I was able to find however after login I saw a crash and uploaded it. not sure if it is related. I would say it happened around the time of (1).

                
                Fatal trap 12: page fault while in kernel mode
                cpuid = 0; apic id = 00
                fault virtual address	= 0x0
                fault code		= supervisor read data, page not present
                instruction pointer	= 0x20:0xffffffff80d22566
                stack pointer	        = 0x28:0xfffffe001a38c590
                frame pointer	        = 0x28:0xfffffe001a38c770
                code segment		= base 0x0, limit 0xfffff, type 0x1b
                			= DPL 0, pres 1, long 1, def32 0, gran 1
                processor eflags	= interrupt enabled, resume, IOPL = 0
                current process		= 12 (irq260: igb2:que 0)
                version.txt06000025412713111367  7616 ustarrootwheelFreeBSD 10.3-RELEASE #31 01118b4(RELENG_2_3): Thu Apr 28 03:57:55 CDT 2016
                    root@ce23-amd64-builder:/builder/pfsense/tmp/obj/builder/pfsense/tmp/FreeBSD-src/sys/pfSense
                

                Regards,
                  JP

                1 Reply Last reply Reply Quote 0
                • J Offline
                  jswope
                  last edited by

                  I have 5 locations and we use the Dell R210 servers all sites except 2 have this issue. Only difference between them are Internet Service Providers.

                  Site 1 R210 Charter Fiber no issues

                  Site 2 AT&T LTE charter fiber bo issues

                  Site 3 Charter Fiber issues every 2-3 days watchdog error no lan routing

                  Site 4 charter coax internet issues every 3-days

                  1 Reply Last reply Reply Quote 0
                  • M Offline
                    marcvb
                    last edited by

                    A other firewall of us got stuck just now, when i looked at the console it was showing a different wan adres.
                    It seems to have been reverted to a private, will take a screenshot next time.

                    1 Reply Last reply Reply Quote 0
                    • C Offline
                      cmb
                      last edited by

                      We seem to have a fix for this. It's not yet merged into the source tree, but will be soon.

                      @jswope:

                      I have 5 locations and we use the Dell R210 servers all sites except 2 have this issue. Only difference between them are Internet Service Providers.

                      It's not universal or even close to it or we wouldn't have released with the issue. Not ISP-specific. Just happens the traffic profile of some systems will encounter it. Certain UDP streams is what triggers it.

                      1 Reply Last reply Reply Quote 0
                      • A Offline
                        afreaken
                        last edited by

                        @cmb:

                        We seem to have a fix for this. It's not yet merged into the source tree, but will be soon.

                        @jswope:

                        I have 5 locations and we use the Dell R210 servers all sites except 2 have this issue. Only difference between them are Internet Service Providers.

                        It's not universal or even close to it or we wouldn't have released with the issue. Not ISP-specific. Just happens the traffic profile of some systems will encounter it. Certain UDP streams is what triggers it.

                        Do you have an ETA?

                        1 Reply Last reply Reply Quote 0
                        • C Offline
                          cmb
                          last edited by

                          Sometime this week.

                          1 Reply Last reply Reply Quote 0
                          • E Offline
                            eeit
                            last edited by

                            Hello,

                            any news on this ? Is there a special setting or a patch to fix the issue available ?

                            Thx.

                            1 Reply Last reply Reply Quote 0
                            • C Offline
                              cmb
                              last edited by

                              Reducing the system to one core is the only workaround. The fix was merged into devel branch today and hopefully into RELENG_2_3 tomorrow, in which case it'll hit snapshots tomorrow.

                              1 Reply Last reply Reply Quote 0
                              • J Offline
                                jonnyaalgaard.org
                                last edited by

                                Then I have upgraded to the new snapshot that came this morning (2.3.1-DEVELOPMENT (amd64) - built on Thu May 12 00:18:20 CDT 2016 - FreeBSD 10.3-RELEASE-p2), and will check it out.
                                I have also removed the temp workaround, from /boot/loader.conf.local
                                The temp workaround for me were to add these lines into loader.conf.local
                                hint.lapic.1.disabled=1
                                hint.lapic.2.disabled=1
                                hint.lapic.3.disabled=1

                                When I entered these lines pfsense were stabile for me, but now I have enabled all cpu cores again, and hopefully the new snapshot will work just fine.
                                Before the workaround I did have 10-12 “crashes” a day.
                                I will post back if the new snapshot works Ok or not.

                                UPDATE! Before i managed to post this my pfsense crashed again with the new snapshot. I will try to disable all but 1 cpu core again, still using todays snapshot.

                                1 Reply Last reply Reply Quote 0
                                • C Offline
                                  cmb
                                  last edited by

                                  There is no fix yet, as I noted. Watch the bug ticket for updates.

                                  1 Reply Last reply Reply Quote 0
                                  • C Offline
                                    cmb
                                    last edited by

                                    The fix for this was merged this morning and is in the most recent available snapshot, Thu May 12 14:01:47 CDT and newer.

                                    My test setups are around 4.5 hours run time in a scenario that never lasted more than 3-4 hours without the fix. Another user who had a circumstance that was much faster to replicate than anything I could duplicate in a lab (a matter of a handful of minutes, rather than hours) has also confirmed it's no longer happening.

                                    Needs more runtime and more feedback from others, but initial results are good.

                                    Those of you impacted, please upgrade to latest 2.3.1 (instructions here), remove the disabling cores workaround if you did that, and let us know how it goes.

                                    1 Reply Last reply Reply Quote 0
                                    • B Offline
                                      byusinger84
                                      last edited by

                                      @cmb:

                                      The fix for this was merged this morning and is in the most recent available snapshot, Thu May 12 14:01:47 CDT and newer.

                                      My test setups are around 4.5 hours run time in a scenario that never lasted more than 3-4 hours without the fix. Another user who had a circumstance that was much faster to replicate than anything I could duplicate in a lab (a matter of a handful of minutes, rather than hours) has also confirmed it's no longer happening.

                                      Needs more runtime and more feedback from others, but initial results are good.

                                      Those of you impacted, please upgrade to latest 2.3.1 (instructions here), remove the disabling cores workaround if you did that, and let us know how it goes.

                                      OP here. Testing this for you now. It usually crashed on me within 6-12 hours but up to 24 hours was the latest. I should know no later than this weekend if it is working or not. I'll update you when I know more. Thank you!

                                      1 Reply Last reply Reply Quote 0
                                      • C Offline
                                        cmb
                                        last edited by

                                        My test environments are now over 11.5 hours and still running fine. That's 7.5+ hours longer than any affected kernel has lasted in the circumstance. Definitely seems to be fixed.

                                        Any feedback from those impacted who upgrade appreciated.

                                        1 Reply Last reply Reply Quote 0
                                        • E Offline
                                          eeit
                                          last edited by

                                          Hello,

                                          guess 2.3.1.a.20160512.2347 works, but now i get permanent crash reports with informations about php-warnings like this:

                                          10.3-RELEASE-p2
                                          FreeBSD 10.3-RELEASE-p2 #68 ac020b1(RELENG_2_3): Fri May 13 00:26:15 CDT 2016    root@ce23-amd64-builder:/builder/pfsense/tmp/obj/builder/pfsense/tmp/FreeBSD-src/sys/pfSense

                                          Crash report details:

                                          PHP Errors:
                                          [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                                          [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                                          [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                                          [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                                          [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                                          [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                                          [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                                          [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                                          [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                                          [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                                          [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                                          [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                                          [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                                          [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                                          [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                                          [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                                          [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                                          [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                                          [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                                          [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                                          [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                                          [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                                          [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                                          [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                                          [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                                          [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                                          [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                                          [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                                          [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                                          [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                                          [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                                          [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                                          [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                                          [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                                          [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                                          [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                                          [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                                          [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                                          [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                                          [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551

                                          This informs are in the front of the Firewall -> Rules -> .. page too.

                                          Any idea to fix this ?

                                          Thx.

                                          1 Reply Last reply Reply Quote 0
                                          • F Offline
                                            fragged
                                            last edited by

                                            @eeit:

                                            Hello,

                                            guess 2.3.1.a.20160512.2347 works, but now i get permanent crash reports with informations about php-warnings like this:

                                            Any idea to fix this ?

                                            Thx.

                                            Fixed in: https://github.com/pfsense/pfsense/commit/4680f6bf755fa7d323beba599ea94646d2d5f3bb

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.