Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PfSense 2.3 LAN interface stops routing traffic - stops working after 2 or 3 day

    General pfSense Questions
    31
    88
    35.9k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      cmb
      last edited by

      Sometime this week.

      1 Reply Last reply Reply Quote 0
      • E
        eeit
        last edited by

        Hello,

        any news on this ? Is there a special setting or a patch to fix the issue available ?

        Thx.

        1 Reply Last reply Reply Quote 0
        • C
          cmb
          last edited by

          Reducing the system to one core is the only workaround. The fix was merged into devel branch today and hopefully into RELENG_2_3 tomorrow, in which case it'll hit snapshots tomorrow.

          1 Reply Last reply Reply Quote 0
          • J
            jonnyaalgaard.org
            last edited by

            Then I have upgraded to the new snapshot that came this morning (2.3.1-DEVELOPMENT (amd64) - built on Thu May 12 00:18:20 CDT 2016 - FreeBSD 10.3-RELEASE-p2), and will check it out.
            I have also removed the temp workaround, from /boot/loader.conf.local
            The temp workaround for me were to add these lines into loader.conf.local
            hint.lapic.1.disabled=1
            hint.lapic.2.disabled=1
            hint.lapic.3.disabled=1

            When I entered these lines pfsense were stabile for me, but now I have enabled all cpu cores again, and hopefully the new snapshot will work just fine.
            Before the workaround I did have 10-12 “crashes” a day.
            I will post back if the new snapshot works Ok or not.

            UPDATE! Before i managed to post this my pfsense crashed again with the new snapshot. I will try to disable all but 1 cpu core again, still using todays snapshot.

            1 Reply Last reply Reply Quote 0
            • C
              cmb
              last edited by

              There is no fix yet, as I noted. Watch the bug ticket for updates.

              1 Reply Last reply Reply Quote 0
              • C
                cmb
                last edited by

                The fix for this was merged this morning and is in the most recent available snapshot, Thu May 12 14:01:47 CDT and newer.

                My test setups are around 4.5 hours run time in a scenario that never lasted more than 3-4 hours without the fix. Another user who had a circumstance that was much faster to replicate than anything I could duplicate in a lab (a matter of a handful of minutes, rather than hours) has also confirmed it's no longer happening.

                Needs more runtime and more feedback from others, but initial results are good.

                Those of you impacted, please upgrade to latest 2.3.1 (instructions here), remove the disabling cores workaround if you did that, and let us know how it goes.

                1 Reply Last reply Reply Quote 0
                • B
                  byusinger84
                  last edited by

                  @cmb:

                  The fix for this was merged this morning and is in the most recent available snapshot, Thu May 12 14:01:47 CDT and newer.

                  My test setups are around 4.5 hours run time in a scenario that never lasted more than 3-4 hours without the fix. Another user who had a circumstance that was much faster to replicate than anything I could duplicate in a lab (a matter of a handful of minutes, rather than hours) has also confirmed it's no longer happening.

                  Needs more runtime and more feedback from others, but initial results are good.

                  Those of you impacted, please upgrade to latest 2.3.1 (instructions here), remove the disabling cores workaround if you did that, and let us know how it goes.

                  OP here. Testing this for you now. It usually crashed on me within 6-12 hours but up to 24 hours was the latest. I should know no later than this weekend if it is working or not. I'll update you when I know more. Thank you!

                  1 Reply Last reply Reply Quote 0
                  • C
                    cmb
                    last edited by

                    My test environments are now over 11.5 hours and still running fine. That's 7.5+ hours longer than any affected kernel has lasted in the circumstance. Definitely seems to be fixed.

                    Any feedback from those impacted who upgrade appreciated.

                    1 Reply Last reply Reply Quote 0
                    • E
                      eeit
                      last edited by

                      Hello,

                      guess 2.3.1.a.20160512.2347 works, but now i get permanent crash reports with informations about php-warnings like this:

                      10.3-RELEASE-p2
                      FreeBSD 10.3-RELEASE-p2 #68 ac020b1(RELENG_2_3): Fri May 13 00:26:15 CDT 2016    root@ce23-amd64-builder:/builder/pfsense/tmp/obj/builder/pfsense/tmp/FreeBSD-src/sys/pfSense

                      Crash report details:

                      PHP Errors:
                      [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                      [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                      [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                      [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                      [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                      [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                      [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                      [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                      [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                      [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                      [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                      [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                      [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                      [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                      [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                      [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                      [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                      [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                      [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                      [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                      [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                      [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                      [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                      [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                      [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                      [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                      [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                      [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                      [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                      [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                      [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                      [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                      [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 5 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                      [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                      [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                      [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551
                      [13-May-2016 04:24:10 America/New_York] PHP Warning:  Missing argument 6 for rule_columns_with_alias(), called in /usr/local/www/firewall_rules.php on line 551 and defined in /usr/local/www/guiconfig.inc on line 1132
                      [13-May-2016 04:24:10 America/New_York] PHP Stack trace:
                      [13-May-2016 04:24:10 America/New_York] PHP  1. {main}() /usr/local/www/firewall_rules.php:0
                      [13-May-2016 04:24:10 America/New_York] PHP  2. rule_columns_with_alias() /usr/local/www/firewall_rules.php:551

                      This informs are in the front of the Firewall -> Rules -> .. page too.

                      Any idea to fix this ?

                      Thx.

                      1 Reply Last reply Reply Quote 0
                      • F
                        fragged
                        last edited by

                        @eeit:

                        Hello,

                        guess 2.3.1.a.20160512.2347 works, but now i get permanent crash reports with informations about php-warnings like this:

                        Any idea to fix this ?

                        Thx.

                        Fixed in: https://github.com/pfsense/pfsense/commit/4680f6bf755fa7d323beba599ea94646d2d5f3bb

                        1 Reply Last reply Reply Quote 0
                        • C
                          cmb
                          last edited by

                          @fragged:

                          @eeit:

                          Hello,

                          guess 2.3.1.a.20160512.2347 works, but now i get permanent crash reports with informations about php-warnings like this:

                          Any idea to fix this ?

                          Thx.

                          Fixed in: https://github.com/pfsense/pfsense/commit/4680f6bf755fa7d323beba599ea94646d2d5f3bb

                          Yeah that was fixed. It's only cosmetic, for those running a version with that issue.

                          1 Reply Last reply Reply Quote 0
                          • A
                            afreaken
                            last edited by

                            @cmb:

                            The fix for this was merged this morning and is in the most recent available snapshot, Thu May 12 14:01:47 CDT and newer.

                            My test setups are around 4.5 hours run time in a scenario that never lasted more than 3-4 hours without the fix. Another user who had a circumstance that was much faster to replicate than anything I could duplicate in a lab (a matter of a handful of minutes, rather than hours) has also confirmed it's no longer happening.

                            Needs more runtime and more feedback from others, but initial results are good.

                            Those of you impacted, please upgrade to latest 2.3.1 (instructions here), remove the disabling cores workaround if you did that, and let us know how it goes.

                            So I followed the instructions on that page, and I get an error if I try to upgrade using the development option on the updates settings.

                            ERROR: Error trying to get packages list. Aborting…
                            pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/packagesite.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/packagesite.txz: Forbidden pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required
                            ERROR: Error trying to get packages list. Aborting...
                            pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/packagesite.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/packagesite.txz: Forbidden pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required

                            Also, afaik the option to upgrade from local file is no longer an option?
                            So unless I save it to a usb, and drive over to the other location, what can I do?

                            1 Reply Last reply Reply Quote 0
                            • A
                              adam65535
                              last edited by

                              Things are looking good so far according to all the posts by others.  Nice job dev team!  It looks like 2.3.1 is getting close to a final release now.  Only issues set for 2.3.1 release are now in feedback status…

                              1 Reply Last reply Reply Quote 0
                              • jimpJ
                                jimp Rebel Alliance Developer Netgate
                                last edited by

                                @afreaken:

                                So I followed the instructions on that page, and I get an error if I try to upgrade using the development option on the updates settings.

                                ERROR: Error trying to get packages list. Aborting…
                                pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/packagesite.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/packagesite.txz: Forbidden pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required
                                ERROR: Error trying to get packages list. Aborting...
                                pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-core/packagesite.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/meta.txz: Forbidden pkg: http://firmware.netgate.com/beta/packages/pfSense_v2_3_amd64-pfSense_v2_3/packagesite.txz: Forbidden pkg: Repository pfSense-core cannot be opened. 'pkg update' required pkg: Repository pfSense cannot be opened. 'pkg update' required

                                Looks like you're on the factory image, not CE, so the snapshots aren't available there currently. I'd suggest waiting for 2.3.1-RELEASE which if all goes as planned, will be first thing next week. You could, potentially, switch to CE snapshots but you'd lose the factory image tweaks/extras.

                                Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                Need help fast? Netgate Global Support!

                                Do not Chat/PM for help!

                                1 Reply Last reply Reply Quote 0
                                • A
                                  afreaken
                                  last edited by

                                  @jimp:

                                  Looks like you're on the factory image, not CE, so the snapshots aren't available there currently. I'd suggest waiting for 2.3.1-RELEASE which if all goes as planned, will be first thing next week. You could, potentially, switch to CE snapshots but you'd lose the factory image tweaks/extras.

                                  Sure, that makes sense. Now I know why I don't see "Community Edition" on my UI, didn't realize there was a tweaked version on my system. In the past I've run the CE edition (obviously not on this machine).

                                  1 Reply Last reply Reply Quote 0
                                  • luckman212L
                                    luckman212 LAYER 8
                                    last edited by

                                    If/when 2.3.1 comes out next week, if I have already botched my Factory system by syncing up with 2.3.1CE is there any way to get back on the Factory track?  Or if I do a backup/format/restore would that work?

                                    1 Reply Last reply Reply Quote 0
                                    • jimpJ
                                      jimp Rebel Alliance Developer Netgate
                                      last edited by

                                      @luckman212:

                                      If/when 2.3.1 comes out next week, if I have already botched my Factory system by syncing up with 2.3.1CE is there any way to get back on the Factory track?  Or if I do a backup/format/restore would that work?

                                      It's not quite that easy. It can be done with some file edits and such but it's not a very clean switch. Overall, less effort to reinstall+restore.

                                      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                      Need help fast? Netgate Global Support!

                                      Do not Chat/PM for help!

                                      1 Reply Last reply Reply Quote 0
                                      • luckman212L
                                        luckman212 LAYER 8
                                        last edited by

                                        @jimp:

                                        Overall, less effort to reinstall+restore.

                                        ok so just to confirm, I can just pave, reinstall, and restore the config.xml – no extra tweaks or edits needed? thanks again

                                        1 Reply Last reply Reply Quote 0
                                        • G
                                          georgeman
                                          last edited by

                                          jimp, could you quickly comment on what caused the issue??

                                          If it ain't broke, you haven't tampered enough with it

                                          1 Reply Last reply Reply Quote 0
                                          • jimpJ
                                            jimp Rebel Alliance Developer Netgate
                                            last edited by

                                            @georgeman:

                                            jimp, could you quickly comment on what caused the issue??

                                            If you mean the original problem from this thread about the traffic stopping, I don't personally have that info. Should be on the tickets and/or in the commit history. It was something in the IPsec code, IIRC.

                                            cmb is on a plane at the moment, he might have some more insight when he's able to respond.

                                            Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                            Need help fast? Netgate Global Support!

                                            Do not Chat/PM for help!

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post