Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    2.4.5 High latency and packet loss, not in a vm

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    81 Posts 22 Posters 19.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      jammcla
      last edited by

      2 Cores worked great for me on my test box, but when I went up to 8 cores like the production box, everything went downhill. luckily I changed the cores on the test box before I tried on the production box.

      1 Reply Last reply Reply Quote 0
      • Q
        q54e3w
        last edited by q54e3w

        Heres my 3 month WAN latency graph showing the clear increase in average latency since the 2.4.5 upgrade. I just added this to my support ticket. I am not seeing a typical short timeframe ISP latency increase.

        da8b12bc-734b-412f-a446-a0aa31cd0798-image.png

        1 Reply Last reply Reply Quote 0
        • provelsP
          provels
          last edited by provels

          No idea if this is related, but I tried to add 2 monitor IPs to my v4 and v6 gateways and immediately I had latency and drops. Even when they were removed, I could barely get control of the web GUI. Had to reboot. Below are clips of top -aSH and proc load.
          e6a9a363-c776-40c3-9974-e2490ddbf403-image.png6305d453-a37d-4d27-b86d-81f559287581-image.png

          Peder

          MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
          BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

          ? 1 Reply Last reply Reply Quote 0
          • ?
            A Former User @provels
            last edited by

            @provels That is exactly the bug, yup! Fix at the moment appears to be to drop to one vCPU if you're able.

            provelsP 1 Reply Last reply Reply Quote 0
            • provelsP
              provels @A Former User
              last edited by

              @muppet Thanks. Think I'd rather keep the procs! My Hyper-V host is pretty lame!

              Peder

              MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
              BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

              ? 1 Reply Last reply Reply Quote 0
              • ?
                A Former User @provels
                last edited by

                @provels Then I suggest rolling back to your 2.4.4p3 snapshot.

                provelsP 1 Reply Last reply Reply Quote 0
                • provelsP
                  provels @A Former User
                  last edited by provels

                  @muppet It's just li'l ole me here. I have 2.4.4-p3, 2.4.5 UFS and 2.4.5 ZFS available on the lab.

                  Peder

                  MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
                  BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

                  1 Reply Last reply Reply Quote 0
                  • Cool_CoronaC
                    Cool_Corona
                    last edited by

                    I get timeouts when filter is reloading.

                    pfsensen_GW_timeout_filterreload.JPG

                    Still running 8 core setup and only minor issues. Its not production ready yet since it has timeouts when this happens:

                    Apr 16 22:06:33 xinetd 13368 Reconfigured: new=0 old=4 dropped=0 (services)
                    Apr 16 22:06:33 xinetd 13368 readjusting service 19001-udp
                    Apr 16 22:06:33 xinetd 13368 readjusting service 19001-tcp
                    Apr 16 22:06:33 xinetd 13368 readjusting service 19000-udp
                    Apr 16 22:06:33 xinetd 13368 readjusting service 19000-tcp
                    Apr 16 22:06:33 xinetd 13368 Swapping defaults
                    Apr 16 22:06:33 xinetd 13368 Starting reconfiguration
                    Apr 16 22:06:32 php-fpm 68319 /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
                    Apr 16 22:06:32 php-fpm 68319 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WANGW'
                    Apr 16 22:06:31 check_reload_status Reloading filter
                    Apr 16 22:06:31 check_reload_status Restarting OpenVPN tunnels/interfaces
                    Apr 16 22:06:31 check_reload_status Restarting ipsec tunnels
                    Apr 16 22:06:31 check_reload_status updating dyndns WANGW
                    Apr 16 22:06:31 rc.gateway_alarm 66212 >>> Gateway alarm: WANGW (Addr:81.19.224.67 Alarm:0 RTT:1.945ms RTTsd:.093ms Loss:14%)
                    Apr 16 22:04:44 xinetd 13368 Reconfigured: new=0 old=4 dropped=0 (services)
                    Apr 16 22:04:44 xinetd 13368 readjusting service 19001-udp
                    Apr 16 22:04:44 xinetd 13368 readjusting service 19001-tcp
                    Apr 16 22:04:44 xinetd 13368 readjusting service 19000-udp
                    Apr 16 22:04:44 xinetd 13368 readjusting service 19000-tcp
                    Apr 16 22:04:44 xinetd 13368 Swapping defaults
                    Apr 16 22:04:44 xinetd 13368 Starting reconfiguration
                    Apr 16 22:04:44 php-fpm 61919 /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
                    Apr 16 22:04:44 php-fpm 61919 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WANGW'
                    Apr 16 22:04:43 check_reload_status Reloading filter
                    Apr 16 22:04:43 check_reload_status Restarting OpenVPN tunnels/interfaces
                    Apr 16 22:04:43 check_reload_status Restarting ipsec tunnels
                    Apr 16 22:04:43 check_reload_status updating dyndns WANGW
                    Apr 16 22:04:43 rc.gateway_alarm 2496 >>> Gateway alarm: WANGW (Addr:81.19.224.67 Alarm:1 RTT:928.897ms RTTsd:5161.512ms Loss:18%)
                    Apr 16 22:04:00 xinetd 13368 Reconfigured: new=0 old=4 dropped=0 (services)
                    Apr 16 22:04:00 xinetd 13368 readjusting service 19001-udp
                    Apr 16 22:04:00 xinetd 13368 readjusting service 19001-tcp
                    Apr 16 22:04:00 xinetd 13368 readjusting service 19000-udp
                    Apr 16 22:04:00 xinetd 13368 readjusting service 19000-tcp
                    Apr 16 22:04:00 xinetd 13368 Swapping defaults
                    Apr 16 22:04:00 xinetd 13368 Starting reconfiguration
                    Apr 16 22:03:27 php-fpm 32951 /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
                    Apr 16 22:03:27 php-fpm 32951 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WANGW'
                    Apr 16 22:03:26 check_reload_status Reloading filter
                    Apr 16 22:03:26 check_reload_status Restarting OpenVPN tunnels/interfaces
                    Apr 16 22:03:26 check_reload_status Restarting ipsec tunnels
                    Apr 16 22:03:26 check_reload_status updating dyndns WANGW
                    Apr 16 22:03:26 rc.gateway_alarm 62528 >>> Gateway alarm: WANGW (Addr:81.19.224.67 Alarm:0 RTT:4.679ms RTTsd:26.791ms Loss:20%)
                    Apr 16 22:03:16 xinetd 13368 Reconfigured: new=0 old=4 dropped=0 (services)
                    Apr 16 22:03:16 xinetd 13368 readjusting service 19001-udp
                    Apr 16 22:03:16 xinetd 13368 readjusting service 19001-tcp
                    Apr 16 22:03:16 xinetd 13368 readjusting service 19000-udp
                    Apr 16 22:03:16 xinetd 13368 readjusting service 19000-tcp
                    Apr 16 22:03:16 xinetd 13368 Swapping defaults
                    Apr 16 22:03:16 xinetd 13368 Starting reconfiguration
                    Apr 16 22:03:16 php-fpm 19187 /rc.openvpn: Gateway, none 'available' for inet6, use the first one configured. ''
                    Apr 16 22:03:16 php-fpm 19187 /rc.openvpn: Gateway, none 'available' for inet, use the first one configured. 'WANGW'
                    Apr 16 22:03:15 check_reload_status Reloading filter
                    Apr 16 22:03:15 check_reload_status Restarting OpenVPN tunnels/interfaces
                    Apr 16 22:03:15 check_reload_status Restarting ipsec tunnels
                    Apr 16 22:03:15 check_reload_status updating dyndns WANGW
                    Apr 16 22:03:15 rc.gateway_alarm 38824 >>> Gateway alarm: WANGW (Addr:81.19.224.67 Alarm:1 RTT:1.923ms RTTsd:.079ms Loss:21%)
                    Apr 16 22:00:52 php [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
                    Apr 16 22:00:00 php [pfBlockerNG] Starting cron process.

                    6 minutes where the network in unreachable.

                    1 Reply Last reply Reply Quote 1
                    • ?
                      A Former User
                      last edited by

                      This is all the same thing. We are past the point of 'is this a problem'. It is. Netgate is working on it. They have solicited configuration data and some users have provided that.

                      The impact of changing gateway configuration (monitor ip, polling freq, averaging time, etc) was reported at least once back on April 4 (look back in this thread) and acknowledged by Netgate.

                      The simplest way to manage this:

                      1. If you can manage on 2.4.5 without disruption, stay put.
                      2. If not, downgrade to 2.4.4-p3. Adjust the settings in System->Upgrade as needed.

                      Now we wait to see what happens, at some point 2.4.5-p1 will come out. When? I don't think anyone one knows at the moment.

                      Not in a passive mood? Open a ticket at Netgate Support and ask what you can provide that they would find helpful.

                      Just a note: When you post system log data with gateway alarms you might want to obscure your public IP, especially if it's a static IP. Just saying...

                      Cool_CoronaC 1 Reply Last reply Reply Quote 0
                      • Cool_CoronaC
                        Cool_Corona @A Former User
                        last edited by Cool_Corona

                        @jwj said in 2.4.5 High latency and packet loss, not in a vm:

                        This is all the same thing. We are past the point of 'is this a problem'. It is. Netgate is working on it. They have solicited configuration data and some users have provided that.

                        The impact of changing gateway configuration (monitor ip, polling freq, averaging time, etc) was reported at least once back on April 4 (look back in this thread) and acknowledged by Netgate.

                        The simplest way to manage this:

                        1. If you can manage on 2.4.5 without disruption, stay put.
                        2. If not, downgrade to 2.4.4-p3. Adjust the settings in System->Upgrade as needed.

                        Now we wait to see what happens, at some point 2.4.5-p1 will come out. When? I don't think anyone one knows at the moment.

                        Not in a passive mood? Open a ticket at Netgate Support and ask what you can provide that they would find helpful.

                        Just a note: When you post system log data with gateway alarms you might want to obscure your public IP, especially if it's a static IP. Just saying...

                        Its the monitor IP and not our own systems :)

                        1 Reply Last reply Reply Quote 0
                        • provelsP
                          provels
                          last edited by provels

                          Hyper-V VM
                          2,000,000 table entries
                          bogons enabled
                          pfBNG enabled
                          monitor IPs enabled
                          single core
                          no problem.

                          Monitor IPs is the only thing that affected my system with 2 procs, all else same.
                          2bc2828a-c758-46df-9c05-34b56a18b29a-image.png

                          Peder

                          MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
                          BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            Yes, that's replicable everywhere I've tried it. The devs working on this are aware.

                            Steve

                            1 Reply Last reply Reply Quote 2
                            • ?
                              A Former User
                              last edited by

                              I have been told that this issue has been replicated in the Netgate lab and they working on a fix.

                              I would also remind all that are impatient for a fix to this issue that the support and development staff at Netgate have been challenged in ways no one could have predicted a few months ago. A massive surge in the need for remote access across their installed customer base and all that goes with that. They have offered up support to non-paying pfsense users as well as their customers on contract to meet these changing needs.

                              So, they are on it.

                              https://www.netgate.com/blog/usns-mercy-steps-up-its-network-for-covid-19-support.html

                              1 Reply Last reply Reply Quote 3
                              • jimpJ
                                jimp Rebel Alliance Developer Netgate
                                last edited by

                                https://forum.netgate.com/post/908806

                                Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                Need help fast? Netgate Global Support!

                                Do not Chat/PM for help!

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post
                                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.