Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Upgrade to 2.4.4-p1 breaks SSH connection

    Scheduled Pinned Locked Moved Firewalling
    12 Posts 6 Posters 1.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      jerrydeasley
      last edited by

      I have a daily borg backup chron job that creates (over SSH) a daily archive of my data to a remote repository at rsync.net. It worked perfectly from April, 2018 until December 3, 2018 when I upgraded my pfsense from 2.4.4 to 2.4.4-p1. Now the borg backup fails. I have done some troubleshooting and I believe have now isolated the problem to pfsense breaking the SSH connection. To test the problem I have used scp (SSH) to attempt to upload a ~100MB file from my computer to rsync.net. If I connect my computer directly to my residential gateway (bypassing pfsense) it uploads the file as expected. If I put the same machine behind the firewall, it fails (times out) after uploading exactly 16MB every time. I have tried a clean install of pfsense 2.4.4.-p1, but it did not help. I would like to try a clean install of 2.4.4 to see if that resolves the problem, but Netgate doesn't host prior versions for download. My gateway is set through passthrough to pfsense. Anyone having a similar problem? Any ideas on how to solve this?

      O 1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        Nothing similar here. I push a ton of rsync data through pfSense regularly.

        Any log messages in the system, gateway, routing logs (or any logs) around the time the transfer fails?

        Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • O
          opoplawski @jerrydeasley
          last edited by

          @jerrydeasley I'm seeing ssh (and other long lived TCP) traffic timeout as well - https://forum.netgate.com/topic/138472/pfsense-periodically-drops-or-misroutes-packets I'd be curious to see if you see similar behavior on your firewall. I have no idea what is up in my case.

          1 Reply Last reply Reply Quote 0
          • S
            SteveITS Galactic Empire
            last edited by

            @jerrydeasley and @opoplawski, two questions. One, in your cases the router is on the source/outbound side of the connection? I had upgraded our office router to 2.4.4_1 but our data center is still on 2.4.4. We run an overnight backup from our DC to our office so have about 90 minutes of rsync each night. Which leads me to question two...this is only for large files correct? So it wouldn't happen when copying lots of small files?

            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
            Upvote ๐Ÿ‘ helpful posts!

            O 1 Reply Last reply Reply Quote 0
            • GrimsonG
              Grimson Banned
              last edited by

              Check the gateway latency/packet-loss during your uploads if it spikes pfSense might mark the gateway as down and depending on your setup reset the states for it. In this case, and if you only have one gateway, you can turn on "Disable Gateway Monitoring Action" in your gateway settings. Additionally you might want to use traffic-shaping/limiting to prevent your WAN connection from being saturated.

              1 Reply Last reply Reply Quote 0
              • O
                opoplawski @SteveITS
                last edited by

                @teamits I'm seeing issues in both directions. rsync establishes a single connection so it wouldn't matter the size of the individual connections. I started seeing my issue on 2.4.4 shortly before updating to 2.4.4p1.

                1 Reply Last reply Reply Quote 0
                • johnpozJ
                  johnpoz LAYER 8 Global Moderator
                  last edited by

                  @jerrydeasley said in Upgrade to 2.4.4-p1 breaks SSH connection:

                  Netgate doesn't host prior versions for download.

                  Huh... Release 2.4.4 and 2.4.4p1 are both available for download..

                  Index of /mirror/downloads/
                  
                  ../
                  old/                                               03-Dec-2018 15:07                   -
                  pfSense-CE-2.4.4-RELEASE-amd64.iso.gz              20-Sep-2018 13:21           342307778
                  pfSense-CE-2.4.4-RELEASE-amd64.iso.gz.sha256       20-Sep-2018 13:22                 114
                  pfSense-CE-2.4.4-RELEASE-p1-amd64.iso.gz           26-Nov-2018 16:53           343733342
                  pfSense-CE-2.4.4-RELEASE-p1-amd64.iso.gz.sha256    26-Nov-2018 16:54                 117
                  pfSense-CE-memstick-2.4.4-RELEASE-amd64.img.gz     20-Sep-2018 13:33           342357493
                  pfSense-CE-memstick-2.4.4-RELEASE-amd64.img.gz...> 20-Sep-2018 13:34                 123
                  pfSense-CE-memstick-2.4.4-RELEASE-p1-amd64.img.gz  26-Nov-2018 17:01           343116906
                  pfSense-CE-memstick-2.4.4-RELEASE-p1-amd64.img...> 26-Nov-2018 17:02                 126
                  pfSense-CE-memstick-ADI-2.4.4-RELEASE-amd64.img.gz 20-Sep-2018 13:47           340978728
                  pfSense-CE-memstick-ADI-2.4.4-RELEASE-amd64.img..> 20-Sep-2018 13:48                 127
                  pfSense-CE-memstick-ADI-2.4.4-RELEASE-p1-amd64...> 26-Nov-2018 17:09           343993128
                  pfSense-CE-memstick-ADI-2.4.4-RELEASE-p1-amd64...> 26-Nov-2018 17:09                 130
                  pfSense-CE-memstick-serial-2.4.4-RELEASE-amd64...> 20-Sep-2018 14:01           343130260
                  pfSense-CE-memstick-serial-2.4.4-RELEASE-amd64...> 20-Sep-2018 14:02                 130
                  pfSense-CE-memstick-serial-2.4.4-RELEASE-p1-amd..> 26-Nov-2018 17:17           342895762
                  pfSense-CE-memstick-serial-2.4.4-RELEASE-p1-amd..> 26-Nov-2018 17:17                 133
                  
                  

                  An intelligent man is sometimes forced to be drunk to spend time with his fools
                  If you get confused: Listen to the Music Play
                  Please don't Chat/PM me for help, unless mod related
                  SG-4860 24.11 | Lab VMs 2.8, 24.11

                  S 1 Reply Last reply Reply Quote 0
                  • jimpJ
                    jimp Rebel Alliance Developer Netgate
                    last edited by

                    Check the state table when the transfer starts and then again when it fails. What does the state for the ssh/rsync connection look like at each point in time?

                    And are you certain nothing else changed on the server side? For example, there is a known issue with FreeBSD 12.0 where if someone manually changed to the TCP Cubic congestion control algorithm they would see network stalls with some traffic, and ssh/rsync is explicitly called out as being affected. pfSense 2.4.4-p1 is on FreeBSD 11.2 and uses the default newreno CC algo, so it wouldn't be pfSense doing that.

                    Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    1 Reply Last reply Reply Quote 0
                    • S
                      SteveITS Galactic Empire @johnpoz
                      last edited by

                      @johnpoz said in Upgrade to 2.4.4-p1 breaks SSH connection:

                      Release 2.4.4 and 2.4.4p1 are both available for download..

                      Yes but if you click on that old/ directory it shows an error now...I'm pretty sure in the past mirrors would host (all?) older versions, since I had saved a link to one.

                      @opoplawski said in Upgrade to 2.4.4-p1 breaks SSH connection:

                      rsync establishes a single connection

                      Yes, of course, d'oh. In that case with 2.4.4_1 on the "remote" side I'm not seeing this.

                      Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                      When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                      Upvote ๐Ÿ‘ helpful posts!

                      1 Reply Last reply Reply Quote 0
                      • johnpozJ
                        johnpoz LAYER 8 Global Moderator
                        last edited by johnpoz

                        @teamits said in Upgrade to 2.4.4-p1 breaks SSH connection:

                        Yes but if you click on that old/ directory it shows an error now

                        Who said to click the OLD dir --- you asked to install 2.4.4 - which is right there for download.. And works just fine!

                        I would like to try a clean install of 2.4.4 to see if that resolves the problem

                        0_1545077354736_works.png

                        Yeah no shit you can not download OLD release.. But 2.4.4 is current, p1 is a patch to that install. So yeah you can download 2.4.4 which is what you asked for. Clean install of 2.4.4

                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                        If you get confused: Listen to the Music Play
                        Please don't Chat/PM me for help, unless mod related
                        SG-4860 24.11 | Lab VMs 2.8, 24.11

                        S 1 Reply Last reply Reply Quote 0
                        • S
                          SteveITS Galactic Empire @johnpoz
                          last edited by

                          @johnpoz said in Upgrade to 2.4.4-p1 breaks SSH connection:

                          Who said to click the OLD dir --- you asked to install 2.4.4

                          I am not Jerry, I was just making a side comment on his question/comment "Netgate doesn't host prior versions for download" vs the answer that 2.4.4 is available (which it is). If I was trying to repair an old router I would probably prefer to install the old version, restore the config from the old version, and then upgrade, rather than restore the old config on the new version. I'm not trying to hijack the thread so I'll stop now.

                          Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                          When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                          Upvote ๐Ÿ‘ helpful posts!

                          1 Reply Last reply Reply Quote 0
                          • johnpozJ
                            johnpoz LAYER 8 Global Moderator
                            last edited by johnpoz

                            If you were wanting to repair an OLD router - you should have a copy of the install media you used to install the OLD router...

                            I have OLD copies of pfsense going back Years... Since I keep a copy of the ISO's that I installed... How do you think I got my banner profile banner with pfsense version 1.0.1 in the background ;) I fired it up on a VM to use specially when they changed to new forum software and could do stuff like that on your profile..

                            An intelligent man is sometimes forced to be drunk to spend time with his fools
                            If you get confused: Listen to the Music Play
                            Please don't Chat/PM me for help, unless mod related
                            SG-4860 24.11 | Lab VMs 2.8, 24.11

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post
                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.