Upgrade to 2.4.4-p1 breaks SSH connection
-
I have a daily borg backup chron job that creates (over SSH) a daily archive of my data to a remote repository at rsync.net. It worked perfectly from April, 2018 until December 3, 2018 when I upgraded my pfsense from 2.4.4 to 2.4.4-p1. Now the borg backup fails. I have done some troubleshooting and I believe have now isolated the problem to pfsense breaking the SSH connection. To test the problem I have used scp (SSH) to attempt to upload a ~100MB file from my computer to rsync.net. If I connect my computer directly to my residential gateway (bypassing pfsense) it uploads the file as expected. If I put the same machine behind the firewall, it fails (times out) after uploading exactly 16MB every time. I have tried a clean install of pfsense 2.4.4.-p1, but it did not help. I would like to try a clean install of 2.4.4 to see if that resolves the problem, but Netgate doesn't host prior versions for download. My gateway is set through passthrough to pfsense. Anyone having a similar problem? Any ideas on how to solve this?
-
Nothing similar here. I push a ton of rsync data through pfSense regularly.
Any log messages in the system, gateway, routing logs (or any logs) around the time the transfer fails?
-
@jerrydeasley I'm seeing ssh (and other long lived TCP) traffic timeout as well - https://forum.netgate.com/topic/138472/pfsense-periodically-drops-or-misroutes-packets I'd be curious to see if you see similar behavior on your firewall. I have no idea what is up in my case.
-
@jerrydeasley and @opoplawski, two questions. One, in your cases the router is on the source/outbound side of the connection? I had upgraded our office router to 2.4.4_1 but our data center is still on 2.4.4. We run an overnight backup from our DC to our office so have about 90 minutes of rsync each night. Which leads me to question two...this is only for large files correct? So it wouldn't happen when copying lots of small files?
-
Check the gateway latency/packet-loss during your uploads if it spikes pfSense might mark the gateway as down and depending on your setup reset the states for it. In this case, and if you only have one gateway, you can turn on "Disable Gateway Monitoring Action" in your gateway settings. Additionally you might want to use traffic-shaping/limiting to prevent your WAN connection from being saturated.
-
@teamits I'm seeing issues in both directions. rsync establishes a single connection so it wouldn't matter the size of the individual connections. I started seeing my issue on 2.4.4 shortly before updating to 2.4.4p1.
-
@jerrydeasley said in Upgrade to 2.4.4-p1 breaks SSH connection:
Netgate doesn't host prior versions for download.
Huh... Release 2.4.4 and 2.4.4p1 are both available for download..
Index of /mirror/downloads/ ../ old/ 03-Dec-2018 15:07 - pfSense-CE-2.4.4-RELEASE-amd64.iso.gz 20-Sep-2018 13:21 342307778 pfSense-CE-2.4.4-RELEASE-amd64.iso.gz.sha256 20-Sep-2018 13:22 114 pfSense-CE-2.4.4-RELEASE-p1-amd64.iso.gz 26-Nov-2018 16:53 343733342 pfSense-CE-2.4.4-RELEASE-p1-amd64.iso.gz.sha256 26-Nov-2018 16:54 117 pfSense-CE-memstick-2.4.4-RELEASE-amd64.img.gz 20-Sep-2018 13:33 342357493 pfSense-CE-memstick-2.4.4-RELEASE-amd64.img.gz...> 20-Sep-2018 13:34 123 pfSense-CE-memstick-2.4.4-RELEASE-p1-amd64.img.gz 26-Nov-2018 17:01 343116906 pfSense-CE-memstick-2.4.4-RELEASE-p1-amd64.img...> 26-Nov-2018 17:02 126 pfSense-CE-memstick-ADI-2.4.4-RELEASE-amd64.img.gz 20-Sep-2018 13:47 340978728 pfSense-CE-memstick-ADI-2.4.4-RELEASE-amd64.img..> 20-Sep-2018 13:48 127 pfSense-CE-memstick-ADI-2.4.4-RELEASE-p1-amd64...> 26-Nov-2018 17:09 343993128 pfSense-CE-memstick-ADI-2.4.4-RELEASE-p1-amd64...> 26-Nov-2018 17:09 130 pfSense-CE-memstick-serial-2.4.4-RELEASE-amd64...> 20-Sep-2018 14:01 343130260 pfSense-CE-memstick-serial-2.4.4-RELEASE-amd64...> 20-Sep-2018 14:02 130 pfSense-CE-memstick-serial-2.4.4-RELEASE-p1-amd..> 26-Nov-2018 17:17 342895762 pfSense-CE-memstick-serial-2.4.4-RELEASE-p1-amd..> 26-Nov-2018 17:17 133 -
Check the state table when the transfer starts and then again when it fails. What does the state for the ssh/rsync connection look like at each point in time?
And are you certain nothing else changed on the server side? For example, there is a known issue with FreeBSD 12.0 where if someone manually changed to the TCP Cubic congestion control algorithm they would see network stalls with some traffic, and ssh/rsync is explicitly called out as being affected. pfSense 2.4.4-p1 is on FreeBSD 11.2 and uses the default newreno CC algo, so it wouldn't be pfSense doing that.
-
@johnpoz said in Upgrade to 2.4.4-p1 breaks SSH connection:
Release 2.4.4 and 2.4.4p1 are both available for download..
Yes but if you click on that old/ directory it shows an error now...I'm pretty sure in the past mirrors would host (all?) older versions, since I had saved a link to one.
@opoplawski said in Upgrade to 2.4.4-p1 breaks SSH connection:
rsync establishes a single connection
Yes, of course, d'oh. In that case with 2.4.4_1 on the "remote" side I'm not seeing this.
-
@teamits said in Upgrade to 2.4.4-p1 breaks SSH connection:
Yes but if you click on that old/ directory it shows an error now
Who said to click the OLD dir --- you asked to install 2.4.4 - which is right there for download.. And works just fine!
I would like to try a clean install of 2.4.4 to see if that resolves the problem
Yeah no shit you can not download OLD release.. But 2.4.4 is current, p1 is a patch to that install. So yeah you can download 2.4.4 which is what you asked for. Clean install of 2.4.4
-
@johnpoz said in Upgrade to 2.4.4-p1 breaks SSH connection:
Who said to click the OLD dir --- you asked to install 2.4.4
I am not Jerry, I was just making a side comment on his question/comment "Netgate doesn't host prior versions for download" vs the answer that 2.4.4 is available (which it is). If I was trying to repair an old router I would probably prefer to install the old version, restore the config from the old version, and then upgrade, rather than restore the old config on the new version. I'm not trying to hijack the thread so I'll stop now.
-
If you were wanting to repair an OLD router - you should have a copy of the install media you used to install the OLD router...
I have OLD copies of pfsense going back Years... Since I keep a copy of the ISO's that I installed... How do you think I got my banner profile banner with pfsense version 1.0.1 in the background ;) I fired it up on a VM to use specially when they changed to new forum software and could do stuff like that on your profile..
