rclone sync command crashes WAN interface with “No buffer space available” errors (endto error: 55)
-
Hey everyone, hoping I can get some help on this one, google foo skills are failing me!
I am trying to upload a folder of about 5Gb to a remote server (tardigrade.io) using sync command in rclone (rclone sync --progress source destination). Everything goes well fort he first few files, up to approximately 1Gb of upload, then the WAN interface stops working with following errors in the gateways log:
Jul 6 14:07:07 dpinger WAN_DHCP 212.51.155.1: Clear latency 12346us stddev 20937us loss 16% Jul 6 14:06:39 dpinger WAN_DHCP 212.51.155.1: sendto error: 55 Jul 6 14:06:38 dpinger WAN_DHCP 212.51.155.1: sendto error: 55 Jul 6 14:06:37 dpinger WAN_DHCP 212.51.155.1: Alarm latency 15720us stddev 23238us loss 21% Jul 6 14:06:36 dpinger WAN_DHCP 212.51.155.1: sendto error: 55 Jul 6 14:06:35 dpinger WAN_DHCP 212.51.155.1: sendto error: 55 Jul 6 14:06:32 dpinger WAN_DHCP 212.51.155.1: sendto error: 55 Jul 6 14:06:31 dpinger WAN_DHCP 212.51.155.1: sendto error: 55 Jul 6 14:06:29 dpinger WAN_DHCP 212.51.155.1: sendto error: 55 Jul 6 14:06:28 dpinger WAN_DHCP 212.51.155.1: sendto error: 55 Jul 6 14:06:24 dpinger WAN_DHCP 212.51.155.1: sendto error: 55 Jul 6 14:06:23 dpinger WAN_DHCP 212.51.155.1: sendto error: 55
I have limited the upload to 10Mbytes/s, but I am still getting the same error, although the interface remains working. When I put the limit up to 20 MB the interface crashes again.
I have a symmetric 1000/1000 connection and what I think a pretty strong system, I rarely see the usage go above 30%:
Intel(R) Core(TM) i3-6100 CPU @ 3.70GHz
8Gb RAMIts a home network, so only some dozens of concurrent connections.
Any ideas on how do diagnose the exact issue and hopefully overcome it would be very much appreciated.
-
It's possible that under heavy loading your gateway stops responding to the pings from
dpinger
. In that casedpinger
will think the gateway is down and start taking action.You could try either turning off gateway monitoring temporarily, or greatly increasing the "setpoints" for packet loss before
dpinger
assumes a gateway is down. My suggestion would be to turn off gateway monitoring completely as a test.It's also possible that the software driver for your NIC has some issue that manifests itself under heavy loads. What is the NIC hardware in your box, and you have done some Google research to see if there any known issues with that NIC under load or if there are some FreeBSD sysctl tunables that might help?
-
Hey bmeeks,
The NIC on the WAN interface is:
igb0@pci0:1:0:0: class=0x020000 card=0x1780103c chip=0x150e8086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82580 Gigabit Network Connection'I had a look at tunables to improve buffer handling for the igb4 cards, as that is what the error code was leading me to on the pfsence site. However I am definitely new to adding tunables, so I wanted to see if there is some other things I missed before I get into that. But, I did not find anything specific about this card's bad performance under load otherwise.
I will try turning of gateway monitoring and see what happens, thanks for the suggestion.
-
@bmeeks said in rclone sync command crashes WAN interface with “No buffer space available” errors (endto error: 55):
It's possible that under heavy loading your gateway stops responding to the pings from
dpinger
. In that casedpinger
will think the gateway is down and start taking action.You could try either turning off gateway monitoring temporarily, or greatly increasing the "setpoints" for packet loss before
dpinger
assumes a gateway is down. My suggestion would be to turn off gateway monitoring completely as a test.That solved it. I ticked "Disable Gateway Monitoring Action" and while the errors still show up, the interface keeps up. It also looks like that even though I limit the upload to 20Mbytes, it still hits near 100Mbytes per second on the interface.