pfSense stops working after some time: no buffer space available

fabiolanza · May 30, 2020, 6:00 PM

Hi,

I am going through a strange problem. My pfSense works perfectly for about 20 hours and then it stops routing traffic by itself, without any touches to pfSense configuration or the network.

After problem kicks in, from the pfSense console:
ping gives me the following error:

ping: sendto: No buffer space available

tracert gives me the following error:

traceroute: sendto: No buffer space available

If I restart the network interfaces, all routing is gone, and network interfaces no longer get an ip address. If I try to add the routes manually, I get the following error:

route: writing to routing socket: Network is unreachable

System:
pfSense is running as a virtual machine on Proxmox. The network ports are USB 3.0 -> Ethernet 1Gbps adapters with USB passthrough: AX88179
The correct drivers are installed, and this adapter is officially supported by FreeBSD.

The problem is fixed by restarting the system entirely. Below are some screenshots of the problem.

Any advice on the cause of the problem is greatly appreciated.

Thanks

DaddyGo · May 30, 2020, 6:56 PM

@fabiolanza said in pfSense stops working after some time: no buffer space available:

Sorry, but it's already a weekend
isn't that just a joke?

The network ports are USB 3.0 -> Ethernet 1Gbps adapters with USB passthrough

by the way, Realtek ethernet controllers are also supported by FreeBSD, but I'll tell you they're not really usable

I have to say, I didn't even think, that the pfSense would start ... - VM + with USB eth. + USB passthrough

huge hurray for the system -

you really don't think that, I'm making to joke with you, I wouldn't have thought it would work in this form

fabiolanza · May 30, 2020, 8:06 PM

I found out that, as it seems, the no buffer space available problem is quite common. There is a page about it in the Netgate documentation

I did what is proposed in this page (set kern.ipc.nmbclusters to 1,000,000): https://docs.netgate.com/pfsense/en/latest/hardware/tuning-and-troubleshooting-network-cards.html

I see that there are other buffers such as net.inet.tcp.recvspace and net.inet.tcp.sendspace. Should I increase their buffers too?

This is what my System Information dashboard is showing up:

From console netstat -m:

I will keep monitoring now to see if the system becomes more stable.

fabiolanza · Jun 1, 2020, 8:44 AM

@DaddyGo are there any official recommendations not to use pfSense in a virtual environment? Your comment is not helpful at all.

fishbone222 · May 30, 2020, 8:33 PM

Not really helpful but sadly true. pfSense in an VM can work but is definitely not newby-friendly.
Many aspects that cause extra trouble and hard to debug if something changes.
I was running it at home successfully for more than one Year in an Proxmox-VE but at least with decent intel pro/1000 NIC'S.
One day it just stopped working and i couldn't find the problem.
Now i have for long time a small Machine with an J3455 and no problems ever again.

So your Project might be possible but without spending any money in dedicated hardware, not even some PCI-E NIC's it's a hard Job for an unstable result.

For me it's obvious if no one has patience to help on that.

But for me i had better results not to passthru the NIC's in my case at PCI-e and i didn't saw a benefit why i should do so.

fabiolanza · May 30, 2020, 9:42 PM

@fishbone222 thanks for your thoughts. I have a mini-itx motherboard with 1 PCIe 16x slot that I am using to connect a NVIDIA Quadro card for hardware transcoding in Plex on my Windows Server VM. Since my motherboard has only 1 network port I had to use the USB adapter to expand. It seems a floppy thing but in reality, it's not and FreeBSD is fully compatible with it. The passthrough works correctly and it's like the card was directly attached to the operating system. I have a ZyXEL firewall appliance but I got tired of the OS and limitations, and decided to use my hypervisor environment to move to pfSense. So far this is the only issue.

bmeeks · Jun 1, 2020, 5:27 AM

@fabiolanza said in pfSense stops working after some time: no buffer space available:

@fishbone222 thanks for your thoughts. I have a mini-itx motherboard with 1 PCIe 16x slot that I am using to connect a NVIDIA Quadro card for hardware transcoding in Plex on my Windows Server VM. Since my motherboard has only 1 network port I had to use the USB adapter to expand. It seems a floppy thing but in reality, it's not and FreeBSD is fully compatible with it. The passthrough works correctly and it's like the card was directly attached to the operating system. I have a ZyXEL firewall appliance but I got tired of the OS and limitations, and decided to use my hypervisor environment to move to pfSense. So far this is the only issue.

USB NICs are, in general, quite lousy. This is especially true when trying to use them in most UNIX/Linux/BSD setups. So your basic root issue is the use of USB NICs. You state that is your only option for your current hardware platform, so you might just be stuck working around the USB NIC problem by doing the reboot now and then. You can attempt some of the tweaks in the document you linked, but honestly I would not have high hopes of full success with the USB NICs.

NollipfSense · Jun 1, 2020, 4:06 AM

@fabiolanza I found this USB 3 NIC that uses Intel chipset and would purchase for another project not related to pfSense, but it was sold out however, I thought to share.

https://www.ebay.com/itm/Syncwire-USB-3-0-to-Ethernet-Adapter-1000-Mbps-RJ45-LAN-Gigabit-Network-Adapter/113718959682?ssPageName=STRK%3AMEBIDX%3AIT&_trksid=p2060353.m1438.l2649

fabiolanza · Jun 1, 2020, 5:25 AM

@NollipfSense Hi I saw somewhere that these use Realtek chipsets. I searched on Syncwire website but did not find the page for this product. Would you have a datasheet to share? Thanks for the answer by the way.

fabiolanza · Jun 1, 2020, 5:27 AM

@bmeeks thanks for the feedback. Dou you mean USB NICs in general or it's more related to the ASIX AX88179 chipset? I did not find an Intel one.

Gertjan · Jun 1, 2020, 6:17 AM

Really ??

edit : and who is 10.10.10.1 ?

Don't tell my you have pfBlockerNG also installed.

fabiolanza · Jun 1, 2020, 8:43 AM

@Gertjan I have a transit link architecture setup. 10.10.10.2 is the firewall and 10.10.10.1 is my L3 switch.

DaddyGo · Jun 1, 2020, 8:49 AM

@fabiolanza
Sorry again!:

I really didn't want to have a say in your ideas, but somehow you have to start pfSense....(and also learning about it)

is there possibly a mini-PCie socket on this miniITX MOBO?
http://www.commell.com.tw/Product/Peripheral/PCI%20Express%20mini%20card/MPX-350.htm

I've been working with pfSense for a long time (also in a VM environment), but so far I wouldn't dare to start a USB NIC installation

a good piece of advice, don't rigidly stick to what FreeBSD supports, listen to the advice of more experienced colleagues,
(I also wrote that, for example for Realtek is also have driver under FreeBSD, but not even recommended by anyone!)

Plus, I know Asix chips well (these chips cannot be used in this environment), I use it in simple portable MODBUS controllers under Windows, that's all it knows, it's not the basis of a firewall interface (ethernet controller):

fabiolanza · Jun 1, 2020, 8:45 AM

I just would like to mention that after the mbuf cluster adjustment that I did, the previous network issue that I had is no longer there. However, I introduced another error, in which any HTTP download would get interrupted after about 50MB. I can resume, but it would still fail over and over until completion of the download. I got sick of this and re-enabled my firewall appliance that is not nearly as nice as the pfSense. I guess I will have to buy one of the chinese pfSense appliances.

DaddyGo · Jun 1, 2020, 8:48 AM

Chinese stuff is not a good idea, you will make a similar mistake!

https://www.netgate.com/solutions/pfsense/sg-1100.html
https://www.pcengines.ch/apu4d4.htm

fabiolanza · Jun 1, 2020, 8:51 AM

@DaddyGo I understand. This is my motherboard: https://www.asrock.com/mb/AMD/X570%20Phantom%20Gaming-ITXTB3/index.asp. I am using the one 1 PCIe for the GPU. Since this mobo has a Thunderbolt 3 I was thinking about using a Thunderbolt PCIe expansion chassis and then installing a PCIe network card. Any thoughts?

fabiolanza · Jun 1, 2020, 8:53 AM

@DaddyGo I am in Italy and there is not a lot of resellers here. I'd go for the https://www.netgate.com/solutions/pfsense/sg-3100.html because I have a 1Gbps WAN link, but can't find where to buy it. It's a pitty Netgate products are not on Amazon.it.

DaddyGo · Jun 1, 2020, 8:54 AM

I live here not far from you and there are many opportunities to get good hardware

DaddyGo · Jun 1, 2020, 8:58 AM

https://www.netgate.com/partners/locator.html#italy

For a 1 Gig ISP you will need: https://www.netgate.com/solutions/pfsense/sg-3100.html

DaddyGo · Jun 1, 2020, 9:11 AM

@fabiolanza
I understand. This is my motherboard: https://www.asrock.com/mb/AMD/X570 Phantom Gaming-ITXTB3/index.asp. I am using the one 1 PCIe for the GPU. Since this mobo has a Thunderbolt 3 I was thinking about using a Thunderbolt PCIe expansion chassis and then installing a PCIe network card. Any thoughts?

Try to think about dedicated hardware, when you building a firewall and consider the needs of your own system.
These miniITX things are good too, but not my favorites.
Consider consumption, as a firewall operates 24 hours a day. (SOHO)
Don't want to run other things on the "iron" where pfSense is running!

Thunderbolt is not a good starting point either

edit: From the fact that you are going to put a PCIe NIC in an external adapter, even the OP system has to handle Thunderbolt.
This is the same as the USB issue.