CIFS transfer performance slow on APU2 board when using VLANs
-
It sounds like there are not only two different VLANs, but they're on two different physical interfaces in pfSense (igb1 and igb2).
As mentioned by Grimson, when they're in the same VLAN there's no routing going on, just packets going through the switch to get from one device to another. When they're in different VLANs, the path goes all the way back to pfSense, which has to route each packet from one network to the other. And don't forget that TCP is a two-way street, as there are acknowledgement packets that go the opposite direction of the data that also need to be routed. Performance here is likely being limited by your CPU.
Since you're using VLANs, and pfSense has to route between them, that's where you're getting slowed down.
-
Thank you for the replies!
Without the use of VLANs I'm getting a solid 106 - 108 MB/s from my Desktop-PC to my (virtualized) FreeNas cifs share (using the same hardware).
Do I understand you right, are server and client on the same device? If yes, are you using two dedicated network ports or do they share a single port to the switch? Try to explain your setup better or add a diagram.
No they are not on the same hardware, they are two different machines. I meant that I did not change the switch or server or hardware on my desktop etc.
@virgiliomi:
It sounds like there are not only two different VLANs, but they're on two different physical interfaces in pfSense (igb1 and igb2).
Yes thats how I set it up.
@virgiliomi:
As mentioned by Grimson, when they're in the same VLAN there's no routing going on, just packets going through the switch to get from one device to another. When they're in different VLANs, the path goes all the way back to pfSense, which has to route each packet from one network to the other. And don't forget that TCP is a two-way street, as there are acknowledgement packets that go the opposite direction of the data that also need to be routed. Performance here is likely being limited by your CPU.
Since you're using VLANs, and pfSense has to route between them, that's where you're getting slowed down.
Yes I understand the concept of VLANs and that now the packets have to be routed. I just could not believe, that the transfer speed would decrease so much. Especially as the CPU is more or less idling around during a transfer.
Edit:
Did some more reading here: https://doc.pfsense.org/index.php/Low_Throughput_Troubleshooting
I would interpret the screenshot from the System Acitivity tab as the cpu core 2 is at 100% and therefore limiting the throughput. Am I interpreting that right?
-
Perhaps should try out a small Layer3 Switch that is then routing all the inter VLAN traffic by itself.
Cisco SG-300/SG350 or SG500/SG550 or D-Link DGS1510-20 might be in the range of your budget
if not they are also often able to get refurbished or used for less money. But a small Layer3 Switch as the
SG300-10 if often able to route all the traffic between the VLANs with nearly wire speed and that with ease! -
" often able to route all the traffic between the VLANs with nearly wire speed and that with ease!"
And now no firewalling between… So why not just put them on the same vlan and not have to route, etc.
A load average of 2.55 doesn't seem like his nothing to me..
Are you having this routing performance between networks on igb1 and igb2 that are not tagged?
-
A load average of 2.55 doesn't seem like his nothing to me..
Are you having this routing performance between networks on igb1 and igb2 that are not tagged?
So you also think the CPU is the limiting factor?
I will try that and report back
Edit:
I set the following tunable:hw.igb.num_queues="1"Now It seems to get a core for every que / nic and I'm getting more or less always 60MB/s. So this would be a slight increase in performance, but I think its the end of whats possible (see attached screenshot)
Edit2:
I think I will install pfsense at my ESXi as vm just to see if more speed is possible.
-
hw.igb.num_queues="1"Ok you can also try out to set 2 or 4 queues for that, nothing matter it can scale up or down
so you have to play around with it. There are also some other things that can be done right.- high up the mbuf size to 125000, 250000, 500000 or 1000000
- Enable PowerD (hi adaptive)
Please read here about that theme: Tuning and troubleshooting network cards
You may be able to set the num.queues higher and the mbuf size lower and for sure also vice versa
it´s a try out or fine tuning so you will be able to get more or less the ideal setting for your set up and
hardware matching all your criteria. -
"I think I will install pfsense at my ESXi as vm just to see if more speed is possible."
How powerful is your esxi host? I had to move my pfsense off my esxi host when I got new inet speed because it was not capable of routing at that speed. But my esxi host is OLD.. It could only manage at best a bit over 200mbps.. When you also did natting only about 120mbps to the internet.. Which was not a big deal when internet was 80.. But when it went to 500 was a big issue.
The server and and my workstation that I move large amounts of data to where put on the same network because of this reason. I did not have any traffic between vlans where the speed reduction was a problem. Wifi clients to server, etc. iot devices don't use much of anything, etc..
But if you have a beefier esxi host then you should see improvement I would think.. Its is a valid test to be sure as a possible solution to your problem. Also curious if the untagged traffic sees the same issue or not.
-
@BlueKobold:
hw.igb.num_queues="1"Ok you can also try out to set 2 or 4 queues for that, nothing matter it can scale up or down
so you have to play around with it. There are also some other things that can be done right.- high up the mbuf size to 125000, 250000, 500000 or 1000000
- Enable PowerD (hi adaptive)
Please read here about that theme: Tuning and troubleshooting network cards
Done & done: I did not notice a difference between the queues and with th mbuf size. PowerD made al little difference (~1,5 MB/s)
I read the article and I will try to play with the values. But to be honest I bought the APU board to not have to play around that much. I thought it would be sufficient for my home network.
"I think I will install pfsense at my ESXi as vm just to see if more speed is possible."
How powerful is your esxi host? I had to move my pfsense off my esxi host when I got new inet speed because it was not capable of routing at that speed. But my esxi host is OLD.. It could only manage at best a bit over 200mbps.. When you also did natting only about 120mbps to the internet.. Which was not a big deal when internet was 80.. But when it went to 500 was a big issue.
The server and and my workstation that I move large amounts of data to where put on the same network because of this reason. I did not have any traffic between vlans where the speed reduction was a problem. Wifi clients to server, etc. iot devices don't use much of anything, etc..
But if you have a beefier esxi host then you should see improvement I would think.. Its is a valid test to be sure as a possible solution to your problem. Also curious if the untagged traffic sees the same issue or not.
My ESXi host ist a xeon e3 1230 (v1) / 32gb ecc / ssd-storage / 4x intel gigabit nic - so it should be powerfull enough. Also my internet is not as fast as yours :) (65/30)
Your setup seems very equivalent to mine (AP with ssid for iot, guest and "normal" users).What exactly do you mean with untagged traffic? The workstation pushing the traffic is already untagged on the switch. The freenas vm is now also on the switch via untagged - there is no difference in the transfer speeds.
Or do you mean I should attach the pfsense via untagged (with just two different subnets on the nics?) -
Talking about untagged to pfsense.
So you have igb1 you have no native network on this interface? Only vlans sitting on top of it?
When you setup an interface untagged would just be a native network on it. It might be say vlan 100 on your switch.. But its not tagged to pfsense kind of like a access port on your switch where your nas is connected… The nas is not aware of the vlan it is on the switch..
When you create a trunk port to pfsense you can set a vlan that is untagged or native.. And then all your other vlans would be tagged. And setup as vlans on pfsense that sit on the igb1 interface.
Same goes for igb2.. Just setup native networks on these - put them in whatever vlan you want on your switch. So when your pc talks to nas pfsense is not dealing with tagged traffic.
-
The issue is your VLAN's are terminated on your firewall, which is offering security at the expense of performance because all of your inter-vlan traffic is traversing and being filtered by PFsense.
For performance, create a transit network between PFsense and your switch, then create your vlans on your switch. This way inter-vlan routing is handled by the switch and it isn't saturating the links to your firewall.
I routinely see ~110 MB/sec transfers between my VLAN's.
-
That is a good point marvosa.. But the way I read the OP issue was that he was routing and firewalling between pfsense before.. He stated using the same hardware, then he added vlans to these interfaces..
But we should prob have him clarify this for sure..
So before when you were seeing higher speeds your pc and nas were on the different networks, just not vlan tagged So there were no vlans on igb1 and igb2..
So pc on say 192.168.0/24 and nas on 192.168.1/24 where you vlan it on the switch but you still routed/firewalled through pfsense… Is pfsense just didn't have any vlans on the interfaces? They were native untagged to pfsense?
Or you saying before both your pc and nas were on the same network 192.168.0/24 lets say and did not have to go through pfsense at all to move files..
-
first of all, thank you for all your replies!
Talking about untagged to pfsense.
So you have igb1 you have no native network on this interface? Only vlans sitting on top of it?
When you setup an interface untagged would just be a native network on it. It might be say vlan 100 on your switch.. But its not tagged to pfsense kind of like a access port on your switch where your nas is connected… The nas is not aware of the vlan it is on the switch..
When you create a trunk port to pfsense you can set a vlan that is untagged or native.. And then all your other vlans would be tagged. And setup as vlans on pfsense that sit on the igb1 interface.
Same goes for igb2.. Just setup native networks on these - put them in whatever vlan you want on your switch. So when your pc talks to nas pfsense is not dealing with tagged traffic.
Ok, now I get it. I will do that when I have a bit time on my hands. So on igb2 there is the native network .20.0 which connects to an untagged port on the switch which is, say vlan 20. On this network there is my nas.
On ig1 is the native network .25.0,a lso connected to an untagged port (vlan 100) on the switch. On this network is my client. On top of igb1 there are multiple vlans which are used for the other low-traffic stuff (iot, etc.)
In this setup routing would be done by pfsense but without vlan-tagging because this is all done by the switch.
Did I understand that correctly?The issue is your VLAN's are terminated on your firewall, which is offering security at the expense of performance because all of your inter-vlan traffic is traversing and being filtered by PFsense.
For performance, create a transit network between PFsense and your switch, then create your vlans on your switch. This way inter-vlan routing is handled by the switch and it isn't saturating the links to your firewall.
I routinely see ~110 MB/sec transfers between my VLAN's.
Thanks for your reply. How would I archive that transit network? From my understanding for that I would need an L3 switch? Or how could pfsense now about the vlans and issue the correct configuration (dhcp etc.) to the different vlans?
Could you explaint that a little further? because that seems like what I want :)
That is a good point marvosa.. But the way I read the OP issue was that he was routing and firewalling between pfsense before.. He stated using the same hardware, then he added vlans to these interfaces..
But we should prob have him clarify this for sure..
So before when you were seeing higher speeds your pc and nas were on the different networks, just not vlan tagged So there were no vlans on igb1 and igb2..
So pc on say 192.168.0/24 and nas on 192.168.1/24 where you vlan it on the switch but you still routed/firewalled through pfsense… Is pfsense just didn't have any vlans on the interfaces? They were native untagged to pfsense?
Or you saying before both your pc and nas were on the same network 192.168.0/24 lets say and did not have to go through pfsense at all to move files..
Sorry, I think I didn't express myself very well in the first post. I meant that I was using the same hardware (same switch, same routerboard with pfsense, same pc etc.) and was able to archive the 110mb/s. I then started using vlans and then noticed the speed penalties. NAS & my client were on the same subnet prior using vlans.
-
"without vlan-tagging because this is all done by the switch"
No it is not done by the switch…
"NAS & my client were on the same subnet prior using vlans"
Well yeah there is going to be performance hit there.. Duh!!! And seems your little box can not route at speed then.. You could try removing the tags and do it with out the tagging.. Might get you a bit more speed.. Tagging does had a tiny performance hit just on its own.
but if you want to route at speed your going to need a bigger box it seems.
-
"without vlan-tagging because this is all done by the switch"
No it is not done by the switch…
"NAS & my client were on the same subnet prior using vlans"
Well yeah there is going to be performance hit there.. Duh!!! And seems your little box can not route at speed then.. You could try removing the tags and do it with out the tagging.. Might get you a bit more speed.. Tagging does had a tiny performance hit just on its own.
but if you want to route at speed your going to need a bigger box it seems.
Yes I was aware that using the vlans would cause a performance hit but I could not believe that it was that much (~ 40 %). Thats the reason I started diggin into it.
I'm going to wait for the answer from marvosa and see if I can get that running with my current configuration. Otherwise I will virtualize a pfsense, setup the routing between the vlan interfaces there and use my physical box as gateway.
-
you can try it with untagged vlans and see if what kind of difference that makes if any.
