CIFS transfer performance slow on APU2 board when using VLANs
-
hw.igb.num_queues="1"Ok you can also try out to set 2 or 4 queues for that, nothing matter it can scale up or down
so you have to play around with it. There are also some other things that can be done right.- high up the mbuf size to 125000, 250000, 500000 or 1000000
- Enable PowerD (hi adaptive)
Please read here about that theme: Tuning and troubleshooting network cards
You may be able to set the num.queues higher and the mbuf size lower and for sure also vice versa
it´s a try out or fine tuning so you will be able to get more or less the ideal setting for your set up and
hardware matching all your criteria. -
"I think I will install pfsense at my ESXi as vm just to see if more speed is possible."
How powerful is your esxi host? I had to move my pfsense off my esxi host when I got new inet speed because it was not capable of routing at that speed. But my esxi host is OLD.. It could only manage at best a bit over 200mbps.. When you also did natting only about 120mbps to the internet.. Which was not a big deal when internet was 80.. But when it went to 500 was a big issue.
The server and and my workstation that I move large amounts of data to where put on the same network because of this reason. I did not have any traffic between vlans where the speed reduction was a problem. Wifi clients to server, etc. iot devices don't use much of anything, etc..
But if you have a beefier esxi host then you should see improvement I would think.. Its is a valid test to be sure as a possible solution to your problem. Also curious if the untagged traffic sees the same issue or not.
-
@BlueKobold:
hw.igb.num_queues="1"Ok you can also try out to set 2 or 4 queues for that, nothing matter it can scale up or down
so you have to play around with it. There are also some other things that can be done right.- high up the mbuf size to 125000, 250000, 500000 or 1000000
- Enable PowerD (hi adaptive)
Please read here about that theme: Tuning and troubleshooting network cards
Done & done: I did not notice a difference between the queues and with th mbuf size. PowerD made al little difference (~1,5 MB/s)
I read the article and I will try to play with the values. But to be honest I bought the APU board to not have to play around that much. I thought it would be sufficient for my home network.
"I think I will install pfsense at my ESXi as vm just to see if more speed is possible."
How powerful is your esxi host? I had to move my pfsense off my esxi host when I got new inet speed because it was not capable of routing at that speed. But my esxi host is OLD.. It could only manage at best a bit over 200mbps.. When you also did natting only about 120mbps to the internet.. Which was not a big deal when internet was 80.. But when it went to 500 was a big issue.
The server and and my workstation that I move large amounts of data to where put on the same network because of this reason. I did not have any traffic between vlans where the speed reduction was a problem. Wifi clients to server, etc. iot devices don't use much of anything, etc..
But if you have a beefier esxi host then you should see improvement I would think.. Its is a valid test to be sure as a possible solution to your problem. Also curious if the untagged traffic sees the same issue or not.
My ESXi host ist a xeon e3 1230 (v1) / 32gb ecc / ssd-storage / 4x intel gigabit nic - so it should be powerfull enough. Also my internet is not as fast as yours :) (65/30)
Your setup seems very equivalent to mine (AP with ssid for iot, guest and "normal" users).What exactly do you mean with untagged traffic? The workstation pushing the traffic is already untagged on the switch. The freenas vm is now also on the switch via untagged - there is no difference in the transfer speeds.
Or do you mean I should attach the pfsense via untagged (with just two different subnets on the nics?) -
Talking about untagged to pfsense.
So you have igb1 you have no native network on this interface? Only vlans sitting on top of it?
When you setup an interface untagged would just be a native network on it. It might be say vlan 100 on your switch.. But its not tagged to pfsense kind of like a access port on your switch where your nas is connected… The nas is not aware of the vlan it is on the switch..
When you create a trunk port to pfsense you can set a vlan that is untagged or native.. And then all your other vlans would be tagged. And setup as vlans on pfsense that sit on the igb1 interface.
Same goes for igb2.. Just setup native networks on these - put them in whatever vlan you want on your switch. So when your pc talks to nas pfsense is not dealing with tagged traffic.
-
The issue is your VLAN's are terminated on your firewall, which is offering security at the expense of performance because all of your inter-vlan traffic is traversing and being filtered by PFsense.
For performance, create a transit network between PFsense and your switch, then create your vlans on your switch. This way inter-vlan routing is handled by the switch and it isn't saturating the links to your firewall.
I routinely see ~110 MB/sec transfers between my VLAN's.
-
That is a good point marvosa.. But the way I read the OP issue was that he was routing and firewalling between pfsense before.. He stated using the same hardware, then he added vlans to these interfaces..
But we should prob have him clarify this for sure..
So before when you were seeing higher speeds your pc and nas were on the different networks, just not vlan tagged So there were no vlans on igb1 and igb2..
So pc on say 192.168.0/24 and nas on 192.168.1/24 where you vlan it on the switch but you still routed/firewalled through pfsense… Is pfsense just didn't have any vlans on the interfaces? They were native untagged to pfsense?
Or you saying before both your pc and nas were on the same network 192.168.0/24 lets say and did not have to go through pfsense at all to move files..
-
first of all, thank you for all your replies!
Talking about untagged to pfsense.
So you have igb1 you have no native network on this interface? Only vlans sitting on top of it?
When you setup an interface untagged would just be a native network on it. It might be say vlan 100 on your switch.. But its not tagged to pfsense kind of like a access port on your switch where your nas is connected… The nas is not aware of the vlan it is on the switch..
When you create a trunk port to pfsense you can set a vlan that is untagged or native.. And then all your other vlans would be tagged. And setup as vlans on pfsense that sit on the igb1 interface.
Same goes for igb2.. Just setup native networks on these - put them in whatever vlan you want on your switch. So when your pc talks to nas pfsense is not dealing with tagged traffic.
Ok, now I get it. I will do that when I have a bit time on my hands. So on igb2 there is the native network .20.0 which connects to an untagged port on the switch which is, say vlan 20. On this network there is my nas.
On ig1 is the native network .25.0,a lso connected to an untagged port (vlan 100) on the switch. On this network is my client. On top of igb1 there are multiple vlans which are used for the other low-traffic stuff (iot, etc.)
In this setup routing would be done by pfsense but without vlan-tagging because this is all done by the switch.
Did I understand that correctly?The issue is your VLAN's are terminated on your firewall, which is offering security at the expense of performance because all of your inter-vlan traffic is traversing and being filtered by PFsense.
For performance, create a transit network between PFsense and your switch, then create your vlans on your switch. This way inter-vlan routing is handled by the switch and it isn't saturating the links to your firewall.
I routinely see ~110 MB/sec transfers between my VLAN's.
Thanks for your reply. How would I archive that transit network? From my understanding for that I would need an L3 switch? Or how could pfsense now about the vlans and issue the correct configuration (dhcp etc.) to the different vlans?
Could you explaint that a little further? because that seems like what I want :)
That is a good point marvosa.. But the way I read the OP issue was that he was routing and firewalling between pfsense before.. He stated using the same hardware, then he added vlans to these interfaces..
But we should prob have him clarify this for sure..
So before when you were seeing higher speeds your pc and nas were on the different networks, just not vlan tagged So there were no vlans on igb1 and igb2..
So pc on say 192.168.0/24 and nas on 192.168.1/24 where you vlan it on the switch but you still routed/firewalled through pfsense… Is pfsense just didn't have any vlans on the interfaces? They were native untagged to pfsense?
Or you saying before both your pc and nas were on the same network 192.168.0/24 lets say and did not have to go through pfsense at all to move files..
Sorry, I think I didn't express myself very well in the first post. I meant that I was using the same hardware (same switch, same routerboard with pfsense, same pc etc.) and was able to archive the 110mb/s. I then started using vlans and then noticed the speed penalties. NAS & my client were on the same subnet prior using vlans.
-
"without vlan-tagging because this is all done by the switch"
No it is not done by the switch…
"NAS & my client were on the same subnet prior using vlans"
Well yeah there is going to be performance hit there.. Duh!!! And seems your little box can not route at speed then.. You could try removing the tags and do it with out the tagging.. Might get you a bit more speed.. Tagging does had a tiny performance hit just on its own.
but if you want to route at speed your going to need a bigger box it seems.
-
"without vlan-tagging because this is all done by the switch"
No it is not done by the switch…
"NAS & my client were on the same subnet prior using vlans"
Well yeah there is going to be performance hit there.. Duh!!! And seems your little box can not route at speed then.. You could try removing the tags and do it with out the tagging.. Might get you a bit more speed.. Tagging does had a tiny performance hit just on its own.
but if you want to route at speed your going to need a bigger box it seems.
Yes I was aware that using the vlans would cause a performance hit but I could not believe that it was that much (~ 40 %). Thats the reason I started diggin into it.
I'm going to wait for the answer from marvosa and see if I can get that running with my current configuration. Otherwise I will virtualize a pfsense, setup the routing between the vlan interfaces there and use my physical box as gateway.
-
you can try it with untagged vlans and see if what kind of difference that makes if any.
