Slow routing between local subnets. How to further troubleshoot?
-
I have two LAN interfaces, connected at Gigabit full duplex. If I to a test SMB transfer on the same subnet, for each separate subnet, I get a solid 100-110MB/s. Through the PFSense across the subnets, I get 10MB/s, but CPU utilization on the PFSense is only at around 20%.
It has Intel 82583V NICs, which use the E1000 driver (em), rather than the igb driver. Surely this should not matter that much, yes?
What is the next step in terms of getting to the bottom of this?
-
describe your setup in more detail. While you prob wont get full gig routing/firewalling traffic through pfsense depending on its hardware. You should be seeing way more than 10mbps.. I run pfsense on old hp n40l as a vm, and I can get 200ish mbps between lan segments.
Is your other lan segment a vlan on the same physical interface? Are you completely physical networks where you have 2 different dumb switches or is there vlans on the smart switch that both networks connect too?
There will always be a performance hit routing/firewalling traffic vs just on the same layer 2 on the same switch, etc. But yes 10 sure seems like something is not right. With such a speed drop I would look to even a possible duplex mismatch somewhere.
-
Sorry, I thought I'd keep the question simple, but yes, I do use VLANs BUT one NIC has only that VLAN tagged on that port and the other NIC has a couple of VLANs also tagged on that port, however the other VLANs are not in use at all at the time of testing.
The next step I was going to try was to plug the test computers directly into each NIC and test again. It's just tough in a production environment, but I do have spare hardware, so I'll restore my config onto that for testing.
Any other tips for troubleshooting?
I've also researched without much success into the real-world performance hit of using multiple tagged VLANs on a port. Any insight into that?
-
" hit of using multiple tagged VLANs on a por"
Yeah your hairpinning when they talk to each other and yeah its going to be a hit.. If you have 2 vlans on 1 physical port and client and they are talking your /2 the possible speed now don't you. All the clients talking are sharing that phy port speed.. And if the vlans are talking to each other then yeah your doing a lot of hairpinning and performance is going to take a hit.
You still have not described your test scenario where you seeing 10mbps.. Are they vlans on the same port? How does the switch connectivity look.. Is there some bottleneck in a uplink somewhere? etc. etc..
-
Sorry about the delay. One subnet is a native port. The other port has just two VLANs configured with one doing absolutely nothing at the time. Just a copy from a subnet on an untagged port on one NIC to a tagged VLAN on the other port. It is all through the Dlink DGS-3100 switch, but I doubt it's an issue with the switch because I can max out the gigabit no worries using that switch to copy to same subnet.
-
So how is the switch configured?
What I can tell you for sure, is 10MBps seems really slow for just being a hardware hit in pfsenes. My pfsense is vm on old hp 40L hardware, me doing speed tests between network segments I see better than that.
10MB would be about what the limit is for a 100mbps connection. You sure you don't have a 100mbps connection somewhere in the setup? Your going to have 4 ports that could have this - maybe an uplink between switches for your other vlan/network?
Here is my test setup.. see attached.
Lan is em1 in my pfsense, goes through a vswitch that is tied to my sg300 and this port is access with my native untagged vlan. Then I have a em2 in my pfsense vm that native untagged is my wlan network and then on top of that are a bunch of vlans. So this connection to different physical esxi nic than the lan nic is trunked all the way to pfsense vm nic, ie it carries tags.
Then I have my desktop (192.168.9.100) that is connect to same sg300 cisco switch to a port that is native untagged lan network. And then I have a laptop (192.168.2.216) plugged in to another switch port that is in my wlan pvid untagged.
If I do a simple file copy from my pc to the laptop and see over 19..
> robocopy c:\test \\192.168.2.216\test push.zip -------------------------------------------------------------------------- ROBOCOPY :: Robust File Copy for Windows -------------------------------------------------------------------------- Started : Saturday, August 13, 2016 6:17:05 AM Source : c:\test\ Dest : \\192.168.2.216\test\ Files : push.zip Options : /DCOPY:DA /COPY:DAT /R:1000000 /W:30 -------------------------------------------------------------------------- 1 c:\test\ 100% New File 3.6 g push.zip -------------------------------------------------------------------------- Total Copied Skipped Mismatch FAILED Extras Dirs : 1 0 0 0 0 0 Files : 1 1 0 0 0 0 Bytes : 3.601 g 3.601 g 0 0 0 0 Times : 0:03:18 0:03:18 0:00:00 0:00:00 Speed : 19446578 Bytes/sec. Speed : 1112.742 MegaBytes/min. Ended : Saturday, August 13, 2016 6:20:24 AMThat is with a really LARGE file.. Have you tested both directions? What OSes are in play are you using smb, smb2, smb3? There could be something just going on in your file copy method that is slowing you down.. What does an Iperf test show?
what does a simple iperf test show
[ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 180 MBytes 151 Mbits/sec sender [ 4] 0.00-10.00 sec 179 MBytes 150 Mbits/sec receiverThat is to same laptop from my pc.. If I put them on the same lan sure I see much higher..
[ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 1.09 GBytes 935 Mbits/sec sender [ 4] 0.00-10.00 sec 1.09 GBytes 935 Mbits/sec receiver> robocopy c:\test \\192.168.9.239\test push.zip ------------------------------------------------------------------------------- ROBOCOPY :: Robust File Copy for Windows ------------------------------------------------------------------------------- Started : Saturday, August 13, 2016 6:35:46 AM Source : c:\test\ Dest : \\192.168.9.239\test\ Files : push.zip Options : /DCOPY:DA /COPY:DAT /R:1000000 /W:30 ------------------------------------------------------------------------------ 1 c:\test\ 100% New File 3.6 g push.zip ------------------------------------------------------------------------------ Total Copied Skipped Mismatch FAILED Extras Dirs : 1 0 0 0 0 0 Files : 1 1 0 0 0 0 Bytes : 3.601 g 3.601 g 0 0 0 0 Times : 0:00:34 0:00:34 0:00:00 0:00:00 Speed : 112137010 Bytes/sec. Speed : 6416.531 MegaBytes/min. Ended : Saturday, August 13, 2016 6:36:21 AMSo while yeah unless your pfsense hardware is capable of routing at your wire speed your not going to see the performance as switched network only.. I find it unlikely that with your hardware the performance hit would be as hard as your seeing. Mine is on vm and see better than yours. New esxi hardware is on my wish list and coming soon. I just love running my pfsense on vm, but yeah its going to be a hit compared to hardware. I might switch to hardware here soon though as saw some posts about psfense running on minnow board, etc.
