New pfSense Install, Upload sucks
-
I am looking to stand up a "new" firewall, that is repurposed old hardware. I have a Dell R610 laying around, so decided to use it to replace my current Protectli FW-2 device that I will use elsewhere.
Now I know that an R610 is "overkill", but my poor FW-2 is getting crushed running pfBlocker, FRR, IPSec tunnels, OpenVPN, softflow, and HAProxy, so a little overkill is ok in my mind.
The issue I'm having is the download through the R610 is great, the upload sucks, starts at ~1Mbps then drops to 0. The FW-2 is not having this issue, and for reference I have a 250/125 circuit, and this is home, not work, so using an older platform is ok with me.
I am not using the on-board Broadcom cards, but rather stuck in a dual Intel card, because everyone states that "Ye shall use Intel with pfSense!". Well, the Intel card may be the issue.
I have gone through the docs at
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#intel-igb-4-and-em-4-cards for the em cards, which the Intel comes up as. I have even disabled the onboard NICs with no avail, and all the C-States are disabled.Here's the specs of the R610 that I have:
Dual Quad-Core Xeon E5520
32GB RAM
Dual 146GB 15k SAS - RAID 1
Intel 82571EB/82571GB Gigabit Ethernet Controller D0/D1
Broadcom NetXtreme II BCM5709 Gigabit EthernetAny thoughts?
-
As a test I would use the onboard NIC to replace the WAN. If the problem persists replace the LAN. With careful choices you can eliminate or point to a port as the cause.
-
@andyrh definitely next steps is to do exactly that, just figured I'd post and see if anything else came up during the day. Won't be able to swap out the connections until tonight.
-
Something throttling that much is usually a bad link somewhere. Check the NIC are actually linked at 1G-FD and not showing errors In Status > Interfaces.
Steve
-
@stephenw10 looking at the Cisco switch, no interface errors on either the WAN or LAN link. Yes, my WAN is a VLAN across my switches.
So if I do an iperf3 from the R610 out to the Internet, great numbers. If I do a iperf3 from an internal box to the LAN interface of the R610, again, great numbers. If I do a speedtest from any client through the R610, crap upload.
I will switch the cables around, but this really feels like something in pf itself when routing/forwarding between the two zones. Now, for completness, the server I tested with is in the LAN zone, while my phone I am testing with is in the DHCP zone, different VLAN/subnet, so there's some routing and rules, they both exhibit the same upload issue. Both are also set as VLAN subinterfaces on the same NIC. I will move them over to one of the Broadcom NICs this evening and see where we stand. -
So got the links switched over to the Broadcom NIC and had to pull the Intel NIC because I have to spoof that MAC address onto the NIC on the Broadcom I am using for WAN so that it even works. Also, did the tuning for the bce cards that is recommended at the link I posted above.
So, long story short, same issue. The R610 can get out, I can ping out from my internal server, but upload is pretty much dead.
I'm not sure what else to look at, but pretty sure it's not a driver problem or NIC issue. Also, the switch still shows 0 errors, but I moved everything to new cables and new interfaces just for giggles.
So yeah, not sure what to look at next...
-
Do you see the same thing both ways between two internal VLANs? Client sending to the server is always bad?
Does it actually 'drop to 0' over some seconds or is that just averaging? We have seen issues where you get basically 1s worth of iperf traffic and then nothing but it can appear as tapering.
Steve
-
@stephenw10 I will have to test that, my phone and the test server are on two different VLANs the R610 is routing for, so that should be an easy test.
Now the crazy thing is, I have a SSH NAT setup into the test server and it's working as expected.
As for the transfer, is exactly what is going on, using ping-ams1.online.net, here's what I am seeing:
[ 5] local 10.27.200.67 port 59678 connected to 163.172.208.7 port 5209 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-5.00 sec 498 KBytes 815 Kbits/sec 3 1.41 KBytes [ 5] 5.00-10.01 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytes [ 5] 10.01-15.01 sec 0.00 Bytes 0.00 bits/sec 1 1.41 KBytesI do notice when, from the test server, I try to ping 8.8.8.8, I get the following results, with .253 being the VLAN interface for the subnet on the R610, so set as default gw on the test server:
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. From x.x.x.253 icmp_seq=1 Redirect Host(New nexthop: 0.0.0.0) 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=19.2 msThe redirect stands out to me, it's not occurring on the old firewall. I suspect that's the main issue, feels like asymmetric routing. I verified the upstream gateway is set to none for the interfaces that are not WAN.
-
@stephenw10 I think I may have found the issue, but can't change it remotely so will need to wait a few before I can confirm.
When I first set this up, there was no WAN connection so I had the LAN set to route out my old firewall. I told the interface to no longer use this gateway, but under Routing I see that the R610 was still set to use the other firewall as the default gateway instead of the WAN interface, so I think that may be the entire issue.
-
Yes, I agree. Feels exactly like asymmetric routing and that ICMP redirect pretty much confirms it.
-
@stephenw10 That's exactly what it was, what a noob mistake on my part!
But, getting sub-par speeds with this rig, and after applying the hardware tuning the WAN will not get a DHCP address, not sure what that's about.
On the old router, I'm getting consistently ~214/~112, whereas on the new rig I get ~95/~95, which is odd that it's so symetrical. This is using the bce for both WAN and internal VLANs. Thinking about going back to the Intel card, but first need to take the MAC spoofing off the bce WAN connection since it's the MAC from the Intel card I'm using. I'm sure that would throw things for a major loop!
Also, the way the R610 is configure, the Broadcom quad port is split between two controllers, and I have the WAN and LAN on different controllers thinking that we wouldn't want to saturate a single controller. Since we are not running at line speed, probably not something to worry about.
I do think it's interesting that I'm pretty much, outside of the 10Gb SFP+ that I'm essentially trying to run the equivalent of a Netgate 1541, on the max RAM size, but it's not too much different except what the motherboard is.
-
@jlw52761 said in New pfSense Install, Upload sucks:
on the new rig I get ~95/~95, which is odd that it's so symetrical
Yes, that 'feels' like something linked at 100M.
I would definitely go back to the Intel NIC. em(4) supported devices are about as tried and trusted as they come,
Steve
-
@stephenw10 One other thing, I'm pretty sure the R610 has that stupid TOE enabler key on the card, doesn't that cause issues with BSD?
-
TCP off-loading? It's probably unsupported but it wouldn't do anything here anyway since your are routing traffic and not terminating TCP connections on the firewall.
It might break iperf tests from pfSense itself of course.Steve
-
@stephenw10 So, moved back to the Intel card and I am seeing the results I expect to see now.
So, the major issue was that I had a hard gateway defined for the LAN interface, which started asymmetric routing issues. While supported, the bce based card was the second issue.
I'm seeing ~230/~110 on a 250/125 circuit, which matches the old firewall almost perfectly, so I think I can put this one to bed. Now, to get BGP and all the other stuff moved to the new firewall, probably gonna setup HA between the old and new and use that to make the switch between the firewalls.
Thanks everyone for the input, it got me in the right direction and hopefully it will help others who come across it.
