New pfSense Install, Upload sucks

jlw52761

So got the links switched over to the Broadcom NIC and had to pull the Intel NIC because I have to spoof that MAC address onto the NIC on the Broadcom I am using for WAN so that it even works. Also, did the tuning for the bce cards that is recommended at the link I posted above.

So, long story short, same issue. The R610 can get out, I can ping out from my internal server, but upload is pretty much dead.

I'm not sure what else to look at, but pretty sure it's not a driver problem or NIC issue. Also, the switch still shows 0 errors, but I moved everything to new cables and new interfaces just for giggles.

So yeah, not sure what to look at next...

stephenw10

Do you see the same thing both ways between two internal VLANs? Client sending to the server is always bad?

Does it actually 'drop to 0' over some seconds or is that just averaging? We have seen issues where you get basically 1s worth of iperf traffic and then nothing but it can appear as tapering.

Steve

jlw52761

@stephenw10 I will have to test that, my phone and the test server are on two different VLANs the R610 is routing for, so that should be an easy test.

Now the crazy thing is, I have a SSH NAT setup into the test server and it's working as expected.

As for the transfer, is exactly what is going on, using ping-ams1.online.net, here's what I am seeing:

[  5] local 10.27.200.67 port 59678 connected to 163.172.208.7 port 5209
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-5.00   sec   498 KBytes   815 Kbits/sec    3   1.41 KBytes
[  5]   5.00-10.01  sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes
[  5]  10.01-15.01  sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes

I do notice when, from the test server, I try to ping 8.8.8.8, I get the following results, with .253 being the VLAN interface for the subnet on the R610, so set as default gw on the test server:

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
From x.x.x.253 icmp_seq=1 Redirect Host(New nexthop: 0.0.0.0)
64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=19.2 ms

The redirect stands out to me, it's not occurring on the old firewall. I suspect that's the main issue, feels like asymmetric routing. I verified the upstream gateway is set to none for the interfaces that are not WAN.

jlw52761

@stephenw10 I think I may have found the issue, but can't change it remotely so will need to wait a few before I can confirm.

When I first set this up, there was no WAN connection so I had the LAN set to route out my old firewall. I told the interface to no longer use this gateway, but under Routing I see that the R610 was still set to use the other firewall as the default gateway instead of the WAN interface, so I think that may be the entire issue.

stephenw10

Yes, I agree. Feels exactly like asymmetric routing and that ICMP redirect pretty much confirms it.

jlw52761

@stephenw10 That's exactly what it was, what a noob mistake on my part!

But, getting sub-par speeds with this rig, and after applying the hardware tuning the WAN will not get a DHCP address, not sure what that's about.

On the old router, I'm getting consistently ~214/~112, whereas on the new rig I get ~95/~95, which is odd that it's so symetrical. This is using the bce for both WAN and internal VLANs. Thinking about going back to the Intel card, but first need to take the MAC spoofing off the bce WAN connection since it's the MAC from the Intel card I'm using. I'm sure that would throw things for a major loop!

Also, the way the R610 is configure, the Broadcom quad port is split between two controllers, and I have the WAN and LAN on different controllers thinking that we wouldn't want to saturate a single controller. Since we are not running at line speed, probably not something to worry about.

I do think it's interesting that I'm pretty much, outside of the 10Gb SFP+ that I'm essentially trying to run the equivalent of a Netgate 1541, on the max RAM size, but it's not too much different except what the motherboard is.

stephenw10

@jlw52761 said in New pfSense Install, Upload sucks:

on the new rig I get ~95/~95, which is odd that it's so symetrical

Yes, that 'feels' like something linked at 100M.

I would definitely go back to the Intel NIC. em(4) supported devices are about as tried and trusted as they come,

Steve

jlw52761

@stephenw10 One other thing, I'm pretty sure the R610 has that stupid TOE enabler key on the card, doesn't that cause issues with BSD?

stephenw10

TCP off-loading? It's probably unsupported but it wouldn't do anything here anyway since your are routing traffic and not terminating TCP connections on the firewall.
It might break iperf tests from pfSense itself of course.

Steve

jlw52761

@stephenw10 So, moved back to the Intel card and I am seeing the results I expect to see now.

So, the major issue was that I had a hard gateway defined for the LAN interface, which started asymmetric routing issues. While supported, the bce based card was the second issue.

I'm seeing ~230/~110 on a 250/125 circuit, which matches the old firewall almost perfectly, so I think I can put this one to bed. Now, to get BGP and all the other stuff moved to the new firewall, probably gonna setup HA between the old and new and use that to make the switch between the firewalls.

Thanks everyone for the input, it got me in the right direction and hopefully it will help others who come across it.