WAN auto-negotiate incorrectly



  • I have been running pfsense for almost 2 years /w FIOS in an ESXI VM with FreeNAS. Recently, my FreeNAS performance has not been good enough for my network in a VM. I decided to move FreeNAS onto its own machine for better performance. Since my only other VM is pfsense I decided to get rid of all the ESXI complications and run pfsense on the same hardware without ESXI.

    Server:
    Supermicro x8dtu-f
    32gigs of RAM ECC
    Intel Xeon L5530
    Intel 82576 Dual-Port Gigabit Ethernet Controller onboard

    So in the ESXI environment everything worked great my WAN came up as 100mbps. Running on the native hardware my WAN will only auto-negotiate to 10mbps. I called FIOS and they weren't much help they just broke my dhcp lease a few times and couldn't do anything else. I did a speed test and only got 9.8mbps up/down on a 50/50mbps link.

    I tested the Ethernet cable running to my ONT and it is good. If I plug in my FIOS router it is negotiates 100mbps just fine same WAN and my pfsense will negotiate to 1000mbps when plugged into my FIOS routers LAN. This is how I'm currently running I would like to get rid of the double NAT ASAP.

    I tried forcing 100mbps but i was getting a lot of packet loss and if I released my dhcp lease I couldn't even get an IP. I tried an HP NC364T PCIe which has Intel NICs and the onboard nics but was still getting 10mbps. I could try more NICs but that would be a bit of a pain.

    So I'm wondering if anyone has any suggestions? I'd hate to go to ESXI!



  • So I'm wondering if anyone has any suggestions? I'd hate to go to ESXI!

    For sure this will be a real pfSense bomb and my question will be first on a personal basis
    so what kind of hardware you where using for the NAS? Are there two Xeon L5330 on the
    board? Is it perhaps better to go with the Supermicro to realize the NAS and take the NAS
    box as a hardware basis for pfSense?

    Are you able to set up on both sides auto-negotiate to force 1000 Mbit/s on the LAN ports for the FIOS router
    and on the other side on the pfSense WAN port?

    As an alternative you could try out to set a small network switch between the both ports, the FIOS LAN port
    and the pfSense WAN port I mean and have a look what is going on there then.

    Another thing I mean is the following, is the WAN port of the pfSense getting a static IP address from
    the internal LAN of the FIOS router? Or is the FIOS router giving a DHCP address to the WAN port of the
    pfSense? I would recommend to set up a static IP address at the WAN port set up on the pfSense site
    and deactivating the DHCP server in the FIOS if this might be able to realize. Or give the pfSense WAN
    port a reserved IP address from the FIOS network, that is outside of its DHCP IP range.

    Are you able to set the FIOS router to the so called bridge mode? Please try it out if you are able to do so.
    And at last you could test it out with an older cross over cable to get the right speed without auto-negotiation.



  • When everything was working my configuration was as follows:

    ESXI Server:
    Supermicro x8dtu-f in a 1u chassis
    72gigs of RAM ECC
    2x Intel Xeon L5530
    Intel 82576 Dual-Port Gigabit Ethernet Controller onboard
    HP NC364T PCIe Quad-Port Gigabit Ethernet

    I had ESXI 5.5 with 2 VMs: FreeNAS and pfsense.

    In this configuration everything was working great, my network looked like the following:
    ONT Ethernet -> pfsense WAN the HP NIC or onboard NIC (designated as my WAN in pfsense)

    In this configuration I wasn't getting the performance I needed out of my FreeNAS so I bought a separate 2u chassis for FreeNAS. Now pfsense is running directly on the above server minus a few sticks of memory that I moved to the FreeNAS machine.

    With pfsense running directly on the above hardware my network config is as follows:
    ONT Ethernet -> Onboard NIC or HP NIC as pfsense WAN

    In this configuration my WAN is getting a DHCP public IP from FIOS and the NICs are negotiating to 10mbps. I tried forcing 100mbps and I was getting traffic loss and other issues.

    As a work-around I put my old FIOS router into the loop as follows:
    ONT Ethernet -> FIOS Router WAN
    FIOS LAN Port 1 -> pfsense WAN

    In this configuration my pfsense WAN is getting a DHCP IP from the FIOS router (192.168.1.2) and they negotiate to 1000mbps. I can force the pfsense wan speed to either 10mbps, 100mpbs, and 1000mbps with no issues. My FIOS Router WAN is negotiating to 100mbps as expected. I am double NATing my internal pfsense network is 192.168.30.0/24. This config works but I'm not a fan of the double NAT.

    As a test I will try the following:
    ONT Ethernet -> Netgear 110TP Port 1
    Netgear 110TP Port 2 -> pfsense WAN

    I would guess that my pfsense WAN will negotiate to 1000mbps and the mystery will be what Port 1 on the Netgear switch negotiates to with FIOS.

    I will also try the static IP address on the pfsense WAN. I'm not a big fan of this configuration because FIOS gives me a new IP about every 30 days. I called FIOS to see if there was anything they could do on their end but all they could seem to do was release my DHCP lease and try giving me a new IP.

    Thanks!



  • In this configuration my WAN is getting a DHCP public IP from FIOS and the NICs are negotiating to 10mbps. I tried forcing 100mbps and I was getting traffic loss and other issues.

    But this might be bringing you another two problems;

    • if in the pfSense configuration DHCP is activated and also on the FIOS router
    • if the FIOS router is serving the WAN port of the pfSense with a DHCP assigned IP address

    As a work-around I put my old FIOS router into the loop as follows:
    ONT Ethernet -> FIOS Router WAN
    FIOS LAN Port 1 -> pfsense WAN

    I am doing exactly this at my home network!

    • DHCP out at the first router
    • DHCP on at the pfSense
    • Static internal IP address from the frist router to the WAN port of the pfSense

    All is fine here.

    In this configuration my pfsense WAN is getting a DHCP IP from the FIOS router (192.168.1.2) and they negotiate to 1000mbps. I can force the pfsense wan speed to either 10mbps, 100mpbs, and 1000mbps with no issues. My FIOS Router WAN is negotiating to 100mbps as expected. I am double NATing my internal pfsense network is 192.168.30.0/24. This config works but I'm not a fan of the double NAT.

    Here is all fine and negotiating well, so where should be the problem?
    Change the WAN set up at the pfSense to static and enter an static IP address from the LAN of the FIOS.
    And all will be done and fine working for you.

    You will be only loosing something between 3% - 5% of the entire throughput by using the double NAT.
    Otherwise I would suggest to buy a pure modem, likes the Draytek Vigor 130 as an example. This would
    be the best then.



  • When I was running with this same hardware and same software (pfsense v2.2.6) in a VM I didn't need to have the FIOS provided router at all. I was able to my FIOS Internet line coming from the OTN directly into my pfsense WAN. It picked up DHCP from the Verizon FIOS server and negotiated 100mbps everything was working properly. When I run pfsense directly on that same hardware without ESXI in the mix the difference in vmnics, drivers, kernels, etc… whatever makes my NICs not play nice with FIOS. This requires me to add the extra hop (FIOS's provided Router) because without it I am capped out at 10mbps instead of 50mbps.

    Yes, having two routers as a solution works, not ideal but works. The other reason I would like to get rid of the Verizon provided router is because of port forwarding and VPNs. Troubleshooting issues is now a nightmare because I have to check to see if my Verizon router dropped my packets or my pfsense dropped my packets. It doubles my overhead dealing with two different routers.

    Thanks :)



  • Yes, having two routers as a solution works, not ideal but works.

    But if you want to get rid of the double NAT situation you are in now, you could try out to set a small switch
    between the ONT and the pfSense VM.

    The other reason I would like to get rid of the Verizon provided router is because of port forwarding and VPNs.

    Ok if VPN will be a part of this game I really understand that you will get rid of the double NAT configuration.

    Troubleshooting issues is now a nightmare because I have to check to see if my Verizon router dropped my packets or my pfsense dropped my packets. It doubles my overhead dealing with two different routers.

    I don´t know your budget, but a small Netgear GS108Tv2 will do this job with ease. Able to get for ~65 € or
    ~$67 here in Europe or in the USA. This might be running fine then like this;
    Internet –- ONT --- GS108Tv2 --- WAN Port pfSense

    The auto negotiation miss match should be gone then often. Perhaps you own a small GB LAN switch
    that will be able to placed between the ONT and the pfSense WAN port.



  • I put a switch infront of my FIOS router just to see if it would get an IP through the switch and it never got an IP. The switch negotiated 100mbps with the ONT and my FIOS router negotiated 1000mbps with the switch. I feel like I'm kind of stuck at this point.



  • Hm,, are you able to connect the ONT to a smaller switch and then only to the WNA port NIC from the pfsense box?
    Without the FIOS router I mean?



  • I put a Netgear GS110TP infront of my pfsense box as described above. I couldn't get an IP, everything negotiated properly, but no IP. I don't adding a switch will work, maybe a hub but I don't have access to one at the moments.

    Last night I installed sophos overtop of my pfsense on the same hardware and what do you know it negotiated properly and gave me an IP when plugged directly into the ONT. So I'm kind of sad that that worked so I popped in a different drive installed the latest pfsense again from scratch and it still came back with 10mbps :-/

    This is mind boggling…

    NICs I've tried:
    HP NC364T PCI Express Quad-Port Gigabit Server Adapter
    Supermicro AOC-SG-I4
    Supermicro X8DTU-F Onboard NICs

    I guess I'll use Sophos for now until pfsense releases v2.3? My family can't take anymore router downtime.

    Thanks for the suggestions!


Log in to reply