New esxi 5 build with pfsense help with NICS

esxiNOOB

new esxi 5 build with pfsense help with NICS

hi community,
Thanks for all the post so far and the great information. Although, I'm unable to find an answer to my question. I have 5 nics on my server. 2 onboard, 2 intel cards, and one ipmi.

i have three vswitchs set up.

1 off the board (general use), 1 intel (for wan), and 1 intel (for lan).

instead of taking a screen shot i'll just use this guy's image below.
notice how the lan only say 100mb? well mine is acting the same way. i get 1gbit to the wan, and only 100mb to the lan even though they are identical brand and model cards. strangely inside pfsense webconfig shows both interfaces as 1000mb uplink.
All my hardware indicators show the connection is truly in a 100mb state. the managed switch and everything else is set to auto negotiate.
can someone PLEASE help?!

extra info:
driver set to e1000 on all nics
32 bit version of pfsense 2.0
no vmware tools installed yet.
esxi v 5.0.0

Please advise if i should scrap this VM and go with a different config.

johnpoz

Do you have a bad cable that is not capable of gig? Did you set the interface speed with esxi? What is the nic connected too?

I run pfsense on esxi, and I get 1000 on both interfaces.

Here is info on how to set it
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004089

esxiNOOB

thanks for the reply.

the nic goes to a netgear prosafe gigabit smart switch.

as i said earlier everything is set to auto negotiate.

so according to the link you sent me (picture taken below). I have it set up in the recommended setting, since both devices are gigabit worthy and in auto negotiate state. It's too bad the netgear won't allow me to set the port to gbit. You can force all settings but gigabit, for that it has to be auto.

but what the hell, I'll go against the recommendations and force esxi 5 to put the card into gigabit. I'll report back when i get a chance.

esxiNOOB

well i manually changed the settings to 1000mb full duplex and all the indicators are showing it's running in gigabit now.

still stumped why the auto wouldn't work, and i'm having to do this non recommended setting.

biggsy

strangely inside pfsense webconfig shows both interfaces as 1000mb uplink

That's because pfSense is only seeing the virtual NIC, which is presenting as a Gb/s vNIC.

notice how the lan only say 100mb?

That actually looks like the guy's WAN, given that he's got it labeled PPPOE. Probably his modem only does 100 Mb/s on its LAN interface.

driver set to e1000 on all nics

Again, this is talking to a vNIC not a real one.

Sorry, all the above doesn't explain why the physical NIC doesn't auto-negotiate to Gb speed. I would be with johnpoz on the bad cable idea. I've had brand new (but cheap) supposedly Cat 6 cables only run up to 100Mb/s before.

esxiNOOB

thanks for your knowledge on the subject biggsy.

everything you said makes sense, except "That's because pfSense is only seeing the virtual NIC, which is presenting as a Gb/s vNIC." How can it be presenting as a Gb/s vNIC when on the config page of the main host it shows 100mb? is that because it's capable of Gb speeds?

I swapped the cables and nics around.

before:
red cable: modem -> intel nic 1 = 1000mb/s
green cable: intel nic 2 -> netgear smart switch = 100mb/s

after:
green cable: modem -> intel nic 1 = 1000 mb/s
red cable: intel nic 2 -> netgear smart switch = 100mb/s

switching the nics produced the same results in all test. (switched the config in pfsense em0/em1)

ugh if i put a vm of win 7 on that nic. it goes to 1000 mb/s.

it almost seems like pfsense is too slow in the auto negotiation. the 100mb light will blink on, then off for a sec, then back on with 100mb lit.
with win 7 it blinks on with 1000 mb connection and never turns off.

i haven't noticed any package losses. is there any way to test this connection i have going from switch to pfsense lan?

at first i thought i could put the management network on this nic and test it, but when i do that it goes to 1000mb/s

:-\

forcing the nic to 1000 mb/s does produce the desired results visually. I just need a way to test it.

johnpoz

"it almost seems like pfsense is too slow in the auto negotiation."

What?


em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:00:00:01
        inet6 fe80::250:56ff:fe00:1%em1 prefixlen 64 scopeid 0x2
        inet 24.13.xxx.xxx netmask 0xfffff800 broadcast 255.255.255.255
        nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active</full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast>

As you can see from above and attached pfsense is running 1000 fully autoselected.

pfsensegig.png_thumb

matguy

The way VMWare virtualizes the networking the virtual NIC that pfSense sees ignores the speed your "real" NIC on your ESX(i) box is connected as.

In practice, internally in ESX(i) the "real" NIC is connected to a virtual switch. The virtual NIC that pfSense sees is connected to that virtual switch. Just like in a real switch, even though the uplink to the outside world may be only 100Mb, another connection to an end device may be Gigabit (or more) and the connection negotiation won't be limited down based on another port's link speed, but the traffic would still be limited.

This explanation doesn't fix your issue, but tries to explain why pfSense can still "link" at Gigabit while your "real" NIC is only linking at 100Mb.

I would still look at the cable, a faulty cable can still link at 100Mb and work fine a that speed, but not be able to auto negotiate for Gigabit if one (or multiple) of the non-transmit/receive pins/wires are bad. At least test a replacement cable.

esxiNOOB

i get it now matguy. thanks for the explanation!!

esxiNOOB

i'll try yet another cable.. and even a trendnet unmanaged gigabit switch.
i'm betting the fault is in the netgear prosafe gs 108t

matguy

@esxiNOOB:

i'll try yet another cable.. and even a trendnet unmanaged gigabit switch.
i'm betting the fault is in the netgear prosafe gs 108t

Oh, I missed the part where you were going to swap the cables/nics around. But, trying a different switch is a good troubleshooting step. I've used the small Netgear switches with ESX and ESXi with good success, although not that model personally. Certainly try the Trendnet, it'll be interesting if something in your chain there is thwarting the GigE auto-negotiation.

biggsy

Maybe a silly question but are you sure all the cables are straight-through (not cross-over) cables?

Maybe you just used "red", "green" an "blue" to identify the cables for your post but (and sorry if you already know this) a lot of places use different colored cables for different things. For example, red for cross-over is fairly common.

If you inherited these cables from somewhere or just chose different colors when you bought them, it might be worth checking.

I have seen a Netgear switch accept a cross-over cable where a straight-through should have been used but it would only negotiate up to 100Mb/s. There might even be something in the standards that says that's what should happen.

pf123user

@biggsy:

notice how the lan only say 100mb?

That actually looks like the guy's WAN, given that he's got it labeled PPPOE. Probably his modem only does 100 Mb/s on its LAN interface.

I'd agree with biggsy. What's on the other end of that NIC displaying only 100mb? The NIC and the cable are probably fine. To test it just take a laptop or PC that is gigabit and plug it into whatever device is on the other end of that link. If the other machines also display 100mb then I'd bet biggsy is correct in guessing that your modem or whatever it is at the other end is only capable of 100mb.

OR

Do you have a physical gigabit router or switch? Plug that into the "vmnic1" interface and see if it can resolve to gigabit from a known good gigabit device on the other end.

matguy

@pf2.0nyc:

@biggsy:

notice how the lan only say 100mb?

That actually looks like the guy's WAN, given that he's got it labeled PPPOE. Probably his modem only does 100 Mb/s on its LAN interface.

I'd agree with biggsy. What's on the other end of that NIC displaying only 100mb? The NIC and the cable are probably fine. To test it just take a laptop or PC that is gigabit and plug it into whatever device is on the other end of that link. If the other machines also display 100mb then I'd bet biggsy is correct in guessing that your modem or whatever it is at the other end is only capable of 100mb.

OR

Do you have a physical gigabit router or switch? Plug that into the "vmnic1" interface and see if it can resolve to gigabit from a known good gigabit device on the other end.

The picture was from someone else's setup to just show basically what he's seeing. His is actually showing 100Mb to a Netgear Prosafe GS108T. Gigabit to his WAN is fine.

esxiNOOB

ya the wan is connected to a Motorola surfboard modem, docsis 3.0… forget the model... but showing a blue light on connection which = gbit.

it is a valid question about the cables and the name i chose for this account "..noob" probably wasn't the best.. but all my cables are patch, aka straight through. i have only one custom made crossover that i hardly use and it's only cat 5.

my problem has been solved with a longer cable and using the 5 port trendnet gbit switch.

although i think i may use the unrecommended settings to connect straight to the netgear.

next question without opening a new thread maybe you couple guys that have kindly helped me can tell me what this hit on the firewall is all about:

WAN 10.20.64.1:67 destination 255.255.255.255:68 UDP

i don't get the 255.255.255.255. it happens almost every min. what the hell is going on!

in the end i think my problem was not esxi related but more netgear switch.

heper

this is related to DHCP traffic … the 255.255.255.255 is the broadcast range it is sending afaik.

it is odd that this happens every minute on your WAN interface. (should only happen when your dhcp lease expires and renews)
http://www.linklogger.com/UDP67_68.htm

biggsy

I'm surprised it only happens every minute. This is DHCP traffic, as heper has said. It's cable modems obtaining or renewing leases.

If I leave on the Log packets blocked by the default rule I see the DHCP requests and replies for every cable modem on the same segment of cable.