9Mbit Limit - ESXi 5 - vSwitch/PfSense 2.0.1

dLockers

Hi guys,

Started a new thread as the last one was regarding PPPoE specifically.

I have phoned my ISP. I am syncing at 40Mbit definitely.
I have a vswitch for WAN with a dedicated NIC for it - there is a cable going from the modem to this. It works great and syncs at 40Mbit when I plug directly in to it with a dedicated PfSense box.
I have a vswitch for LAN, which works great for everything else.
I am using VMXNET2 Enhanced adapters.

However I hit a brick wall at 9Mbps WAN/internet traffic.

Images:

Second:

Help? Please!

heper

i'm running a couple of pfsenses on esxi4-5 system, they hit the 1Gbit interface limit without issues … it must be said these are on low end 1-socket xeons from early last year.
note: this is using the e1000 vm adapter

have you installing a different OS in a vm ? does that have similar issues ?
If yes then its probably hardware or driver issues on hypervisor

if no ... then maybe hitting the cpu limit on a bsd vm ?

dLockers

I have tried a fresh VM, 32bit and 64bit flavours.

I have tried with e1000 and VMXNET2 (Enhanced).

Both times I get the problem. My vswitch config:

It also does it when I pass through the NIC completely, i.e. no VMXNET driver at all.

biggsy

I'd have to agree with heper on this but here are some other possibilities and suggestions:
What hardware platform are you using for ESXi?
Are you using Intel NICs?
Stick with E1000 it won't do you any harm if your WAN speed is 40Mb/s.
Have you changed the cable between the modem and ESXi?
Did you lock the speed of the WAN NIC to 100Mb/s or is it autonegotiate? A mismatch between that NIC and the modem could be slowing you right down to 10Mb/s.
Anything strange in the ESXi or pfSense logs?

Supermule

Check your trafficshaping setup on your vswitch….

dLockers

@biggsy:

I'd have to agree with heper on this but here are some other possibilities and suggestions:
What hardware platform are you using for ESXi?
Are you using Intel NICs?
Stick with E1000 it won't do you any harm if your WAN speed is 40Mb/s.
Have you changed the cable between the modem and ESXi?
Did you lock the speed of the WAN NIC to 100Mb/s or is it autonegotiate? A mismatch between that NIC and the modem could be slowing you right down to 10Mb/s.
Anything strange in the ESXi or pfSense logs?

Hardware is:
Intel DQ67SW (with onboard Intel 8275LM NIC)
Intel Xeon 1230
5TB passthrough on onboard controller (Intel)
1.25TB datastores for VM's on SIL3114
2x Intel 10/100/1000CT adapter (8279L or something - 1 for LAN, 1 for WAN)

One vSwitch is PPPoE only with a straight cable to the modem.
If I plug a dedicated MaxTerm 8300B terminal PC into it, I sync fine at 40Mbit.

One vSwitch is for management traffic and LAN traffic.

E1000 and VNXNET2 Enhanced both have the same performance (identical).

I have locked the vSwitch configuration to 100Mbit FULL - I'm not sure if that controls the NIC, but I assume so.

Traffic shaping I have configured nothing, but likewise it can't be this because I get the same results even if I passthrough the adapter using vt-d.

Supermule

Change it to autonegotiation instead on the PPPoE NIC…

dLockers

@Supermule:

Change it to autonegotiation instead on the PPPoE NIC…

It was previously on auto negotiation, I set it to 100MB FULL just in case.

There are no collisions or anything to indicate mismatch.

Pass through gives me the same problem.

podilarius

On what kind of device did you run speed test on?

dLockers

@podilarius:

On what kind of device did you run speed test on?

My host, an Intel Core i7 920 D0 stepping.

I also ran a download direct from the PfSense shell (check the OP, its the bottom line of the output)

Edit: Unless packet shaping is enabled by default on the LAN interface?

podilarius

Traffic shaping is not enabled by default within pfSense. Though, have you installed the VMWare software/tool from the packages?

dLockers

@podilarius:

Traffic shaping is not enabled by default within pfSense. Though, have you installed the VMWare software/tool from the packages?

Yes I have installed VMWare Tools using the package and the vmxnet_load thing is in loader.conf

Maybe if I install it from the CD?

Edit; But I get the problem using vt-d passthrough of the NIC as well, so I can't see it being that.

dLockers

Bump!

Help save the environment and fix this with me! Saves another dedicated box!

biggsy

Someone please correct me if this is wrong but I think a mismatch, between autonegotiate on one end and and fixed 100 full on the other, may not show up as collisions or other errors but the physical NICs could be forced down to 10Mb/s (and maybe even half duplex) - the lowest common denominator.

Could you build another VM on vSwitch1 (maybe a WHS) and see how it does talking direct to the WAN
Can you set the interface speed on the modem or is it just auto?
I couldn't tell from your earlier response whether you had tried changing the physical LAN cable to the modem.
Tried reversing the NICs between the two vSwitches? If only to rule out a faulty NIC.

Maybe clutching at straws now but I can't help thinking there will be a lower layer answer to this.

Still thinking :)

dLockers

Maybe someone running ESXI can post their example configs?

I am going to try ipcop/smoothwall or something tonight and see if I can get that working at full speed.

They are all surprisingly more difficult to install compared to pfsense though.

Supermule

Try M0n0wall instead….easy peasy top install and get up running!

dLockers

Monowall - same symptons.

What the hell is going on here!

johnpoz

What I can tell you is I am running 32bit on esxi 5 with e1000, was running vmxnet3 but ran into a weird snag with making ipsec vpn connection to outside server from a client on the local network.. Have not had time to work that out yet, but going back to e1000 nics and working fine.

Running the same version of esxi 5 as you. But running 2.1 version of pfsense

2.1-BETA0 (i386)
built on Fri Jun 8 00:26:36 EDT 2012
FreeBSD 8.3-RELEASE-p2

With gitsync of a couple of days ago.

Here is my my traffic graph showing some downloads earlier - you can clearly see hitting my around 16Mbps limit.

Oh and this is on some really CHEAP hardware!! HP N40L box, running multiple VMs one being my router. With a cheapass $35 secondary nic being used for the wan connection.

speed.jpg_thumb

Metu69salemi

@johnpoz:

Oh and this is on some really CHEAP hardware!! HP N40L box, running multiple VMs one being my router. With a cheapass $35 secondary nic being used for the wan connection.

[OT]
Thank you about this post. I was puzzled if ESXi v5 would work with HP microserver, i think i got my answer
[/OT]

podilarius

I have a test one running on a Dell R710 with dell optimizations for ESX. I am getting 53.39 MBits down and 8.59 Mbits up on a moderately used connection (50/10 Cable Connection from Comcast). I am using E1000 nics. I am also double NATing and I don't have any problems. I think it has something to do with your ESX setup.