ESXi 5.5 packet loss

arozanov

Hello! I have a problem with packet loss on ESXi 5.5. PFsense 2.2.4 (works like nat, fw) openvm-tools, vmxnet3.
In pfsense performance tab i saw packet loss (picture 1, don't watch on the highest peaks, its when rebooted)
It's always receive loss, transmit is ok.
After ENABLING(!) i.e untick "Disable hardware TCP segmentation offload" and "Disable hardware large receive offload" the performance tab looks like picture 4, it seems like loss now rare, but it happens.
According with http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010071
i want to try edit RX Ring buffers, but don't know how :) And how to view current parameters of buffers? How to monitor where is bottleneck on pfsense and what i need to increase/tune?
Your help very appreciated, thanks.

1.jpg_thumb

4.jpg_thumb

johnpoz

how many packets? I am not seeing this at all..

nodropps.png_thumb

arozanov

@johnpoz:

how many packets? I am not seeing this at all..

Sorry, don't know why, but its problem with screenshot from vsphere client. Something around 50-300 packets every 30-60 mins.
Before tuning, it was 200-400 every minute.

drops.JPG_thumb

johnpoz

no not how many drops how many TOTAL packets – ie the ones your not dropping. See my screen shot.

arozanov

@johnpoz:

no not how many drops how many TOTAL packets – ie the ones your not dropping. See my screen shot.

Ah, ok.

pckts.JPG_thumb

Halvsvenskeren

Try to see if E1000 network adapter works better for comparison.

johnpoz

So your seeing a LOT of packets!!! Yeah I think adjusting the ring value or ring 2 value might be in order.. But sorry not sure how to do that in freebsd or pfsense.. linux normally just a simple ethtool -G

With intel something with sysctl hw.em.rxd= but not sue with using the vmx3??

arozanov

@johnpoz:

So your seeing a LOT of packets!!! Yeah I think adjusting the ring value or ring 2 value might be in order.. But sorry not sure how to do that in freebsd or pfsense.. linux normally just a simple ethtool -G

With intel something with sysctl hw.em.rxd= but not sue with using the vmx3??

Yes, same problem for me. Not sure, can i write just something like hw.vmx.rxd=4096? And which parameter for ring value, and which one for ring 2?…..

arozanov

@Halvsvenskeren:

Try to see if E1000 network adapter works better for comparison.

Its a little problematic for me. But can be done, if needed. I thought, it will be the same problem.

Halvsvenskeren

We saw much better performance from the E1000 than VMXnet2 and 3.

Try it and report back.

arozanov

@Halvsvenskeren:

We saw much better performance from the E1000 than VMXnet2 and 3.

Try it and report back.

Ok, will try later today and will report back.

arozanov

So, the problem was with receiving buffers. According with https://www.freebsd.org/cgi/man.cgi?query=vmx&sektion=4 i increased buffers of vmx adapters and problem is gone (em works well too, with tuned buffers). After, jumbo frames was overloaded, so i increased this parameter too.
The same with receiving buffers on windows clients, but this is another story…

Thanks, guys! :)

anzak84

Which commands were executed on pfsense?
System: Advanced: System Tunables
hw.em.txd value
hw.em.rxd value
right?

System 2 Nics Wan -em1000 and Lan -em1000