pfSense on ESXi | Best Practices



  • Dear All,
    I need to take your recommendation for the best practices for VMware ESXi networking to work with pfSense. I'm using normal vSwitch and tagged my Firewall VLAN as the LAN interface and Tagged my WAN VLAN as the WAN interface. everything is working fine. sometimes when downloading any file from the internet I noticed that the download stops from a while and continue and also not with the full internet bandwidth.
    So, My Questions:

    • 1- Is my ESXi network method is correct or I should make something special ?
    • 2- Is there any special configuration for pfSense to work with ESXi ?

    Note: all my physical NICs are 1GB


  • LAYER 8 Rebel Alliance



  • @Rico That's typically what I did! but I'm still feel something wrong ?



  • Any Help ?


  • LAYER 8

    are you sure is a pfsense/esxi problem and not the ISP throttling large downloads ?
    maybe with some screenshot of your config we can better find out if there is something strange/wrong
    also use wireshark/tcpdump to analyze the traffic when your download slow down
    if it's only "sometimes" (define sometimes) maybe the server you are downloading from is congested


  • LAYER 8 Rebel Alliance

    VMware Tools installed?
    Try to switch from vmxnet3 to e1000 or vice versa.

    -Rico



  • Dear all,
    Thanks for your replies guys. Today, I did a ping test and that was the results:

    Ping Results

    Pinging micosoft.com [208.91.197.26] with 1500 bytes of data:
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=219ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=222ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=221ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=219ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=220ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=219ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=220ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=219ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=221ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=233ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=315ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=219ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=219ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=219ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=221ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=221ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=220ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=307ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=219ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=220ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=221ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=268ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=217ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=219ms TTL=230
    Reply from 208.91.197.26: bytes=1464 (sent 1500) time=218ms TTL=230
    Request timed out.
    
    Ping statistics for 208.91.197.26:
        Packets: Sent = 125, Received = 116, Lost = 9 (7% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 217ms, Maximum = 315ms, Average = 220ms
    
    Pinging micosoft.com [208.91.197.26] with 32 bytes of data:
    Reply from 208.91.197.26: bytes=32 time=216ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=282ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=263ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=219ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=254ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=271ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=252ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=233ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=270ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=273ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=257ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=218ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=313ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=286ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=235ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=217ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=297ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=318ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=216ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=271ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=216ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=237ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=275ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=220ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=251ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=281ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=216ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=229ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=232ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=310ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=288ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=213ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=237ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=241ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=233ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=324ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=216ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=273ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=294ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=216ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=240ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=216ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=272ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=270ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=288ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=291ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=216ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=213ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=238ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=300ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=286ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=218ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=213ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=273ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=216ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=268ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=220ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=271ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=216ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=319ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=265ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=216ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=258ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=275ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=256ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=299ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=216ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Request timed out.
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=215ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=251ms TTL=230
    Reply from 208.91.197.26: bytes=32 time=214ms TTL=230
    
    Ping statistics for 208.91.197.26:
        Packets: Sent = 279, Received = 245, Lost = 34 (12% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 213ms, Maximum = 324ms, Average = 225ms
    

    So, I noticed that there is a Packet Loss !! Please Advise where excellently the problem ?
    Thanks


  • LAYER 8

    imho probably your ISP, ping your isp gateway and see if you have the same problem, if it work without problem do a traceroute and ping every single hop until you find what is timing out, after that call your ISP
    also check if all your cable are in good condition



  • I tried without pfSense directly connected to WAN VLAN, everything working fine without any problem which means it's pfSense problem 100%


  • LAYER 8

    esxcli network nic tso set --enable=0 -n vmnic0
    esxcli network nic cso set --enable=0 -n vmnic0
    ?
    i'm pretty sure it's not a pfsense problem
    try to use another physical network interface card to connect the virtual switch to the physical switch to eliminate physical problems.
    Immagine.jpg

    esxi version? ESXi 6.0 is known to have host loses network connectivity randomly
    check /var/log/vmkernel.log file, if there is evidence of transmit timeouts

    also check if this article can help you
    https://kb.vmware.com/s/article/1004109



  • Thanks @kiokoman for you help, really appropriated. But why when I directly connected the WAN without pfSense everything goes fine ? also I tried to change the adapter type from e1000 to VMNET3 and the same problem !!


  • LAYER 8 Rebel Alliance

    Again: Latest VMware Tools installed?

    -Rico



  • @Rico yes sure



  • @Rico Open-VM-Tools v10.1.0_2,1


  • LAYER 8

    did you set traffic shaping or load balancing on the vswitch ?




  • LAYER 8

    i'm tring to mess as much as i can with my vm but i'm unable to reproduce it
    can you do a test with pfsense 2.5.0-devel ?
    also i found this about the vmx driver
    https://www.freebsd.org/cgi/man.cgi?query=vmx&sektion=4
    ethernet0.virtualDev="vmxnet3" i have it inside the virtual machine configuration and
    The hw.pci.honor_msi_blacklist tunable must be disabled to enable MSI-X support.
    i have it set on my
    /boot/loader.conf.local

    hw.pci.honor_msi_blacklist="0"
    

    also i found this but idk if it's still relevant
    https://forum.netgate.com/topic/88082/esxi-5-5-packet-loss/12



  • I have a bunch of pfsense instances running in ESXi. No issues. All 2.4.4-p3 and 2.4.5-RC.


  • LAYER 8 Rebel Alliance

    Yeah I don't think this problem is pfSense related.
    Can you spin up two more VMs for testing? One with vanilla FreeBSD 11.2 and another with Linux or Windows. Check if the timeouts happen in one or both test VMs.

    -Rico



  • problem SOLVED after "Disabling hardware checksum offload"


  • LAYER 8

    🤦 🤦
    i told you 4 days ago to disable cso "esxcli network nic cso set --enable=0 -n vmnic0"
    but i forgot to tell you to disable it from the pfsense gui





  • @mohkhalifa said in pfSense on ESXi | Best Practices:

    problem SOLVED after "Disabling hardware checksum offload"

    Awesome. I poked around on a few of mine and didn't find any with that enabled. Mostly Dell hardware here. Good find.