Pfsense 2.3.2 problem on esxi 6.0 build 4192238 (vcloud 8.10 with nsx)



  • Hello,
    i tried to setup a pfsense 2.3.2 in our new vcloud cell based on vsphere 6(fully updated) and we observed a strange behaviour.
    The problem is pretty simple to reproduce, just install a pfsense and a server in lan(direct network), set a nat, ssh is the best thing to see the problem, connect to the server behind pfsense via ssh and wait….after a few seconds/minutes session will be disconnected and won't reconnect, after a couple of minutes you will be able to connect again and then connection drops in an unexpected way.
    I tried to every type on network card (e1000,e1000e,vmxnet3) with no luck, tried with or without openvm-tools still no change.
    Same pfsense moved to an infrastructure based on vsphere 5.5 seems working fine.
    Also installing a pfsense 2.2.6 and importing conf from 2.3.2 on the vsphere 6 infrastructure seems working without problems.

    For now we will use version 2.2.6 on vcloud 8.10, anyway we're looking forward for a solution, i still have the test environment so if someone have some ideas i can try and check what happens.



  • I am not experiencing any of the problems you have.  I do, however have a fully-patched ESXi running my pfSense…  (VMware ESXi, 6.0.0, 2494685)



  • No problems here. ESXi 6.0U2, pfsense 2.3.2p1

    Check Gateway monitoring and the option to kill states if the gateway goes down.

    This change in 2.3 bit me for a while as I had left the old ISP gateway and when my DHCP changed networks, the old gateway was still there, but once you started to send enough traffic, the gateway monitor would mark that gateway as down and kill all the states.

    I bet ping to the LAN interface doesn't stop. How about the WAN interface? Traceroute?

    @ironashram:

    Hello,
    i tried to setup a pfsense 2.3.2 in our new vcloud cell based on vsphere 6(fully updated) and we observed a strange behaviour.
    The problem is pretty simple to reproduce, just install a pfsense and a server in lan(direct network), set a nat, ssh is the best thing to see the problem, connect to the server behind pfsense via ssh and wait….after a few seconds/minutes session will be disconnected and won't reconnect, after a couple of minutes you will be able to connect again and then connection drops in an unexpected way.
    I tried to every type on network card (e1000,e1000e,vmxnet3) with no luck, tried with or without openvm-tools still no change.
    Same pfsense moved to an infrastructure based on vsphere 5.5 seems working fine.
    Also installing a pfsense 2.2.6 and importing conf from 2.3.2 on the vsphere 6 infrastructure seems working without problems.

    For now we will use version 2.2.6 on vcloud 8.10, anyway we're looking forward for a solution, i still have the test environment so if someone have some ideas i can try and check what happens.



  • Gateway monitoring indeed was my problem, we have nexus 9000 in our new setup and they bring this fantastic feauture thak makes gateway respond to ping only sometimes :(

    Thanks pppfsense for pointing me in the right direction.



  • Glad my not-so glamorous 2 day troubleshooting experience with this helped you out :-)

    When this happened, I had just moved my equipment from a shelf to a rack on wheels in my basement (due to construction of french drain).
    My guess is that the pfsense/equipment was down for long enough time (full day before I rigged a consumer grade router to get temp Internet),
    that the ISP decided to put me on a different subnet when I reconnected.

    This drove me crazy as, with the move, I didn't introduce any new variables, but there was a physical change, none the less.
    The symptoms would be that once I was able to turn things back on, I would get Internet connectivity, but then, once I would download a file and semi-saturate the link, the gateway monitor would check the old gateway from the original DHCP subnet that I was part of (I knew my IP could change at any time, but never imagined that they would also change your subnet).

    I saw a bunch of WAN dropped packets in the managed switch that I use to connect everything, so I followed that route for an afternoon and changed cables, RJ-45 couplers, etc.  I was almost to the point of suspecting AC interference due to the new cable routing!   
    Of course this was simply because the WAN would reject packets while the NAT states were being reset, but I had no idea of that yet.

    It was not until the weekend when I was able to do more testing and debugging, that I realized what was happening.
    This never came up when I did the upgrade to 2.3, as my WAN gateway had not changed, so I just could not imagine what could have changed
    from the equipment being on a static wooden shelf, to being on a mobile wire shelf :-).

    Good lesson, just hope I don't get more of these crazy ones!

    @ironashram:

    Gateway monitoring indeed was my problem, we have nexus 9000 in our new setup and they bring this fantastic feauture thak makes gateway respond to ping only sometimes :(

    Thanks pppfsense for pointing me in the right direction.