PfSense stops forwarding traffic under ESX5.x

[benoshea]

Hi,

I have run into an issue where pfSense (latest vApp images, both amd64 and i386) will stop forwarding traffic through the appliance. This is triggered by changing anything to do with the physical interfaces on the appliance - Setting MTU, MSS, re-assigining interfaces, etc. As soon as one of those changes are applied, inbound and outbound NATs cease to function, traffic to the appliance still wokrks (SSH, webConfigurator) it just seems to be all traffic through the appliance that stops working until the appliance is rebooted.

Exporting the configuration and applying it to a physical server does not have the same issues. I have now seen this in 2 separate customers environments, both esx 5.0. I have deployed several appliances under esx4 that did not suffer from this issue.

Has anyone else seen this issue? Any suggestions on how to resolve it (other than leaving it on physical hardware)?

Thanks,
Ben

[cmb]

Sounds like you probably have an outdated ESX host with a timekeeping bug that causes FreeBSD's clock to stop advancing, which stops traffic from passing. Upgrade ESX for the fix. See:
http://forum.pfsense.org/index.php/topic,54206.0.html

[benoshea]

Hi,
  I've already checked this. I can reliably make it fail whenever I want to and I have verified that the time does in fact keep ticking when it fails.

Thanks,
Ben

[johnpoz]

so I am running pfsense on esxi 5.1 – can you give an example of what you do to make it fail whenever you want.

And I can try and duplicate it on mine.

You are talking esxi right, I was not aware of any actual esx in the 5.x line..4.1 is latest isn't it?

[cmb]

We do exactly as described frequently, nearly all our production firewalls in several colos, our office, homes, etc. run in ESX (ESXi technically, I and most use ESXi and ESX interchangeably these days). Never so much as a blip. So it's far from a general problem, tons of people do what you're doing with no issues.

Need some more troubleshooting, packet capture to see what gets where, check firewall states for what's getting passed, etc.

[benoshea]

@johnpoz:

so I am running pfsense on esxi 5.1 – can you give an example of what you do to make it fail whenever you want.

And I can try and duplicate it on mine.

You are talking esxi right, I was not aware of any actual esx in the 5.x line..4.1 is latest isn't it?

Yes, I meant esxi :) Specifically, ESXi 5.0 (I've not tried 5.1)

Any of these things will reliably make it fail:
  1. Change physical interface properties (MSS, MTU, IP, etc)
  2. Re-assign physical interfaces
  3. Enable PPTP server and connect a client

After any of these things are performed, traffic through pfSense will stop flowing (on the 2 different customer setups I've tried), though traffic to the box (SSH, webConfigurator) will be fine.

Thanks,
Ben

[benoshea]

@cmb:

We do exactly as described frequently, nearly all our production firewalls in several colos, our office, homes, etc. run in ESX (ESXi technically, I and most use ESXi and ESX interchangeably these days). Never so much as a blip. So it's far from a general problem, tons of people do what you're doing with no issues.

Need some more troubleshooting, packet capture to see what gets where, check firewall states for what's getting passed, etc.

What version of ESXi are you running? The systems I have tried it on have all been ESXi 5.0 (I've not tried 5.1 yet). We have many of the same setup on ESXi 4.x without issue.