PfSense stops forwarding traffic under ESX5.x

benoshea

Hi,

I have run into an issue where pfSense (latest vApp images, both amd64 and i386) will stop forwarding traffic through the appliance. This is triggered by changing anything to do with the physical interfaces on the appliance - Setting MTU, MSS, re-assigining interfaces, etc. As soon as one of those changes are applied, inbound and outbound NATs cease to function, traffic to the appliance still wokrks (SSH, webConfigurator) it just seems to be all traffic through the appliance that stops working until the appliance is rebooted.

Exporting the configuration and applying it to a physical server does not have the same issues. I have now seen this in 2 separate customers environments, both esx 5.0. I have deployed several appliances under esx4 that did not suffer from this issue.

Has anyone else seen this issue? Any suggestions on how to resolve it (other than leaving it on physical hardware)?

Thanks,
Ben

cmb

Sounds like you probably have an outdated ESX host with a timekeeping bug that causes FreeBSD's clock to stop advancing, which stops traffic from passing. Upgrade ESX for the fix. See:
http://forum.pfsense.org/index.php/topic,54206.0.html

benoshea

Hi,
I've already checked this. I can reliably make it fail whenever I want to and I have verified that the time does in fact keep ticking when it fails.

Thanks,
Ben

johnpoz

so I am running pfsense on esxi 5.1 – can you give an example of what you do to make it fail whenever you want.

And I can try and duplicate it on mine.

You are talking esxi right, I was not aware of any actual esx in the 5.x line..4.1 is latest isn't it?

cmb

We do exactly as described frequently, nearly all our production firewalls in several colos, our office, homes, etc. run in ESX (ESXi technically, I and most use ESXi and ESX interchangeably these days). Never so much as a blip. So it's far from a general problem, tons of people do what you're doing with no issues.

Need some more troubleshooting, packet capture to see what gets where, check firewall states for what's getting passed, etc.

benoshea

@johnpoz:

so I am running pfsense on esxi 5.1 – can you give an example of what you do to make it fail whenever you want.

And I can try and duplicate it on mine.

You are talking esxi right, I was not aware of any actual esx in the 5.x line..4.1 is latest isn't it?

Yes, I meant esxi :) Specifically, ESXi 5.0 (I've not tried 5.1)

Any of these things will reliably make it fail:
1. Change physical interface properties (MSS, MTU, IP, etc)
2. Re-assign physical interfaces
3. Enable PPTP server and connect a client

After any of these things are performed, traffic through pfSense will stop flowing (on the 2 different customer setups I've tried), though traffic to the box (SSH, webConfigurator) will be fine.

Thanks,
Ben

benoshea

@cmb:

We do exactly as described frequently, nearly all our production firewalls in several colos, our office, homes, etc. run in ESX (ESXi technically, I and most use ESXi and ESX interchangeably these days). Never so much as a blip. So it's far from a general problem, tons of people do what you're doing with no issues.

Need some more troubleshooting, packet capture to see what gets where, check firewall states for what's getting passed, etc.

What version of ESXi are you running? The systems I have tried it on have all been ESXi 5.0 (I've not tried 5.1 yet). We have many of the same setup on ESXi 4.x without issue.