NAT port forward fails the first time

kevindd992002

Now, my setup is WAN <-> FrontEnd FW <-> DMZ <-> Backend FW <-> LAN with two pfsense 2.2 (latest) boxes. I setup a port forward rule on both firewalls (on their WAN side, of course) and after applying the changes, the LAN side of the boxes just isn't accessible. I had to restart the boxes and the port forward rules work and both boxes from their LAN sides were accessible. What could cause that? This did not happen with pre-2.2 releases.

kevindd992002

BUMP!

kevindd992002

Can anybody acknowledge this problem please? I can recreate this issue multiples times on two freshly-installed pfsense 2.2 boxes.

doktornotor

Flush states after changing the rules.

kevindd992002

@doktornotor:

Flush states after changing the rules.

Well, after I hit apply it already hangs up on me? How will I flush the states if that happens? And this doesn't happen with the older version.

RacingRalph

I am having the same issue, which makes remote administration of pfsense impossible.

Making changes in the nat->port-forwarding setup (adding, modifying or deletion of a rule) and after
hitting 'Apply Changes',
pfsense is completely unreachable (also from the internet.. :-X). No error message is issued into any logfile.

The only fixes I found so far:

After (manual at the console, since the firewall is not reachable via any network connection) rebooting, pfsense functions normal and the port-forwarding changes got correctly applied.
Also manually issuing '/etc/rc.filter_configure_sync' at the shell-console fixes the problem and network connection are working again.
This (automatic reboot, if firewall cant reach internet) might be a workaround, but I hope there is a better fix.

Tried many different settings for >12h now, nothing helped.
Hope anybody has an idea, whats going on here.

RacingRalph

BUMP

This problem is really a showstopper, what info do you need to help with this problem? pfsense runs under vmware esxi 4.1 and I can regenerate this problem under virtualbox. Perhaps this has something to do with virtualization?

johnpoz

I run pfsense under esxi, current 5.5 build 2456374, I make changes to it all the time via a openvpn connection and never have any issues. Setting up a port forward should not make you loose connections

I just now added a port forward to 22, and right after the apply it was available. My openvpn connection never dropped and 22 was available from public right after hitting apply. Even a RDP to a box on the inside via never skipped a beat..

RacingRalph

Hi,

I made a screen capture movie. This is a fresh pfsense installation, with 2 network interface added for testing purposes:

http://youtu.be/F4ccXKhRlOo

johnpoz

Well what did you use to make that fancy video and I could do the same.. But what I can tell you is via openvpn connection.

In my browser I hit pfsense 192.168.1.253 web gui, made port forward, hit apply, then removed port forward and hit apply all while pinging from the remote client a box inside pfsense at 192.168.1.100, and pfsense lan interface itself at 192.168.1.253

not 1 ping was lost during the process

And this vpn has to go through a proxy that is in JAX, FL while I'm in downers grove, IL to my home connection in schaumburg, il

pfsensemakenatchange.png_thumb
vpninfo.png_thumb

kevindd992002

@johnpoz, I'm not sure why you can't reproduce this issue but we can reproduce it every single time we hit apply after adding a port forward. This is happening on two of my firewalls and I already did a fresh re-installation on them. Perhaps there's a set of settings that's causing this?

RacingRalph

@johnpoz Are you using latest pfsense (2.2-RELEASE amd64)? AFAIK it has nothing to do with OpenVPN.

There seems to be a setting or a constellation, that produces this error.

johnpoz

Yes I am using latest

2.2-RELEASE (amd64)
built on Thu Jan 22 14:03:54 CST 2015
FreeBSD 10.1-RELEASE-p4

I didn't say it had to do with openvpn, just stating that to show I am in remote. There is nothing special you should have to do, clearly you have something wrong.. Do you have vmware tools installed? Your on a really OLD build of esxi. OP didn't say either way, might be on hardware?

Your saying it happens on both VB and ESXI?? What is common to those to tests? I can fire up a clean box for testing if need be, but just recently did that to play with actual vmware tools vs opentools and did not see such an issue, etc..

RacingRalph

@johnpoz:

Your on a really OLD build of esxi. OP didn't say either way, might be on hardware?

Your saying it happens on both VB and ESXI?? What is common to those to tests? I can fire up a clean box for testing if need be, but just recently did that to play with actual vmware tools vs opentools and did not see such an issue, etc..

We are using latest build of ESXi 4.1 (1682698), opentools package installed. The testing vm uses latest VirtualBox, also tried with https://wiki.freebsd.org/VirtualBox#Installing_Guest_Additions_for_FreeBSD_guests.
Good question, what these two platforms have in common, so that they produce this failure.. Perhaps its an issue with esxi 4.1/VirtualBox and freebsd 10.1.
What vNics cards do you use for virtualization?
Everything works perfect and pfsense is really great in many aspects. The only problem is this port-forwarding thing, which I can not resolve. :o

johnpoz

Huh, you say your running esxi then you say your using virtualbox?? So your running virtual box on a VM? What does vb have to do with it if your on esxi??

4.1 is OLD, 5.5 is current.

I use vmxnet3

RacingRalph

@johnpoz:

Huh, you say your running esxi then you say your using virtualbox?? So your running virtual box on a VM? What does vb have to do with it if your on esxi??

4.1 is OLD, 5.5 is current.

I use vmxnet3

Sorry for the confusion. The esxi 4.1 (or at least 5.1) has to be used, because the machine esxi is installed on is a Dell PowerEdge T300, which does not support 5.5.
The Virtual Box installation is just a test environment, which is running independently on my local workstation, to cross check and debug this problem.

Next I will install pfsense on an old server (DELL PowerEdge SC430) with the config.xml from the esxi, adapt the interfaces and see, if the problem also exists without any usage of any virtualization technology.

johnpoz

Can you explain your setup for the pfsense vm setup both esxi and vb. I can fire up a copy of vb and install pfsense.

So for example in my esxi setup related to your setup I have a Wan vswitch that is connected to 1 physical nic. The wan, cable modem is connected to this nic directly. I then have a vswitch Lan. This is connected to a different nic, and this nic connects to my physical switch.

pfsense as wan interface connected to wan vswitch, vmxnet3 and gets public IP from my isp. Its lan vnic is connected to the lan vswitch and has IP on my lan network 192.168.1.253/24 while my physical pc I use to access pfsense webgui is on 192.168.1.100.

I have the openvm tools installed, but not really required for testing of this issue since pfsense currently supports vmxnet3 out of the box. The tools would have some vm memory management stuff, allow you to send shutdown to the vm via esxi, etc.

I can fire up vb, and its setup would be my lan interface on workstation would be bridged to lan and pfsense wan would be connected and get an IP from my 192.168.1.0/24 network. I would then use a host only network in vb and that would be lan side and connect to pfsense from my workstation running vb on this network. Since my current lan is the pfsense default I would have to change the pfsense lan interface to something different.

Does this match up with your setup? If not how are you setup?

RacingRalph

@johnpoz:

Does this match up with your setup? If not how are you setup?

Hi. I made further tests, to debug the problem. I installed pfsense on a test-PC (real hardware, not virtual) and restored a backup from the ESXi pfsense setup on this test-PC. Only thing I did to the config.xml is search/replace em0 with vr0 network interface name.
After restore the test-PC came up with no error and this machine acted as a 1:1 clone of the ESXi machine. I made some nat port forwardings on the test-PC and they worked without any error and pfsense worked as expected.
So this brings me to the conclusion, that this problem is not related to pfsense or my setup of pfsense, but with the virtualization technology used.

Here are some screenshots of my setup on the ESXi 4.1 server:
(openVM tools package is installed)

Using 'E1000' adapter. Was not able to see vmxnet2 adapter in pfsense. Should vmxnet2 be possible?

The adapter as pfsense sees it.

vSwitch

johnpoz

How are you doing port forwarding when your pfsense has 1 interface? So your doing vlans..

Why would you use vmxnet2, pfsense 2.2 has native support for vmxnet3.. Oh maybe your ancient version of esxi does not have vmx3?

RacingRalph

@johnpoz:

How are you doing port forwarding when your pfsense has 1 interface? So your doing vlans..

Why would you use vmxnet2, pfsense 2.2 has native support for vmxnet3.. Oh maybe your ancient version of esxi does not have vmx3?

Yes, using VLans, but only since yesterday. Before using VLans, I had multiple virtual network adapters. Switching to VLans had no influence on the problem, discussed in this thread.

Yes, vmxnet3 is not available under Esxi 4.1. But after reading this http://kb.vmware.com/kb/1001805 I am not sure if this is true. Will have to research more, why vmxnet3 is not showing up in my ESXi 4.1 server.

Edit: ok, seems vmxnet3 adapter is not available, when 'Other'->'Freebsd 64-bit' is chosen as the guest-OS. Seems I have to migrate to ESXi 5.1 (http://kb.vmware.com/kb/2007240) to make use of vmx-9.