NAT port forward fails the first time
-
Your on a really OLD build of esxi. OP didn't say either way, might be on hardware?
Your saying it happens on both VB and ESXI?? What is common to those to tests? I can fire up a clean box for testing if need be, but just recently did that to play with actual vmware tools vs opentools and did not see such an issue, etc..
We are using latest build of ESXi 4.1 (1682698), opentools package installed. The testing vm uses latest VirtualBox, also tried with https://wiki.freebsd.org/VirtualBox#Installing_Guest_Additions_for_FreeBSD_guests.
Good question, what these two platforms have in common, so that they produce this failure.. Perhaps its an issue with esxi 4.1/VirtualBox and freebsd 10.1.
What vNics cards do you use for virtualization?
Everything works perfect and pfsense is really great in many aspects. The only problem is this port-forwarding thing, which I can not resolve. :o -
Huh, you say your running esxi then you say your using virtualbox?? So your running virtual box on a VM? What does vb have to do with it if your on esxi??
4.1 is OLD, 5.5 is current.
I use vmxnet3
-
Huh, you say your running esxi then you say your using virtualbox?? So your running virtual box on a VM? What does vb have to do with it if your on esxi??
4.1 is OLD, 5.5 is current.
I use vmxnet3
Sorry for the confusion. The esxi 4.1 (or at least 5.1) has to be used, because the machine esxi is installed on is a Dell PowerEdge T300, which does not support 5.5.
The Virtual Box installation is just a test environment, which is running independently on my local workstation, to cross check and debug this problem.Next I will install pfsense on an old server (DELL PowerEdge SC430) with the config.xml from the esxi, adapt the interfaces and see, if the problem also exists without any usage of any virtualization technology.
-
Can you explain your setup for the pfsense vm setup both esxi and vb. I can fire up a copy of vb and install pfsense.
So for example in my esxi setup related to your setup I have a Wan vswitch that is connected to 1 physical nic. The wan, cable modem is connected to this nic directly. I then have a vswitch Lan. This is connected to a different nic, and this nic connects to my physical switch.
pfsense as wan interface connected to wan vswitch, vmxnet3 and gets public IP from my isp. Its lan vnic is connected to the lan vswitch and has IP on my lan network 192.168.1.253/24 while my physical pc I use to access pfsense webgui is on 192.168.1.100.
I have the openvm tools installed, but not really required for testing of this issue since pfsense currently supports vmxnet3 out of the box. The tools would have some vm memory management stuff, allow you to send shutdown to the vm via esxi, etc.
I can fire up vb, and its setup would be my lan interface on workstation would be bridged to lan and pfsense wan would be connected and get an IP from my 192.168.1.0/24 network. I would then use a host only network in vb and that would be lan side and connect to pfsense from my workstation running vb on this network. Since my current lan is the pfsense default I would have to change the pfsense lan interface to something different.
Does this match up with your setup? If not how are you setup?
-
Does this match up with your setup? If not how are you setup?
Hi. I made further tests, to debug the problem. I installed pfsense on a test-PC (real hardware, not virtual) and restored a backup from the ESXi pfsense setup on this test-PC. Only thing I did to the config.xml is search/replace em0 with vr0 network interface name.
After restore the test-PC came up with no error and this machine acted as a 1:1 clone of the ESXi machine. I made some nat port forwardings on the test-PC and they worked without any error and pfsense worked as expected.
So this brings me to the conclusion, that this problem is not related to pfsense or my setup of pfsense, but with the virtualization technology used.Here are some screenshots of my setup on the ESXi 4.1 server:
(openVM tools package is installed)
Using 'E1000' adapter. Was not able to see vmxnet2 adapter in pfsense. Should vmxnet2 be possible?
The adapter as pfsense sees it.
vSwitch -
How are you doing port forwarding when your pfsense has 1 interface? So your doing vlans..
Why would you use vmxnet2, pfsense 2.2 has native support for vmxnet3.. Oh maybe your ancient version of esxi does not have vmx3?
-
How are you doing port forwarding when your pfsense has 1 interface? So your doing vlans..
Why would you use vmxnet2, pfsense 2.2 has native support for vmxnet3.. Oh maybe your ancient version of esxi does not have vmx3?
Yes, using VLans, but only since yesterday. Before using VLans, I had multiple virtual network adapters. Switching to VLans had no influence on the problem, discussed in this thread.
Yes, vmxnet3 is not available under Esxi 4.1. But after reading this http://kb.vmware.com/kb/1001805 I am not sure if this is true. Will have to research more, why vmxnet3 is not showing up in my ESXi 4.1 server.
Edit: ok, seems vmxnet3 adapter is not available, when 'Other'->'Freebsd 64-bit' is chosen as the guest-OS. Seems I have to migrate to ESXi 5.1 (http://kb.vmware.com/kb/2007240) to make use of vmx-9.
-
And why wouldn't just go with current 5.5?
-
And why wouldn't just go with current 5.5?
Our DELL PowerEdge T300 Server is not supported by ESXi 5.5
http://www.vmware.com/resources/compatibility/search.phpWith the free version of ESXi the new 5.5 features, esp. creating/admin of vmx-10 are not accessible without the Web Client, which is is not available for free.
-
you can admin vmx-10, you can't use vmx-10 with 4.1 can you? All of my machines are vmx10, you get a nag screen says you can not edit/create vmx10 specific features. None of which I am using.
Its not supported why? Because they don't have it on their supported list? they don't have my hp n40l on there either, and works GREAT!
-
you can admin vmx-10, you can't use vmx-10 with 4.1 can you? All of my machines are vmx10, you get a nag screen says you can not edit/create vmx10 specific features. None of which I am using.
Its not supported why? Because they don't have it on their supported list? they don't have my hp n40l on there either, and works GREAT!
OK - inserted a 5.5 CD into the Dell server. It warns that there are custom VIBs installed (oem-dell-Configuration-VIB, etc). Fear that if I ignore this warning, the migration will fail and ESXi wont boot anymore and so my organisation is fucked (I have .ova backups of all virtual machines and a backup of the esxi config, done with 'vicfg-cfgbackup.pl')
This thread becomes offtpic.. pfsense really pushes me hard, but seems to make me a better sysadmin.. :o :D
-
well if you have backups what does it matter? I am not a fan of upgrading such a huge jump in tech, there is one thing going from 5.5u1 to update2, etc.. But your jumping from 4.1 to 5.5? I would do clean if it was me.
Worse case you restore to your 4.1 setup.
-
OK… finally I could solve the problem..
First I upgraded ESXi from 4.1 to 5.1U3. Upgraded pfsense machine to vmx-9. Changed the network adapter to vmixnic3. Did portforwarding test and again same failure.. :/
But ok. I knew, its something with the virtualization and it seems it was not the network adapter... Then I saw, the pfsense machine only has 512MB on memory (also see screenshot above..), which is really low. I increased it to 4GB und Nat port-forwarding works now. -
I doubt the memory was an issue.. My pfsense vm only has 512 and no issues. So why don't you update to 5.5?? I just don't understand why your running old version?
-
I doubt the memory was an issue.. My pfsense vm only has 512 and no issues. So why don't you update to 5.5?? I just don't understand why your running old version?
I also gave the pfsense machine a 2nd vCPU. Before it only had 1. The Host-machine would offer 4 cpus. Will test tomorrow again, when I am at the office, with/without >=512MB RAM and 1/2 vCPUs.
So why don't you update to 5.5?? I just don't understand why your running old version?
5.1 is supported by VMWare till 2016 (http://www.vmware.com/files/pdf/support/Product-Lifecycle-Matrix.pdf). But ok - will try migration to 5.5 tomorrow. Do you really think it would make that much of a difference? Of course we can better match and compare our installation, if we are running the same hypervisor.
-
I use ESX 5.5 and E1000 as the vNIC. I also have opentools installed. What else can I do to troubleshoot this? It is constantly happening. Could it be possibly because of opentools?
-
What else can I do to troubleshoot this? It is constantly happening. Could it be possibly because of opentools?
On the esxi install it is/was happening constantly. On the virtual box testing setup it was only 50% of the time.
I was able to reproduce it on the VB setup. Here is 'rules.debug.old' (4909 Bytes), which seems to be active, after nat-portforwarding was changed. This looks like a minimal default pf setup to me.. The 'rules.debug', when the firewall works has 16985 Bytes.
Btw: The backup config.xml has 94kb. Do I get this right? When I change something in the webconfigurator, these changes are first saved to a config.xml, then, when 'Apply Changes' is pressed a message is sent to a php daemon, that reads the config.xml and reloads parts of the firewall? What happens if 'Apply Changes' is triggered, befor a complete config.xml is written?
[2.2-RELEASE][admin@pfSense.intern]/tmp: cat rules.debug.old set optimization normal set timeout { adaptive.start 0, adaptive.end 0 } set limit states 47000 set limit src-nodes 47000 #System aliases loopback = "{ lo0 }" #SSH Lockout Table table <sshlockout>persist table <webconfiguratorlockout>persist #Snort tables table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons" table <negate_networks># User Aliases # Gateways set skip on pfsync0 no nat proto carp no rdr proto carp nat-anchor "natearly/*" nat-anchor "natrules/*" # Outbound NAT rules (automatic) # Subnets to NAT tonatsubnets = "{ 127.0.0.0/8 }" # Load balancing anchor rdr-anchor "relayd/*" # TFTP proxy rdr-anchor "tftp-proxy/*" # UPnPd rdr anchor rdr-anchor "miniupnpd" anchor "relayd/*" anchor "openvpn/*" anchor "ipsec/*" # Allow IPv6 on loopback pass in quick on $loopback inet6 all tracker 1000000001 label "pass IPv6 loopback" pass out quick on $loopback inet6 all tracker 1000000002 label "pass IPv6 loopback" # Block all IPv6 block in log quick inet6 all tracker 1000000003 label "Block all IPv6" block out log quick inet6 all tracker 1000000004 label "Block all IPv6" # block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device, # and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but # route-to can override that, causing problems such as in redmine #2073 block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local" block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local" #--------------------------------------------------------------------------- # default deny rules #--------------------------------------------------------------------------- block in log inet all tracker 1000000103 label "Default deny rule IPv4" block out log inet all tracker 1000000104 label "Default deny rule IPv4" block in log inet6 all tracker 1000000105 label "Default deny rule IPv6" block out log inet6 all tracker 1000000106 label "Default deny rule IPv6" # IPv6 ICMP is not auxilary, it is required for operation # See man icmp6(4) # 1 unreach Destination unreachable # 2 toobig Packet too big # 128 echoreq Echo service request # 129 echorep Echo service reply # 133 routersol Router solicitation # 134 routeradv Router advertisement # 135 neighbrsol Neighbor solicitation # 136 neighbradv Neighbor advertisement pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state # Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep) pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state # We use the mighty pf, we cannot be fooled. block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 # Snort package block log quick from <snort2c>to any tracker 1000000117 label "Block snort2c hosts" block log quick from any to <snort2c>tracker 1000000118 label "Block snort2c hosts" # SSH lockout block in log quick proto tcp from <sshlockout>to (self) port 22 tracker 1000000301 label "sshlockout" # webConfigurator lockout block in log quick proto tcp from <webconfiguratorlockout>to (self) port 443 tracker 1000000351 label "webConfiguratorlockout" block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table" # loopback pass in on $loopback inet all tracker 1000000561 label "pass IPv4 loopback" pass out on $loopback inet all tracker 1000000562 label "pass IPv4 loopback" pass in on $loopback inet6 all tracker 1000000563 label "pass IPv6 loopback" pass out on $loopback inet6 all tracker 1000000564 label "pass IPv6 loopback" # let out anything from the firewall host itself and decrypted IPsec traffic pass out inet all keep state allow-opts tracker 1000000565 label "let out anything IPv4 from firewall host itself" pass out inet6 all keep state allow-opts tracker 1000000566 label "let out anything IPv6 from firewall host itself" # VPN Rules anchor "tftp-proxy/*"</virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout> -
What else can I do to troubleshoot this? It is constantly happening. Could it be possibly because of opentools?
On the esxi install it is/was happening constantly. On the virtual box testing setup it was only 50% of the time.
I was able to reproduce it on the VB setup. Here is 'rules.debug.old' (4909 Bytes), which seems to be active, after nat-portforwarding was changed. This looks like a minimal default pf setup to me.. The 'rules.debug', when the firewall works has 16985 Bytes.
Btw: The backup config.xml has 94kb. Do I get this right? When I change something in the webconfigurator, these changes are first saved to a config.xml, then, when 'Apply Changes' is pressed a message is sent to a php daemon, that reads the config.xml and reloads parts of the firewall? What happens if 'Apply Changes' is triggered, befor a complete config.xml is written?
[2.2-RELEASE][admin@pfSense.intern]/tmp: cat rules.debug.old set optimization normal set timeout { adaptive.start 0, adaptive.end 0 } set limit states 47000 set limit src-nodes 47000 #System aliases loopback = "{ lo0 }" #SSH Lockout Table table <sshlockout>persist table <webconfiguratorlockout>persist #Snort tables table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons" table <negate_networks># User Aliases # Gateways set skip on pfsync0 no nat proto carp no rdr proto carp nat-anchor "natearly/*" nat-anchor "natrules/*" # Outbound NAT rules (automatic) # Subnets to NAT tonatsubnets = "{ 127.0.0.0/8 }" # Load balancing anchor rdr-anchor "relayd/*" # TFTP proxy rdr-anchor "tftp-proxy/*" # UPnPd rdr anchor rdr-anchor "miniupnpd" anchor "relayd/*" anchor "openvpn/*" anchor "ipsec/*" # Allow IPv6 on loopback pass in quick on $loopback inet6 all tracker 1000000001 label "pass IPv6 loopback" pass out quick on $loopback inet6 all tracker 1000000002 label "pass IPv6 loopback" # Block all IPv6 block in log quick inet6 all tracker 1000000003 label "Block all IPv6" block out log quick inet6 all tracker 1000000004 label "Block all IPv6" # block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device, # and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but # route-to can override that, causing problems such as in redmine #2073 block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local" block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local" #--------------------------------------------------------------------------- # default deny rules #--------------------------------------------------------------------------- block in log inet all tracker 1000000103 label "Default deny rule IPv4" block out log inet all tracker 1000000104 label "Default deny rule IPv4" block in log inet6 all tracker 1000000105 label "Default deny rule IPv6" block out log inet6 all tracker 1000000106 label "Default deny rule IPv6" # IPv6 ICMP is not auxilary, it is required for operation # See man icmp6(4) # 1 unreach Destination unreachable # 2 toobig Packet too big # 128 echoreq Echo service request # 129 echorep Echo service reply # 133 routersol Router solicitation # 134 routeradv Router advertisement # 135 neighbrsol Neighbor solicitation # 136 neighbradv Neighbor advertisement pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state # Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep) pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state # We use the mighty pf, we cannot be fooled. block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 # Snort package block log quick from <snort2c>to any tracker 1000000117 label "Block snort2c hosts" block log quick from any to <snort2c>tracker 1000000118 label "Block snort2c hosts" # SSH lockout block in log quick proto tcp from <sshlockout>to (self) port 22 tracker 1000000301 label "sshlockout" # webConfigurator lockout block in log quick proto tcp from <webconfiguratorlockout>to (self) port 443 tracker 1000000351 label "webConfiguratorlockout" block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table" # loopback pass in on $loopback inet all tracker 1000000561 label "pass IPv4 loopback" pass out on $loopback inet all tracker 1000000562 label "pass IPv4 loopback" pass in on $loopback inet6 all tracker 1000000563 label "pass IPv6 loopback" pass out on $loopback inet6 all tracker 1000000564 label "pass IPv6 loopback" # let out anything from the firewall host itself and decrypted IPsec traffic pass out inet all keep state allow-opts tracker 1000000565 label "let out anything IPv4 from firewall host itself" pass out inet6 all keep state allow-opts tracker 1000000566 label "let out anything IPv6 from firewall host itself" # VPN Rules anchor "tftp-proxy/*"</virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout>Where do I find the solution in your post? I'm not sure if it is a direct reply to my question?
I'm also using VLAN's on my backend firewall but not on my frontend one. They both exhibit the same issue though. And yes, I only have 512MB on both them.
-
Where do I find the solution in your post? I'm not sure if it is a direct reply to my question?
My apologies, thought your posting was from johnpoz.
-
I doubt the memory was an issue.. My pfsense vm only has 512 and no issues.
OK - I am 100% sure this is a memory issue - maybe in conjunction with virtualization in use. Not sure if VB officially supports Freebsd 10, its true that VMWare officially supports Freebsd 10.1 only after ESXi 5.5U2 (http://blogs.vmware.com/guestosguide/guest-os/unix-and-others/freebsd), but kevindd992002 seems to have the same problem with ESXi 5.5.
I replayed the whole situation again in VB. When the machine had 512MB Ram, I was able to reproduce the problem. After I gave it 4GB (might also work with 1GB, will have to try), I could not reproduce the problem.
I just dont understand, why no error message is produced. Is it possible to increase the log level somewhere?
