NAT port forward fails the first time
-
I use ESX 5.5 and E1000 as the vNIC. I also have opentools installed. What else can I do to troubleshoot this? It is constantly happening. Could it be possibly because of opentools?
-
What else can I do to troubleshoot this? It is constantly happening. Could it be possibly because of opentools?
On the esxi install it is/was happening constantly. On the virtual box testing setup it was only 50% of the time.
I was able to reproduce it on the VB setup. Here is 'rules.debug.old' (4909 Bytes), which seems to be active, after nat-portforwarding was changed. This looks like a minimal default pf setup to me.. The 'rules.debug', when the firewall works has 16985 Bytes.
Btw: The backup config.xml has 94kb. Do I get this right? When I change something in the webconfigurator, these changes are first saved to a config.xml, then, when 'Apply Changes' is pressed a message is sent to a php daemon, that reads the config.xml and reloads parts of the firewall? What happens if 'Apply Changes' is triggered, befor a complete config.xml is written?
[2.2-RELEASE][admin@pfSense.intern]/tmp: cat rules.debug.old set optimization normal set timeout { adaptive.start 0, adaptive.end 0 } set limit states 47000 set limit src-nodes 47000 #System aliases loopback = "{ lo0 }" #SSH Lockout Table table <sshlockout>persist table <webconfiguratorlockout>persist #Snort tables table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons" table <negate_networks># User Aliases # Gateways set skip on pfsync0 no nat proto carp no rdr proto carp nat-anchor "natearly/*" nat-anchor "natrules/*" # Outbound NAT rules (automatic) # Subnets to NAT tonatsubnets = "{ 127.0.0.0/8 }" # Load balancing anchor rdr-anchor "relayd/*" # TFTP proxy rdr-anchor "tftp-proxy/*" # UPnPd rdr anchor rdr-anchor "miniupnpd" anchor "relayd/*" anchor "openvpn/*" anchor "ipsec/*" # Allow IPv6 on loopback pass in quick on $loopback inet6 all tracker 1000000001 label "pass IPv6 loopback" pass out quick on $loopback inet6 all tracker 1000000002 label "pass IPv6 loopback" # Block all IPv6 block in log quick inet6 all tracker 1000000003 label "Block all IPv6" block out log quick inet6 all tracker 1000000004 label "Block all IPv6" # block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device, # and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but # route-to can override that, causing problems such as in redmine #2073 block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local" block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local" #--------------------------------------------------------------------------- # default deny rules #--------------------------------------------------------------------------- block in log inet all tracker 1000000103 label "Default deny rule IPv4" block out log inet all tracker 1000000104 label "Default deny rule IPv4" block in log inet6 all tracker 1000000105 label "Default deny rule IPv6" block out log inet6 all tracker 1000000106 label "Default deny rule IPv6" # IPv6 ICMP is not auxilary, it is required for operation # See man icmp6(4) # 1 unreach Destination unreachable # 2 toobig Packet too big # 128 echoreq Echo service request # 129 echorep Echo service reply # 133 routersol Router solicitation # 134 routeradv Router advertisement # 135 neighbrsol Neighbor solicitation # 136 neighbradv Neighbor advertisement pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state # Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep) pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state # We use the mighty pf, we cannot be fooled. block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 # Snort package block log quick from <snort2c>to any tracker 1000000117 label "Block snort2c hosts" block log quick from any to <snort2c>tracker 1000000118 label "Block snort2c hosts" # SSH lockout block in log quick proto tcp from <sshlockout>to (self) port 22 tracker 1000000301 label "sshlockout" # webConfigurator lockout block in log quick proto tcp from <webconfiguratorlockout>to (self) port 443 tracker 1000000351 label "webConfiguratorlockout" block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table" # loopback pass in on $loopback inet all tracker 1000000561 label "pass IPv4 loopback" pass out on $loopback inet all tracker 1000000562 label "pass IPv4 loopback" pass in on $loopback inet6 all tracker 1000000563 label "pass IPv6 loopback" pass out on $loopback inet6 all tracker 1000000564 label "pass IPv6 loopback" # let out anything from the firewall host itself and decrypted IPsec traffic pass out inet all keep state allow-opts tracker 1000000565 label "let out anything IPv4 from firewall host itself" pass out inet6 all keep state allow-opts tracker 1000000566 label "let out anything IPv6 from firewall host itself" # VPN Rules anchor "tftp-proxy/*"</virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout> -
What else can I do to troubleshoot this? It is constantly happening. Could it be possibly because of opentools?
On the esxi install it is/was happening constantly. On the virtual box testing setup it was only 50% of the time.
I was able to reproduce it on the VB setup. Here is 'rules.debug.old' (4909 Bytes), which seems to be active, after nat-portforwarding was changed. This looks like a minimal default pf setup to me.. The 'rules.debug', when the firewall works has 16985 Bytes.
Btw: The backup config.xml has 94kb. Do I get this right? When I change something in the webconfigurator, these changes are first saved to a config.xml, then, when 'Apply Changes' is pressed a message is sent to a php daemon, that reads the config.xml and reloads parts of the firewall? What happens if 'Apply Changes' is triggered, befor a complete config.xml is written?
[2.2-RELEASE][admin@pfSense.intern]/tmp: cat rules.debug.old set optimization normal set timeout { adaptive.start 0, adaptive.end 0 } set limit states 47000 set limit src-nodes 47000 #System aliases loopback = "{ lo0 }" #SSH Lockout Table table <sshlockout>persist table <webconfiguratorlockout>persist #Snort tables table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons" table <negate_networks># User Aliases # Gateways set skip on pfsync0 no nat proto carp no rdr proto carp nat-anchor "natearly/*" nat-anchor "natrules/*" # Outbound NAT rules (automatic) # Subnets to NAT tonatsubnets = "{ 127.0.0.0/8 }" # Load balancing anchor rdr-anchor "relayd/*" # TFTP proxy rdr-anchor "tftp-proxy/*" # UPnPd rdr anchor rdr-anchor "miniupnpd" anchor "relayd/*" anchor "openvpn/*" anchor "ipsec/*" # Allow IPv6 on loopback pass in quick on $loopback inet6 all tracker 1000000001 label "pass IPv6 loopback" pass out quick on $loopback inet6 all tracker 1000000002 label "pass IPv6 loopback" # Block all IPv6 block in log quick inet6 all tracker 1000000003 label "Block all IPv6" block out log quick inet6 all tracker 1000000004 label "Block all IPv6" # block IPv4 link-local. Per RFC 3927, link local "MUST NOT" be forwarded by a routing device, # and clients "MUST NOT" send such packets to a router. FreeBSD won't route 169.254./16, but # route-to can override that, causing problems such as in redmine #2073 block in log quick from 169.254.0.0/16 to any tracker 1000000101 label "Block IPv4 link-local" block in log quick from any to 169.254.0.0/16 tracker 1000000102 label "Block IPv4 link-local" #--------------------------------------------------------------------------- # default deny rules #--------------------------------------------------------------------------- block in log inet all tracker 1000000103 label "Default deny rule IPv4" block out log inet all tracker 1000000104 label "Default deny rule IPv4" block in log inet6 all tracker 1000000105 label "Default deny rule IPv6" block out log inet6 all tracker 1000000106 label "Default deny rule IPv6" # IPv6 ICMP is not auxilary, it is required for operation # See man icmp6(4) # 1 unreach Destination unreachable # 2 toobig Packet too big # 128 echoreq Echo service request # 129 echorep Echo service reply # 133 routersol Router solicitation # 134 routeradv Router advertisement # 135 neighbrsol Neighbor solicitation # 136 neighbradv Neighbor advertisement pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000107 keep state # Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep) pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000108 keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000109 keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000111 keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000112 keep state # We use the mighty pf, we cannot be fooled. block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000113 block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000114 block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000115 block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000116 # Snort package block log quick from <snort2c>to any tracker 1000000117 label "Block snort2c hosts" block log quick from any to <snort2c>tracker 1000000118 label "Block snort2c hosts" # SSH lockout block in log quick proto tcp from <sshlockout>to (self) port 22 tracker 1000000301 label "sshlockout" # webConfigurator lockout block in log quick proto tcp from <webconfiguratorlockout>to (self) port 443 tracker 1000000351 label "webConfiguratorlockout" block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table" # loopback pass in on $loopback inet all tracker 1000000561 label "pass IPv4 loopback" pass out on $loopback inet all tracker 1000000562 label "pass IPv4 loopback" pass in on $loopback inet6 all tracker 1000000563 label "pass IPv6 loopback" pass out on $loopback inet6 all tracker 1000000564 label "pass IPv6 loopback" # let out anything from the firewall host itself and decrypted IPsec traffic pass out inet all keep state allow-opts tracker 1000000565 label "let out anything IPv4 from firewall host itself" pass out inet6 all keep state allow-opts tracker 1000000566 label "let out anything IPv6 from firewall host itself" # VPN Rules anchor "tftp-proxy/*"</virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout>Where do I find the solution in your post? I'm not sure if it is a direct reply to my question?
I'm also using VLAN's on my backend firewall but not on my frontend one. They both exhibit the same issue though. And yes, I only have 512MB on both them.
-
Where do I find the solution in your post? I'm not sure if it is a direct reply to my question?
My apologies, thought your posting was from johnpoz.
-
I doubt the memory was an issue.. My pfsense vm only has 512 and no issues.
OK - I am 100% sure this is a memory issue - maybe in conjunction with virtualization in use. Not sure if VB officially supports Freebsd 10, its true that VMWare officially supports Freebsd 10.1 only after ESXi 5.5U2 (http://blogs.vmware.com/guestosguide/guest-os/unix-and-others/freebsd), but kevindd992002 seems to have the same problem with ESXi 5.5.
I replayed the whole situation again in VB. When the machine had 512MB Ram, I was able to reproduce the problem. After I gave it 4GB (might also work with 1GB, will have to try), I could not reproduce the problem.
I just dont understand, why no error message is produced. Is it possible to increase the log level somewhere? -
Valid points freebsd 10 not supported on esxi.. So all bets are off if you ask me with any sort of issue if your not running vm software that supports the os your trying to run.
Again why can you not update to 5.5 - because your hardware is not listed? Your running FREE version of esxi? Or do you have support from vmware that would require you to run on what they will support? If not I would go to current 5.5 since you want to run a freebsd 10 vm (pfsense)
As to any issues you have with vb.. can not help sorry, have not used that in any sort of sense for years. Your not running pfsense in your "production" setup in VB are you? Thought you were using esxi, etc.. So what does it matter what vb does or not do?
-
To reproduce the problem in VB (4.3.22-98236 used):
-
Create new vanilla machine. Freebsd 64bit, give it 512MB RAM, 10gb disk storage, everything else default settings
-
I create 2nd network adapter, make both network-bridges (depends on your LAN setup)
-
Install pfsense the regular way, after reboot configure it, so that the webconfigurator can be accessed
-
Make some port forwardings and delete the again. I got a blocked firewall after 3-4 tries.
-
Shutdown machine, change RAM to >512MB, try making port forwarding again
-
-
How do you explain my case? I'm using esx 5.5 already but I have the same issue and I use 512MB RAM for both of my firewalls.
-
Again what does VB have to do with it?? Are you using VB in "production" – your just trying to reproduce an issue that also doesn't support freebsd 10.1 does it?
-
What case are you talking about - you hijacking more threads?
I am getting confused on what thread starter, what the cases are ;) heheeh Need more coffee.
Are you using u2 of 5.5? Where are you details?
-
What case are you talking about - you hijacking more threads?
Are you using u2 of 5.5? Where are you details?
Again, this is MY thread. And please read my replies before you accuse me of doing anything. I have the same EXACT ISSUE as RacingRalph.
Both my firewalls only has 512MB. I'm using esxi 5.5 but I experience the same issue. So this is not a matter of esxi version.
-
Not accusing anyone of anything ;)
This is why you shouldn't mix up thread unless they are the same.. If your running current 5.5, and he is running 4.1 and VB sorry but those are different.. Let me relook over the thread. Tell you I have current 5.5 with opentools pfsense 64bit, only 512 with 2 cpu and have seen no issues create nats, changing firewall rules, etc. etc.
edit:
Ok yeah he really took over your thread filling it with stuff that has nothing do with yours - I got confused, sorry.You really have provided little detail.. So looking over do you have 32 bit or 64 bit, why are you running e1000? Have you tried it with vmx3 native? Before and after you install the open tools.. What other packages do you have installed if any? What build of esxi, 10.1 I do believe needs at min update 2.. I am on 5.5.0 build 2456374
yeah sure looks like you need update 2, this is when they added freebsd 10.1 support.
-
Well here is the thing.. Freebsd 10.1 was not added until 5.5u2 – so why would you expect it to function properly. Until you move to 5.5u2 your going to have to use an older version of pfsense..
edit: this was directed at rachingralph - but seems that post is gone about following kb from vmware, etc.
-
Until you move to 5.5u2 your going to have to use an older version of pfsense..
OK - Updated to ESXi 5.5u2 today. Works like a charm.. :P Reduced pfsense memory to 512kB and removed 2nd vCPU. Could reproduce the firewall block symptome.. >:(
But the I added a 2nd vCPU (and let the 512kB RAM as it was) and with these settings I am not able to reproduce the problem.
How many cores das your virtual pfsense machine have?
-
My vm has 2 cores, and 512MB of ram – yeah I would think it wouldn't even boot with 512KB of ram ;)
So did you do a clean install when you changed from 2 cpu to 1? Not sure how freebsd handles having 2 cpus and then booting and only seeing 1, etc.
Is this with or without open tools? Using vmx3 or e1000 ?
-
My vm has 2 cores, and 512MB of ram – yeah I would think it wouldn't even boot with 512KB of ram ;)
So did you do a clean install when you changed from 2 cpu to 1? Not sure how freebsd handles having 2 cpus and then booting and only seeing 1, etc.
Is this with or without open tools? Using vmx3 or e1000 ?
oh yes.. of course 'MB's. :)
Didnt do a clean install - but that shouldnt be a problen - can do that tomorrow, pfsense is very flexible in that aspect, with the config.xml backup/restore mechanism. According to the FreeBSD book the amd64 kernel is compatible with most Xeon cpus (https://www.freebsd.org/doc/handbook/bsdinstall-hardware.html). But since not many cpus with only 1 socket/core exist anymore, it makes sense that a multicore cpu is expected by the system. (in my case: http://en.wikipedia.org/wiki/List_of_Intel_Xeon_microprocessors#.22Yorkfield-CL.22_.2845_nm.29)
It is with open tools and vmxnet3.
-
I have the same EXACT ISSUE as RacingRalph.
Did you check the number of vCPUs in your ESXi setup for pfsense?
-
Not accusing anyone of anything ;)
This is why you shouldn't mix up thread unless they are the same.. If your running current 5.5, and he is running 4.1 and VB sorry but those are different.. Let me relook over the thread. Tell you I have current 5.5 with opentools pfsense 64bit, only 512 with 2 cpu and have seen no issues create nats, changing firewall rules, etc. etc.
edit:
Ok yeah he really took over your thread filling it with stuff that has nothing do with yours - I got confused, sorry.You really have provided little detail.. So looking over do you have 32 bit or 64 bit, why are you running e1000? Have you tried it with vmx3 native? Before and after you install the open tools.. What other packages do you have installed if any? What build of esxi, 10.1 I do believe needs at min update 2.. I am on 5.5.0 build 2456374
yeah sure looks like you need update 2, this is when they added freebsd 10.1 support.
Well here is the thing.. Freebsd 10.1 was not added until 5.5u2 – so why would you expect it to function properly. Until you move to 5.5u2 your going to have to use an older version of pfsense..
edit: this was directed at rachingralph - but seems that post is gone about following kb from vmware, etc.
I have a 64bit pfsense 2.2. I'm using e1000 because it is the default and I didn't think it would give me any problems. Is it problematic? Haven't tried vmx3 native yet. And I only applied NAT port forwards after installing open tools so I haven't tested that yet. No packages installed other than open tools. I'm on 5.5.0 1331820.
Oh ok. So since pfsense is Freebsd 10.1 based, I would have to update esxi to 5.5u2 to make sure that everything is compatible. Why didn't pfsense tell us that during installation? Lol.
-
Yeah freebsd 10.1 isn't officially supported on vmware until 5.5u2 – 1331820 that is initial release is it not? You have not applied any patches?
-
Yeah freebsd 10.1 isn't officially supported on vmware until 5.5u2 – 1331820 that is initial release is it not? You have not applied any patches?
Oh ok. I'm not the admin of our vcenter, actually. That's why I can't apply patches easily. But I'll let the admin know about that then.
Thanks.
