a somewhat strange problem with VIP 1:1 NAT reachability

eleitl

I've got a /28 and 3x /24 the latter of which I want to map
1:1 to private address space (e.g. each /24 to 10.0.x.0/24).

My pfSense version is 2.1-DEVELOPMENT (i386)
built on Fri Oct 21 12:51:56 EDT 2011

I've got a setup where hosts and firewalls look like this

WAN LAN
switch1 - fw1 - switch2
| - fw2 - |
| - ho1 - |
| - … - |

The hosts have two NICs (for WAN and LAN) and have currently
virtual guests bound the WAN interface, thus bypassing the
firewalls. I now want to renumber the guests to private
address space, and let the firewall deal with network
mapping and traffic filtering.

The lan switch is managed, and is assigned an address
10.0.0.x via fw1 DHCP.

I've defined VIPs (type Proxy ARP, matching one public /24)
and created a Firewall NAT 1:1 mapping as well as firewall
rules to pass the relevant traffic. I've disabled the
WAN switch port (by putting it on an unreachable VLAN)
so that virtual guests bound to the same /24 are out of
the loop for testing. I have not disabled any
other hosts on the switch.

Now the strange thing is that I can see the switch IP via
the external mapped IP (two of them, in fact), but nothing
else (but the gateway). nmap from within the network sees
the other IPs fine.

This makes absolutely no sense. There must be something simple
I'm missing. Any idea how to debug this? Thanks!

My rules look like this:

cat /tmp/rules.debug

#System aliases

loopback = "{ lo0 }"
LAN = "{ igb0 }"
WAN = "{ em0 }"
OPT1 = "{ igb1 }"

#SSH Lockout Table
table <sshlockout>persist
table <webconfiguratorlockout>persist
#Snort tables
table <snort2c>table <virusprot># User Aliases

Gateways

GWGW_WAN = " route-to ( em0 88.198.239.113 ) "
GWWANGWv6 = " route-to ( em0 2a01:4f8:7d:300::1 ) "

set loginterface igb0
set optimization normal
set limit states 299000
set limit src-nodes 299000

set skip on pfsync0

scrub in on $LAN all fragment reassemble
scrub in on $WAN all fragment reassemble
scrub in on $OPT1 all fragment reassemble

no nat proto carp
no rdr proto carp
nat-anchor "natearly/"
nat-anchor "natrules/"

binat on em0 from 10.0.0.0/24 to any -> 88.198.222.0/24
binat on em0 from 10.0.0.20 to any -> 88.198.239.118

Outbound NAT rules

nat on $WAN from 10.0.0.0/24 to any port 500 -> 88.198.239.114/32 static-port
nat on $WAN from 10.0.0.0/24 to any -> 88.198.239.114/32 port 1024:65535
nat on $WAN from 127.0.0.0/8 to any -> 88.198.239.114/32 port 1024:65535
nat on $WAN from 172.16.1.0/24 to any port 500 -> 88.198.239.114/32 static-port
nat on $WAN from 172.16.1.0/24 to any -> 88.198.239.114/32 port 1024:65535
nat on $WAN from 127.0.0.0/8 to any -> 88.198.239.114/32 port 1024:65535

Load balancing anchor

rdr-anchor "relayd/*"

TFTP proxy

rdr-anchor "tftp-proxy/*"
table <negate_networks>{ 10.0.0.0/24 88.198.239.112/28 172.16.1.0/24 }

UPnPd rdr anchor

rdr-anchor "miniupnpd"

anchor "relayd/*"
#---------------------------------------------------------------------------

default deny rules

#---------------------------------------------------------------------------
block in log inet all label "Default deny rule IPv4"
block out log inet all label "Default deny rule IPv4"
block in log inet6 all label "Default deny rule IPv6"
block out log inet6 all label "Default deny rule IPv6"

IPv6 ICMP is not auxilary, it is required for operation

See man icmp6(4)

1 unreach Destination unreachable

2 toobig Packet too big

128 echoreq Echo service request

129 echorep Echo service reply

133 routersol Router solicitation

134 routeradv Router advertisement

135 neighbrsol Neighbor solicitation

136 neighbradv Neighbor advertisement

pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} keep state

Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)

pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} keep state
pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} keep state
pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} keep state
pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {129,133,134,135,136} keep state
pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} keep state

We use the mighty pf, we cannot be fooled.

block quick inet proto { tcp, udp } from any port = 0 to any
block quick inet proto { tcp, udp } from any to any port = 0
block quick inet6 proto { tcp, udp } from any port = 0 to any
block quick inet6 proto { tcp, udp } from any to any port = 0

Snort package

block quick from <snort2c>to any label "Block snort2c hosts"
block quick from any to <snort2c>label "Block snort2c hosts"
block in log quick proto carp from (self) to any
pass quick proto carp
pass quick proto pfsync

SSH lockout

block in log quick proto tcp from <sshlockout>to any port 22 label "sshlockout"

webConfigurator lockout

block in log quick proto tcp from <webconfiguratorlockout>to any port 80 label "webConfiguratorlockout"
block in quick from <virusprot>to any label "virusprot overload table"
antispoof for igb0

allow access to DHCP server on LAN

pass in on $LAN proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server"
pass in on $LAN proto udp from any port = 68 to 10.0.0.1 port = 67 label "allow access to DHCP server"
pass out on $LAN proto udp from 10.0.0.1 port = 67 to any port = 68 label "allow access to DHCP server"
table <bogons>persist file "/etc/bogons"
table <bogonsv6>persist file "/etc/bogonsv6"

block bogon networks

http://www.cymru.com/Documents/bogon-bn-nonagg.txt

http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt

block in log quick on $WAN from <bogons>to any label "block bogon IPv4 networks from WAN"
block in log quick on $WAN from <bogonsv6>to any label "block bogon IPv6 networks from WAN"
antispoof for em0

block anything from private networks on interfaces with the option set

antispoof for $WAN
block in log quick on $WAN from 10.0.0.0/8 to any label "Block private networks from WAN block 10/8"
block in log quick on $WAN from 127.0.0.0/8 to any label "Block private networks from WAN block 127/8"
block in log quick on $WAN from 172.16.0.0/12 to any label "Block private networks from WAN block 172.16/12"
block in log quick on $WAN from 192.168.0.0/16 to any label "Block private networks from WAN block 192.168/16"
block in log quick on $WAN from fc00::/7 to any label "Block ULA networks from WAN block fc00::/7"
antispoof for igb1

loopback

pass in on $loopback inet all label "pass IPv4 loopback"
pass out on $loopback inet all label "pass IPv4 loopback"
pass in on $loopback inet6 all label "pass IPv6 loopback"
pass out on $loopback inet6 all label "pass IPv6 loopback"

let out anything from the firewall host itself and decrypted IPsec traffic

pass out inet all keep state allow-opts label "let out anything IPv4 from firewall host itself"
pass out inet6 all keep state allow-opts label "let out anything IPv6 from firewall host itself"
pass out route-to ( em0 88.198.239.113 ) from 88.198.239.114 to !88.198.239.112/28 keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( em0 2a01:4f8:7d:300::1 ) inet6 from 2a01:4f8:7d:300::2 to !2a01:4f8:7d:300:0:0:0:0/56 keep state allow-opts label "let out anything from firewall host itself"

make sure the user cannot lock himself out of the webConfigurator or SSH

pass in quick on igb0 proto tcp from any to (igb0) port { 80 22 } keep state label "anti-lockout rule"

User-defined rules follow

anchor "userrules/*"
pass in quick on $WAN reply-to ( em0 88.198.239.113 ) proto icmp from any to any keep state label "USER_RULE: allow ICMP ping from WAN"
pass in quick on $WAN reply-to ( em0 2a01:4f8:7d:300::1 ) inet6 proto ipv6-icmp from any to any keep state label "USER_RULE: allow IPv6 ICMP ping from WAN"
pass in quick on $WAN reply-to ( em0 88.198.239.113 ) proto tcp from any to 88.198.239.114 port 22 flags S/SA keep state label "USER_RULE: allow SSH administration on WAN"
pass in quick on $WAN reply-to ( em0 88.198.239.113 ) proto tcp from any to 88.198.239.114 port 80 flags S/SA keep state label "USER_RULE: allow HTTP administration on WAN"
pass in quick on $WAN reply-to ( em0 88.198.239.113 ) proto tcp from any to 88.198.239.114 port 443 flags S/SA keep state label "USER_RULE: allow HTTPS administration on WAN"
pass in quick on $WAN reply-to ( em0 88.198.239.113 ) inet proto { tcp udp } from any to 88.198.239.118 port 80 keep state label "USER_RULE: NAT "
pass in quick on $WAN reply-to ( em0 88.198.239.113 ) inet proto { tcp udp } from any to 88.198.222.0/24 keep state label "USER_RULE: NAT "
pass in quick on $WAN reply-to ( em0 88.198.239.113 ) inet proto { tcp udp } from any to 10.0.0.20 keep state label "USER_RULE"
pass in quick on $WAN reply-to ( em0 88.198.239.113 ) inet proto { tcp udp } from any to 10.0.0.2 keep state label "USER_RULE"
pass in quick on $WAN reply-to ( em0 88.198.239.113 ) inet proto { tcp udp } from any to 10.0.0.3 keep state label "USER_RULE"
pass in quick on $OPT1 from any to any keep state label "USER_RULE"

VPN Rules

anchor "tftp-proxy/*"</bogonsv6></bogons></bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></virusprot></snort2c></webconfiguratorlockout></sshlockout>