Virtual IP alias as gateway for a subnet

jimp

You didn't disable the firewall under System > Advanced, did you? (That would also disable the ability to do NAT…)

Switch back to automatic outbound NAT, save/apply, and then get a copy of /tmp/rules.debug and copy/paste it here.

The NAT rule you had looked right, that should have been working from what I could see.

mmerlone

@jimp:

You didn't disable the firewall under System > Advanced, did you? (That would also disable the ability to do NAT…)

Nope.

@jimp:

Switch back to automatic outbound NAT, save/apply, and then get a copy of /tmp/rules.debug and copy/paste it here.

#System aliases

loopback = "{ lo0 }"
WAN = "{ xl0 }"
LAN = "{ bge1 }"
ADSL = "{ bge0 }"

#SSH Lockout Table
table <sshlockout>persist
table <webconfiguratorlockout>persist
#Snort tables
table <snort2c>table <virusprot># User Aliases 
table <lanvisitantes>{   10.0.0.0/24 } 
LanVisitantes = "<lanvisitantes>"

# Gateways
GWGWADSL = " route-to ( bge0 192.168.1.100 ) "
GWGWWAN = " route-to ( xl0 x.x.x.153 ) "
GWGWLANNS100 = " route-to ( bge1 192.168.0.254 ) "
GWGWLANRV042 = " route-to ( bge1 192.168.0.251 ) "
GWGW_Group = "  route-to { ( bge0 192.168.1.100 )  }  "

set loginterface bge1
set optimization normal
set limit states 96000
set limit src-nodes 96000

set skip on pfsync0

scrub in on $WAN all    fragment reassemble
scrub in on $LAN all    fragment reassemble
scrub in on $ADSL all    fragment reassemble

no nat proto carp
no rdr proto carp
nat-anchor "natearly/*"
nat-anchor "natrules/*"

# Outbound NAT rules

# Subnets to NAT 

# Load balancing anchor
rdr-anchor "relayd/*"
# TFTP proxy
rdr-anchor "tftp-proxy/*"
table <negate_networks>{ x.x.x.152/29 192.168.0.0/24 192.168.1.0/24 }
# UPnPd rdr anchor
rdr-anchor "miniupnpd"

anchor "relayd/*"
#---------------------------------------------------------------------------
# default deny rules
#---------------------------------------------------------------------------
block in log all label "Default deny rule"
block out log all label "Default deny rule"

# We use the mighty pf, we cannot be fooled.
block quick proto { tcp, udp } from any port = 0 to any
block quick proto { tcp, udp } from any to any port = 0

# Block all IPv6
block in quick inet6 all
block out quick inet6 all

# Snort package
block quick from <snort2c>to any label "Block snort2c hosts"
block quick from any to <snort2c>label "Block snort2c hosts"
block in log quick proto carp from (self) to any
pass quick proto carp
pass quick proto pfsync

# SSH lockout
block in log quick proto tcp from <sshlockout>to any port 22 label "sshlockout"

# webConfigurator lockout
block in log quick proto tcp from <webconfiguratorlockout>to any port 80 label "webConfiguratorlockout"
block in quick from <virusprot>to any label "virusprot overload table"
antispoof for xl0
antispoof for bge1
antispoof for bge0

# loopback
pass in on $loopback all label "pass loopback"
pass out on $loopback all label "pass loopback"
# let out anything from the firewall host itself and decrypted IPsec traffic
pass out all keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( xl0 x.x.x.153 ) from x.x.x.154 to !x.x.x.152/29 keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( bge1 192.168.0.254 ) from 192.168.0.252 to !192.168.0.0/24 keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( bge0 192.168.1.100 ) from 192.168.1.101 to !192.168.1.0/24 keep state allow-opts label "let out anything from firewall host itself"
# make sure the user cannot lock himself out of the webConfigurator or SSH
pass in quick on bge1 proto tcp from any to (bge1) port { 80 22 } keep state label "anti-lockout rule"

# User-defined rules follow

anchor "userrules/*"
pass  on {  xl0  bge1  bge0  }  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $WAN reply-to ( xl0 x.x.x.153 )  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $LAN reply-to ( bge1 192.168.0.254 )  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $LAN reply-to ( bge1 192.168.0.254 )  from 192.168.0.0/24 to any keep state  label "USER_RULE: Default allow LAN to any rule"
pass  in  quick  on $LAN reply-to ( bge1 192.168.0.254 )  from   $LanVisitantes to any keep state  label "USER_RULE"
pass  in  quick  on $LAN reply-to ( bge1 192.168.0.254 )  from any to   $LanVisitantes keep state  label "USER_RULE"
pass  in  quick  on $ADSL reply-to ( bge0 192.168.1.100 )  inet proto icmp  from any to any keep state  label "USER_RULE"

# VPN Rules
anchor "tftp-proxy/*"</virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></lanvisitantes></lanvisitantes></virusprot></snort2c></webconfiguratorlockout></sshlockout>

@jimp:

The NAT rule you had looked right, that should have been working from what I could see.

Oh, God. I'll leave on vacation in 54 minutes anyway. :) Would be so happy if I could make it work this year….

jimp

Hmm there are no NAT rules in that file at all.

The only way that can happen on Automatic Outbound NAT is if the firewall doesn't know there is a WAN, meaning Interfaces > WAN has no gateway selected from the drop-down on that page.

mmerlone

@jimp:

The only way that can happen on Automatic Outbound NAT is if the firewall doesn't know there is a WAN, meaning Interfaces > WAN has no gateway selected from the drop-down on that page.

Seems there is another way…. see attached WAN does have a gateway selected.

2012-12-20_16-11-51.png_thumb

jimp

Ah, well there is one more I forgot - if the LAN interface does have a gateway set, it would think that the LAN is a WAN so it wouldn't do NAT.

You don't need nor want to have a gateway set on the LAN interface, and if you have a gateway defined that is actually your LAN IP, remove it.

mmerlone

@jimp:

Ah, well there is one more I forgot - if the LAN interface does have a gateway set, it would think that the LAN is a WAN so it wouldn't do NAT.

Makes total sense, it did have a gateway set.

@jimp:

You don't need nor want to have a gateway set on the LAN interface, and if you have a gateway defined that is actually your LAN IP, remove it.

Sure, sure. Selected 'none' as gateway for LAN interface, reset states, rebooted pfsense and …. (still rebooting) .... nope! No luck, no echo reply from 8.8.8.8 to 10.0.0.8. Partial success though: my workstation on 192.168.0.0/24 does ping 8.8.8.8 and am using it now to post to the forum :)

What the heck! Let me reboot 10.0.0.8 and see what happens, it is a Win XP anyway and my workstation is linux. Rebooted and no luck yet. Dont' forget 10.0.0.8 uses 10.0.0.5 as default gateway, which is a VIP Alias on pfsense.

States and packet capture on WAN interface attached.


16:34:10.530106 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 9216, length 40
16:34:16.030458 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 9472, length 40
16:34:21.532543 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 9728, length 40

2012-12-20_16-31-56.png_thumb

mmerlone

Forgot to mention: also deselected a gateway for ADSL interface, just to be sure.

jimp

So it's still not getting NAT, do you see nat rules in /tmp/rules.debug now?

jimp

@mmerlone:

Forgot to mention: also deselected a gateway for ADSL interface, just to be sure.

Why?

WANs must have a gateway set.
LANs must not have a gateway set.

Otherwise automatic outbound NAT will not work.

mmerlone

@jimp:

So it's still not getting NAT, do you see nat rules in /tmp/rules.debug now?

Not yet (oops, regarding 10.0.0.0/24):

#System aliases

loopback = "{ lo0 }"
WAN = "{ xl0 }"
LAN = "{ bge1 }"
ADSL = "{ bge0 }"

#SSH Lockout Table
table <sshlockout>persist
table <webconfiguratorlockout>persist
#Snort tables
table <snort2c>table <virusprot># User Aliases 
table <lanvisitantes>{   10.0.0.0/24 } 
LanVisitantes = "<lanvisitantes>"

# Gateways
GWGWADSL = " route-to ( bge0 192.168.1.100 ) "
GWGWWAN = " route-to ( xl0 177.135.88.153 ) "
GWGWLANNS100 = " route-to ( bge1 192.168.0.254 ) "
GWGWLANRV042 = " route-to ( bge1 192.168.0.251 ) "
GWGW_Group = "  route-to { ( bge0 192.168.1.100 )  }  "

set loginterface bge1
set optimization normal
set limit states 96000
set limit src-nodes 96000

set skip on pfsync0

scrub in on $WAN all    fragment reassemble
scrub in on $LAN all    fragment reassemble
scrub in on $ADSL all    fragment reassemble

no nat proto carp
no rdr proto carp
nat-anchor "natearly/*"
nat-anchor "natrules/*"

# Outbound NAT rules

# Subnets to NAT 
tonatsubnets	= "{ 192.168.0.0/24 192.168.1.0/24 127.0.0.0/8  }"
nat on $WAN  from $tonatsubnets port 500 to any port 500 -> 177.135.88.154/32 port 500  
nat on $WAN  from $tonatsubnets to any -> 177.135.88.154/32 port 1024:65535  

# Load balancing anchor
rdr-anchor "relayd/*"
# TFTP proxy
rdr-anchor "tftp-proxy/*"
table <negate_networks>{ 177.135.88.152/29 192.168.0.0/24 192.168.1.0/24 }
# UPnPd rdr anchor
rdr-anchor "miniupnpd"

anchor "relayd/*"
#---------------------------------------------------------------------------
# default deny rules
#---------------------------------------------------------------------------
block in log all label "Default deny rule"
block out log all label "Default deny rule"

# We use the mighty pf, we cannot be fooled.
block quick proto { tcp, udp } from any port = 0 to any
block quick proto { tcp, udp } from any to any port = 0

# Block all IPv6
block in quick inet6 all
block out quick inet6 all

# Snort package
block quick from <snort2c>to any label "Block snort2c hosts"
block quick from any to <snort2c>label "Block snort2c hosts"
block in log quick proto carp from (self) to any
pass quick proto carp
pass quick proto pfsync

# SSH lockout
block in log quick proto tcp from <sshlockout>to any port 22 label "sshlockout"

# webConfigurator lockout
block in log quick proto tcp from <webconfiguratorlockout>to any port 80 label "webConfiguratorlockout"
block in quick from <virusprot>to any label "virusprot overload table"
antispoof for xl0
antispoof for bge1
antispoof for bge0

# loopback
pass in on $loopback all label "pass loopback"
pass out on $loopback all label "pass loopback"
# let out anything from the firewall host itself and decrypted IPsec traffic
pass out all keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( xl0 177.135.88.153 ) from 177.135.88.154 to !177.135.88.152/29 keep state allow-opts label "let out anything from firewall host itself"
# make sure the user cannot lock himself out of the webConfigurator or SSH
pass in quick on bge1 proto tcp from any to (bge1) port { 80 22 } keep state label "anti-lockout rule"

# User-defined rules follow

anchor "userrules/*"
pass  on {  xl0  bge1  bge0  }  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $WAN reply-to ( xl0 177.135.88.153 )  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $LAN  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $LAN  from 192.168.0.0/24 to any keep state  label "USER_RULE: Default allow LAN to any rule"
pass  in  quick  on $LAN  from   $LanVisitantes to any keep state  label "USER_RULE"
pass  in  quick  on $LAN  from any to   $LanVisitantes keep state  label "USER_RULE"
pass  in  quick  on $ADSL  inet proto icmp  from any to any keep state  label "USER_RULE"

# VPN Rules
anchor "tftp-proxy/*"</virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></lanvisitantes></lanvisitantes></virusprot></snort2c></webconfiguratorlockout></sshlockout>

So, I erased the VIP and created again, without a gateway on LAN interface, reset states, and now 10.0.0.8 cannot even ping 10.0.0.5. Packets and state table:

16:45:12.039107 ARP, Request who-has 10.0.0.8 tell 10.0.0.5, length 28
16:45:17.539196 ARP, Request who-has 10.0.0.8 tell 10.0.0.5, length 28

2012-12-20_16-44-59.png_thumb

jimp

Closer..

# Subnets to NAT 
tonatsubnets	= "{ 192.168.0.0/24 192.168.1.0/24 127.0.0.0/8  }"

It is getting your other subnets but not the 10.x.x.x one.

Make sure the IP alias VIP has the proper subnet mask set (not /32).

mmerlone

@jimp:

It is getting your other subnets but not the 10.x.x.x one.
Make sure the IP alias VIP has the proper subnet mask set (not /32).

/24

2012-12-20_16-52-58.png_thumb

jimp

And that IP alias is on the correct interface? (LAN)

Check Diagnostics > Routes, paste the output here, it might give some more clues.

mmerlone

@jimp:

And that IP alias is on the correct interface? (LAN)

Oops, well noted. Was not. Corrected and now 10.0.0.8 can ping 10.0.0.5, but not yet 8.8.8.8.

@jimp:

Check Diagnostics > Routes, paste the output here, it might give some more clues.

Attached.

2012-12-20_17-01-23.png_thumb

jimp

OK that all looks good now - and if you check /tmp/rules.debug and look for tonatsubnets - the 10.x.x.x network still doesn't show up?

If not, then you probably will need to go to outbound NAT, delete any rules there, and switch to manual outbound NAT once more. Then add a rule for 10.x.x.x.

mmerlone

@jimp:

OK that all looks good now - and if you check /tmp/rules.debug and look for tonatsubnets - the 10.x.x.x network still doesn't show up?

Nope:

# Subnets to NAT 
tonatsubnets	= "{ 192.168.0.0/24 192.168.1.0/24 127.0.0.0/8  }"
nat on $WAN  from $tonatsubnets port 500 to any port 500 -> x.x.x.154/32 port 500  
nat on $WAN  from $tonatsubnets to any -> x.x.x.154/32 port 1024:65535

@jimp:

If not, then you probably will need to go to outbound NAT, delete any rules there, and switch to manual outbound NAT once more. Then add a rule for 10.x.x.x.

Why can't that be automatic? Why life can't be easy? Why there's no Santa Claus?
Did not delete existing rules, they look good for me. I just added new NAT rule, reset states, and this time got luck, 10.0.0.8 now pings 8.8.8.8.

Now the questions:
Is that a bug or feature?
If not a feature, where can I request it?

This box will replace an old Netscreen, 1:1 NATing a bunch of servers, responsible for our internet presence, connect two remote offices, provide IPsec VPN for mobile warriors, and some more I don't remember now.

Will leave on vacation now and get back on january 7 to finish this. Happy holidays, merry Christmas, happy end-of-the-world, happy new year, and many thanks for your almost-chat support. I was almost ditching pfsense.

Best regards,

–
Marcio Merlone

jimp

I had thought it was automatic, but it's apparently not (at least on the version you're using, I don't know about 2.1)

Using multiple subnets on a single interface in that was is still mostly an edge case. Most people put distinct subnets on separate VLANs or interfaces.