Virtual IP alias as gateway for a subnet

mmerlone · Dec 20, 2012, 5:23 PM

I suspect if you check the state table for 8.8.8.8 you'll see that NAT isn't being applied. Normally with NAT being applied you'd see two states, one in, one out, and one of them has three IPs: client -> external -> target and target -> client. If no NAT is being applied, the middle IP doesn't show up.

Indeed, filtering state tables for 8.8.8.8 returns nothing. Also tried to check www.whatismyip.com. from that box, and got the attached states, but yet no luck accessing the address. Checked firewall rules and it should allow anything from 10.0.0.0/24 to access everything.

mmerlone · Dec 20, 2012, 5:32 PM

Some packet capture on WAN interface of pfsense:

15:34:21.369931 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 61191, length 40
15:34:26.870028 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 61447, length 40
15:34:32.370169 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 61703, length 40
15:34:33.106802 IP 10.0.0.8.138 > 10.0.0.255.138: UDP, length 201
15:34:33.106811 IP 10.0.0.8.138 > 10.0.0.255.138: UDP, length 201
15:34:37.870094 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 61959, length 40
15:34:43.370047 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 62215, length 40
15:34:48.870229 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 62471, length 40
15:34:54.370440 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 62727, length 40
15:34:59.870426 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 62983, length 40

jimp · Dec 20, 2012, 5:33 PM

If you are seeing that capture on the WAN interface, then NAT isn't happening.

Make sure the gateway is selected on the WAN interface
Make sure outbound NAT is set to automatic
Then reset the state table again

If it still doesn't work, switch to manual outbound NAT and make your own rules.

mmerlone · Dec 20, 2012, 5:46 PM

@jimp:

If you are seeing that capture on the WAN interface, then NAT isn't happening.

Good to learn.

@jimp:

Make sure the gateway is selected on the WAN interface
Make sure outbound NAT is set to automatic
Then reset the state table again

All done.

@jimp:

If it still doesn't work, ….

Still doesn't work. Why is that? Could some other advanced option interfere? As I said at first, I did not mess anything, started over with default config. Is there any option other than automatic outbound NAT to check?

@jimp:

… switch to manual outbound NAT and make your own rules.

Can you point me some RTFM or assist me? The attached setup did not work, also tried a few variations.

Side info: I also have a NAT'ed ADSL interface, but no matter which gateway I set as default (WAN or ADSL) the problem persists. No interface has the option to block bogon networks checked.

jimp · Dec 20, 2012, 5:53 PM

You didn't disable the firewall under System > Advanced, did you? (That would also disable the ability to do NAT…)

Switch back to automatic outbound NAT, save/apply, and then get a copy of /tmp/rules.debug and copy/paste it here.

The NAT rule you had looked right, that should have been working from what I could see.

mmerlone · Dec 20, 2012, 6:01 PM

@jimp:

You didn't disable the firewall under System > Advanced, did you? (That would also disable the ability to do NAT…)

Nope.

@jimp:

Switch back to automatic outbound NAT, save/apply, and then get a copy of /tmp/rules.debug and copy/paste it here.

#System aliases

loopback = "{ lo0 }"
WAN = "{ xl0 }"
LAN = "{ bge1 }"
ADSL = "{ bge0 }"

#SSH Lockout Table
table <sshlockout>persist
table <webconfiguratorlockout>persist
#Snort tables
table <snort2c>table <virusprot># User Aliases 
table <lanvisitantes>{   10.0.0.0/24 } 
LanVisitantes = "<lanvisitantes>"

# Gateways
GWGWADSL = " route-to ( bge0 192.168.1.100 ) "
GWGWWAN = " route-to ( xl0 x.x.x.153 ) "
GWGWLANNS100 = " route-to ( bge1 192.168.0.254 ) "
GWGWLANRV042 = " route-to ( bge1 192.168.0.251 ) "
GWGW_Group = "  route-to { ( bge0 192.168.1.100 )  }  "

set loginterface bge1
set optimization normal
set limit states 96000
set limit src-nodes 96000

set skip on pfsync0

scrub in on $WAN all    fragment reassemble
scrub in on $LAN all    fragment reassemble
scrub in on $ADSL all    fragment reassemble

no nat proto carp
no rdr proto carp
nat-anchor "natearly/*"
nat-anchor "natrules/*"

# Outbound NAT rules

# Subnets to NAT 

# Load balancing anchor
rdr-anchor "relayd/*"
# TFTP proxy
rdr-anchor "tftp-proxy/*"
table <negate_networks>{ x.x.x.152/29 192.168.0.0/24 192.168.1.0/24 }
# UPnPd rdr anchor
rdr-anchor "miniupnpd"

anchor "relayd/*"
#---------------------------------------------------------------------------
# default deny rules
#---------------------------------------------------------------------------
block in log all label "Default deny rule"
block out log all label "Default deny rule"

# We use the mighty pf, we cannot be fooled.
block quick proto { tcp, udp } from any port = 0 to any
block quick proto { tcp, udp } from any to any port = 0

# Block all IPv6
block in quick inet6 all
block out quick inet6 all

# Snort package
block quick from <snort2c>to any label "Block snort2c hosts"
block quick from any to <snort2c>label "Block snort2c hosts"
block in log quick proto carp from (self) to any
pass quick proto carp
pass quick proto pfsync

# SSH lockout
block in log quick proto tcp from <sshlockout>to any port 22 label "sshlockout"

# webConfigurator lockout
block in log quick proto tcp from <webconfiguratorlockout>to any port 80 label "webConfiguratorlockout"
block in quick from <virusprot>to any label "virusprot overload table"
antispoof for xl0
antispoof for bge1
antispoof for bge0

# loopback
pass in on $loopback all label "pass loopback"
pass out on $loopback all label "pass loopback"
# let out anything from the firewall host itself and decrypted IPsec traffic
pass out all keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( xl0 x.x.x.153 ) from x.x.x.154 to !x.x.x.152/29 keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( bge1 192.168.0.254 ) from 192.168.0.252 to !192.168.0.0/24 keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( bge0 192.168.1.100 ) from 192.168.1.101 to !192.168.1.0/24 keep state allow-opts label "let out anything from firewall host itself"
# make sure the user cannot lock himself out of the webConfigurator or SSH
pass in quick on bge1 proto tcp from any to (bge1) port { 80 22 } keep state label "anti-lockout rule"

# User-defined rules follow

anchor "userrules/*"
pass  on {  xl0  bge1  bge0  }  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $WAN reply-to ( xl0 x.x.x.153 )  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $LAN reply-to ( bge1 192.168.0.254 )  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $LAN reply-to ( bge1 192.168.0.254 )  from 192.168.0.0/24 to any keep state  label "USER_RULE: Default allow LAN to any rule"
pass  in  quick  on $LAN reply-to ( bge1 192.168.0.254 )  from   $LanVisitantes to any keep state  label "USER_RULE"
pass  in  quick  on $LAN reply-to ( bge1 192.168.0.254 )  from any to   $LanVisitantes keep state  label "USER_RULE"
pass  in  quick  on $ADSL reply-to ( bge0 192.168.1.100 )  inet proto icmp  from any to any keep state  label "USER_RULE"

# VPN Rules
anchor "tftp-proxy/*"</virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></lanvisitantes></lanvisitantes></virusprot></snort2c></webconfiguratorlockout></sshlockout>

@jimp:

The NAT rule you had looked right, that should have been working from what I could see.

Oh, God. I'll leave on vacation in 54 minutes anyway. :) Would be so happy if I could make it work this year….

jimp · Dec 20, 2012, 6:05 PM

Hmm there are no NAT rules in that file at all.

The only way that can happen on Automatic Outbound NAT is if the firewall doesn't know there is a WAN, meaning Interfaces > WAN has no gateway selected from the drop-down on that page.

mmerlone · Dec 20, 2012, 6:09 PM

@jimp:

The only way that can happen on Automatic Outbound NAT is if the firewall doesn't know there is a WAN, meaning Interfaces > WAN has no gateway selected from the drop-down on that page.

Seems there is another way…. see attached WAN does have a gateway selected.

jimp · Dec 20, 2012, 6:16 PM

Ah, well there is one more I forgot - if the LAN interface does have a gateway set, it would think that the LAN is a WAN so it wouldn't do NAT.

You don't need nor want to have a gateway set on the LAN interface, and if you have a gateway defined that is actually your LAN IP, remove it.

mmerlone · Dec 20, 2012, 6:30 PM

@jimp:

Ah, well there is one more I forgot - if the LAN interface does have a gateway set, it would think that the LAN is a WAN so it wouldn't do NAT.

Makes total sense, it did have a gateway set.

@jimp:

You don't need nor want to have a gateway set on the LAN interface, and if you have a gateway defined that is actually your LAN IP, remove it.

Sure, sure. Selected 'none' as gateway for LAN interface, reset states, rebooted pfsense and …. (still rebooting) .... nope! No luck, no echo reply from 8.8.8.8 to 10.0.0.8. Partial success though: my workstation on 192.168.0.0/24 does ping 8.8.8.8 and am using it now to post to the forum :)

What the heck! Let me reboot 10.0.0.8 and see what happens, it is a Win XP anyway and my workstation is linux. Rebooted and no luck yet. Dont' forget 10.0.0.8 uses 10.0.0.5 as default gateway, which is a VIP Alias on pfsense.

States and packet capture on WAN interface attached.


16:34:10.530106 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 9216, length 40
16:34:16.030458 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 9472, length 40
16:34:21.532543 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 9728, length 40

mmerlone · Dec 20, 2012, 6:31 PM

Forgot to mention: also deselected a gateway for ADSL interface, just to be sure.

jimp · Dec 20, 2012, 6:35 PM

So it's still not getting NAT, do you see nat rules in /tmp/rules.debug now?

jimp · Dec 20, 2012, 6:37 PM

@mmerlone:

Forgot to mention: also deselected a gateway for ADSL interface, just to be sure.

Why?

WANs must have a gateway set.
LANs must not have a gateway set.

Otherwise automatic outbound NAT will not work.

mmerlone · Dec 20, 2012, 6:45 PM

@jimp:

So it's still not getting NAT, do you see nat rules in /tmp/rules.debug now?

Not yet (oops, regarding 10.0.0.0/24):

#System aliases

loopback = "{ lo0 }"
WAN = "{ xl0 }"
LAN = "{ bge1 }"
ADSL = "{ bge0 }"

#SSH Lockout Table
table <sshlockout>persist
table <webconfiguratorlockout>persist
#Snort tables
table <snort2c>table <virusprot># User Aliases 
table <lanvisitantes>{   10.0.0.0/24 } 
LanVisitantes = "<lanvisitantes>"

# Gateways
GWGWADSL = " route-to ( bge0 192.168.1.100 ) "
GWGWWAN = " route-to ( xl0 177.135.88.153 ) "
GWGWLANNS100 = " route-to ( bge1 192.168.0.254 ) "
GWGWLANRV042 = " route-to ( bge1 192.168.0.251 ) "
GWGW_Group = "  route-to { ( bge0 192.168.1.100 )  }  "

set loginterface bge1
set optimization normal
set limit states 96000
set limit src-nodes 96000

set skip on pfsync0

scrub in on $WAN all    fragment reassemble
scrub in on $LAN all    fragment reassemble
scrub in on $ADSL all    fragment reassemble

no nat proto carp
no rdr proto carp
nat-anchor "natearly/*"
nat-anchor "natrules/*"

# Outbound NAT rules

# Subnets to NAT 
tonatsubnets	= "{ 192.168.0.0/24 192.168.1.0/24 127.0.0.0/8  }"
nat on $WAN  from $tonatsubnets port 500 to any port 500 -> 177.135.88.154/32 port 500  
nat on $WAN  from $tonatsubnets to any -> 177.135.88.154/32 port 1024:65535  

# Load balancing anchor
rdr-anchor "relayd/*"
# TFTP proxy
rdr-anchor "tftp-proxy/*"
table <negate_networks>{ 177.135.88.152/29 192.168.0.0/24 192.168.1.0/24 }
# UPnPd rdr anchor
rdr-anchor "miniupnpd"

anchor "relayd/*"
#---------------------------------------------------------------------------
# default deny rules
#---------------------------------------------------------------------------
block in log all label "Default deny rule"
block out log all label "Default deny rule"

# We use the mighty pf, we cannot be fooled.
block quick proto { tcp, udp } from any port = 0 to any
block quick proto { tcp, udp } from any to any port = 0

# Block all IPv6
block in quick inet6 all
block out quick inet6 all

# Snort package
block quick from <snort2c>to any label "Block snort2c hosts"
block quick from any to <snort2c>label "Block snort2c hosts"
block in log quick proto carp from (self) to any
pass quick proto carp
pass quick proto pfsync

# SSH lockout
block in log quick proto tcp from <sshlockout>to any port 22 label "sshlockout"

# webConfigurator lockout
block in log quick proto tcp from <webconfiguratorlockout>to any port 80 label "webConfiguratorlockout"
block in quick from <virusprot>to any label "virusprot overload table"
antispoof for xl0
antispoof for bge1
antispoof for bge0

# loopback
pass in on $loopback all label "pass loopback"
pass out on $loopback all label "pass loopback"
# let out anything from the firewall host itself and decrypted IPsec traffic
pass out all keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( xl0 177.135.88.153 ) from 177.135.88.154 to !177.135.88.152/29 keep state allow-opts label "let out anything from firewall host itself"
# make sure the user cannot lock himself out of the webConfigurator or SSH
pass in quick on bge1 proto tcp from any to (bge1) port { 80 22 } keep state label "anti-lockout rule"

# User-defined rules follow

anchor "userrules/*"
pass  on {  xl0  bge1  bge0  }  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $WAN reply-to ( xl0 177.135.88.153 )  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $LAN  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $LAN  from 192.168.0.0/24 to any keep state  label "USER_RULE: Default allow LAN to any rule"
pass  in  quick  on $LAN  from   $LanVisitantes to any keep state  label "USER_RULE"
pass  in  quick  on $LAN  from any to   $LanVisitantes keep state  label "USER_RULE"
pass  in  quick  on $ADSL  inet proto icmp  from any to any keep state  label "USER_RULE"

# VPN Rules
anchor "tftp-proxy/*"</virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></lanvisitantes></lanvisitantes></virusprot></snort2c></webconfiguratorlockout></sshlockout>

So, I erased the VIP and created again, without a gateway on LAN interface, reset states, and now 10.0.0.8 cannot even ping 10.0.0.5. Packets and state table:

16:45:12.039107 ARP, Request who-has 10.0.0.8 tell 10.0.0.5, length 28
16:45:17.539196 ARP, Request who-has 10.0.0.8 tell 10.0.0.5, length 28

jimp · Dec 20, 2012, 6:45 PM

Closer..

# Subnets to NAT 
tonatsubnets	= "{ 192.168.0.0/24 192.168.1.0/24 127.0.0.0/8  }"

It is getting your other subnets but not the 10.x.x.x one.

Make sure the IP alias VIP has the proper subnet mask set (not /32).

mmerlone · Dec 20, 2012, 6:48 PM

@jimp:

It is getting your other subnets but not the 10.x.x.x one.
Make sure the IP alias VIP has the proper subnet mask set (not /32).

/24

jimp · Dec 20, 2012, 6:52 PM

And that IP alias is on the correct interface? (LAN)

Check Diagnostics > Routes, paste the output here, it might give some more clues.

mmerlone · Dec 20, 2012, 6:58 PM

@jimp:

And that IP alias is on the correct interface? (LAN)

Oops, well noted. Was not. Corrected and now 10.0.0.8 can ping 10.0.0.5, but not yet 8.8.8.8.

@jimp:

Check Diagnostics > Routes, paste the output here, it might give some more clues.

Attached.

jimp · Dec 20, 2012, 7:00 PM

OK that all looks good now - and if you check /tmp/rules.debug and look for tonatsubnets - the 10.x.x.x network still doesn't show up?

If not, then you probably will need to go to outbound NAT, delete any rules there, and switch to manual outbound NAT once more. Then add a rule for 10.x.x.x.

mmerlone · Dec 20, 2012, 7:16 PM

@jimp:

OK that all looks good now - and if you check /tmp/rules.debug and look for tonatsubnets - the 10.x.x.x network still doesn't show up?

Nope:

# Subnets to NAT 
tonatsubnets	= "{ 192.168.0.0/24 192.168.1.0/24 127.0.0.0/8  }"
nat on $WAN  from $tonatsubnets port 500 to any port 500 -> x.x.x.154/32 port 500  
nat on $WAN  from $tonatsubnets to any -> x.x.x.154/32 port 1024:65535

@jimp:

If not, then you probably will need to go to outbound NAT, delete any rules there, and switch to manual outbound NAT once more. Then add a rule for 10.x.x.x.

Why can't that be automatic? Why life can't be easy? Why there's no Santa Claus?
Did not delete existing rules, they look good for me. I just added new NAT rule, reset states, and this time got luck, 10.0.0.8 now pings 8.8.8.8.

Now the questions:
Is that a bug or feature?
If not a feature, where can I request it?

This box will replace an old Netscreen, 1:1 NATing a bunch of servers, responsible for our internet presence, connect two remote offices, provide IPsec VPN for mobile warriors, and some more I don't remember now.

Will leave on vacation now and get back on january 7 to finish this. Happy holidays, merry Christmas, happy end-of-the-world, happy new year, and many thanks for your almost-chat support. I was almost ditching pfsense.

Best regards,

–
Marcio Merlone