Virtual IP alias as gateway for a subnet

jimp · Dec 20, 2012, 1:20 PM

Yes, the only considerations you'd have would be:

Add the IP Alias VIP to the firewall (Make sure to use the correct subnet mask)
Adjust your firewall rules to pass traffic from a source of the second subnet
If you are on manual outbound NAT, make sure your outbound NAT rules cover a source of the new subnet

mmerlone · Dec 20, 2012, 5:02 PM

So, I just wiped out the discs and started over with a clean config. This time did not hit the bug where install hangs on 38% of /usr (it is a Dell Poweredge 860, SAS5/i mirror on SATA discs). Nice…

@jimp:

Add the IP Alias VIP to the firewall (Make sure to use the correct subnet mask)

Done.

@jimp:

Adjust your firewall rules to pass traffic from a source of the second subnet

Done. Also made a top rule on each interface plus a floating one to allow ICMP in any direction.

@jimp:

If you are on manual outbound NAT, make sure your outbound NAT rules cover a source of the new subnet

Did not touch this, Outbound NAT is automatic. All config defaults.

Result:
1. pfsense is able to ping 8.8.8.8.
2. A client machine on 10.0.0.0/24 segment IS able to ping another on 192.168.0.0/24, so routing is working.
3. A client machine on 10.0.0.0/24 segment is NOT able to ping 8.8.8.8.
4. A client machine on 192.168.0.0/24 (my workstation) segment is NOT able to ping 8.8.8.8.

Both 10.0.0.0/24 and 192.168.0.0/24 are on the same ethernet bus and broadcast domain, does it matter?
I have reset states table (Diagnostics -> States -> Reset states), no luck in any box.

What next?

jimp · Dec 20, 2012, 5:04 PM

Shouldn't matter, so long as the WAN interface has a gateway selected, which indicates to the system that it needs to NAT out on there, it should work with automatic outbound NAT.

I suspect if you check the state table for 8.8.8.8 you'll see that NAT isn't being applied. Normally with NAT being applied you'd see two states, one in, one out, and one of them has three IPs: client -> external -> target and target -> client. If no NAT is being applied, the middle IP doesn't show up.

mmerlone · Dec 20, 2012, 5:23 PM

@jimp:

I suspect if you check the state table for 8.8.8.8 you'll see that NAT isn't being applied. Normally with NAT being applied you'd see two states, one in, one out, and one of them has three IPs: client -> external -> target and target -> client. If no NAT is being applied, the middle IP doesn't show up.

Indeed, filtering state tables for 8.8.8.8 returns nothing. Also tried to check www.whatismyip.com. from that box, and got the attached states, but yet no luck accessing the address. Checked firewall rules and it should allow anything from 10.0.0.0/24 to access everything.

mmerlone · Dec 20, 2012, 5:32 PM

Some packet capture on WAN interface of pfsense:

15:34:21.369931 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 61191, length 40
15:34:26.870028 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 61447, length 40
15:34:32.370169 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 61703, length 40
15:34:33.106802 IP 10.0.0.8.138 > 10.0.0.255.138: UDP, length 201
15:34:33.106811 IP 10.0.0.8.138 > 10.0.0.255.138: UDP, length 201
15:34:37.870094 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 61959, length 40
15:34:43.370047 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 62215, length 40
15:34:48.870229 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 62471, length 40
15:34:54.370440 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 62727, length 40
15:34:59.870426 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 62983, length 40

jimp · Dec 20, 2012, 5:33 PM

If you are seeing that capture on the WAN interface, then NAT isn't happening.

Make sure the gateway is selected on the WAN interface
Make sure outbound NAT is set to automatic
Then reset the state table again

If it still doesn't work, switch to manual outbound NAT and make your own rules.

mmerlone · Dec 20, 2012, 5:46 PM

@jimp:

If you are seeing that capture on the WAN interface, then NAT isn't happening.

Good to learn.

@jimp:

Make sure the gateway is selected on the WAN interface
Make sure outbound NAT is set to automatic
Then reset the state table again

All done.

@jimp:

If it still doesn't work, ….

Still doesn't work. Why is that? Could some other advanced option interfere? As I said at first, I did not mess anything, started over with default config. Is there any option other than automatic outbound NAT to check?

@jimp:

… switch to manual outbound NAT and make your own rules.

Can you point me some RTFM or assist me? The attached setup did not work, also tried a few variations.

Side info: I also have a NAT'ed ADSL interface, but no matter which gateway I set as default (WAN or ADSL) the problem persists. No interface has the option to block bogon networks checked.

jimp · Dec 20, 2012, 5:53 PM

You didn't disable the firewall under System > Advanced, did you? (That would also disable the ability to do NAT…)

Switch back to automatic outbound NAT, save/apply, and then get a copy of /tmp/rules.debug and copy/paste it here.

The NAT rule you had looked right, that should have been working from what I could see.

mmerlone · Dec 20, 2012, 6:01 PM

@jimp:

You didn't disable the firewall under System > Advanced, did you? (That would also disable the ability to do NAT…)

Nope.

@jimp:

Switch back to automatic outbound NAT, save/apply, and then get a copy of /tmp/rules.debug and copy/paste it here.

#System aliases

loopback = "{ lo0 }"
WAN = "{ xl0 }"
LAN = "{ bge1 }"
ADSL = "{ bge0 }"

#SSH Lockout Table
table <sshlockout>persist
table <webconfiguratorlockout>persist
#Snort tables
table <snort2c>table <virusprot># User Aliases 
table <lanvisitantes>{   10.0.0.0/24 } 
LanVisitantes = "<lanvisitantes>"

# Gateways
GWGWADSL = " route-to ( bge0 192.168.1.100 ) "
GWGWWAN = " route-to ( xl0 x.x.x.153 ) "
GWGWLANNS100 = " route-to ( bge1 192.168.0.254 ) "
GWGWLANRV042 = " route-to ( bge1 192.168.0.251 ) "
GWGW_Group = "  route-to { ( bge0 192.168.1.100 )  }  "

set loginterface bge1
set optimization normal
set limit states 96000
set limit src-nodes 96000

set skip on pfsync0

scrub in on $WAN all    fragment reassemble
scrub in on $LAN all    fragment reassemble
scrub in on $ADSL all    fragment reassemble

no nat proto carp
no rdr proto carp
nat-anchor "natearly/*"
nat-anchor "natrules/*"

# Outbound NAT rules

# Subnets to NAT 

# Load balancing anchor
rdr-anchor "relayd/*"
# TFTP proxy
rdr-anchor "tftp-proxy/*"
table <negate_networks>{ x.x.x.152/29 192.168.0.0/24 192.168.1.0/24 }
# UPnPd rdr anchor
rdr-anchor "miniupnpd"

anchor "relayd/*"
#---------------------------------------------------------------------------
# default deny rules
#---------------------------------------------------------------------------
block in log all label "Default deny rule"
block out log all label "Default deny rule"

# We use the mighty pf, we cannot be fooled.
block quick proto { tcp, udp } from any port = 0 to any
block quick proto { tcp, udp } from any to any port = 0

# Block all IPv6
block in quick inet6 all
block out quick inet6 all

# Snort package
block quick from <snort2c>to any label "Block snort2c hosts"
block quick from any to <snort2c>label "Block snort2c hosts"
block in log quick proto carp from (self) to any
pass quick proto carp
pass quick proto pfsync

# SSH lockout
block in log quick proto tcp from <sshlockout>to any port 22 label "sshlockout"

# webConfigurator lockout
block in log quick proto tcp from <webconfiguratorlockout>to any port 80 label "webConfiguratorlockout"
block in quick from <virusprot>to any label "virusprot overload table"
antispoof for xl0
antispoof for bge1
antispoof for bge0

# loopback
pass in on $loopback all label "pass loopback"
pass out on $loopback all label "pass loopback"
# let out anything from the firewall host itself and decrypted IPsec traffic
pass out all keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( xl0 x.x.x.153 ) from x.x.x.154 to !x.x.x.152/29 keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( bge1 192.168.0.254 ) from 192.168.0.252 to !192.168.0.0/24 keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( bge0 192.168.1.100 ) from 192.168.1.101 to !192.168.1.0/24 keep state allow-opts label "let out anything from firewall host itself"
# make sure the user cannot lock himself out of the webConfigurator or SSH
pass in quick on bge1 proto tcp from any to (bge1) port { 80 22 } keep state label "anti-lockout rule"

# User-defined rules follow

anchor "userrules/*"
pass  on {  xl0  bge1  bge0  }  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $WAN reply-to ( xl0 x.x.x.153 )  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $LAN reply-to ( bge1 192.168.0.254 )  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $LAN reply-to ( bge1 192.168.0.254 )  from 192.168.0.0/24 to any keep state  label "USER_RULE: Default allow LAN to any rule"
pass  in  quick  on $LAN reply-to ( bge1 192.168.0.254 )  from   $LanVisitantes to any keep state  label "USER_RULE"
pass  in  quick  on $LAN reply-to ( bge1 192.168.0.254 )  from any to   $LanVisitantes keep state  label "USER_RULE"
pass  in  quick  on $ADSL reply-to ( bge0 192.168.1.100 )  inet proto icmp  from any to any keep state  label "USER_RULE"

# VPN Rules
anchor "tftp-proxy/*"</virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></lanvisitantes></lanvisitantes></virusprot></snort2c></webconfiguratorlockout></sshlockout>

@jimp:

The NAT rule you had looked right, that should have been working from what I could see.

Oh, God. I'll leave on vacation in 54 minutes anyway. :) Would be so happy if I could make it work this year….

jimp · Dec 20, 2012, 6:05 PM

Hmm there are no NAT rules in that file at all.

The only way that can happen on Automatic Outbound NAT is if the firewall doesn't know there is a WAN, meaning Interfaces > WAN has no gateway selected from the drop-down on that page.

mmerlone · Dec 20, 2012, 6:09 PM

@jimp:

The only way that can happen on Automatic Outbound NAT is if the firewall doesn't know there is a WAN, meaning Interfaces > WAN has no gateway selected from the drop-down on that page.

Seems there is another way…. see attached WAN does have a gateway selected.

jimp · Dec 20, 2012, 6:16 PM

Ah, well there is one more I forgot - if the LAN interface does have a gateway set, it would think that the LAN is a WAN so it wouldn't do NAT.

You don't need nor want to have a gateway set on the LAN interface, and if you have a gateway defined that is actually your LAN IP, remove it.

mmerlone · Dec 20, 2012, 6:30 PM

@jimp:

Ah, well there is one more I forgot - if the LAN interface does have a gateway set, it would think that the LAN is a WAN so it wouldn't do NAT.

Makes total sense, it did have a gateway set.

@jimp:

You don't need nor want to have a gateway set on the LAN interface, and if you have a gateway defined that is actually your LAN IP, remove it.

Sure, sure. Selected 'none' as gateway for LAN interface, reset states, rebooted pfsense and …. (still rebooting) .... nope! No luck, no echo reply from 8.8.8.8 to 10.0.0.8. Partial success though: my workstation on 192.168.0.0/24 does ping 8.8.8.8 and am using it now to post to the forum :)

What the heck! Let me reboot 10.0.0.8 and see what happens, it is a Win XP anyway and my workstation is linux. Rebooted and no luck yet. Dont' forget 10.0.0.8 uses 10.0.0.5 as default gateway, which is a VIP Alias on pfsense.

States and packet capture on WAN interface attached.


16:34:10.530106 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 9216, length 40
16:34:16.030458 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 9472, length 40
16:34:21.532543 IP 10.0.0.8 > 8.8.8.8: ICMP echo request, id 512, seq 9728, length 40

mmerlone · Dec 20, 2012, 6:31 PM

Forgot to mention: also deselected a gateway for ADSL interface, just to be sure.

jimp · Dec 20, 2012, 6:35 PM

So it's still not getting NAT, do you see nat rules in /tmp/rules.debug now?

jimp · Dec 20, 2012, 6:37 PM

@mmerlone:

Forgot to mention: also deselected a gateway for ADSL interface, just to be sure.

Why?

WANs must have a gateway set.
LANs must not have a gateway set.

Otherwise automatic outbound NAT will not work.

mmerlone · Dec 20, 2012, 6:45 PM

@jimp:

So it's still not getting NAT, do you see nat rules in /tmp/rules.debug now?

Not yet (oops, regarding 10.0.0.0/24):

#System aliases

loopback = "{ lo0 }"
WAN = "{ xl0 }"
LAN = "{ bge1 }"
ADSL = "{ bge0 }"

#SSH Lockout Table
table <sshlockout>persist
table <webconfiguratorlockout>persist
#Snort tables
table <snort2c>table <virusprot># User Aliases 
table <lanvisitantes>{   10.0.0.0/24 } 
LanVisitantes = "<lanvisitantes>"

# Gateways
GWGWADSL = " route-to ( bge0 192.168.1.100 ) "
GWGWWAN = " route-to ( xl0 177.135.88.153 ) "
GWGWLANNS100 = " route-to ( bge1 192.168.0.254 ) "
GWGWLANRV042 = " route-to ( bge1 192.168.0.251 ) "
GWGW_Group = "  route-to { ( bge0 192.168.1.100 )  }  "

set loginterface bge1
set optimization normal
set limit states 96000
set limit src-nodes 96000

set skip on pfsync0

scrub in on $WAN all    fragment reassemble
scrub in on $LAN all    fragment reassemble
scrub in on $ADSL all    fragment reassemble

no nat proto carp
no rdr proto carp
nat-anchor "natearly/*"
nat-anchor "natrules/*"

# Outbound NAT rules

# Subnets to NAT 
tonatsubnets	= "{ 192.168.0.0/24 192.168.1.0/24 127.0.0.0/8  }"
nat on $WAN  from $tonatsubnets port 500 to any port 500 -> 177.135.88.154/32 port 500  
nat on $WAN  from $tonatsubnets to any -> 177.135.88.154/32 port 1024:65535  

# Load balancing anchor
rdr-anchor "relayd/*"
# TFTP proxy
rdr-anchor "tftp-proxy/*"
table <negate_networks>{ 177.135.88.152/29 192.168.0.0/24 192.168.1.0/24 }
# UPnPd rdr anchor
rdr-anchor "miniupnpd"

anchor "relayd/*"
#---------------------------------------------------------------------------
# default deny rules
#---------------------------------------------------------------------------
block in log all label "Default deny rule"
block out log all label "Default deny rule"

# We use the mighty pf, we cannot be fooled.
block quick proto { tcp, udp } from any port = 0 to any
block quick proto { tcp, udp } from any to any port = 0

# Block all IPv6
block in quick inet6 all
block out quick inet6 all

# Snort package
block quick from <snort2c>to any label "Block snort2c hosts"
block quick from any to <snort2c>label "Block snort2c hosts"
block in log quick proto carp from (self) to any
pass quick proto carp
pass quick proto pfsync

# SSH lockout
block in log quick proto tcp from <sshlockout>to any port 22 label "sshlockout"

# webConfigurator lockout
block in log quick proto tcp from <webconfiguratorlockout>to any port 80 label "webConfiguratorlockout"
block in quick from <virusprot>to any label "virusprot overload table"
antispoof for xl0
antispoof for bge1
antispoof for bge0

# loopback
pass in on $loopback all label "pass loopback"
pass out on $loopback all label "pass loopback"
# let out anything from the firewall host itself and decrypted IPsec traffic
pass out all keep state allow-opts label "let out anything from firewall host itself"
pass out route-to ( xl0 177.135.88.153 ) from 177.135.88.154 to !177.135.88.152/29 keep state allow-opts label "let out anything from firewall host itself"
# make sure the user cannot lock himself out of the webConfigurator or SSH
pass in quick on bge1 proto tcp from any to (bge1) port { 80 22 } keep state label "anti-lockout rule"

# User-defined rules follow

anchor "userrules/*"
pass  on {  xl0  bge1  bge0  }  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $WAN reply-to ( xl0 177.135.88.153 )  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $LAN  inet proto icmp  from any to any keep state  label "USER_RULE"
pass  in  quick  on $LAN  from 192.168.0.0/24 to any keep state  label "USER_RULE: Default allow LAN to any rule"
pass  in  quick  on $LAN  from   $LanVisitantes to any keep state  label "USER_RULE"
pass  in  quick  on $LAN  from any to   $LanVisitantes keep state  label "USER_RULE"
pass  in  quick  on $ADSL  inet proto icmp  from any to any keep state  label "USER_RULE"

# VPN Rules
anchor "tftp-proxy/*"</virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></lanvisitantes></lanvisitantes></virusprot></snort2c></webconfiguratorlockout></sshlockout>

So, I erased the VIP and created again, without a gateway on LAN interface, reset states, and now 10.0.0.8 cannot even ping 10.0.0.5. Packets and state table:

16:45:12.039107 ARP, Request who-has 10.0.0.8 tell 10.0.0.5, length 28
16:45:17.539196 ARP, Request who-has 10.0.0.8 tell 10.0.0.5, length 28

jimp · Dec 20, 2012, 6:45 PM

Closer..

# Subnets to NAT 
tonatsubnets	= "{ 192.168.0.0/24 192.168.1.0/24 127.0.0.0/8  }"

It is getting your other subnets but not the 10.x.x.x one.

Make sure the IP alias VIP has the proper subnet mask set (not /32).

mmerlone · Dec 20, 2012, 6:48 PM

@jimp:

It is getting your other subnets but not the 10.x.x.x one.
Make sure the IP alias VIP has the proper subnet mask set (not /32).

/24

jimp · Dec 20, 2012, 6:52 PM

And that IP alias is on the correct interface? (LAN)

Check Diagnostics > Routes, paste the output here, it might give some more clues.