Cannot NAT trough OPT1 interface on multiwan

[villavai]

i cannot give the debug because I've already revert back to 2.1.5, and since pf is my only fw, and is being use in production site, so i hope someone kindly test it in 2.2 latest snapshot to give the answer is this problem still exist or not.

thank you.

[naras]

@villavai:

i cannot give the debug because I've already revert back to 2.1.5, and since pf is my only fw, and is being use in production site, so i hope someone kindly test it in 2.2 latest snapshot to give the answer is this problem still exist or not.

thank you.

Unfortunately, it's still there.

[eSpezi]

Hi everybody,

this is not only a NAT problem.
It seems that the routing in general does not work properly with multiwan. (in my case WAN_KD=cable router, WAN_TO=PPPoE modem)
I'm running the Oct 03 10:14:50 CDT 2014 BETA, but also had the same problem with a previous version (Oct 01).

I can't even access the pfS Webgui from the WAN_TO side, although in the firewall log the request is shown as PASS.

My config:
set optimization normal
set timeout { adaptive.start 0, adaptive.end 0 }
set limit states 198000
set limit src-nodes 198000

#System aliases

loopback = "{ lo0 }"
WAN_KD = "{ re0 }"
LAN = "{ re2 }"
WAN_TO = "{ pppoe0 }"
OpenVPN = "{ openvpn }"

#SSH Lockout Table
table <sshlockout>persist
table <webconfiguratorlockout>persist
#Snort tables
table <snort2c>table <virusprot>table <bogons>persist file "/etc/bogons"
table <vpn_networks>{ 192.168.254.0/24 }
table <negate_networks>{ 192.168.254.0/24 }

User Aliases

admin_ports = "{  456  4562 }"
ovpn_ports = "{  1194  1199 }"
table <ovpn_remote_networks>{  192.168.254.0/24  192.168.222.0/24 }
OVPN_Remote_Networks = "<ovpn_remote_networks>"
table <rfc1918_private_networks>{  10.0.0.0/8  172.16.0.0/12  192.168.0.0/16 }
RFC1918_Private_Networks = "<rfc1918_private_networks>"
table <server>{  172.16.100.100 }
Server = "<server>"

Gateways

GWWAN_KD_GW = " route-to ( re0 192.168.0.1 ) "
GWWAN_TO_PPPOE = " route-to ( pppoe0 XXX.WAN.TO.GW ) "
GWGWG_LAN = "  route-to { ( pppoe0 XXX.WAN.TO.GW )  }  "

set loginterface re2

set skip on pfsync0

scrub on $WAN_KD all    fragment reassemble
scrub on $LAN all    fragment reassemble
scrub on $WAN_TO all    fragment reassemble

no nat proto carp
no rdr proto carp
nat-anchor "natearly/"
nat-anchor "natrules/"

Outbound NAT rules (automatic)

Subnets to NAT

tonatsubnets = "{ 127.0.0.0/8 172.16.0.0/16 192.168.254.0/24 }"
nat on $WAN_KD  from $tonatsubnets to any port 500 -> 192.168.0.2/32  static-port
nat on $WAN_KD  from $tonatsubnets to any -> 192.168.0.2/32 port 1024:65535 
nat on $WAN_TO  from $tonatsubnets to any port 500 -> XXX.WAN.TO.IP/32  static-port
nat on $WAN_TO  from $tonatsubnets to any -> XXX.WAN.TO.IP/32 port 1024:65535

Load balancing anchor

rdr-anchor "relayd/*"

TFTP proxy

rdr-anchor "tftp-proxy/*"

NAT Inbound Redirects

rdr on re0 proto tcp from any to 192.168.0.2 port 443 -> $Server
rdr on re0 proto tcp from any to 192.168.0.2 port 8182 -> $Server port 5900

UPnPd rdr anchor

rdr-anchor "miniupnpd"

anchor "relayd/"
anchor "openvpn/"
anchor "ipsec/*"

Block all IPv6

block in log quick inet6 all tracker 1000000001 label "Block all IPv6"
block out log quick inet6 all tracker 1000000002 label "Block all IPv6"
#–-------------------------------------------------------------------------

default deny rules

#---------------------------------------------------------------------------
block in log inet all tracker 1000000101 label "Default deny rule IPv4"
block out log inet all tracker 1000000102 label "Default deny rule IPv4"
block in log inet6 all tracker 1000000103 label "Default deny rule IPv6"
block out log inet6 all tracker 1000000104 label "Default deny rule IPv6"

IPv6 ICMP is not auxilary, it is required for operation

See man icmp6(4)

1    unreach        Destination unreachable

2    toobig          Packet too big

128  echoreq        Echo service request

129  echorep        Echo service reply

133  routersol      Router solicitation

134  routeradv      Router advertisement

135  neighbrsol      Neighbor solicitation

136  neighbradv      Neighbor advertisement

pass  quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} tracker 1000000105 keep state

Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)

pass out  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} tracker 1000000106 keep state
pass out  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} tracker 1000000107 keep state
pass in  quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000108 keep state
pass in  quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} tracker 1000000109 keep state
pass in  quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} tracker 1000000110 keep state

We use the mighty pf, we cannot be fooled.

block log quick inet proto { tcp, udp } from any port = 0 to any tracker 1000000111
block log quick inet proto { tcp, udp } from any to any port = 0 tracker 1000000112
block log quick inet6 proto { tcp, udp } from any port = 0 to any tracker 1000000113
block log quick inet6 proto { tcp, udp } from any to any port = 0 tracker 1000000114

Snort package

block log quick from <snort2c>to any tracker 1000000115 label "Block snort2c hosts"
block log quick from any to <snort2c>tracker 1000000116 label "Block snort2c hosts"

SSH lockout

block in log quick proto tcp from <sshlockout>to (self) port 4562 tracker 1000000301 label "sshlockout"

webConfigurator lockout

block in log quick proto tcp from <webconfiguratorlockout>to (self) port 456 tracker 1000000351 label "webConfiguratorlockout"
block in log quick from <virusprot>to any tracker 1000000400 label "virusprot overload table"

block bogon networks (IPv4)

http://www.cymru.com/Documents/bogon-bn-nonagg.txt

block in log quick on $WAN_KD from <bogons>to any tracker 1000001551 label "block bogon IPv4 networks from WAN_KD"
antispoof log for $WAN_KD tracker 1000001570
antispoof log for $LAN tracker 1000002620
antispoof log for $WAN_TO tracker 1000003670

loopback

pass in  on $loopback inet all tracker 1000004761 label "pass IPv4 loopback"
pass out  on $loopback inet all tracker 1000004762 label "pass IPv4 loopback"
pass in  on $loopback inet6 all tracker 1000004763 label "pass IPv6 loopback"
pass out  on $loopback inet6 all tracker 1000004764 label "pass IPv6 loopback"

let out anything from the firewall host itself and decrypted IPsec traffic

pass out  inet all keep state allow-opts tracker 1000004765 label "let out anything IPv4 from firewall host itself"
pass out  inet6 all keep state allow-opts tracker 1000004766 label "let out anything IPv6 from firewall host itself"
pass out  route-to ( re0 192.168.0.1 ) from 192.168.0.2 to !192.168.0.0/24 tracker 1000004861 keep state allow-opts label "let out anything from firewall host itself"
pass out  route-to ( pppoe0 XXX.WAN.TO.GW ) from XXX.WAN.TO.IP to !XXX.WAN.TO.IP/32 tracker 1000004862 keep state allow-opts label "let out anything from firewall host itself"

User-defined rules follow

anchor "userrules/*"
pass  in  quick  on $OpenVPN inet from $OVPN_Remote_Networks to 172.16.0.0/16 tracker 1412417430 keep state  label "USER_RULE"
pass  in  quick  on $WAN_KD reply-to ( re0 192.168.0.1 ) inet proto tcp  from any to 192.168.0.2 port $admin_ports tracker 1412198572 flags S/SA keep state  label "USER_RULE"
pass  in  quick  on $WAN_KD reply-to ( re0 192.168.0.1 ) inet proto udp  from any to 192.168.0.2 port $ovpn_ports tracker 1412415560 keep state  label "USER_RULE"
pass  in  quick  on $WAN_KD reply-to ( re0 192.168.0.1 )  proto tcp  from any to $Server port 443 flags S/SA keep state  label "USER_RULE: NAT "
pass  in  quick  on $WAN_KD reply-to ( re0 192.168.0.1 )  proto tcp  from any to $Server port 5900 flags S/SA keep state  label "USER_RULE: NAT "
pass  in  quick  on $LAN inet proto tcp  from any to 172.16.100.1 port $admin_ports tracker 1412207357 flags S/SA keep state  label "USER_RULE"
pass  in  quick  on $LAN inet from 172.16.0.0/16  to <negate_networks>tracker 0100000101 keep state  label "NEGATE_ROUTE: Negate policy routing for destination"
pass  in  quick  on $LAN  $GWGWG_LAN inet from 172.16.0.0/16 to any tracker 0100000101 keep state  label "USER_RULE: Default allow LAN to any rule"
pass  in  quick  on $WAN_TO reply-to ( pppoe0 XXX.WAN.TO.GW ) inet proto tcp  from any to XXX.WAN.TO.IP port $admin_ports tracker 1412448206 flags S/SA keep state  label "USER_RULE"

VPN Rules

anchor "tftp-proxy/*"

Any news on the issue?

Thanks,
Harry</negate_networks></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></server></server></rfc1918_private_networks></rfc1918_private_networks></ovpn_remote_networks></ovpn_remote_networks></negate_networks></vpn_networks></bogons></virusprot></snort2c></webconfiguratorlockout></sshlockout>

[mikeisfly]

I can confirm, I am having the same issue using the Oct 4th 64bit build. I can provide configurations if needed but I know I have configured everything correctly and the firewall blocks show no blocks. Downgrading back to 2.1.5 fixes the issues with the exact same configuration file. Just to give an over view of what I did.

created a rule under lan to have my plex machine use the second WAN.
Created a NAT (port Forward for TCP 32400) to plex machine
Rule was automatically generated.

Some fixes that I have found:

Changing default gateways to second WAN (Puts everything on second WAN), Not the desired behavior
Downgrading to 2.1.5 (everything works as expected)

I have this running on my home router so I can flash firmware again if Needed.

[eri--]

Actually seems that the one that have posted here have some issues in their configuration.

They have set a generic policy routing rule on their lan which overrides the reply-to(NAT of their OPTx) tha would make things work.

[eSpezi]

Hi Ermal,

on my Dual WAN (WAN1 = Cable Router, WAN2 = PPPoE Modem) installation I successfully can ping each WAN IF.
But I only can reach pfSenses WebGui or anything else on the current Default Gateway.
By current I mean, that if the default Default Gateway (WAN1) is marked as down (unfortunately still way too often false alarm by apinger) then I am able to reach the WebGui via WAN2.

I followed your suggestions, switched to manual outbound and added the propsed rule on top of everything, but that doesn't change anything.

Has anybody had success with this workaround?
Any other suggestions to make it work?

Thanks,
Harry

Edit:
Ermal in my case, you're right. I've policy routing on lan to a failover gateway group. That same scenario used to work fine with 2.15

[mattbunce]

Excuse my ignorance, but I can't quite figure out if there is a resolution to this yet? I see from Gloom this should be working if you set manual NAT rules on the OPT1 interface (not sure exactly how to, or what this means?).

Also, ermal suggests that the problem is that we're using a "generic policy routing rule" - and this is overriding the reply-to. Again, I'm not exactly sure what is meant by "generic policy routing rule" and if there's anything we can do to work around this?

I have run Wireshark on the NAT target machine, and whilst I don't understand what I'm looking at, you can clearly see that the http request I am making to the NAT forwarded port (from an external network) is hitting the target machine, so it does seem to be the reply that is getting lost somewhere.

My setup

WAN = DHCP client
OPT1 = PIA VPN

I have set-up forwarding via Firewall > NAT and manually created a floating rule for TCP/UDP traffic on OPT1 coming in the my target IP.

I have two LAN rules, the first is for target machine and sets the default gateway as OPT1. The second is for all devices and has the default gateway as the router default (which would be WAN).

The floating rule seems to make no difference, I see the traffic within Wireshark whether or not the floating rule is enabled.

[cmb]

It's not working, none of the workarounds are worth trying (short of for experimentation purposes if you want). This is the bug:
https://redmine.pfsense.org/issues/3760

[mattbunce]

Thanks cmb - That saves me spending any more time on this for now!

[eri--]

Please try a next coming snapshot it should be fixed.

[eSpezi]

Hi Ermal,

thanks a lot for addressing the issue!
Is the fix already in the Wed Oct 15 14:37:51 CDT 2014 Build?

Cheers,
Harry

[eri--]

Yeah it should be on that build or newer.

[mattbunce]

Unfortunately, this still doesn't seem to be working for me. I've tried used the automatically generated linked rule and manually creating a floating rule, but nomatter what I try it is still not working for me on the October 16th 09:56:37 version.

Is there anything else I should be looking at?

[phil.davis]

@mattbunce:

Unfortunately, this still doesn't seem to be working for me. I've tried used the automatically generated linked rule and manually creating a floating rule, but nomatter what I try it is still not working for me on the October 16th 09:56:37 version.

Is there anything else I should be looking at?

Follow the bug report also: https://redmine.pfsense.org/issues/3760
On that, cmb indicates he has done some testing and it is still not fixed.

[mattbunce]

Cheers Phil

I have already commented on the bug report too, I just didn't want to get too 'chatty' on there, because it seems the folks on there have done some pretty advanced diagnostics and I didn't want to distract from the good work they were doing :)

[eri--]

Another round of fixes.
Next snapshot should have the missing piece for fixing this.

[mattbunce]

I still can't seem to get this working on the 17th October, 11:27:45 version.

Have you had a chance to try it out ermal? Perhaps I am not configuring the Nat and Rules pages correctly?

M

[mikeisfly]

This is not working for me and I'm using 2.2-BETA (amd64) built on Fri Oct 17 20:02:23 CDT 2014 FreeBSD 10.1-RC2

I have attached my LAN, NAT (Port Forwrd), Manual Outbound NAT Rules.

I have also captured packets on both my WAN, WAN2 and LAN interfaces. On my WAN2 and LAN I can see the Syn, and the Syn Ack packets however I don't see the ack packet. Also there are some retransmissions. When I capture packets on my WAN and filter for port 32400 the capture is blank which tells me that the packet is not being sent out the default gateway. I have attached the .pcap files here in the form of a .txt file if you would like to look at them in wireshark the extension just needs to be changed back to .pcap . Hope this helps

Thanks,

P.S. Not sure why I was trying to embed these images from my onedrive folder but it didn't seem to work so I had to add attachments.

[capture on LAN.txt](/public/imported_attachments/1/capture on LAN.txt)
[capture on wan2.txt](/public/imported_attachments/1/capture on wan2.txt)

[Wolf666]

I think my problem is the same.
If someone has some spare time to read: https://forum.pfsense.org/index.php?topic=82944.0

I am running:

Version 2.2-BETA (amd64)
built on Fri Oct 17 20:02:23 CDT 2014
FreeBSD 10.1-RC2

I am available to test any solution and report back.

[mikeisfly]

It's a known bug and the developers are working on it. Until then if this is a deal breaker you can try going back to 2.1.5 until the issue is resolved. That is what I'm doing.