You will have to disable gateway monitoring for the second WAN, so it can't detect if it's down, but otherwise it may function OK.

I think this has something to do with connecting pfsense wan to my ISPmodem(home hub 3000) and using the Advance DMZ? raceroute shows the second hop is my ISP router 192.168.0.11 however, pfsense is still getting a external IP 142.134.91.xx maybe i am going down the wrong path here? pfsense IP from advance DMZ:142.134.91.xx gateway: 142.134.88.1 Monitor IP: 142.134.88.1 Bell router IP: 192.168.11 pfsense router IP: 192.168.0.2 Not sure what 192.168.2.1 is below in the dhcp catt from pfsense traceroute to google.ca (172.217.10.131), 30 hops max, 60 byte packets 1 pfsense.WORKGROUP (192.168.0.2) 0.696 ms 0.661 ms 0.640 ms 2 bell router 192.168.0.11 (192.168.0.11) 1.858 ms 1.850 ms 1.854 ms 3 loop0.6cw.ba17.hlfx.ns.aliant.net (142.176.50.10) 1.928 ms 1.864 ms 1.804 ms 4 ae15-182.cr02.hlfx.ns.aliant.net (142.166.181.141) 1.868 ms 1.794 ms 1.803 ms [2.4.3-RELEASE][root@router.WORKGROUP]/root: cat /var/db/dh dhclient.leases.em0 dhclient.leases.em0.35 dhclient.leases.em0.34 dhclient.leases.em0_vlan35 [2.4.3-RELEASE][root@router.WORKGROUP]/root: cat /var/db/dhclient.leases.em0 lease { interface "em0"; fixed-address 192.168.0.57; option subnet-mask 255.255.255.0; option routers 192.168.0.11; option domain-name-servers 192.168.0.11,142.166.166.166; option domain-name "home"; option broadcast-address 192.168.0.255; option dhcp-lease-time 259200; option dhcp-message-type 5; option dhcp-server-identifier 192.168.0.11; renew 0 2018/7/15 01:58:08; rebind 1 2018/7/16 04:58:08; expire 1 2018/7/16 13:58:08; } lease { interface "em0"; fixed-address 192.168.0.55; option subnet-mask 255.255.255.0; option routers 192.168.0.11; option domain-name-servers 192.168.0.11,142.166.166.166; option domain-name "home"; option broadcast-address 192.168.0.255; option dhcp-lease-time 259200; option dhcp-message-type 5; option dhcp-server-identifier 192.168.0.11; renew 0 2018/7/15 01:43:10; rebind 1 2018/7/16 04:43:10; expire 1 2018/7/16 13:43:10; } lease { interface "em0"; fixed-address 142.134.91.xx; option subnet-mask 255.255.252.0; option routers 142.134.88.1; option domain-name-servers 47.55.55.55,142.166.166.166; option dhcp-lease-time 600; option dhcp-message-type 5; option dhcp-server-identifier 192.168.2.1; renew 1 2018/7/16 20:32:41; rebind 1 2018/7/16 20:36:26; expire 1 2018/7/16 20:37:41; } lease { interface "em0"; fixed-address 142.134.91.xx; option subnet-mask 255.255.252.0; option routers 142.134.88.1; option domain-name-servers 47.55.55.55,142.166.166.166; option dhcp-lease-time 600; option dhcp-message-type 5; option dhcp-server-identifier 192.168.2.1; renew 1 2018/7/16 20:42:44; rebind 1 2018/7/16 20:46:29; expire 1 2018/7/16 20:47:44; } lease { interface "em0"; fixed-address 142.134.91.xx; option subnet-mask 255.255.252.0; option routers 142.134.88.1; option domain-name-servers 47.55.55.55,142.166.166.166; option dhcp-lease-time 600; option dhcp-message-type 5; option dhcp-server-identifier 192.168.2.1; renew 1 2018/7/16 20:52:47; rebind 1 2018/7/16 20:56:32; expire 1 2018/7/16 20:57:47; } lease { interface "em0"; fixed-address 142.134.91.xx; option subnet-mask 255.255.252.0; option routers 142.134.88.1; option domain-name-servers 47.55.55.55,142.166.166.166; option dhcp-lease-time 600; option dhcp-message-type 5; option dhcp-server-identifier 192.168.2.1; renew 1 2018/7/16 21:02:50; rebind 1 2018/7/16 21:06:35; expire 1 2018/7/16 21:07:50; } lease { interface "em0"; fixed-address 142.134.91.69; option subnet-mask 255.255.252.0; option routers 142.134.88.1; option domain-name-servers 47.55.55.55,142.166.166.166; option dhcp-lease-time 600; option dhcp-message-type 5; option dhcp-server-identifier 192.168.2.1; renew 1 2018/7/16 21:12:53; rebind 1 2018/7/16 21:16:38; expire 1 2018/7/16 21:17:53; }

Thank you very much for your help, I already solved by removing the NAT. regards

I recently upgrade to 2.4.3-RELEASE-p1 and this is working properly now. Not sure if the upgrade fixed it or maybe it just needed a reboot.

Timoteo test this setting and I will say to youcolored text

@azmodeuz said in Routing 2 Router LANs under a Third Router: Firewall Rules > LAN: PASS - Source: LAN NET - Destination: 192.168.8.0/24 - Gateway: 192.168.88.7 No. You need to pass sources 192.168.2.0/24 and 192.168.3.0/24 into LAN. Do NOT set a gateway on those rules. Imagine yourself sitting in one of the routers. You say "I have a packet for 192.168.X.X. What next hop do I need to send it to? Consult my routing table. I have a route for 192.168.X.X - I send that traffic to next-hop Y.Y.Y.Y (the route's gateway)." If you are unfamiliar with all of this why are you making it so complicated? Please get it working with one then move to the second. Far less to look at and communicate.

What is the point of this? Your wanting to load share to 2 different vpn connections off the same physical interface? And the same TUN interface as well? Have no clue to what is the use case here... What is the point of the complexity - what does it get you? Your worried that r44 or r45 go down? What is the point of the loadsharing across the connection.. NHRP - with just the 2 connections.. With GRE and IPsec involved as well?? Is this some sort of class work - seems like nonsense waste of time, I see no real world application here. And down the rabbit hole we go...

I have (hopefully) solved my immediate problem by marking the Tier 2 gateway in the group that we use most as "Disable Gateway Monitoring Action" so that if the Tier 1 gateway is down pfSense will never take the Tier 2 gateway down. This should be fine for our most used gateway group but it is inappropriate for other groups that we occasionally use. If/when we switch to using another gateway group I'll have to remember and change that setting on that gateway. It seems to me that the various monitoring and threshold settings should be defined in the gateway group and would override those on the gateway, when the gateway is used as part of a group. That would allow me to configure each group as it makes sense and then switch between them with ease.

sound like you have asymmetrical mess if your gateway is going to be out your lan interface. Why don't you draw up your network and point out exactly what your trying to do.. ption use non-local gateway through interface specific route How is it you would be hitting a "gateway" that is not on the same network?

@kpa Here is parts of my configs: Client [image: 1530578082042-5d158d98-e6c7-4eb1-ae50-31fef1ec71e6-%D0%B8%D0%B7%D0%BE%D0%B1%D1%80%D0%B0%D0%B6%D0%B5%D0%BD%D0%B8%D0%B5-resized.png] Server ccd [image: 1530578172008-e3ed8f4c-afaf-4e63-808a-c1c90d55680d-%D0%B8%D0%B7%D0%BE%D0%B1%D1%80%D0%B0%D0%B6%D0%B5%D0%BD%D0%B8%D0%B5.png] Server server.conf [image: 1530578205194-a56d5260-f8e8-45d2-9835-f8f3b837f919-%D0%B8%D0%B7%D0%BE%D0%B1%D1%80%D0%B0%D0%B6%D0%B5%D0%BD%D0%B8%D0%B5.png] Server log before client connected [image: 1530578284083-ced1dbe8-b9e6-48de-ac6f-f855f726f04c-%D0%B8%D0%B7%D0%BE%D0%B1%D1%80%D0%B0%D0%B6%D0%B5%D0%BD%D0%B8%D0%B5-resized.png] Server log after client connected [image: 1530578685871-c9afca24-742f-47a6-a23e-ca8501004b07-%D0%B8%D0%B7%D0%BE%D0%B1%D1%80%D0%B0%D0%B6%D0%B5%D0%BD%D0%B8%D0%B5-resized.png] Client log Jul 3 07:45:33 openvpn 60153 Initialization Sequence Completed Jul 3 07:45:33 openvpn 60153 /usr/local/sbin/ovpn-linkup ovpnc2 1500 1570 10.10.0.2 255.255.255.0 init Jul 3 07:45:33 openvpn 60153 /sbin/route add -net 10.10.0.0 10.10.0.1 255.255.255.0 Jul 3 07:45:33 openvpn 60153 /sbin/ifconfig ovpnc2 10.10.0.2 10.10.0.1 mtu 1500 netmask 255.255.255.0 up Jul 3 07:45:33 openvpn 60153 do_ifconfig, tt->did_ifconfig_ipv6_setup=0 Jul 3 07:45:33 openvpn 60153 TUN/TAP device /dev/tun2 opened Jul 3 07:45:33 openvpn 60153 TUN/TAP device ovpnc2 exists previously, keep at program end Jul 3 07:45:33 openvpn 60153 ROUTE_GATEWAY CLIENT_EX_IP/255.255.255.192 IFACE=em0 HWADDR=00:0c:29:6c:7e:79 Jul 3 07:45:33 openvpn 60153 Incoming Data Channel: Using 256 bit message hash 'SHA256' for HMAC authentication Jul 3 07:45:33 openvpn 60153 Incoming Data Channel: Cipher 'AES-128-CBC' initialized with 128 bit key Jul 3 07:45:33 openvpn 60153 Outgoing Data Channel: Using 256 bit message hash 'SHA256' for HMAC authentication Jul 3 07:45:33 openvpn 60153 Outgoing Data Channel: Cipher 'AES-128-CBC' initialized with 128 bit key Jul 3 07:45:33 openvpn 60153 OPTIONS IMPORT: route-related options modified Jul 3 07:45:33 openvpn 60153 OPTIONS IMPORT: --ifconfig/up options modified Jul 3 07:45:33 openvpn 60153 OPTIONS IMPORT: timers and/or timeouts modified Jul 3 07:45:33 openvpn 60153 PUSH: Received control message: 'PUSH_REPLY,route-gateway 10.10.0.1,topology subnet,ping 10,ping-restart 120,ifconfig 10.10.0.2 255.255.255.0' Jul 3 07:45:33 openvpn 60153 SENT CONTROL [server]: 'PUSH_REQUEST' (status=1) Jul 3 07:45:32 openvpn 60153 [server] Peer Connection Initiated with [AF_INET]VPN_SERVER_EXT_IP:PORT Jul 3 07:45:32 openvpn 60153 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 DHE-RSA-AES256-GCM-SHA384, 2048 bit RSA Jul 3 07:45:32 openvpn 60153 VERIFY OK: depth=0, C=RU, ST=TO, L=Tomsk, O=Kireva, OU=IT_dept, CN=server, name=oneandoneserver, emailAddress=winmasta@kireva.com Jul 3 07:45:32 openvpn 60153 VERIFY EKU OK Jul 3 07:45:32 openvpn 60153 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication Jul 3 07:45:32 openvpn 60153 Validating certificate extended key usage Jul 3 07:45:32 openvpn 60153 VERIFY KU OK Jul 3 07:45:32 openvpn 60153 VERIFY OK: depth=1, C=RU, ST=TO, L=Tomsk, O=Kireva, OU=IT_dept, CN=Kireva CA, name=oneandoneserver, emailAddress=winmasta@kireva.com Jul 3 07:45:31 openvpn 60153 TLS: Initial packet from [AF_INET]VPN_SERVER_EXT_IP:PORT, sid=446f96a7 9c4b7ab0 Jul 3 07:45:31 openvpn 60153 UDPv4 link remote: [AF_INET]VPN_SERVER_EXT_IP:PORT Jul 3 07:45:31 openvpn 60153 UDPv4 link local (bound): [AF_INET]CLIENT_EXT_IP:0 Jul 3 07:45:31 openvpn 60153 Socket Buffers: R=[42080->42080] S=[57344->57344] Jul 3 07:45:31 openvpn 60153 TCP/UDP: Preserving recently used remote address: [AF_INET]VPN_SERVER_EXT_IP:PORT Jul 3 07:45:31 openvpn 60153 Incoming Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication Jul 3 07:45:31 openvpn 60153 Outgoing Control Channel Authentication: Using 256 bit message hash 'SHA256' for HMAC authentication Jul 3 07:45:31 openvpn 60153 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts Jul 3 07:45:31 openvpn 60153 MANAGEMENT: unix domain socket listening on /var/etc/openvpn/client2.sock Jul 3 07:45:31 openvpn 60082 library versions: OpenSSL 1.0.2m-freebsd 2 Nov 2017, LZO 2.10 Jul 3 07:45:31 openvpn 60082 OpenVPN 2.4.4 amd64-portbld-freebsd11.1 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Oct 8 2017 Jul 3 07:45:31 openvpn 60825 SIGTERM[hard,] received, process exiting Jul 3 07:45:31 openvpn 60825 /usr/local/sbin/ovpn-linkdown ovpnc2 1500 1570 10.10.0.2 255.255.255.0 init Jul 3 07:45:31 openvpn 60825 Closing TUN/TAP interface Jul 3 07:45:31 openvpn 60825 event_wait : Interrupted system call (code=4) Jul 3 07:45:29 openvpn 60825 MANAGEMENT: Client disconnected Jul 3 07:45:29 openvpn 60825 MANAGEMENT: CMD 'status 2' Jul 3 07:45:29 openvpn 60825 MANAGEMENT: CMD 'state 1' Jul 3 07:45:29 openvpn 7621 MANAGEMENT: Client disconnected Jul 3 07:45:29 openvpn 60825 MANAGEMENT: Client connected from /var/etc/openvpn/client2.sock Jul 3 07:45:29 openvpn 7621 MANAGEMENT: CMD 'status 2' Jul 3 07:45:29 openvpn 7621 MANAGEMENT: CMD 'state 1' Jul 3 07:45:29 openvpn 7621 MANAGEMENT: Client connected from /var/etc/openvpn/client1.sock 111

Are the lines bonded?

Finally! For some reason it didn't work to set a rule allowing traffic to the destination IP for the proxy. Opening for port 80 to any destination fixed it!

@kpa said in Routing table with policy-based routing: It's more like that the routing process uses information tagged on to the packets traversing the system to detect if a set of packets need special handling and bypasses the normal routing table when it sees those special tags. The firewall rules that match the incoming traffic apply these special tags to the incoming packets. Gotcha, that makes more sense. Thanks for the explanation!