Solved, it was indeed just a simple tweak. I unchecked the System -> Routing -> Miscellaneous -> Failover default gateway option I had checked, and updated any outgoing firewall rule to explicitly use the advanced settings. Under advanced settings I changed Gateway from default to the name of my Gateway group. Now, outgoing firewall rules have a black sprocket icon next to them to indicate advanced settings which in my case is the Gateway group. Internal traffic rules, any LAN -> LAN for example, do not need this updated setting. Only traffic using a Gateway should be updated to use a Gateway group.

To answer my initial questions explicitly: yes, this has to apply to each firewall rule that allows WAN traffic. If you have an outgoing firewall rule with the Advanced Settings -> Gateway left on 'default' it will NOT allow traffic to leave via the secondary WAN. I had to update my SSH rule and HTTPS rules independently for both to work. This is all clear in retrospect, but this might help someone else in the future to learn from my confusion.

Once again, thanks heper!


Are the network settings and the gateway set correct on the destination host?

Check the local firewall and default gateway on that target computer then.

A Feature-Request for PfSense:

I jut configured Openstack to deploy an mtu using DHCP Option 26, but PFsense seems to ignore this one consequently.

dhcpdump -i tapb8659f7c-df
  TIME: 2017-06-29 13:22:00.059
    IP: 10.40.50.3 (fa:16:3e:2c:e0:61) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)
    OP: 1 (BOOTPREQUEST)
HTYPE: 1 (Ethernet)
  HLEN: 6
  HOPS: 0
  XID: 11ac2ce2
  SECS: 0
FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 0.0.0.0
SIADDR: 0.0.0.0
GIADDR: 0.0.0.0
CHADDR: fa:16:3e:2c:e0:61:00:00:00:00:00:00:00:00:00:00
SNAME: .
FNAME: .
OPTION:  53 (  1) DHCP message type        3 (DHCPREQUEST)
OPTION:  50 (  4) Request IP address        10.40.50.3
OPTION:  61 (  7) Client-identifier        01:fa:16:3e:2c:e0:61
OPTION:  12 (  9) Host name                xxxxxxxxx
OPTION:  55 (  9) Parameter Request List      1 (Subnet mask)
                                            28 (Broadcast address)
                                              2 (Time offset)
                                            121 (Classless Static Route)
                                              3 (Routers)
                                            15 (Domainname)
                                              6 (DNS server)
                                            12 (Host name)
                                            119 (Domain Search)

–-------------------------------------------------------------------------

TIME: 2017-06-29 13:22:00.059
    IP: 10.40.50.2 (fa:16:3e:63:19:c0) > 10.40.50.3 (fa:16:3e:2c:e0:61)
    OP: 2 (BOOTPREPLY)
HTYPE: 1 (Ethernet)
  HLEN: 6
  HOPS: 0
  XID: 11ac2ce2
  SECS: 0
FLAGS: 0
CIADDR: 0.0.0.0
YIADDR: 10.40.50.3
SIADDR: 10.40.50.2
GIADDR: 0.0.0.0
CHADDR: fa:16:3e:2c:e0:61:00:00:00:00:00:00:00:00:00:00
SNAME: .
FNAME: .
OPTION:  53 (  1) DHCP message type        5 (DHCPACK)
OPTION:  54 (  4) Server identifier        10.40.50.2
OPTION:  51 (  4) IP address leasetime      86400 (24h)
OPTION:  58 (  4) T1                        43200 (12h)
OPTION:  59 (  4) T2                        75600 (21h)
OPTION:  1 (  4) Subnet mask              255.255.255.0
OPTION:  28 (  4) Broadcast address        10.40.50.255
OPTION:  15 ( 14) Domainname                openstacklocal
OPTION:  12 ( 15) Host name                host-10-40-50-3
OPTION:  3 (  4) Routers                  10.40.50.3
OPTION: 121 ( 14) Classless Static Route    20a9fea9fe0a2832  .....(2
                                            02000a283203    ...(2.
OPTION:  6 (  8) DNS server                xxxxxxxxxxxxx
OPTION:  26 (  2) Interface MTU            1450
–-------------------------------------------------------------------------

This shouldn't be ignored because it'll result in fragmented packets / incorrect checksums since OS itself adds ~50 Bytes to a VXLAN-paket anyway.

In 1:1 you can set the NAT for the whole subnet if you enter 172.16.0.0 at "External subnet IP" and at "Internal IP" select network and 172.16.100.0/24

It doesn't matter if this also includes IPs assigned to computers in group A, since you haven't add an IP alias for these addresses to WAN.

@Valley:

When I check my public IP address my dynamic Xfinity is shown - how can I setup up my Comcast static IP address as the default address?

One way is to bridge a third interface to your WAN port and put the static IP's on devices on that interface.

Other is to use VIP's and then port forwarding or 1:1 NAT to the intended devices.

https://doc.pfsense.org/index.php/What_are_Virtual_IP_Addresses

Not wanting to hijack the OP's thread but, yeah, it took quite a lot of web searching to find that solution - not really knowing what to look for.

Then, I had to find a workaround for the fact that the table is cleared on a firewall rule change.

Doktornotor pointed me to the shellcmd package and its "afterfilterchangeshellcmd" option.  I use that to call a tiny PHP script that checks whether the table is empty.  If it is, the connection to the sending host is restarted and the table is rebuilt.

Lots of fun :)

I think both agree on bad design.

@dotdash:

It appears you are not getting any routes from the peer router.
The fact that the pinger showed the remote router down seems to indicate the transit layer to 100.100.100.100 failed.
Is that IP provider-assigned, or did you obfuscate the real IP? That block is not a traditional private space, but is supposed to be used by service providers for their internal networks.

I've obfuscate the real ip.

The only way to do that is to get Multilink PPP from your ISP.

Maybe with pfsense it is hard but it is easy to setup using a layer 3 switch.  All you have to do is point the local traffic to the layer 3 switch.  It knows where everything is and will route or switch to the device.  Nothing hard.  It is a good way to bring a layer 3 switch into the fold without disrupting normal operations.

Check the system log for hints.

Just set a static route for 192.168.45.0/24 on the Exchange pointing to pfSense:

Replace "<gateway>" with the LAN IP of pfSense within 192.168.44.0/24.</gateway>

If it is critical to get this working I suggest you buy some support hours.  Contact support to discuss first.  For done things which were very complicated this is what I have done.

It is been more than a year but I eventually gave up on this.  I think using openvpn might be easier but not sure.  I wish you luck with this!

Not sure what this document refers to but it is innacurate.

You must configure 3G/LTE setting on interface if using as a wan connection or at least multi wan.  Directions say not to make any changes to the interface but you have to in order to make it work.

https://doc.pfsense.org/index.php/Configuring_3G_modems

I will reply to myself as I've found the problem so anyone with the same hardware can use this as a solution.

The problem was that the VLAN3 on the WDR3600 was incorrectly set to the interface eth1.3 and on the WDR-3600 the switch is on the eth0 interface, so the VLAN3 had to be set to eth0.3 and voila! everything works!!!

The only thing that should be added to this setup are the firewall rules, that I have set as the following screenshot shows.

VLAN3_fw_rules.png_thumb
VLAN3_fw_rules.png