@johnpoz: Oh you caught me never nothing bigger than a /24.. <rolleyes>Been managing global networks for years.. Actually mange a /16 with Arin - but sure just small networks for me ;)  I have personally migrated large plant networks with 1000's of nodes to new networks.. Not once have needed to run multiple layer 3 on the same layer 2.. Now if you had lots of static devices on a network that had to each be touched - ok might take you more than a few minutes.. But would be your own fault or the guy before you not thinking ahead to have large amounts of devices with statics in the first place. While it is "possible" suggesting it to new user even in a LAB is just Borked plain and simple..  There is ZERO reason to ever do this other then the time needed to migrate, which is planned correctly should be very short amount of time..  Minutes if done correctly!! Not in the smallest of labs or the smallest of home networks just starting out is this ever a good idea!  Production, not production, lab - whatever this is just plain Borked.. Period! BTW - are you having fun smiting me every time you login?  Seems odd my count just went up again, minutes if not exactly when you logged into the forum..</rolleyes> You are being a bit forceful and nasty in how you reply. Whatever. Yawn. Agree to disagree.

its not a modem if your pfsense wan is not public - its a gateway (modem/router combo) doing nat.  So you either need to put it in bridge mode so pfsense gets your public IP.  Or you would have to forward any ports you want to forward on pfsense - this would be double nat.  Your cox isp device might also support dmz host, where it forwards all ports to an inside IP.. ie your pfsense wan IP.

Hmm even disabling ipv6 on WAN on pfsense makes me lose access to it. I regain access after a reboot. There is no special configuration I am doing on pfsense's side and ipv6 worked fine before adding the openwrt router. It seems pfsense becomes quite buggy behind another router for ipv6. I don't need v6, but it would be nice. I clearly don't have much knowledge on the subject, but it seems weird to me that ipv6 is more complicated in these scenarios compared to double nat, when v6 was supposed to make this simpler.

That is on 10/100 and no does not do vlans..

@Fabio72: Every openvpn client is using the failover group as outbound interface. Every openvpn client has the pulled routes disabled. I configured outbound NAT rules for every VPN and disabled the IPV6 gateways for the VPN interfaces. This is most critical part of your configuration.

@Derelict: Bridging is incompatible with CARP/HA interfaces. Good to know, then switches are the only option anyway. Thanks for your help

a router (pfsense) is not a switch. switches are much cheaper ($80) & perform much better realtek nics are known for their bad behaviour 2Gbit/s wirespeed requires serverclass hardware, not low power (at this point in time)

You're still using GroupWise? That's your problem there… I'd guess you are using a load balanced gateway. Try putting a rule for GroupWise and ssl traffic on a failover gateway. (Before the balancer) Lots of secure sites don't like the IP changing.

Solved, it was indeed just a simple tweak. I unchecked the System -> Routing -> Miscellaneous -> Failover default gateway option I had checked, and updated any outgoing firewall rule to explicitly use the advanced settings. Under advanced settings I changed Gateway from default to the name of my Gateway group. Now, outgoing firewall rules have a black sprocket icon next to them to indicate advanced settings which in my case is the Gateway group. Internal traffic rules, any LAN -> LAN for example, do not need this updated setting. Only traffic using a Gateway should be updated to use a Gateway group. To answer my initial questions explicitly: yes, this has to apply to each firewall rule that allows WAN traffic. If you have an outgoing firewall rule with the Advanced Settings -> Gateway left on 'default' it will NOT allow traffic to leave via the secondary WAN. I had to update my SSH rule and HTTPS rules independently for both to work. This is all clear in retrospect, but this might help someone else in the future to learn from my confusion. Once again, thanks heper!    

Are the network settings and the gateway set correct on the destination host?

Check the local firewall and default gateway on that target computer then.

A Feature-Request for PfSense: I jut configured Openstack to deploy an mtu using DHCP Option 26, but PFsense seems to ignore this one consequently. dhcpdump -i tapb8659f7c-df   TIME: 2017-06-29 13:22:00.059     IP: 10.40.50.3 (fa:16:3e:2c:e0:61) > 255.255.255.255 (ff:ff:ff:ff:ff:ff)     OP: 1 (BOOTPREQUEST) HTYPE: 1 (Ethernet)   HLEN: 6   HOPS: 0   XID: 11ac2ce2   SECS: 0 FLAGS: 0 CIADDR: 0.0.0.0 YIADDR: 0.0.0.0 SIADDR: 0.0.0.0 GIADDR: 0.0.0.0 CHADDR: fa:16:3e:2c:e0:61:00:00:00:00:00:00:00:00:00:00 SNAME: . FNAME: . OPTION:  53 (  1) DHCP message type        3 (DHCPREQUEST) OPTION:  50 (  4) Request IP address        10.40.50.3 OPTION:  61 (  7) Client-identifier        01:fa:16:3e:2c:e0:61 OPTION:  12 (  9) Host name                xxxxxxxxx OPTION:  55 (  9) Parameter Request List      1 (Subnet mask)                                             28 (Broadcast address)                                               2 (Time offset)                                             121 (Classless Static Route)                                               3 (Routers)                                             15 (Domainname)                                               6 (DNS server)                                             12 (Host name)                                             119 (Domain Search) –------------------------------------------------------------------------- TIME: 2017-06-29 13:22:00.059     IP: 10.40.50.2 (fa:16:3e:63:19:c0) > 10.40.50.3 (fa:16:3e:2c:e0:61)     OP: 2 (BOOTPREPLY) HTYPE: 1 (Ethernet)   HLEN: 6   HOPS: 0   XID: 11ac2ce2   SECS: 0 FLAGS: 0 CIADDR: 0.0.0.0 YIADDR: 10.40.50.3 SIADDR: 10.40.50.2 GIADDR: 0.0.0.0 CHADDR: fa:16:3e:2c:e0:61:00:00:00:00:00:00:00:00:00:00 SNAME: . FNAME: . OPTION:  53 (  1) DHCP message type        5 (DHCPACK) OPTION:  54 (  4) Server identifier        10.40.50.2 OPTION:  51 (  4) IP address leasetime      86400 (24h) OPTION:  58 (  4) T1                        43200 (12h) OPTION:  59 (  4) T2                        75600 (21h) OPTION:  1 (  4) Subnet mask              255.255.255.0 OPTION:  28 (  4) Broadcast address        10.40.50.255 OPTION:  15 ( 14) Domainname                openstacklocal OPTION:  12 ( 15) Host name                host-10-40-50-3 OPTION:  3 (  4) Routers                  10.40.50.3 OPTION: 121 ( 14) Classless Static Route    20a9fea9fe0a2832  .....(2                                             02000a283203    ...(2. OPTION:  6 (  8) DNS server                xxxxxxxxxxxxx OPTION:  26 (  2) Interface MTU            1450 –------------------------------------------------------------------------- This shouldn't be ignored because it'll result in fragmented packets / incorrect checksums since OS itself adds ~50 Bytes to a VXLAN-paket anyway.

In 1:1 you can set the NAT for the whole subnet if you enter 172.16.0.0 at "External subnet IP" and at "Internal IP" select network and 172.16.100.0/24 It doesn't matter if this also includes IPs assigned to computers in group A, since you haven't add an IP alias for these addresses to WAN.

@Valley: When I check my public IP address my dynamic Xfinity is shown - how can I setup up my Comcast static IP address as the default address? One way is to bridge a third interface to your WAN port and put the static IP's on devices on that interface. Other is to use VIP's and then port forwarding or 1:1 NAT to the intended devices. https://doc.pfsense.org/index.php/What_are_Virtual_IP_Addresses

Not wanting to hijack the OP's thread but, yeah, it took quite a lot of web searching to find that solution - not really knowing what to look for. Then, I had to find a workaround for the fact that the table is cleared on a firewall rule change. Doktornotor pointed me to the shellcmd package and its "afterfilterchangeshellcmd" option.  I use that to call a tiny PHP script that checks whether the table is empty.  If it is, the connection to the sending host is restarted and the table is rebuilt. Lots of fun :)

I think both agree on bad design.