Anyone else got any input on this? Would it be worthwhile to maybe post a bug-report?

Me, I would make a transit network between the WAN pfSense and the proxy pfSense and disable NAT on the proxy. I would not try to put the same subnet on both sides of the proxy.

| $ route get 10.200.100.0 | | route to: 10.10.100.0 | | destination: 10.200.100.0 | | mask: 255.255.255.0 | | gateway: 10.10.0.1 | | fib: 0 | | interface: re0 | | flags:<up,gateway,done,static></up,gateway,done,static> | | recvpipe | sendpipe | ssthresh | rtt,msec | mtu | weight | expire | | 0 | 0 | 0 | 0 | 1500 | 1 | 0 | | $ route get 10.200.100.100 | | route to: 10.200.100.100 | | destination: 10.10.100.0 | | mask: 255.255.255.0 | | gateway: 10.10.0.1 | | fib: 0 | | interface: re0 | | flags:<up,gateway,done,static></up,gateway,done,static> | | recvpipe | sendpipe | ssthresh | rtt,msec | mtu | weight | expire | | 0 | 0 | 0 | 0 | 1500 | 1 | 0 | Resgard, Rodrigo Prazim

Right.. I've figured it out. Basically I'm committing some sins which are causing unpredictable behaviour. I'm not too stressed about them now as I'll be moving away from this setup relatively soon and I have tested with a new, working setup. For anyone else reading, my sins are: Mixing VLAN and untagged traffic on the same interface That interface is a VirtIO (which doesn't really work with VLANs I believe) Once I tested a new build running under ESXi, with VMXNET3 drivers and all separate interfaces (so to pfSense), problems went away and behaviour was as expected.

Hi, For starters,  you should have your WANs in separate networks, not the same.. Then, for policy routing, you need to create IP Aliases with the hosts you want to use the specific WANs and set their gateway accordingly, in a LAN rule.. Put that rule on top of the other LAN rules. Best regards Kostas

I think I can setup multiple routers and have them online all at the same time.  I will be able to swap real easy and add devices easy.  I worked with EIGRP for 15 years so I have an idea of what a basic routing protocol can do.  BGP is not what I need. So the big question is how stable is RIPv2?  I know RIP will not work in a large network but at my house it should do what I need. If I go down this road and spend the money.  I don't want to find out pfsense does not work otherwise I will have to dump pfsense for something else that does work.  I feel like once I spend the money I am committed.

Either make it the default gateway or with the advanced option s It's something like 'tcp outgoing address'. … Been too long since I bothered with it. Try browsing the squid documentation

i just wasted an hour after setting up gateway groups wondering why NAT reflection broke… i strongly suggest this side effect should be mentioned in the pfsense book - which i didnt find or overlooked.

Start by updating to the latest stable version. (read the upgrade notes)

roundrobin

Yes you can! Create a Rule In a Firewall > rules > Lan, you can specify a gateway in Advanced Options from a specific source (this should be the IPs statically mapped, of your roommates. Create an aliases IPs). Put this rule on top of the generic pass all in LAN.

I'll check for the asymmetrical, it would mean that the 10.20.0.250 GW is bypassed on the return traffic, why not … Messy my diagram, really? :( I've updated a new version showing up the physical links and couple of updates to make it clearer. I've got 3 physical NICs, 1 for the core, 1 for LAB A and 1 for LAB B. The ports are set to trunk mode with correct allowed VLANs. The slowness is from 10.20.0.250 (red line) or from 10.30.0.250 (yellow line), both LAB use the same configuration, it makes sense they have the same issue. LAB A and LAB B talk between each other using VLAN 101, when copying data this way (green line), I have no performance issue, it works well. There is static route set in pfsense LAB A for 172.16.30.0/24 to use the core gateway 10.20.0.254, the LAB A gateway is 10.20.0.250 (VLAN 200), same for LAB B (VLAN 300). LAB A and LAB B can't talk to each other though the core, I don't want it, it is only to give access to my Citrix sources. Hopefully it clarify the situation. [image: Pfsense_design_issue_slowness.jpg] [image: Pfsense_design_issue_slowness.jpg_thumb]

@kimkhan: 6. I have default gateway switching enabled in System/Advanced/Miscellaneous (If I don't have this checked the failover does not work) It would seem that this is the only setting needed to do what you and I are attempting to accomplish. If you don't use this, the more advanced way would be to use the gateway groups and not set the automatic gateway switching. Examples show gateway groups are necessary if you are looking to balance load across one more more WANs. So far, I have not been able to get this to work well regardless of using the default gateway enabled setting or using the numerous multi-wan setup instructions posted in this forum and all over the internet. Another issue I have noticed is when either WAN goes down or down and back up, the user experience is high latency to internet connections and the pfSense GUI becomes unresponsive or incredibly slow. Restarting the firewall or restarting PHP-FPM seems to restore normal operation until the next time one of the WAN connections bounces or goes down. It's been about two months since you posted this as a possible solution, I would be interested to know how it's working out for you not that some time has passed? Thanks, Markn455

Further testing shows that the 'flapping" of the satellite link really has little to do with it. Simple failover from WAN1 to WAN2 results in poor performance and loss of gui responsiveness. So far, I have not been able to get Multi-Wan failover to work properly. I have tried all the suggested configurations and even the most simple automatic gateway switching builtin to pfSense. If anyone has a working configure Dual WAN failover setup that works it would work. I don't even care about load balancing across the two WANs, just needing a failover configuration that actually works. I have tried the suggested configurations here and some on Youtube multiple times.

"Why would you be natting to internal rfc1918 networks?" Because I connect to a wireless network that I don't manage that uses rfc1918 IPs.  Each wireless node (router) in the network gets configured with a random 10.0.0.0/29 network address during initial setup on each node.  The routing for these nodes is managed with OLSR on the wireless network.  Pfsense apparently used to have a plugin for OLSR, but doesn't any longer and I cannot add routes for my internal LAN to OLSR.  Nodes can come and go without notification or coordination on this network, so I can't reasonably maintain an accurate static route list, so I have a generic static 10.0.0.0/8 route out to that network interface to cover all wireless networks.    I'm only allocated a /29 on the wireless network, and I provide services from multiple internal LAN IPs, so I have NAT configured so it only consumes one wireless IP.  This is on my OPT1 interface, and the IP and gateway are provided via DHCP from the wireless node. I'm open to suggestions for better ways to do this, but this is the only way I could see getting it to work with the restrictions I have. My internal LAN is 10.10.6.0/24.  This works fine because the LAN interface's /24 route is more specific than the wireless /8, so things route properly. The WAN port connects to my ISP, and is a 73.x.x.x/24 which is provided via DHCP from my cable modem. So to recap: To internet 73.x.x.1 (gateway)   | 73.x.x.x/24   WAN +–---------+ | pfsense    | LAN--10.10.6.1/24----To internal LAN +-----------+ OPT1 10.117.100.157/29   | 10.117.100.153 (gateway)   To a couple dozen or so random 10.x.x.x/29 networks routed by OLSR "Do you have both gateways you get via dhcp as "default"?"  "Post up your gateway section" The only gateway that is set default is the WAN (internet) side. This is from my /conf/config.xml file: <gateways><gateway_item><interface>opt1</interface>                         <gateway>dynamic</gateway>                         <name>MESH_NMT_DHCP</name>                         <weight>1</weight>                         <ipprotocol>inet</ipprotocol> <monitor_disable></monitor_disable></gateway_item>                 <gateway_item><interface>wan</interface>                         <gateway>dynamic</gateway>                         <name>WAN_DHCP</name>                         <weight>1</weight>                         <ipprotocol>inet</ipprotocol> <monitor_disable><defaultgw><latencyhigh>1500</latencyhigh>                         <losshigh>100</losshigh></defaultgw></monitor_disable></gateway_item>                 <gateway_item><interface>wan</interface>                         <gateway>dynamic</gateway>                         <name>WAN_DHCP6</name>                         <weight>1</weight>                         <ipprotocol>inet6</ipprotocol> <monitor_disable><defaultgw></defaultgw></monitor_disable></gateway_item></gateways> and just for info: <staticroutes><route><network>10.0.0.0/8</network>                         <gateway>MESH_NMT_DHCP</gateway></route></staticroutes> Normally netstat -nr shows this: Internet: Destination        Gateway            Flags      Netif Expire default            73.x.x.1        UGS        em0 10.0.0.0/8        10.117.100.153    UGS        em2 10.10.6.0/24      link#2            U          em1 10.10.6.1          link#2            UHS        lo0 10.117.100.152/29  link#3            U          em2 10.117.100.153    10.117.100.153    UGHS        em2 10.117.100.157    link#3            UHS        lo0 73.x.x.0/24    link#1            U          em0 73.x.x.x      link#1            UHS        lo0 75.75.75.75        73.x.x.1        UGHS        em0 75.75.76.76        73.x.x.1        UGHS        em0 127.0.0.1          link#8            UH          lo0 172.16.0.0/12      10.117.100.153    UGS        em2 When it goes bad I see this: Internet: Destination        Gateway            Flags      Netif Expire default            10.117.100.153    UGS        em2 10.0.0.0/8        10.117.100.153    UGS        em2 10.10.6.0/24      link#2            U          em1 10.10.6.1          link#2            UHS        lo0 10.117.100.152/29  link#3            U          em2 10.117.100.153    10.117.100.153    UGHS        em2 10.117.100.157    link#3            UHS        lo0 73.x.x.0/24    link#1            U          em0 73.x.x.x      link#1            UHS        lo0 75.75.75.75        73.x.x.1        UGHS        em0 75.75.76.76        73.x.x.1        UGHS        em0 127.0.0.1          link#8            UH          lo0 172.16.0.0/12      10.117.100.153    UGS        em2 I've looked through the various logs when the problem happens, and I don't see anything obviously wrong.  I've played with various values and ultimately disabled gateway monitoring to make sure that isn't causing the problem.

Just to be clear, you just want your mobile IPsec clients to be able to communicate with an endpoint device across an OpenVPN tunnel?  Or is there more to it then that?