I solved the HP part, running the following on the switch CLI:

****************************************************************************** * Copyright (c) 2010-2016 Hewlett Packard Enterprise Development LP          * * Without the owner's prior written consent,                                * * no decompiling or reverse-engineering shall be allowed.                    * ****************************************************************************** <hp>system-view System View: return to User View with Ctrl+Z. [HP]ip ttl-expires enable [HP]ip unreachables enable [HP]</hp>

Reference: https://community.hpe.com/t5/Switches-Hubs-and-Modems/Troubles-with-traceroute-in-Switch-HP-5500g/td-p/5880679

Now, tracing the route to Google Public DNS ( 8.8.8.8 ) my router appears:

# traceroute 8.8.8.8 traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets 1  10.100.132.1 (10.100.132.1)  0.551 ms  0.773 ms  0.940 ms 2  * * * 3  187.86.158.121 (187.86.158.121)  6.439 ms  6.440 ms  6.437 ms 4  172.21.1.133 (172.21.1.133)  7.674 ms  7.676 ms  7.672 ms 5  172.22.100.137 (172.22.100.137)  7.667 ms  7.663 ms  7.659 ms 6  172.22.100.121 (172.22.100.121)  7.654 ms  2.738 ms  2.578 ms 7  ip-187-86-128-93.vetorialnet.com.br (187.86.128.93)  2.638 ms  2.641 ms  3.039 ms 8  177-101-203-189.static.stech.net.br (177.101.203.189)  8.913 ms  9.578 ms  10.419 ms 9  xgborder-rs-pae-01-xe-0-0-0.3300.stech.net.br (200.152.253.252)  11.026 ms  11.136 ms  11.506 ms 10  * * * 11  108.170.245.161 (108.170.245.161)  37.144 ms 108.170.245.129 (108.170.245.129)  36.718 ms * 12  209.85.242.119 (209.85.242.119)  36.232 ms * 72.14.238.221 (72.14.238.221)  36.333 ms 13  google-public-dns-a.google.com (8.8.8.8)  55.787 ms  55.838 ms  55.688 ms

Maybe there is something like ttl-expires and/or unreachables for pfSense?

Anyone else got any input on this? Would it be worthwhile to maybe post a bug-report?

Me, I would make a transit network between the WAN pfSense and the proxy pfSense and disable NAT on the proxy. I would not try to put the same subnet on both sides of the proxy.

| $ route get 10.200.100.0 |

| route to: 10.10.100.0 |
| destination: 10.200.100.0 |
| mask: 255.255.255.0 |
| gateway: 10.10.0.1 |
| fib: 0 |
| interface: re0 |
| flags:<up,gateway,done,static></up,gateway,done,static> |

| recvpipe | sendpipe | ssthresh | rtt,msec | mtu | weight | expire |
| 0 | 0 | 0 | 0 | 1500 | 1 | 0 |

| $ route get 10.200.100.100 |

| route to: 10.200.100.100 |
| destination: 10.10.100.0 |
| mask: 255.255.255.0 |
| gateway: 10.10.0.1 |
| fib: 0 |
| interface: re0 |
| flags:<up,gateway,done,static></up,gateway,done,static> |

| recvpipe | sendpipe | ssthresh | rtt,msec | mtu | weight | expire |
| 0 | 0 | 0 | 0 | 1500 | 1 | 0 |

Resgard,

Rodrigo Prazim

Right.. I've figured it out. Basically I'm committing some sins which are causing unpredictable behaviour. I'm not too stressed about them now as I'll be moving away from this setup relatively soon and I have tested with a new, working setup.

For anyone else reading, my sins are:

Mixing VLAN and untagged traffic on the same interface That interface is a VirtIO (which doesn't really work with VLANs I believe)

Once I tested a new build running under ESXi, with VMXNET3 drivers and all separate interfaces (so to pfSense), problems went away and behaviour was as expected.

Hi,

For starters,  you should have your WANs in separate networks, not the same..
Then, for policy routing, you need to create IP Aliases with the hosts you want to use the specific WANs and set their gateway accordingly, in a LAN rule..

Put that rule on top of the other LAN rules.

Best regards

Kostas

I think I can setup multiple routers and have them online all at the same time.  I will be able to swap real easy and add devices easy.  I worked with EIGRP for 15 years so I have an idea of what a basic routing protocol can do.  BGP is not what I need.

So the big question is how stable is RIPv2?  I know RIP will not work in a large network but at my house it should do what I need.

If I go down this road and spend the money.  I don't want to find out pfsense does not work otherwise I will have to dump pfsense for something else that does work.  I feel like once I spend the money I am committed.

Either make it the default gateway or with the advanced option s

It's something like 'tcp outgoing address'. … Been too long since I bothered with it.
Try browsing the squid documentation

i just wasted an hour after setting up gateway groups wondering why NAT reflection broke…

i strongly suggest this side effect should be mentioned in the pfsense book - which i didnt find or overlooked.

Start by updating to the latest stable version. (read the upgrade notes)

roundrobin

Yes you can!

Create a Rule In a Firewall > rules > Lan, you can specify a gateway in Advanced Options from a specific source (this should be the IPs statically mapped, of your roommates. Create an aliases IPs). Put this rule on top of the generic pass all in LAN.

I'll check for the asymmetrical, it would mean that the 10.20.0.250 GW is bypassed on the return traffic, why not …

Messy my diagram, really? :( I've updated a new version showing up the physical links and couple of updates to make it clearer. I've got 3 physical NICs, 1 for the core, 1 for LAB A and 1 for LAB B. The ports are set to trunk mode with correct allowed VLANs.

The slowness is from 10.20.0.250 (red line) or from 10.30.0.250 (yellow line), both LAB use the same configuration, it makes sense they have the same issue.
LAB A and LAB B talk between each other using VLAN 101, when copying data this way (green line), I have no performance issue, it works well.

There is static route set in pfsense LAB A for 172.16.30.0/24 to use the core gateway 10.20.0.254, the LAB A gateway is 10.20.0.250 (VLAN 200), same for LAB B (VLAN 300). LAB A and LAB B can't talk to each other though the core, I don't want it, it is only to give access to my Citrix sources.

Hopefully it clarify the situation.

Pfsense_design_issue_slowness.jpg
Pfsense_design_issue_slowness.jpg_thumb

@kimkhan:

6. I have default gateway switching enabled in System/Advanced/Miscellaneous (If I don't have this checked the failover does not work)

It would seem that this is the only setting needed to do what you and I are attempting to accomplish. If you don't use this, the more advanced way would be to use the gateway groups and not set the automatic gateway switching. Examples show gateway groups are necessary if you are looking to balance load across one more more WANs. So far, I have not been able to get this to work well regardless of using the default gateway enabled setting or using the numerous multi-wan setup instructions posted in this forum and all over the internet. Another issue I have noticed is when either WAN goes down or down and back up, the user experience is high latency to internet connections and the pfSense GUI becomes unresponsive or incredibly slow. Restarting the firewall or restarting PHP-FPM seems to restore normal operation until the next time one of the WAN connections bounces or goes down.

It's been about two months since you posted this as a possible solution, I would be interested to know how it's working out for you not that some time has passed?

Thanks,

Markn455

Further testing shows that the 'flapping" of the satellite link really has little to do with it. Simple failover from WAN1 to WAN2 results in poor performance and loss of gui responsiveness. So far, I have not been able to get Multi-Wan failover to work properly. I have tried all the suggested configurations and even the most simple automatic gateway switching builtin to pfSense. If anyone has a working configure Dual WAN failover setup that works it would work. I don't even care about load balancing across the two WANs, just needing a failover configuration that actually works. I have tried the suggested configurations here and some on Youtube multiple times.

"Why would you be natting to internal rfc1918 networks?"

Because I connect to a wireless network that I don't manage that uses rfc1918 IPs.  Each wireless node (router) in the network gets configured with a random 10.0.0.0/29 network address during initial setup on each node.  The routing for these nodes is managed with OLSR on the wireless network.  Pfsense apparently used to have a plugin for OLSR, but doesn't any longer and I cannot add routes for my internal LAN to OLSR.  Nodes can come and go without notification or coordination on this network, so I can't reasonably maintain an accurate static route list, so I have a generic static 10.0.0.0/8 route out to that network interface to cover all wireless networks.    I'm only allocated a /29 on the wireless network, and I provide services from multiple internal LAN IPs, so I have NAT configured so it only consumes one wireless IP.  This is on my OPT1 interface, and the IP and gateway are provided via DHCP from the wireless node.

I'm open to suggestions for better ways to do this, but this is the only way I could see getting it to work with the restrictions I have.

My internal LAN is 10.10.6.0/24.  This works fine because the LAN interface's /24 route is more specific than the wireless /8, so things route properly.

The WAN port connects to my ISP, and is a 73.x.x.x/24 which is provided via DHCP from my cable modem.

So to recap:

To internet
73.x.x.1 (gateway)
  |
73.x.x.x/24
  WAN
+–---------+
| pfsense    | LAN--10.10.6.1/24----To internal LAN
+-----------+
OPT1
10.117.100.157/29
  |
10.117.100.153 (gateway)
  To a couple dozen or so random 10.x.x.x/29 networks routed by OLSR

"Do you have both gateways you get via dhcp as "default"?"  "Post up your gateway section"
The only gateway that is set default is the WAN (internet) side.

This is from my /conf/config.xml file:
<gateways><gateway_item><interface>opt1</interface>
                        <gateway>dynamic</gateway>
                        <name>MESH_NMT_DHCP</name>
                        <weight>1</weight>
                        <ipprotocol>inet</ipprotocol>

<monitor_disable></monitor_disable></gateway_item>
                <gateway_item><interface>wan</interface>
                        <gateway>dynamic</gateway>
                        <name>WAN_DHCP</name>
                        <weight>1</weight>
                        <ipprotocol>inet</ipprotocol>

<monitor_disable><defaultgw><latencyhigh>1500</latencyhigh>
                        <losshigh>100</losshigh></defaultgw></monitor_disable></gateway_item>
                <gateway_item><interface>wan</interface>
                        <gateway>dynamic</gateway>
                        <name>WAN_DHCP6</name>
                        <weight>1</weight>
                        <ipprotocol>inet6</ipprotocol>

<monitor_disable><defaultgw></defaultgw></monitor_disable></gateway_item></gateways>

and just for info:
<staticroutes><route><network>10.0.0.0/8</network>
                        <gateway>MESH_NMT_DHCP</gateway></route></staticroutes>

Normally netstat -nr shows this:
Internet:
Destination        Gateway            Flags      Netif Expire
default            73.x.x.1        UGS        em0
10.0.0.0/8        10.117.100.153    UGS        em2
10.10.6.0/24      link#2            U          em1
10.10.6.1          link#2            UHS        lo0
10.117.100.152/29  link#3            U          em2
10.117.100.153    10.117.100.153    UGHS        em2
10.117.100.157    link#3            UHS        lo0
73.x.x.0/24    link#1            U          em0
73.x.x.x      link#1            UHS        lo0
75.75.75.75        73.x.x.1        UGHS        em0
75.75.76.76        73.x.x.1        UGHS        em0
127.0.0.1          link#8            UH          lo0
172.16.0.0/12      10.117.100.153    UGS        em2

When it goes bad I see this:
Internet:
Destination        Gateway            Flags      Netif Expire
default            10.117.100.153    UGS        em2
10.0.0.0/8        10.117.100.153    UGS        em2
10.10.6.0/24      link#2            U          em1
10.10.6.1          link#2            UHS        lo0
10.117.100.152/29  link#3            U          em2
10.117.100.153    10.117.100.153    UGHS        em2
10.117.100.157    link#3            UHS        lo0
73.x.x.0/24    link#1            U          em0
73.x.x.x      link#1            UHS        lo0
75.75.75.75        73.x.x.1        UGHS        em0
75.75.76.76        73.x.x.1        UGHS        em0
127.0.0.1          link#8            UH          lo0
172.16.0.0/12      10.117.100.153    UGS        em2

I've looked through the various logs when the problem happens, and I don't see anything obviously wrong. 
I've played with various values and ultimately disabled gateway monitoring to make sure that isn't causing the problem.