Strange Multi-WAN behaviour after upgrading to v26.03 from v25.11.1
-
After upgrading pfSense+ to v26.03, I've noticed that my multi-wan configuration isn't behaving as it did in the previous version. My router is a Netgate 6100.
I'm in Canada (irrelevant) and using two internet providers - Cogeco & Bell. Cogeco is a cable connection, and Bell is a fiber optic connection which is faster. Both connections are configured as a gateway group, with Bell configured as Tier 1 priority, and Cogeco as Tier 2. I've only been using v26,03 for a couple of days, and have noticed odd behaviour.
Despite Bell being my primary connection, I notice most traffic passes through the Cogeco connection, regardless of how I configure the default gateway in the routing options. The only way to stop traffic from being passed through the Cogeco connection is if I manually disable that interface. At that point traffic then passes through the Bell connection. The real oddity - if I go to a website to display my current IP address, it shows the correct address as expected when I change the default gateway.
To ensure I wasn't going mad, I rebooted the router in the previous v25.11.1 environment, and the odd behaviour disappears, and the gateway group works as expected, as well as changing the default gateway. Traffic will properly flow through to whatever I set the default gateway to.
Have I encountered a bug? Have I inadvertently changed a setting somewhere I wasn't aware of? After upgrading, I did alter some firewall rules, but did not touch any settings related to routing or interfaces. Has anyone else observed similar behaviour with this version of pfSense?
I discovered this anomaly quite by accident after I connected a new machine to my 10G LAN segment. I went to speedtest.net to check internet speeds and noticed I was only achieving the maximum speeds offered by the Cogeco connection. I logged into the pfSense interface and observed the traffic graph - sure enough, traffic was going in & out on the Cogeco connection. IP address showed it had detected the Bell connection. That's what led me down this road, and when I disabled the Cogeco interface, I was able to achieve the higher throughput offered by the Bell connection (almost 1.5Gbps down, and around 1Gbps up).
Open to listening to ideas. If I have encountered a bug, it's not a big deal to roll back to the older version. I'd rather look for a resolution to the issue, because something is different between the versions.
-
After more tinkering, I've discovered that if I restore an older configuration file from before the upgrade, pfSense+ v26.03 functions as it should. The problem only returns if I restore the "bad" configuration file I saved after the update.
My apologies - this appears to be either a configuration issue, or corruption somewhere in the configuration. I will likely waste more time diagnosing than to simply restore from a known good configuration state. This will be the route I will take, and re-create any new firewall rules needed.
-
@NetTechBrad You are not alone on this.
I also saw strange behaviour in multiwan routing that cannot be explained but I couldn't reliably describe it.
It was related with stickyness too.
After readjusting the outbound policy routing rules, (wih many additons and deletions) I cannot replicate it any more.
But also worked fine in 25.11 without "adjustments" -
@netblues Interesting. Perhaps you have a different problem. During this past week, I had time to dig deeper into the cause of my issue. I can confirm, with absolute certainty, my issue was caused by a rule I created. My issue wasn't caused by a bug in v26.03. But, that doesn't mean you haven't discovered issues.
The rule that I created was on the LAN side, directing all traffic bound for port 8080 for the destination address to pass through the Cogeco connection. Unknown to me at the time I initially posted the issue, many sites that measure internet speed, such as speedtest.net, use port 8080 in the background. Thus, when I measured speed, regardless of my default gateway setting, traffic was forced to talk on port 8080. This explains why it would always provide speed measurements for the Cogeco WAN connection. I encountered one internet speed test that didn't use port 8080, and it properly measured speed based on my default gateway configuration when the rule was enabled. The base routing was always working as it should when the rule was enabled, because the "what's my ip address" lookup would always be accurate throughout the entire process. It was the speed tests that were throwing me off for a bit until I realized what was happening.
Naturally, if I disabled or deleted the rule, the speed test would work as expected on all sites. For clarity, I created this rule after updating to v26.03. I didn't have this rule configured in v25.11.1, which explains why I could boot between the different environments, and receive different results.
I learned a few things while chasing this problem I created for myself, so all was not lost. The rule isn't really a problem, I just need to remember the consequences.
Hope you are able to resolve your issue.
-
@NetTechBrad Glad you nailed it.
In my case, as I said, I cannot replicate it anymore, so it is also "fixed"
-
@NetTechBrad fwiw I think I had the same situation after switching from 25.11 to 26.03RC ... ended up picking through my XML config with a fine tooth comb to uncover some bad firewall rules that referenced invalid interfaces or gateways. Once I surgically removed those everything was fine. Guess 26.03 is just stricter about the way it parses the rules.
