CE 2.7.2 to CE 2.7.2 routing issue
-
I had to 2.6.x virtualized installations running in two separate buildings, each with it's own internet connection.
A network was created to allow traffic between the buildings for management and backup.I converted one of the sites to physical hardware and at the same time installed 2.7.2 and restored the 2.6.x config.
(That was and adventure on it's own ...)
Got the 2.7.2 setup working with routing between the buildings working fine.I installed 2.7.2 in the second building and restored the 2.6 backup from the old install.
Worked through the exact same issues in the second building as in the first.
Internet activity works fine in building two, I can reach building one just fine from building two.However.... building one can't access building two.
Gateway shows as offline.I have compared all of the gateway, routing and rebooted both pfsense instances.
What's the magic here?
Edit: The dashboard traffic graph shows traffic in both directions on the interface in question in building one.
-
Hmm, no magic should be required.
If it worked in 2.6 I'd expect it to work in 2.7.2.It sounds like there were some issues with the hardware change. Possible the link interface doesn't have the gateway defined?
Is the link on separate NICs at each end? Are the firewall rules passing the monitoring pings?
-
@stephenw10
Link is working.
In building 2 I can open the management interface on pfsense from building one.
From building one, I see 100% packet loss on the dashboard gateway monitoring widget.VM interfaces are sometimes fun(tm) to correlate to physical ports, so I double checked that on every port in use.
It's like there is a firewall rule allowing traffic only one direction - but the interfaces on both sides of this link are configured with "any to any" rules.
Edit:
I even turned off pfblocker in the hopes that rules were being generated for that interface when they shouldn't be.Will pfblocker rules survive after pfblocker is disabled?
-
If there are any pfBlocker rules you would see them in the floating rules tab.
You could have a missing route on one end. Reply packets still work because the interface acts as a WAN with reply-to tags applies to incoming traffic.
-
Something is just .. flaky.
pfblocker was configured for two specific networks and I don't think there were any floating rules.
After turning it off, there are no floating rules at all and the only rules on WAN now are the "block bogon" and "block private" rules.I'll try turning it back on see what happens.
Still seeing 100% packet loss on the gateway monitor.
Bldg1 LAN interface 192.168.10.1
Bldg1 Link interface 192.168.30.1
Bldg2 Link interface 192.168.30.2
Bldg2 LAN interface 192.168.20.1from bldg1 I can't ping 192.168.30.2 or 192.168.20.1
(192.168.20.1 is the monitor ip address for Link gateway.)I can get to things on the 192.168.20.0 subnet if adressing it via IP -- name resolution isn't working apparently.
So the any to any rule on the link isn't allowing some protocols... -
You should not have 'block private networks' set on the link interface on either side. The source traffic there is all from private subnets.
-
@stephenw10 It isn't - that's only on the WAN interface -- incoming if I read it correctly.
-
Did you check the routing table on each side?
-
-
Rules on the interface where the remote system is showing down:
Above is for bldg1 -- blow is the corresponding interface in bldg 2
-
Are they using the internal LAN IP addresses as the monitoring IP on each side? You seem to have static routes for those in the table which would otherwise be unexpected.
Do pings also fail simply to the remote side of the link?
Are you NATing that traffic? Do your Outbound NAT rules include the link interface?
-
the monitor ip is the LAN network internal gateway of the opposite pfsense.
Local Ip -> remote IP -> remote LAN gateway.
30.1 30.2 20.1Previously, (2.6x) this didn't work unless static routes were manually defined.
It worked between 2.7.2 and 2.6, but this issue arose when I went to 2.7.2 on both ends.The traffic across that .30 network is primarily between the buildings with no restrictions that I can see.
Any protocol, any to any on both ends.Pings fail only one way. Some traffic is getting through both ways.
It's like the firewall rule on one end isn't really operating correctly.
-
And that was it -- I opened and resaved both rules -- no changes mind you -- and boom -- it works as expected.
Sooo... lesson learned:
Don't restore backups.
Don't save them encrypted so you can read and manually transcribe the settings because restores are ...Well, it did restore the settings -- they just didn't take effect. :/
-
Hmm, odd. I assume no alerts were shown about failures to load the ruleset?
-
@stephenw10
No alerts in Bldg 2.
Bldg 1 ... I have pfblocker installed and it spams alerts so hard I'm thinking about just removing it :/Are there any alternatives?
I can go back to a pi-hole elsewhere on the network if there isn't. -
Hmm, odd. Sometimes if the running ruleset doesn't match the configured rules it's because it's unloadable. But when that happens you see an alert in the GUI. That should be different to anything pfBlocker is alerting you to.
Otherwise it appears it simply hadn't loaded. I assume the firewall rebooted when you restored the config? -
I did not see any related alerts - and the restore in bldg 2 caused a reboot and it was rebooted manually before and after the cutover.
-
Is it spamming alerts on the IP rules or the DNS rules? You could just disable the rules causing issues. I was never happy with the rulesets we could come across so we do IP via pfBlocker and DNS filtering via a third party service. It also would have helped to do some packet capturing to see if packets are even making it to the other side. That would verify if it were a routing or firewall issue.
-
@Stewart
pfBlockerNG MaxMind - MaxMind now requires a License Key! Review the IP tab: MaxMind settings for more information. @ 2024-02-29 16:15:40Once an hour.
I have turned off everything I can find about IP. -
@MakOwner said in CE 2.7.2 to CE 2.7.2 routing issue:
@Stewart
pfBlockerNG MaxMind - MaxMind now requires a License Key! Review the IP tab: MaxMind settings for more information. @ 2024-02-29 16:15:40Once an hour.
I have turned off everything I can find about IP.I think it now also requires an Account ID in addition to the License Key be provided to download updates. I think this new requirement took effect in January of this year.
I had to make a recent change in the Suricata IDS/IPS package because of the MaxMind authentication API change.
