Weird routing issue
-
Hello,
we have a very weird routing issue in pfsense.
Using pfsense in 2 locations which are interconnected via dedicated network interface. Both locations have separate internet provider.
in location1 there is network 172.16.26.0/23 and in location 2 there are networks 172.18.28.0/23, 192.168.1.0/24
In routing table on pfsense1/location1 , we have correct routes to the remote networks in location2
netstat -rn (excerpt):
172.18.28.0/23 172.16.40.1 UGS igb2
192.168.1.0/24 172.16.40.1 UGS igb2though, we can access 192.168.1.0/24 in location2 from location1, we can NOT access 172.168.28.0/23 because packets do not get passed to the interconnect interface (gw 172.16.40.1) but they leave on the Internet-Provider WAN Interface, i.e. apparently they go the wrong route.
What we don`t understand is why it "just works" with 192.168.1.0 but not with 172.18.28.0 - we see no apparent difference in configuration regarding these 2 networks.
Any clue how we can troubleshoot this?
regards
Roland -
i made some other weird observation:
normally, we have added our routing via an ALIAS of network definitions, i.e. ALIASNAME - SUB1/MASK, SUB2/MASK, SUB3/MASK
if i add another network to the ALIASNAME and add that in system->routing->ROUTES-TAB , then the newly added network does not get added.
I need to explicitly add the network in the ROUTES-TAB, though other routing from the alias works.Maybe i need to flush/reload something elserwere so i can use ALIAS Names for routing?
-
"Using pfsense in 2 locations which are interconnected via dedicated network interface"
This how you would normally do it..
So in your site 1 you would have routes
172.18.28.0/23 172.16.40.2
192.168.1.0/24 172.16.40.2in site 2 you would have
172.16.26.0/23 172.16.40.1
Do have gateway to the other side of the transit in each site?
What are your rules on your interfaces. If your forcing traffic out a gateway say your internet gateway before you allow traffic using the normal pfsense routing then yeah your going to have problems.
-
yes, it`s like you visualized!
we have a rule on the 172.16.26.0 lan which does policy based routing into WAN/Internet for ANY destination, but what i don´t understand is why for target 192.168.1.0/24 packets being correctly routed via the interconnect and for target 172.16.28.0/23 they are being routed via WAN/Internet - imho they should go the same way and that is what i don`t understand.
if we add extra-rule for 172.18.28.0 to do policy based routing via Interconnect, it works - but why don`t we need that rule for 192.168.1.0, too ?
-
dude if you want helping finding what is wrong in you rules - post them..
-
here we go
Rules for LAN interface on network 172.16.26.0/23
SERVERLAN net (172.16.28.0/23)
MGMTLAN net (172.16.37.0/24)
VMLAN net (172.16.30.0/23)
DMZ_TIER2 net (172.16.36.0/24)
DMZ_UM_EXT net (public ip subnet, ip/subnet not shown here because of data protection)IPv4+6 * LAN net * This Firewall * * none Pass any to this Firewall
IPv4 * LAN net * SERVERLAN net * * none LAN to SERVERLAN
IPv4 * LAN net * MGMTLAN net * * none LAN to MGMTLAN
IPv4 * LAN net * VMLAN net * * none LAN to VMLAN
IPv4 * LAN net * 192.168.81.0/24 * * none LAN to OpenVPN
IPv4 * LAN net * DMZ_UM_EXT net * * none LAN to DMZ1
IPv4 * LAN net * DMZ_TIER2 net * * none LAN to DMZ2
IPv4 * LAN net * * * WAN_UM none everything else to internet -
is a screenshot really that hard??
I don't see any rules that allow any traffic to the other sites 192.168.1/24 or 172.18.28/23
is serverlan really 172.18.28??
This is what you said this network was
"172.18.28.0/23 172.16.40.1 UGS igb2"From those rules I don't see how you could get across to anything..
-
sorry, there is some infos in the description fields i do not want to have on the internet and i
m new to this forum so it didnt come to my mind.yes, from the ruleset i should not be able to connect to 192.168.0.x IP adresses, but i CAN.
i can login via ssh to 192.168.1.50 for example and on the remote system, i see client`s ip 172.16.27.45 in netstat. so no nat or anything else in place.
i don`t understand how this is possible as there is no rule which would allow or policyroute this.
-
Well if wan_um gateway can get there then you could get there.
Do you have any floating rules?
What other routes do you have? When you do a traceroute to this 192.168.1.50 from your client 172.16.27.45 what do you show?
Have seen users have any any in their floating and then wonder why stuff is working even though they have a block on the interface ;)
-
no floating rules in place.
i investigated further and apparently the thing is all about "negate_networks"
https://forum.pfsense.org/index.php?topic=66776.45
i can see with "pfctl -T show -t negate_networks that it contains 192.168.1.0 (and others) but not 172.18.28.0. the question is , why.
will read into it further.
thanks for help so far
-
Because 192.168.1.0/24 is defined in a VPN somewhere, most likely.
You need to bypass policy routing for the 172.18.28.0/23 subnet.
Your problem is not routing in general, it is that you are policy routing out the WAN_UM gateway, which means everything not explicitly exempted in the rules above gets shoved that way without regard to the routing table. Why are you doing that? Is WAN_UM not the default gateway?
-
yes, 192.168.1.0 is also defined in a deactivated ipsec tunnel definition - apparently that
s the reason why it exists in negate_networks, though - and thats the reason why 192.168.1.0 is (by chance) being routed the correct way and 172.18.28.0 not -
Dude as Derelict said and I stated in post 1 you need to allow rule above your rule that shoves everything down that gateway..
"If your forcing traffic out a gateway say your internet gateway before you allow traffic using the normal pfsense routing then yeah your going to have problems." -
yes, i know.
but i was more curious why 192.168.1.0 was working THOUGH (i.e. without explicit allow rule).
you should know how your firewall works and how things behave.
you should be able to explain things and do not wonder about miraculous firewall behaviour.
that kann kill your security. -
Agreed, but since you have figured that out.. Now its time to correct your rules.
-
Your problem is not routing in general, it is that you are policy routing out the WAN_UM gateway, which
means everything not explicitly exempted in the rules above gets shoved that way without regard to
the routing table. Why are you doing that? Is WAN_UM not the default gateway?WAN_UM is a gateway group, you can
t set that as a default gateway and you can only route to a gw-group via policy routing, cant you ? -
if you force a gateway, be it default or a group or whatever.. You have to allow rules above that if you want your clients to talk to other networks off pfsense that are not reachable through that gateway your forcing traffic through. Is that simple!
