Update pfsense 2.0.1 stable to 2.1 problem with routes
-
The issue I was experiencing was limited to the two LANs communicating. I have many rules on my WANs, but my LANs are rather simple (by design). Have you done any troubleshooting to know specifically where the issue is occurring and which routes are affected?
I always start with a ping and traceroute, and then check the firewall logs. I can usually determine rather quickly why things aren't getting from point A to point B using those three tools. If you need assistance interpreting those things, just post the data here. Another suggestion; I usually time my testing on 5-minute intervals so I can match up different logs with one another. Sometimes if you test, make a change, and test again, you can miss things in the logs.
-
The behavior of policy based routing, which is what we're dealing with here, changed between 1.2.X and 2.0.X. At that time automatic 'negate' rules were added behind the scenes that allowed traffic to negate a policy based rule when the destination is a local subnet. I was burnt by that change because I had rule sets that had previously not allowed traffic between local subnets that now suddenly did. Worse, I didn't realise anything had changed for some months. That'll teach me to read the release notes properly. ::) Anyway after some requests a check box to disable the negate rules was added in System: Advanced: Networking: I always have it unchecked, I don't want the negate rules catching me out again. I wonder if the behaviour changed again in 2.1, it seems to fit the symptoms.
Steve
-
This thread saved me over the weekend when I did my upgrade :)
Also worth noting, you need some new rules if you're using NAT reflection and policy based routing. I had to add rules for LAN->LAN, LAN2->LAN2, VLAN1->VLAN1 etc for NAT reflection to work.
EDIT - Oops, post corrected to say NAT reflection, not 1:1 NAT. I had a 1:1 NAT config page open at the time :)
-
do not compliqueis more, I can not lose more time with this because in pfsense 2.1 BETA works. Something as simple as a route, I can not lose more time. There is a bug and I just want to help you solve … A shame since pfsense 2.1 stable comes with a amount of improvements as MultiWAN dynamic dns and a bunch of bug fixes.
The real problem is that instead routes of enrutarte to the host that is, it takes you out to the wan and not because there is anything wrong, but simply routed bad this version of pfsense 2.1
We will have to wait ...
Just to say I can confirm a route problem on 2.1-RELASE.
For maykel535 and admins: On 2.1-RC1 (15aug snapshot) I got no problem, hence "bug had been introduced between 15 august and 16 sept (2.1 release date))
In my case I can not route one OpenVPN LAN. See http://forum.pfsense.org/index.php/topic,68031.0.html
Maybe jimp can help us….Thanks
-
I can also confirm that upgrading from 2.0.3 to 2.1 release has a routing issue. The LAN interface appears to route traffic fine but other interfaces only route to the default gateway. I've confirmed that the router ignores its static routes and locally connected subnets (for interfaces other than LAN). It just sends everything to the default gateway. All the interfaces in this setup are vlan interfaces (not sure if that matters). I will reiterate that all of this worked prior to upgrading to 2.1 release.
-
I had the same problem while upgrading from 2.0.1 to 2.1.0. After some digging on the subject I've realized that the problem is probably associated with <negate_networks>table. Quick comparison of /etc/inc/filter.inc shows that version 2.0.1 inserts directly connected networks into <negate_networks>table:
/* add a Negate_networks table */ $natrules .= "table <negate_networks>{"; if($direct_networks_list) $natrules .= " $direct_networks_list "; if($vpns_list) $natrules .= " $vpns_list "; $natrules .= "}\n";</negate_networks>
while in 2.1.0 this piece of code is missing:
/* add a Negate_networks table */ $aliases .= "table <negate_networks>"; if($vpns_list) $aliases .= "{ $vpns_list }"; $aliases .= "\n";</negate_networks>
I suspect that this is the cause of problem. The change isn't mentioned at "New Feature and Changes" (https://doc.pfsense.org/index.php/2.1_New_Features_and_Changes#NAT.2FFirewall_Rules.2FAlias) but one can see that it was made by user ermal on Nov 16, 2012 (https://github.com/pfsense/pfsense/commit/b4227df690fb7a989ead9b3928ebaaaa34b495eb
Edit: To clarify the role of <negate_networks>: pfSense is using this table in auto created rules that are connected to user rules with policy routing.
Regards,
John</negate_networks></negate_networks></negate_networks> -
Great job John!
I think we have just to wait and see… and hope some developer take care of it...
I had the same problem while upgrading from 2.0.1 to 2.1.0. ….
why "had"?
How did you solve it? Maybe replacing code on filter.inc? -
We do not know the reason of modifying <negate_networks>so it's probably better to add rule using GUI. For example if you have custom rule:
pass in quick on $LAN $GWFailOverGW from $AllowedLocalNets to any keep state label "USER_RULE: allow ... "
add the following one just BEFORE above rule:
pass in quick on $LAN from $AllowedLocalNets to $DirectlyConnectedNets keep state label "My negate policy routing for destination"
where
$AllowedLocalNets and $DirectlyConnectedNets are custom aliases.Regards,
John</negate_networks> -
why "had"?
How did you solve it? Maybe replacing code on filter.inc?I've simply reverted to redundant live copy of 2.0.1 release. English isn't my native language, so forgive me for any inaccuracy.
Regards,
john -
How did you solve it? Maybe replacing code on filter.inc?
ok - I've booted 2.1 release again, but I have plenty of rules with policy routing - so I ended with this quick (but very ugly) fix involving /etc/inc/filter.inc modification:
# diff /etc/inc/filter.inc /etc/inc/filter.inc_org 591,595d590 < /* xxxxxx 20131102 */ < $my_direct_nets_list = "192.168.1.0/24 172.16.1.0/24"; < if($my_direct_nets_list) < $aliases .= "table <my_direct_nets>{ $my_direct_nets_list }\n"; < 600,602d594 < /* xxxxxx 20131102 add direct nets */ < if($my_direct_nets_list) < $aliases .= "{ $my_direct_nets_list }";</my_direct_nets>
Regards
John -
Great job John!
I think we have just to wait and see… and hope some developer take care of it...
....Indeed I was thinking….
As I stated here http://forum.pfsense.org/index.php/topic,68031.msg375663.html#msg375663
my problem is for sure due to multiwan gw...I "diffed" /etc/inc/filter.inc 2.1-RC1 (working) and 2.1-Release... looking for differences affecting vpns and/or gw... And I found out this:
2.1-RC1/* exception(s) to a user rules can go here. */ /* rules with a gateway or pool should create another rule for routing to vpns */ if((($aline['route'] <> "") && (trim($aline['type']) == "pass") && strstr($dst, "any")) && (!isset($config['system']['disablenegate']))) { /* negate VPN/PPTP/PPPoE/Static Route networks for load balancer/gateway rules */ $negate_networks = " to <negate_networks>";</negate_networks>
2.1-Release
/* exception(s) to a user rules can go here. */ /* rules with a gateway or pool should create another rule for routing to vpns */ if((($aline['route'] <> "") && (trim($aline['type']) == "pass") && strstr($dst, "any")) && (!isset($config['system']['disablenegate']))) { /* negate VPN/PPTP/PPPoE/Static Route networks for load balancer/gateway rules */ $negate_networks = " to <negate_networks>" . filter_generate_port($rule, "destination");</negate_networks>
But I do not think that's the issue…
Maybe I can try to modify filter.inc and see...
-
$negate_networks = " to <negate_networks> " . filter_generate_port($rule, "destination");</negate_networks>
From memory, the filter_generate_port code was added because before that, a rule that fed traffic for a particular port(s) into a gateway group would result in a wider pass rule to Negate_Networks that passed traffic on any port. So this is actually a good fix to tighten up the behaviour.
-
Here is my take on the situation, using an example config like:
LAN1: 10.42.1.0/24
LAN2: 10.42.2.0/24
WAN1: some public IP
WAN2: some other public IP
OpenVPN tunnel 10.99.1.0/24 to a remote LAN 10.99.2.0/24
Gateway group: LoadBalance - WAN1 and WAN2 on tier1.User rule on LAN1: Pass source LAN1 subnet destination any gateway LoadBalance
The "negate_networks" thing in 2.0.n is saying:
"Oh no, a rule like that cannot possibly be what you want, because it just won't work to stuff traffic for LAN2, the OpenVPN tunnel and remote LAN into the LoadBalance gateway group. You (the sysadmin) have said you are happy to pass to destination any, so I (pfSense code) will help you by putting an extra rule in place that passes traffic for destination LAN2, OpenVPN tunnel and remote LAN without putting it into LoadBalance gateway group."Generally this is a good thing - traffic to destination any actually gets to its destination. Some across a VPN, some delivered locally direct to LAN2 and some fed into LoadBalance gateway group.
In 2.1 this "helping" is no longer done for locally connected networks (LAN2), so now it says:
"Oh no, a rule like that cannot possibly be what you want, because it just won't work to stuff traffic for the OpenVPN tunnel and remote LAN into the LoadBalance gateway group. You (the sysadmin) have said you are happy to pass to destination any, so I (pfSense code) will help you by putting an extra rule in place that passes traffic for destination OpenVPN tunnel and remote LAN without putting it into LoadBalance gateway group."So, in 2.1, the user rules need to actually say what traffic they want to pass between local LANs - nothing comes "for free" behind the scenes. Before a rule that feeds "destination any" traffic into a particular gateway or gateway group, you need to put any rules to pass traffic to other local LANs. It seems to me that this is a change in behaviour from 2.0.n to 2.1 and people with multi-LAN installs need to be aware of it and add user rules to their LAN(s) appropriately.
It is probably a good thing, as now in 2.1 you can use "destination any" into a gateway (group) and not also open up traffic from LAN1 to LAN2… But I wonder why it does not go further and stop doing the negate_networks for VPN? Maybe that would create too many issues for users who don't understand their rules? Pass rules on LANn to destination "VPN-networks" would have to be added before "destination any" rules into a gateway (group), people would feed "destination any" into a gateway (group) and then wonder why their VPN traffic died.
-
Really sorry but… I do not understand....
In 2.1 this "helping" is no longer done for locally connected networks (LAN2), so now it says:
"Oh no, a rule like that cannot possibly be what you want, because it just won't work to stuff traffic for the OpenVPN tunnel and remote LAN into the LoadBalance gateway group. You (the sysadmin) have said you are happy to pass to destination any, so I (pfSense code) will help you by putting an extra rule in place that passes traffic for destination OpenVPN tunnel and remote LAN without putting it into LoadBalance gateway group."If I do not get wrong With this "pfSense sw help" I should be able to get traffic pass between VPN LANs but not from LAN1 to LAN2 (LAN to DMZ in my case: indeed I do not remember but it seems to me like it was ok…)
Instead I can not pass traffic through mainsite LAN and VPN2 remote LAN (VPN1 is ok) .
But I guess, There's something simiilar in my case… because as I stated here http://forum.pfsense.org/index.php/topic,68031.15.html it seems pfsense tries "routing/loadbalancing" traffic directed to VPN2 remote LAN using default gw.So, in 2.1, the user rules need to actually say what traffic they want to pass between local LANs - nothing comes "for free" behind the scenes. Before a rule that feeds "destination any" traffic into a particular gateway or gateway group, you need to put any rules to pass traffic to other local LANs. It seems to me that this is a change in behaviour from 2.0.n to 2.1 and people with multi-LAN installs need to be aware of it and add user rules to their LAN(s) appropriately.
Correct me if i get wrong, but I doubt this is 2.1 feature… as Everything works flawless on 2.1-RCX otherwise it should not work also on BETA, RC0, RC1 etc.
Thanks
-
I should be able to get traffic pass between VPN LANs but not from LAN1 to LAN2
Yes, that is the way the code appears to work. So maybe there is something else also going on. The way to track it down is to look in /tmp/rules.debug and see:
a) What networks get included in <negate_networks>.
b) What rules direct traffic into a gateway or gateway group.
c) Which of those rules also have a pass rule above to allow traffic straight through to <negate_networks>.If you want to post your /tmp/rules.debug and an overview of which subnets are LAN1, LAN2, VPN tunnels, at the other end of a VPN etc, then it should be possible to analyse what is going on, and then find the code that is generating an unexpected/unwanted ruleset.</negate_networks></negate_networks>
-
If you want to post your /tmp/rules.debug and an overview of which subnets are LAN1, LAN2, VPN tunnels, at the other end of a VPN etc, then it should be possible to analyse what is going on, and then find the code that is generating an unexpected/unwanted ruleset.
Thank you… you have opened my eyes!!!
Those are the differences on /tmp/rules.debug
2.1-RC1 (online: production fw)
.... table <vpn_networks>{ 10.106.100.0/24 192.168.11.0/24 10.106.100.0/24 192.168.12.0/24 10.116.100.0/24 192.168.12.0/24 192.168.21.0/24 10.106.100.0/24 192.168.12.0/24 10.116.100.0/24 192.168.12.0/24 } table <negate_networks>{ 10.106.100.0/24 192.168.11.0/24 10.106.100.0/24 192.168.12.0/24 10.116.100.0/24 192.168.12.0/24 192.168.21.0/24 10.106.100.0/24 192.168.12.0/24 10.116.100.0/24 192.168.12.0/24 } ....</negate_networks></vpn_networks>
2.1-RELEASE (offline: test machine with just lan netif up)
... table <vpn_networks>{ 10.116.100.0/24 192.168.12.0/24 192.168.21.0/24} table <negate_networks>{ 10.116.100.0/24 192.168.12.0/24 192.168.21.0/24} ....</negate_networks></vpn_networks>
maybe (maybe!! ;) ) RC1 is a lil'bit redundant… but it works...
Hence This is the final proof: That's a bug.
Tomorrow I'll try to be "verbose"... -
Hmm, interesting stuff. As I think I said earlier I have 'negate networks' unchecked and added my own rules. Probably why I've not seen any issues.
Are you sure this is an issue with what ends up in the <negate_networks>table rather than what rules are generated from it?
Steve</negate_networks>
-
Hmm, interesting stuff. As I think I said earlier I have 'negate networks' unchecked and added my own rules. Probably why I've not seen any issues.
Are you sure this is an issue with what ends up in the <negate_networks>table rather than what rules are generated from it?
Steve</negate_networks>
As you can see above, probably, the issue is due to the <vpn_networks>because It does not list all vpn networks (e.g. there isn't 10.106.100.0) and (I think) <negate_networks>is made up of <vpn_networks>+other stuff…</vpn_networks></negate_networks></vpn_networks>
-
From my reading of the code, <negate_networks>now only contains the same as <vpn_networks>- other stuff is no longer put in it.
What is special about the VPN networks that are not listed in the 2.1-RELEASE rules.debug?
Are they not listed in "IPv4 Remote Network/s" box in the server/client config?The Remote Network/s box can now have a comma-separated list of networks. I just discovered that if I have more than 1 network there, then none of them appear in vpn_networks. If I cut it down to just 1 network, then it appears in vpn_networks. That is a bug. Got to go right now, back can look at it in a day or so, or maybe someone else can look at the code before then…</vpn_networks></negate_networks>
-
In my conf, I specify 2nd remote networks in OpenVPN server "advanced configuration" field as "in the previous (old) way", and hence this is because I can get just the first VPN to work.
Fake care of the following:( in my conf) disabling multiwan get both vpn to work!Meantime I submitted a bug on https://redmine.pfsense.org/issues/3309