Update pfsense 2.0.1 stable to 2.1 problem with routes

vielfede

Great job John!
I think we have just to wait and see… and hope some developer take care of it...
....

Indeed I was thinking….
As I stated here http://forum.pfsense.org/index.php/topic,68031.msg375663.html#msg375663
my problem is for sure due to multiwan gw...

I "diffed" /etc/inc/filter.inc 2.1-RC1 (working) and 2.1-Release... looking for differences affecting vpns and/or gw... And I found out this:
2.1-RC1

	/* exception(s) to a user rules can go here. */
	/* rules with a gateway or pool should create another rule for routing to vpns */
	if((($aline['route'] <> "") && (trim($aline['type']) == "pass") && strstr($dst, "any")) && (!isset($config['system']['disablenegate']))) {
		/* negate VPN/PPTP/PPPoE/Static Route networks for load balancer/gateway rules */
		$negate_networks = " to <negate_networks>";</negate_networks>

2.1-Release


	/* exception(s) to a user rules can go here. */
	/* rules with a gateway or pool should create another rule for routing to vpns */
	if((($aline['route'] <> "") && (trim($aline['type']) == "pass") && strstr($dst, "any")) && (!isset($config['system']['disablenegate']))) {
		/* negate VPN/PPTP/PPPoE/Static Route networks for load balancer/gateway rules */
		$negate_networks = " to <negate_networks>" . filter_generate_port($rule, "destination");</negate_networks>

But I do not think that's the issue…

Maybe I can try to modify filter.inc and see...

phil.davis

$negate_networks = " to <negate_networks> " . filter_generate_port($rule, "destination");</negate_networks>

From memory, the filter_generate_port code was added because before that, a rule that fed traffic for a particular port(s) into a gateway group would result in a wider pass rule to Negate_Networks that passed traffic on any port. So this is actually a good fix to tighten up the behaviour.

phil.davis

Here is my take on the situation, using an example config like:
LAN1: 10.42.1.0/24
LAN2: 10.42.2.0/24
WAN1: some public IP
WAN2: some other public IP
OpenVPN tunnel 10.99.1.0/24 to a remote LAN 10.99.2.0/24
Gateway group: LoadBalance - WAN1 and WAN2 on tier1.

User rule on LAN1: Pass source LAN1 subnet destination any gateway LoadBalance

The "negate_networks" thing in 2.0.n is saying:
"Oh no, a rule like that cannot possibly be what you want, because it just won't work to stuff traffic for LAN2, the OpenVPN tunnel and remote LAN into the LoadBalance gateway group. You (the sysadmin) have said you are happy to pass to destination any, so I (pfSense code) will help you by putting an extra rule in place that passes traffic for destination LAN2, OpenVPN tunnel and remote LAN without putting it into LoadBalance gateway group."

Generally this is a good thing - traffic to destination any actually gets to its destination. Some across a VPN, some delivered locally direct to LAN2 and some fed into LoadBalance gateway group.

In 2.1 this "helping" is no longer done for locally connected networks (LAN2), so now it says:
"Oh no, a rule like that cannot possibly be what you want, because it just won't work to stuff traffic for the OpenVPN tunnel and remote LAN into the LoadBalance gateway group. You (the sysadmin) have said you are happy to pass to destination any, so I (pfSense code) will help you by putting an extra rule in place that passes traffic for destination OpenVPN tunnel and remote LAN without putting it into LoadBalance gateway group."

So, in 2.1, the user rules need to actually say what traffic they want to pass between local LANs - nothing comes "for free" behind the scenes. Before a rule that feeds "destination any" traffic into a particular gateway or gateway group, you need to put any rules to pass traffic to other local LANs. It seems to me that this is a change in behaviour from 2.0.n to 2.1 and people with multi-LAN installs need to be aware of it and add user rules to their LAN(s) appropriately.

It is probably a good thing, as now in 2.1 you can use "destination any" into a gateway (group) and not also open up traffic from LAN1 to LAN2… But I wonder why it does not go further and stop doing the negate_networks for VPN? Maybe that would create too many issues for users who don't understand their rules? Pass rules on LANn to destination "VPN-networks" would have to be added before "destination any" rules into a gateway (group), people would feed "destination any" into a gateway (group) and then wonder why their VPN traffic died.

vielfede

Really sorry but… I do not understand....

@phil.davis:

In 2.1 this "helping" is no longer done for locally connected networks (LAN2), so now it says:
"Oh no, a rule like that cannot possibly be what you want, because it just won't work to stuff traffic for the OpenVPN tunnel and remote LAN into the LoadBalance gateway group. You (the sysadmin) have said you are happy to pass to destination any, so I (pfSense code) will help you by putting an extra rule in place that passes traffic for destination OpenVPN tunnel and remote LAN without putting it into LoadBalance gateway group."

If I do not get wrong With this "pfSense sw help" I should be able to get traffic pass between VPN LANs but not from LAN1 to LAN2 (LAN to DMZ in my case: indeed I do not remember but it seems to me like it was ok…)

Instead I can not pass traffic through mainsite LAN and VPN2 remote LAN (VPN1 is ok) .
But I guess, There's something simiilar in my case… because as I stated here http://forum.pfsense.org/index.php/topic,68031.15.html it seems pfsense tries "routing/loadbalancing" traffic directed to VPN2 remote LAN using default gw.

@phil.davis:

So, in 2.1, the user rules need to actually say what traffic they want to pass between local LANs - nothing comes "for free" behind the scenes. Before a rule that feeds "destination any" traffic into a particular gateway or gateway group, you need to put any rules to pass traffic to other local LANs. It seems to me that this is a change in behaviour from 2.0.n to 2.1 and people with multi-LAN installs need to be aware of it and add user rules to their LAN(s) appropriately.

Correct me if i get wrong, but I doubt this is 2.1 feature… as Everything works flawless on 2.1-RCX otherwise it should not work also on BETA, RC0, RC1 etc.

Thanks

phil.davis

I should be able to get traffic pass between VPN LANs but not from LAN1 to LAN2

Yes, that is the way the code appears to work. So maybe there is something else also going on. The way to track it down is to look in /tmp/rules.debug and see:
a) What networks get included in <negate_networks>.
b) What rules direct traffic into a gateway or gateway group.
c) Which of those rules also have a pass rule above to allow traffic straight through to <negate_networks>.

If you want to post your /tmp/rules.debug and an overview of which subnets are LAN1, LAN2, VPN tunnels, at the other end of a VPN etc, then it should be possible to analyse what is going on, and then find the code that is generating an unexpected/unwanted ruleset.</negate_networks></negate_networks>

vielfede

@phil.davis:

If you want to post your /tmp/rules.debug and an overview of which subnets are LAN1, LAN2, VPN tunnels, at the other end of a VPN etc, then it should be possible to analyse what is going on, and then find the code that is generating an unexpected/unwanted ruleset.

Thank you… you have opened my eyes!!!

Those are the differences on /tmp/rules.debug

2.1-RC1 (online: production fw)


....
table <vpn_networks>{ 10.106.100.0/24 192.168.11.0/24 10.106.100.0/24 192.168.12.0/24 10.116.100.0/24 192.168.12.0/24 192.168.21.0/24 10.106.100.0/24 192.168.12.0/24 10.116.100.0/24 192.168.12.0/24 }
table <negate_networks>{ 10.106.100.0/24 192.168.11.0/24 10.106.100.0/24 192.168.12.0/24 10.116.100.0/24 192.168.12.0/24 192.168.21.0/24 10.106.100.0/24 192.168.12.0/24 10.116.100.0/24 192.168.12.0/24 }
....</negate_networks></vpn_networks>

2.1-RELEASE (offline: test machine with just lan netif up)


...
table <vpn_networks>{ 10.116.100.0/24 192.168.12.0/24 192.168.21.0/24}
table <negate_networks>{ 10.116.100.0/24 192.168.12.0/24 192.168.21.0/24}
....</negate_networks></vpn_networks>

maybe (maybe!! ;) ) RC1 is a lil'bit redundant… but it works...
Hence This is the final proof: That's a bug.
Tomorrow I'll try to be "verbose"...

stephenw10

Hmm, interesting stuff. As I think I said earlier I have 'negate networks' unchecked and added my own rules. Probably why I've not seen any issues.

Are you sure this is an issue with what ends up in the <negate_networks>table rather than what rules are generated from it?

Steve</negate_networks>

vielfede

@stephenw10:

Hmm, interesting stuff. As I think I said earlier I have 'negate networks' unchecked and added my own rules. Probably why I've not seen any issues.

Are you sure this is an issue with what ends up in the <negate_networks>table rather than what rules are generated from it?

Steve</negate_networks>

As you can see above, probably, the issue is due to the <vpn_networks>because It does not list all vpn networks (e.g. there isn't 10.106.100.0) and (I think) <negate_networks>is made up of <vpn_networks>+other stuff…</vpn_networks></negate_networks></vpn_networks>

phil.davis

From my reading of the code, <negate_networks>now only contains the same as <vpn_networks>- other stuff is no longer put in it.
What is special about the VPN networks that are not listed in the 2.1-RELEASE rules.debug?
Are they not listed in "IPv4 Remote Network/s" box in the server/client config?

The Remote Network/s box can now have a comma-separated list of networks. I just discovered that if I have more than 1 network there, then none of them appear in vpn_networks. If I cut it down to just 1 network, then it appears in vpn_networks. That is a bug. Got to go right now, back can look at it in a day or so, or maybe someone else can look at the code before then…</vpn_networks></negate_networks>

vielfede

In my conf, I specify 2nd remote networks in OpenVPN server "advanced configuration" field as "in the previous (old) way", and hence this is because I can get just the first VPN to work.
Fake care of the following:( in my conf) disabling multiwan get both vpn to work!

Meantime I submitted a bug on https://redmine.pfsense.org/issues/3309

phil.davis

The Remote Network/s box can now have a comma-separated list of networks. I just discovered that if I have more than 1 network there, then none of them appear in vpn_networks. If I cut it down to just 1 network, then it appears in vpn_networks. That is a bug. Got to go right now, back can look at it in a day or so, or maybe someone else can look at the code before then…

https://github.com/pfsense/pfsense/pull/850 fixes the problem when there is a comma-separated list in "Remote Network(s)".

In my conf, I specify 2nd remote networks in OpenVPN server "advanced configuration" field as "in the previous (old) way"

Hmmm - perhaps there used to be code that found the entries in the advanced configuration also? But I don't think so. Anyway, with the fix above you will be able to put lists of Remote Network(s) and have it come through correctly to vpn_networks and negate_networks.

vielfede

@phil.davis:

The Remote Network/s box can now have a comma-separated list of networks. I just discovered that if I have more than 1 network there, then none of them appear in vpn_networks. If I cut it down to just 1 network, then it appears in vpn_networks. That is a bug. Got to go right now, back can look at it in a day or so, or maybe someone else can look at the code before then…

https://github.com/pfsense/pfsense/pull/850 fixes the problem when there is a comma-separated list in "Remote Network(s)".

In my conf, I specify 2nd remote networks in OpenVPN server "advanced configuration" field as "in the previous (old) way"

Hmmm - perhaps there used to be code that found the entries in the advanced configuration also? But I don't think so. Anyway, with the fix above you will be able to put lists of Remote Network(s) and have it come through correctly to vpn_networks and negate_networks.

Thanks Phil.
I fixed the problem on filter.inc on pfSense test server and now vpn_networks and negates_networks seem ok (using comma separated remote network on vpn_server config).
Now I should to test it on production…. I hope to be able to do it on next monday....
I'll let you know...

frhack

After upgrading to 2.1 I encountered routes problem.

My configuration is:

2 WAN:

WAN
WAN_VOIP

1 DMZ with:
- 1 web server host forced to use WAN gw (WAN gw specified in DMZ firewall rule)
- 1 asterisk host forced to use WAN VOIP gw (WAN_VOIP gw specified in DMZ firewall rule)

2 LAN:

LAN forced to to use WAN VOIP (WAN GW specified in LAN FW rule)
LAN_VOIP forced to use WAN (WAN VOIP GW specified in LAN_VOIP FW rule)

After upgrade to 2.1: no connection from internet to DMZ asterisk, DMZ webserver

At this time I downgraded to 2.03

Thanks

–
Francesco

maykel535

I solved this problem. In the version pfsense 2.1 stable, also add static routes, you also have to add in paragraph rules to allow traffic to that route.

This well because security have gotten more but if you do not know the routes stop working when updating from 2.0.1 to 2.1 pfsense

Thanks for all.

vielfede

Fix successfully tested!
Now both vpns work.

Indeed now I discovered anothter issue on 1:1 NAT and hence I'm still on 2.1-RC1….
But this will be another thread... we can change this one to [SOLVED]

skenter

Hi I am facing the same problem. My configuration includes 3 Wan (Wan1,Wan2,Wan3) in one gateway group and 2 lans. My static is like x.x.x.x -> Gw2 through is ignored. Can you give me some more information about the solution? Do I have to add a floating firewall rule?
Thank you very much.

phil.davis

@skenter:

Hi I am facing the same problem. My configuration includes 3 Wan (Wan1,Wan2,Wan3) in one gateway group and 2 lans. My static is like x.x.x.x -> Gw2 through is ignored. Can you give me some more information about the solution? Do I have to add a floating firewall rule?
Thank you very much.

The fix in my December 09 post is only relevant to configs that have OpenVPN instances with comma-separated lists of subnets in the "remote network/s" field. That fix involves changing /etc/inc/filter.inc
Post more info about your configuration, the order of the rules on each of your LANs and the content of /tmp/rules.debug - that should help sort out where and why the traffic is not being directed as expected.

skenter

Hi thank you for your reply. As you can see from the static route i am trying to route packets with destination 193.193.185.90 from GW OTENET - 192.200.9.1 on interface Wan2. Here https://skydrive.live.com/redir?resid=82D07EFA9D38DD9A!326 you can also see firewall rules and the gateway group regarding multi wan configuration. I found also this post about similar problem http://forum.pfsense.org/index.php/topic,49963.0.html
Thank you

phil.davis

@skenter:

Hi thank you for your reply. As you can see from the static route i am trying to route packets with destination 193.193.185.90 from GW OTENET - 192.200.9.1 on interface Wan2. Here https://skydrive.live.com/redir?resid=82D07EFA9D38DD9A!326 you can also see firewall rules and the gateway group regarding multi wan configuration. I found also this post about similar problem http://forum.pfsense.org/index.php/topic,49963.0.html
Thank you

I think this is now a "feature" of pfSense 2.1 - in older versions of pfSense, when you sent destination all to a gateway [group], it added a rule just before the gateway rule. That extra rule passed other traffic that it thought was local, without pushing it into the gateway [group]. Now this behaviour is limited to only OpenVPN tunnel networks and the remote networks at the end of those tunnels.

You need to add a rule before the rule that sends destination * to Gateway:
Pass source LANnet port * destination 193.193.185.90/31 port * gateway none (*)

That will pass the matching traffic to the ordinary routing table, where your static route will be used.

The general principle is to explicitly put in pass rules to match and pass any traffic that needs to use the ordinary routing table (traffic to other local subnets, or subnets reached by static routes, or even private subnets reached across site-to-site VPN links). After these pass rule(s) then put general "destination all" rules that are intended to feed general internet traffic into gateway [groups].

For example, I my systems, I have all my internal networks across all offices within the 10.42.0.0/16 network (10.42.1.0/24 10.42.2.0/24 …). I put a pass rule on each LAN... from LANnet to 10.42.0.0/16. Then put the rules directing traffic to gateway [groups].

skenter

Hi thank you very much , i think i did it correctly but unfortunately still seems to be a problem.

https://skydrive.live.com/redir?resid=82D07EFA9D38DD9A!332&authkey=!AKSOKDzmOxpv6N0

Any idea ? Thank you again for your time.