PfSense 2.1 Floating rules for Multi Wan doesn't work.

Reiner030

Hi,

nice idea with the different tcp_outgoing addresses…
I found yesterday also a site which show this option (before I found this thread ^^):
http://linuxaria.com/pills/setup-squid-to-use-multiple-outgoing-ip-addresses

This thread seems the only one (with "content") who mentioned that there is a bug in pfSense 2.1 with floating rules.
Because of right topic I add my information here, too.

Since update from 2.0.1 to 2.1.0 we have in remote office also the problem that only our default WAN interface is accessible from outside... :( Here my test from yesterday night; we are using there 1st WAN with PPPoE, 2nd WAN with local modem-router.

Here each a test to http://ident.me :

21:48:14.600710 IP 177.103.xxx.xxx.51527 > 176.58.123.25.32362: Flags [s], seq 1591963423, win 65228, options [mss 1452,nop,wscale 7,sackOK,TS val 1064842937 ecr 0], length 0
21:48:17.599925 IP 192.168.0.2.4478 > 176.58.123.25.80: Flags [s], seq 1591963423, win 65228, options [mss 1452,nop,wscale 7,sackOK,TS val 1064845937 ecr 0], length 0
21:48:20.799857 IP 192.168.0.2.4478 > 176.58.123.25.80: Flags [s], seq 1591963423, win 65228, options [mss 1452,nop,wscale 7,sackOK,TS val 1064849137 ecr 0], length 0
21:48:23.999792 IP 192.168.0.2.4478 > 176.58.123.25.80: Flags [s], seq 1591963423, win 65228, options [mss 1452,sackOK,eol], length 0
21:48:27.199723 IP 192.168.0.2.4478 > 176.58.123.25.80: Flags [s], seq 1591963423, win 65228, options [mss 1452,sackOK,eol], length 0
21:48:30.399660 IP 192.168.0.2.4478 > 176.58.123.25.80: Flags [s], seq 1591963423, win 65228, options [mss 1452,sackOK,eol], length 0

[code]21:51:46.841211 IP 192.168.0.2.1286 > 176.58.123.25.24577: Flags [s], seq 187921402, win 65228, options [mss 1452,nop,wscale 7,sackOK,TS val 1065055182 ecr 0], length 0
21:51:49.840521 IP 177.103.xxx.xxx.23922 > 176.58.123.25.80: Flags [s], seq 187921402, win 65228, options [mss 1452,nop,wscale 7,sackOK,TS val 1065058182 ecr 0], length 0
21:51:53.040452 IP 177.103.xxx.xxx.23922 > 176.58.123.25.80: Flags [s], seq 187921402, win 65228, options [mss 1452,nop,wscale 7,sackOK,TS val 1065061382 ecr 0], length 0
21:51:56.240386 IP 177.103.xxx.xxx.23922 > 176.58.123.25.80: Flags [s], seq 187921402, win 65228, options [mss 1452,sackOK,eol], length 0
21:51:59.440321 IP 177.103.xxx.xxx.23922 > 176.58.123.25.80: Flags [s], seq 187921402, win 65228, options [mss 1452,sackOK,eol], length 0

Interesting part: the 1st line connects over backup WAN Interface with right IP but to wrong destination port... and every repeated initializing packet goes to right destination port but over wrong main WAN interface ?

I was thinking about testing today if there is a difference between localhost and lan interface by accessing remote pbx but I forgot to deactivate floating rules so the employees had tested it before I had time for it ^^.

Using normal rules works nice so it must be a floating rule bug... 
=> Is it perhaps possible easy to fix it with a "remote github" patch ?

First I was not sure if it's really a floating rule bug because
[list]
[li] incoming connections to firewall on slave WAN interface are unusable since update, too.
If you have same problems here the solution:
=> I fixed it with NATting:[list]
[li]WAN1:22   => localhost:22,[/li]
[li]WAN1:443 => localhost:443,[/li]
[li]WAN2:22   => localhost:22 and[/li]
[li]WAN2:443 => localhost:443.[/li]
[/list][/li]
[li]DynDNS service can't be updated anymore, too...
But the service should be bound on 2nd WAN interface directly?[/li]
[/list]

Bests[/s][/s][/s][/s][/s][/code][/s][/s][/s][/s][/s][/s]

Reiner030

mmh… addititional strange behavior... I manually run / later re-checked on webgui.
The registered dynamic name service is registered to use WAN2 ... but got IP from WAN1 and switched here to wrong IP

Dec 3 14:55:58	php: rc.dyndns.update: phpDynDNS (xxx.no-ip.org): (Success) DNS hostname update successful.
Dec 3 14:55:58	php: rc.dyndns.update: phpDynDNS: updating cache file /conf/dyndns_wannoip'xxx.no-ip.org'0.cache: 177.103.xxx.xxx
Dec 3 14:55:57	php: rc.dyndns.update: DynDns (xxx.no-ip.org): 177.103.xxx.xxx extracted from checkip.dyndns.org
Dec 3 14:55:57	php: rc.dyndns.update: DynDNS (xxx.no-ip.org): Current Service: noip
Dec 3 14:55:57	php: rc.dyndns.update: DynDNS (xxx.no-ip.org): DynDns _checkStatus() starting.
Dec 3 14:55:56	php: rc.dyndns.update: DynDNS (xxx.no-ip.org): DynDns _update() starting.
Dec 3 14:55:56	php: rc.dyndns.update: DynDns (xxx.no-ip.org): DynDns: cacheIP != wan_ip. Updating. Cached IP: 201.83.xxx.xxx WAN IP: 177.103.xxx.xxx
Dec 3 14:55:56	php: rc.dyndns.update: DynDns (xxx.no-ip.org): Current WAN IP: 177.103.xxx.xxx Cached IP: 201.83.xxx.xxx
Dec 3 14:55:56	php: rc.dyndns.update: DynDns (xxx.no-ip.org): 177.103.xxx.xxx extracted from checkip.dyndns.org
Dec 3 14:55:53	php: rc.dyndns.update: DynDNS (xxx.no-ip.org): running get_failover_interface for wan. found re0
Dec 3 14:55:53	php: rc.dyndns.update: DynDns (xxx.no-ip.org): 177.103.xxx.xxx extracted from checkip.dyndns.org
Dec 3 14:55:53	php: rc.dyndns.update: DynDns: updatedns() starting

Perhaps better for a new thread:
Why is checkip.dyndns.org used for other dyndns services ?
Would be nice if checking service can be made selectable.

I knew its is / was a nice service. But since yesterday I don't think so anymore…
I used them for my SQuiD loadbalancing tests... and got 127.0.0.1 back :D

Also DynDNS won't check IPv6 addresses because they say in their supportforum it would always stay same IP.
Ok this should be normally the case (especially for firewalls) but it would be nice service to get them either in response for re-check settings.

http://ident.me offer IPv6/IPv4 address and showed me yesterday 127.0.0.1 and the right local IP address in answer page.
http://v4.ident.me only IPv4
http://v6.ident.me only IPv6

Reiner030

Hi,

on normal usage a very nice option ;)

@ruggero:

i think i have a solution :
instead of squid use squid3_dev .

in custom options :
"
acl venticinque random 1/4
acl settantacinquediv2 random 0.5

tcp_outgoing_address 192.168.4.1 venticinque
tcp_outgoing_address 192.168.3.1 settantacinquediv2
tcp_outgoing_address 192.168.2.1

I justed tested it with this additional ACL RegEx (needs perhaps some improvements) so we can let Youtube videos through the good line and block it in the not so good backup line:

acl_youtube urlpath_regex http://www.youtube.com/watch\?v=.*
acl wan2 random 1/2

tcp_outgoing_address <wan1 ip=""> youtube
tcp_outgoing_address <wan2 ip=""> wan2
tcp_outgoing_address <wan1 ip=""></wan1></wan2></wan1>

Pitily there is a problem if one of the lines are down… then every x request is timing out... :(

Is it perhaps possible to let SQuiD know by local files/checks what lines are up?

Not very RAM friendly would be a solution to run for each WAN interface a SQuiD instance and in front of it the main SQuiD with these SQuiDs as parents... then he can auto-select/fallback as he is allowed to ask them by rule.

Reiner030

@Reiner030:

acl_youtube urlpath_regex http://www.youtube.com/watch\?v=.*
acl wan2 random 1/2

tcp_outgoing_address <wan1 ip=""> youtube
tcp_outgoing_address <wan2 ip=""> wan2
tcp_outgoing_address <wan1 ip=""></wan1></wan2></wan1>

ah, squid3-dev matches irregulary in transparent mode also urlpath_regex - correct is url_regex.
And I added further sniffed URLs also for HTML5 blocking (perhaps too much, but better than too few ;)):

acl Youtube_Streams url_regex (youtube.com|youtube.be|m.ytimg.com|s.ytimg.com|s2.ytimg.com|s3.ytimg.com|googlevideo.com)/(.*\.(flv|swf)|player204|stream_204|watchtime|generate_204|videoplayback)

tcp_outgoing_address <wan2 ip=""> Youtube_Streams</wan2>

Actual my testing "live" pfSense goes out only on WAN1 even I set tcp_outgoing_address onto WAN2 IP ???.
So I can't test it correctly but I guess it can be made shorter like this with ReGex:

acl Youtube_Streams url_regex (youtube.com|youtube.be|m.ytimg.com|s\d*.ytimg.com|i\d*.ytimg.com|googlevideo.com)/(.*\.(flv|swf)|(player|stream_|generate_)204|watchtime|videoplayback)

basitkhan

I downloaded the lib modules for squid 3.3.8 from the following link

http://e-sac.siteseguro.ws/pfsense/8/amd64/All/ldd/

please let me know that the acl random method solve the issue? and if the one Wan goes down what will happen then?
it will send all the traffic to valid interface or it will still keep sending the packets to failed wan also?

need clarification on this.

Thanks

Reiner030

Hi,

@Basit:

I downloaded the lib modules for squid 3.3.8 from the following link

http://e-sac.siteseguro.ws/pfsense/8/amd64/All/ldd/

please let me know that the acl random method solve the issue? and if the one Wan goes down what will happen then?
it will send all the traffic to valid interface or it will still keep sending the packets to failed wan also?

need clarification on this.

Thanks

Its for us not an urgent task to use loadbalanced proxy and I had problems to getting SQuidGuard+SQuid3-dev both run.
So I downgraded to stable SQuid and we use the fibre line as normal (with youtube bound to this wan ip).
If the gateway failover then only non-youtube video content can be loaded.

I used the i386 one on an Soekris appliance http://e-sac.siteseguro.ws/pfsense/8/All/ldd/ added like here described:
http://forum.pfsense.org/index.php/topic,62256.msg373587.html#msg373587

… ah just updated.... updates fetched... I must see If I had time for testing it out next weeks when less work is to do.

Bests

hyrol

Finally, after a long time trying pfSense Squid Package + Multi Wan and I have managed to find its way in a deadlock.
pfSense 2.1 Squid Package + Multi Wan, no longer using the Floating Rules, but using the Interface Groups.
Good Luck Everyone.

![Interface Groups.jpg](/public/imported_attachments/1/Interface Groups.jpg)
![Interface Groups.jpg_thumb](/public/imported_attachments/1/Interface Groups.jpg_thumb)
![Proxy Server.jpg](/public/imported_attachments/1/Proxy Server.jpg)
![Proxy Server.jpg_thumb](/public/imported_attachments/1/Proxy Server.jpg_thumb)
![Internet Rules.jpg](/public/imported_attachments/1/Internet Rules.jpg)
![Internet Rules.jpg_thumb](/public/imported_attachments/1/Internet Rules.jpg_thumb)
![Floating Rules.jpg](/public/imported_attachments/1/Floating Rules.jpg)
![Floating Rules.jpg_thumb](/public/imported_attachments/1/Floating Rules.jpg_thumb)

niebla

Thanks, and congratulations!

What version of squid and squidguard are you using?

Please send a screenshot of your system->routing screen showing your default route.

hyrol

Still configure the same as pfSense 2.0.3 Squid Package Multi Wan, only changes to the Floating Rules to Interface Group.

![Lan Rules.jpg_thumb](/public/imported_attachments/1/Lan Rules.jpg_thumb)
![Lan Rules.jpg](/public/imported_attachments/1/Lan Rules.jpg)
Groups.jpg_thumb

Gateways.jpg_thumb

SaFi

@hyrol
Thank you ..
I wondering what's squid version you talking about 2.7 or 3.8 or it doesn't matter?
secondly where you use interface named "internet" I saw it has no rules, will you be more detailed

regards
SaFi

hyrol

Under the Menu=> Interface, you can see Interface Group.

Interface.jpg_thumb

niebla

Created the interface group. Squid uses the default gateway only with 2.1.

hyrol

@SaFi:

@hyrol
Thank you ..
I wondering what's squid version you talking about 2.7 or 3.8 or it doesn't matter?
secondly where you use interface named "internet" I saw it has no rules, will you be more detailed

regards
SaFi

Im try for squid 2.7, you can try for squid 3.8 you can tell me later for the result.
If you expert configure for inbound/outbound for WANs under Interface Group.. Actually that use for WANs rules not for LANs rules.

niebla

The problem is squid is using the default route and does not care about interface groups. What am I missing?

hyrol

Actually this is not Load-Balance Round Robin, this is Load-Balance Bandwith Agreggation and you can see all the WANs its working.
It is worth it from nothing.

Load-Balance.jpg_thumb

niebla

How does Squid know to use the interface group instead of the default gateway?

hyrol

I do not know how Squid works, most importantly it works. ;)

niebla

Squid works with multi-wan on 2.0.3 by using floating rules. Users have reported that it is not working with 2.1 with multi-wan even when using floating rules.

Many users are looking for a way to use multi-wan and squid using 2.1.

When this is solved many of us who have squid and multi wan will be able to use 2.1, and be grateful to the person who provides the solution.

basitkhan

@hyrol:

Finally, after a long time trying pfSense Squid Package + Multi Wan and I have managed to find its way in a deadlock.
pfSense 2.1 Squid Package + Multi Wan, no longer using the Floating Rules, but using the Interface Groups.
Good Luck Everyone.

I have followed the same steps but all in vain,
squid still uses only default gateway :( does not work with Interface Group

still waiting to fix it…

hyrol

If you check whatmyip you can see only default WAN, but you can see all the WANs work together.

Note: This Load-Balance Bandwith Agreggation not Load-Balance Round Robin.