Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Multi WAN seems to be poorly implemented

    Scheduled Pinned Locked Moved Routing and Multi WAN
    48 Posts 10 Posters 18.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      dhatz
      last edited by

      Thx jimp.

      What would happen in a load-balancing pfsense setup with two gateways with sticky enabled, if one gateway failed? What would prevent pf from remembering the old (finished) sessions tracks for src.track (e.g. 30 minutes) and keep sending traffic its way?

      I am having in mind the issue raised in
      http://lists.freebsd.org/pipermail/freebsd-pf/2006-December/002860.html
      http://lists.freebsd.org/pipermail/freebsd-pf/2006-December/002861.html

      In that example they suggest using pfctl -K to remove source tracking entries.

      1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        I believe the way our multi-wan works none of that info applies, as it does some behind-the-scenes magic when a WAN fails that may be doing the right thing here already. Ermal would have to say for sure though.

        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • F
          frater
          last edited by

          But what code does that checkbox execute?
          Can I do something from the console….?

          1 Reply Last reply Reply Quote 0
          • D
            dhatz
            last edited by

            To see the code, do a search, e.g.

            fgrep lb_use_sticky /etc/inc/*

            Basically it enables the "sticky-address" feature of pf.

            1 Reply Last reply Reply Quote 0
            • jimpJ
              jimp Rebel Alliance Developer Netgate
              last edited by

              The code triggered by the checkbox to activate sticky can be found in /etc/inc/interfaces.inc

              As for the GUI field to set the timeout manually, here is what does that:
              https://github.com/bsdperimeter/pfsense/commit/4573641589d50718b544b778cea864cfd725078a

              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              1 Reply Last reply Reply Quote 0
              • F
                frater
                last edited by

                I really ment…
                can I write some value to some pseudo-file in /proc or doesn't it work that way?

                The checkbox / input field in the php-site does something...
                Can you tell us what it does or can this be backported to 2.0

                I'm insisting because it's a bit of a show stopper for my implementation....

                1 Reply Last reply Reply Quote 0
                • jimpJ
                  jimp Rebel Alliance Developer Netgate
                  last edited by

                  Read my last post again, I provided everything you'd need to know to make the change on 2.0.1 by hand.

                  Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                  Need help fast? Netgate Global Support!

                  Do not Chat/PM for help!

                  1 Reply Last reply Reply Quote 0
                  • F
                    frater
                    last edited by

                    Did you edit your previous post?….
                    Anyhow

                    I applied those 2 patches (I wasn't able to create a patch file from it, but downloaded the 2 raw files with fetch)
                    I set the timeout to 3600 seconds which should be sufficient.
                    If this works I will have the best of both worlds (equal spreading of bandwidth and stability)

                    Tomorrow I will know if it does its job better.

                    sticky.pfsense.png
                    sticky.pfsense.png_thumb

                    1 Reply Last reply Reply Quote 0
                    • D
                      dhatz
                      last edited by

                      @frater:

                      Tomorrow I will know if it does its job better.

                      So, what's your impression of the new src.track feature (i.e. sticky with a timeout)?

                      And have you tested how it handles failure of one of the two gateways? (see my question in the previous page)

                      TIA.

                      1 Reply Last reply Reply Quote 0
                      • F
                        frater
                        last edited by

                        Because there was still one site that gave problems I didn't dare to post immediately.
                        But I never had any problems accessing the 2 hosting servers of our own.

                        I think it's working…
                        I assume it's properly working when your line goes down. The behaviour will depend a lot on what it should do with the current states (flush them or keep them open).

                        I just updated to 2.01, so I needed to patch it again.
                        I now did it in a way that is easy to share with the community.
                        It's a bit less work now....

                        I do think it should have been backported to 2.01

                        Cheers and happy new year

                        
                        cd /usr/local/www
                        cp -p system_advanced_misc.php system_advanced_misc.php.2.01
                        patch system_advanced_misc.php <system_advanced_misc.php.patch<br>cd /etc/inc
                        cp -p filter.inc filter.inc.201
                        patch filter.inc <filter.inc.patch< pre="">/etc/inc/filter.inc.patch
                        

                        281a282,284

                        if (isset($config['system']['lb_use_sticky']) && is_numeric($config['system']['srctrack']) && ($config['system']['srctrack'] > 0))
                              $rules .= "set timeout src.track {$config['system']['srctrack']}\n";

                        
                        /usr/local/www/system_advanced_misc.php.patch
                        

                        58a59

                        $pconfig['srctrack'] = $config['system']['srctrack'];
                        105c106
                        <              if($_POST['lb_use_sticky'] == "yes")


                        if($_POST['lb_use_sticky'] == "yes") {
                        107c108,109
                        <              else


                        $config['system']['srctrack'] = $_POST['srctrack'];
                                      } else
                        192a195,200
                        function sticky_checked(obj) {
                              if (obj.checked)
                                      jQuery('#srctrack').attr('disabled',false);
                              else
                                      jQuery('#srctrack').attr('disabled','true');
                        }
                        269c277
                        <                                                                      />


                        onClick="sticky_checked(this)" />
                        278a287,292

                        " class="formfld unknown" >

                        >                                                                      "By default this is 0, so source tracking is removed as soon as the state expires. " .
                                                                                              "Setting this timeout higher will cause the source/destination relationship to persist for longer periods of time."); ?>
                        387c401
                        <                                                                      ---

                        1 Reply Last reply Reply Quote 0
                        • F
                          frater
                          last edited by

                          I upgraded to 2.01 and re-applied the patch….
                          It's not working  :-(

                          I am getting kicked from our hosting server from time to time. 'tcpdump' is running at the same time and as you can see it all of a sudden switches to the other connection.
                          This is not sticky at all......

                          I did think it was working properly before I upgraded, although I did get some strange behaviour on other servers....

                          Can I check if srctrack is really set properly?
                          I have little knowledge of FreeBSD / pfilter

                          tcpdump -i eth0 -n 'port 8443'

                          14:33:33.041319 IP 89.250.179.117.29754 > 46.243.24.60.pcsync-https: P 1949:2786(837) ack 2746 win 16425
                          14:33:33.041658 IP 46.243.24.60.pcsync-https > 89.250.179.117.29754: . 2746:4206(1460) ack 2786 win 96
                          14:33:33.041676 IP 46.243.24.60.pcsync-https > 89.250.179.117.29754: P 4206:4254(48) ack 2786 win 96
                          14:33:33.092396 IP 89.250.179.117.19784 > 46.243.24.60.pcsync-https: . ack 98209 win 16425
                          14:33:33.102207 IP 89.250.179.117.19784 > 46.243.24.60.pcsync-https: P 8928:9765(837) ack 98209 win 16425
                          14:33:33.102225 IP 89.250.180.164.13586 > 46.243.24.60.pcsync-https: S 929859587:929859587(0) win 8192 <mss 1460,nop,wscale="" 2,nop,nop,sackok="">14:33:33.102315 IP 46.243.24.60.pcsync-https > 89.250.180.164.13586: S 1120370230:1120370230(0) ack 929859588 win 5840 <mss 7="" 1460,nop,nop,sackok,nop,wscale="">14:33:33.102667 IP 46.243.24.60.pcsync-https > 89.250.179.117.19784: . 98209:99669(1460) ack 9765 win 228
                          14:33:33.102681 IP 46.243.24.60.pcsync-https > 89.250.179.117.19784: P 99669:99685(16) ack 9765 win 228
                          14:33:33.108718 IP 89.250.180.164.38584 > 46.243.24.60.pcsync-https: S 1307358421:1307358421(0) win 8192 <mss 1460,nop,wscale="" 2,nop,nop,sackok="">14:33:33.108751 IP 46.243.24.60.pcsync-https > 89.250.180.164.38584: S 2972341496:2972341496(0) ack 1307358422 win 5840 <mss 7="" 1460,nop,nop,sackok,nop,wscale="">14:33:33.114151 IP 89.250.180.164.16897 > 46.243.24.60.pcsync-https: S 2970339047:2970339047(0) win 8192 <mss 1460,nop,wscale="" 2,nop,nop,sackok="">14:33:33.114183 IP 46.243.24.60.pcsync-https > 89.250.180.164.16897: S 1688705918:1688705918(0) ack 2970339048 win 5840 <mss 7="" 1460,nop,nop,sackok,nop,wscale="">14:33:33.120833 IP 89.250.180.164.48746 > 46.243.24.60.pcsync-https: S 794117234:794117234(0) win 8192 <mss 1460,nop,wscale="" 2,nop,nop,sackok="">14:33:33.120865 IP 46.243.24.60.pcsync-https > 89.250.180.164.48746: S 3021423565:3021423565(0) ack 794117235 win 5840 <mss 7="" 1460,nop,nop,sackok,nop,wscale="">14:33:33.126519 IP 89.250.180.164.52196 > 46.243.24.60.pcsync-https: S 1506349235:1506349235(0) win 8192 <mss 1460,nop,wscale="" 2,nop,nop,sackok="">14:33:33.126550 IP 46.243.24.60.pcsync-https > 89.250.180.164.52196: S 3390423845:3390423845(0) ack 1506349236 win 5840</mss></mss></mss></mss></mss></mss></mss></mss></mss> 
                          

                          again….  can't this feature be backported?????

                          1 Reply Last reply Reply Quote 0
                          • D
                            dhatz
                            last edited by

                            @frater:

                            Can I check if srctrack is really set properly?

                            You can check if there is a rule. From shell prompt run:

                            pfctl -sr | fgrep src.track

                            1 Reply Last reply Reply Quote 0
                            • jimpJ
                              jimp Rebel Alliance Developer Netgate
                              last edited by

                              Also check:

                              pfctl -st
                              

                              To see the current timer values.

                              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                              Need help fast? Netgate Global Support!

                              Do not Chat/PM for help!

                              1 Reply Last reply Reply Quote 0
                              • F
                                frater
                                last edited by

                                pfctl -st | grep src.track
                                src.track 1800s
                                pfctl -sr | grep src.track

                                So, I guess I didn't implement the patch properly this time….

                                Can you supply a patch file or just 2 the files already patched?

                                1 Reply Last reply Reply Quote 0
                                • jimpJ
                                  jimp Rebel Alliance Developer Netgate
                                  last edited by

                                  The fun thing about git is each commit is a patch. If you have the commit id, you have the patch. If you do

                                  git show 4573641589d50718b544b778cea864cfd725078a
                                  

                                  Then you get something usable as a patch file.

                                  commit 4573641589d50718b544b778cea864cfd725078a
                                  Author: jim-p <jimp@pfsense.org>Date:   Tue Nov 15 16:28:45 2011 -0500
                                  
                                      Add a gui field to set the source tracking timeout for sticky connections.
                                  
                                  diff --git a/etc/inc/filter.inc b/etc/inc/filter.inc
                                  index 29864df..fdd43b7 100644
                                  --- a/etc/inc/filter.inc
                                  +++ b/etc/inc/filter.inc
                                  @@ -280,6 +280,8 @@ function filter_configure_sync($delete_states_if_needed = true) {
                                   		/* User defined maximum table entries in Advanced menu. */
                                   		$rules .= "set limit table-entries {$config['system']['maximumtableentries']}\n";
                                   	}
                                  +	if (isset($config['system']['lb_use_sticky']) && is_numeric($config['system']['srctrack']) && ($config['system']['srctrack'] > 0))
                                  +		$rules .= "set timeout src.track {$config['system']['srctrack']}\n";
                                  
                                   	// Configure flowtable support if enabled.
                                   	flowtable_configure();
                                  diff --git a/usr/local/www/system_advanced_misc.php b/usr/local/www/system_advanced_misc.php
                                  index d25c96d..e1da772 100644
                                  --- a/usr/local/www/system_advanced_misc.php
                                  +++ b/usr/local/www/system_advanced_misc.php
                                  @@ -56,6 +56,7 @@ $pconfig['proxyuser'] = $config['system']['proxyuser'];
                                   $pconfig['proxypass'] = $config['system']['proxypass'];
                                   $pconfig['harddiskstandby'] = $config['system']['harddiskstandby'];
                                   $pconfig['lb_use_sticky'] = isset($config['system']['lb_use_sticky']);
                                  +$pconfig['srctrack'] = $config['system']['srctrack'];
                                   $pconfig['gw_switch_default'] = isset($config['system']['gw_switch_default']);
                                   $pconfig['preferoldsa_enable'] = isset($config['ipsec']['preferoldsa']);
                                   $pconfig['racoondebug_enable'] = isset($config['ipsec']['racoondebug']);
                                  @@ -102,9 +103,10 @@ if ($_POST) {
                                   		else
                                   			unset($config['system']['proxypass']);
                                  
                                  -		if($_POST['lb_use_sticky'] == "yes")
                                  +		if($_POST['lb_use_sticky'] == "yes") {
                                   			$config['system']['lb_use_sticky'] = true;
                                  -		else
                                  +			$config['system']['srctrack'] = $_POST['srctrack'];
                                  +		} else
                                   			unset($config['system']['lb_use_sticky']);
                                  
                                   		if($_POST['gw_switch_default'] == "yes")
                                  @@ -190,6 +192,12 @@ include("head.inc");
                                   		print_info_box($savemsg);
                                   ?></jimp@pfsense.org> 
                                  

                                  Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                  Need help fast? Netgate Global Support!

                                  Do not Chat/PM for help!

                                  1 Reply Last reply Reply Quote 0
                                  • F
                                    frater
                                    last edited by

                                    Should there really be any output for

                                    pfctl -sr | grep src.track  ???

                                    And what would this output be?
                                    I can't find anything wrong with my /etc/inc/filter.inc nor the /usr/local/www/system_advanced_misc.php

                                    Replacing "/usr/local/www/system_advanced_misc.php" with the 2.10 version doesn't make a difference….

                                    Did you apply the patch?

                                    I really thought it was working in my 2.0 setup.

                                    1 Reply Last reply Reply Quote 0
                                    • jimpJ
                                      jimp Rebel Alliance Developer Netgate
                                      last edited by

                                      If pftctl -st shows anything except 0s for src.track, then it's working. 0s (zero seconds) is the default value of src.track.

                                      pfctl -sr wouldn't show it, but it would be in /tmp/rules.debug

                                      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                      Need help fast? Netgate Global Support!

                                      Do not Chat/PM for help!

                                      1 Reply Last reply Reply Quote 0
                                      • F
                                        frater
                                        last edited by

                                        I currently have a situation in which I'm sure that I'm not having a continuing output over the same interface…
                                        This has been proven by doing a tcpdump on the target server...
                                        A new connection to that same server should go over the same interface.....

                                        That's the behaviour one needs in a multi-WAN setup (I believe it should be that way always).
                                        If not, you'll get kicked....

                                        I don't understand how one can live with a multi-WAN if all of a sudden it suddenly decides to talk using a different interface....
                                        This is asking for trouble and will result in people getting constantly kicked...
                                        I therefore don't understand why this subject is treated with such a low priority....

                                        Unless I'm totally wrong, of course...
                                        But no-one has said that thus far either....

                                        1 Reply Last reply Reply Quote 0
                                        • K
                                          kevindd992002
                                          last edited by

                                          I don't quite get how sticky works. If I analyzed correctly, with sticky enabled multi-thread http downloading or usenet downloading won't be lOad balanced at all?

                                          1 Reply Last reply Reply Quote 0
                                          • F
                                            frater
                                            last edited by

                                            It's because you only look at it from your point of view.
                                            If you want multiple connections to the same server, this isn't for you….

                                            I'm using pfsense in multi-LAN, multi-WAN environment....
                                            traffic will then be equally spread over the different intranet/Internet connections, but once a source address is using a certain target, I don't want it to switch suddenly.

                                            Many targets don't like this and you will get kicked constantly from websites because you're all of a sudden coming from somewhere else....

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.