Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    HOWTO: Traffic Shaping with Squid Cache

    Scheduled Pinned Locked Moved Traffic Shaping
    12 Posts 6 Posters 14.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      damur
      last edited by

      @netsysadmin:

      In your rules, why do you specify the ACK queue for non-TCP traffic?
      There are no ACK packets for UDP & ICMP.

      You are right, that isn't necessary. Thanks

      1 Reply Last reply Reply Quote 0
      • M
        mamat_do
        last edited by

        Thank you in advance for your share mr. damur

        Can you explain in more detail about the above screenshot …
        And where to put the settings

        zph_mode tos;
zph_local 0x10;

        in the squid2

        I beg for enlightenment …
        Sorry for my bad english

        1 Reply Last reply Reply Quote 0
        • D
          damur
          last edited by

          If you have installed the squid package, go to Services -> Proxy Server and paste the options into the "Custom Options" field.

          The screenshot shows my filter rules. The most important is the first one, because it catches all the traffic going through your proxy. If you look at your queues under Status -> Queues, you will see traffic in your ack-queue, whenever you get a cache hit. Now you can limit your LAN interface in the traffic shaper to e.g. 100 MBit/s and separate it into two queues: one with your actual inbound bandwidth (e.g. 6 MBit/s) and one with the rest (94 MBit/s). Use the 94 MBit/s queue only for cache hits and add the rest of the queues as childs of the 6 MBit/s queue.

          1 Reply Last reply Reply Quote 0
          • E
            Ecnerwal
            last edited by

            While I guess, from reading some other posts that basically claim you can't run shaper and squid on the same box, some folks want to shape the cache hits(?!), this (shape the actual traffic, and DON'T shape the cache hits) is certainly the behavior I want on my box. As usual, documentation is sparse, scattered and half the time written by people who want to shape cache hits, it seems.

            I run a moderate (my standards - disk is cheap, and bandwidth is expensive) huge (other people's standards - they only cache tiny files?) disk cache of 340 GB or so, 8GB RAM cache, with a max cached object size of 4GB to try and handle most of the piggish system updates from cache. Hit percentage is often low, but with a large cache even a low percentage is a significant amount of data that gets re-served locally without having to load up the outside line. Fired up my second pfsense box finally and gave the shaper a whirl, (1 wan, 1 lan at present - set up with the multi-wan wizard and setting wan to one - start off the choice of wizards with having to check the documentation to find which one to use for the most common case since it's not there…)

            "Naturally" the shaper promptly killed all the advantage of the cache by limiting the LAN to WAN traffic speed (the WAN is a 100Mb connection to a 10Mb/1Mb cable modem - the LAN is gigabit - the LAN was effectively 10Mb with the shaper on.) I removed it this morning and cache served performance improved again; trying to sort out what's actually going on with the method described here (I feel like there's a step missing in the description - or I haven't seen something yet that's in front of my nose) is the first step in bringing the shaper back online, if that is going to happen. This certainly seems to be the best bet I can find searching the forum for answers other than "you can't do that."

            pfsense 2.0.3 and squid 2.7.9

            <edit-add>This http://forum.pfsense.org/index.php/topic,50337.msg275275.html#msg275275 also sheds some dim light on the subject.</edit-add>

            pfSense on i5 3470/DQ77MK/16GB/500GB

            1 Reply Last reply Reply Quote 0
            • D
              damur
              last edited by

              @Ecnerwal:

              […], this (shape the actual traffic, and DON'T shape the cache hits) is certainly the behavior I want on my box. […]

              Just to clarify: The cache hits do get shaped since they pass the shaper. The goal is to filter them and put them into a queue where they are not getting throttled.

              I don't know where you are stuck, so I will post the steps for a minimalistic setup:

              • add the zph_* settings to squid

              • create a traffic shaper (dont use the wizard). In your case 1Mb WAN and 1Gb LAN

              • create 2 queues on yout LAN: one (A) limited to 10 Mb. The other one (B) gets the remaining 990Mb

              • create one queue (A) for your WAN: 1Mb

              • now create a filter rule that catches all TCP traffic and set the queuing to B/A

              • Always reset your states in Diagnostic->States before testing

              Now you should see all traffic going to queue A by default. When you get a cache hit (test it on a large picture) you can see the traffic in B.

              Hope that helps!

              1 Reply Last reply Reply Quote 0
              • A
                abunuha
                last edited by

                Can you give the detail setup for your traffic shaping?

                1 Reply Last reply Reply Quote 0
                • D
                  damur
                  last edited by

                  This are the queues from my config:

                  
	 <shaper><queue><interface>lan</interface>
			<name>lan</name>
			<scheduler>HFSC</scheduler>
			<bandwidth>1</bandwidth>
			<bandwidthtype>Gb</bandwidthtype>
			 <queue><name>proxy</name>
				<interface>lan</interface>
				<priority>3</priority>
				<bandwidth>950</bandwidth>
				<bandwidthtype>Mb</bandwidthtype>
				<enabled>on</enabled></queue> 
			 <queue><name>internet</name>
				<interface>lan</interface>
				<priority>3</priority>
				<bandwidth>50</bandwidth>
				<bandwidthtype>Mb</bandwidthtype>
				<enabled>on</enabled>
				 <queue><name>ack</name>
					<interface>lan</interface>
					<priority>6</priority>
					<bandwidth>2</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled></queue> 
				 <queue><name>real_time</name>
					<interface>lan</interface>
					<priority>7</priority>
					<bandwidth>23</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					<realtime3>8%</realtime3>
					<realtime>on</realtime></queue> 
				 <queue><name>non_real_time</name>
					<interface>lan</interface>
					<priority>3</priority>
					<bandwidth>75</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					 <queue><name>p2p</name>
						<interface>lan</interface>
						<priority>1</priority>
						<bandwidth>10</bandwidth>
						<bandwidthtype>%</bandwidthtype>
						<enabled>on</enabled>
						<ecn>ecn</ecn>
						<rio>rio</rio></queue> 
					 <queue><name>others</name>
						<interface>lan</interface>
						<priority>3</priority>
						<bandwidth>40</bandwidth>
						<bandwidthtype>%</bandwidthtype>
						<enabled>on</enabled>
						<default>default</default>
						<ecn>ecn</ecn>
						<rio>rio</rio></queue> 
					 <queue><name>http</name>
						<interface>lan</interface>
						<priority>4</priority>
						<bandwidth>50</bandwidth>
						<bandwidthtype>%</bandwidthtype>
						<enabled>on</enabled>
						<rio>rio</rio>
						<ecn>ecn</ecn></queue> 
					<ecn>ecn</ecn>
					<rio>rio</rio></queue> 
				<upperlimit3>50Mb</upperlimit3>
				<upperlimit>on</upperlimit></queue> 
			<enabled>on</enabled></queue> 
		 <queue><interface>wan</interface>
			<name>wan</name>
			<scheduler>HFSC</scheduler>
			<bandwidth>10</bandwidth>
			<bandwidthtype>Mb</bandwidthtype>
			 <queue><name>non_real_time</name>
				<interface>wan</interface>
				<priority>3</priority>
				<bandwidth>50</bandwidth>
				<bandwidthtype>%</bandwidthtype>
				<enabled>on</enabled>
				 <queue><name>p2p</name>
					<interface>wan</interface>
					<priority>1</priority>
					<bandwidth>10</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					<red>red</red>
					<ecn>ecn</ecn>
					<rio>rio</rio></queue> 
				 <queue><name>others</name>
					<interface>wan</interface>
					<priority>3</priority>
					<bandwidth>40</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					<default>default</default>
					<red>red</red>
					<ecn>ecn</ecn>
					<rio>rio</rio></queue> 
				 <queue><name>http</name>
					<interface>wan</interface>
					<priority>4</priority>
					<bandwidth>50</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					<red>red</red>
					<rio>rio</rio>
					<ecn>ecn</ecn></queue> 
				<red>red</red>
				<rio>rio</rio>
				<ecn>ecn</ecn></queue> 
			 <queue><name>ack</name>
				<interface>wan</interface>
				<priority>6</priority>
				<bandwidth>25</bandwidth>
				<bandwidthtype>%</bandwidthtype>
				<enabled>on</enabled></queue> 
			 <queue><name>real_time</name>
				<interface>wan</interface>
				<priority>7</priority>
				<bandwidth>25</bandwidth>
				<bandwidthtype>%</bandwidthtype>
				<enabled>on</enabled>
				<realtime3>25%</realtime3>
				<realtime>on</realtime></queue> 
			<enabled>on</enabled></queue></shaper>
                  1 Reply Last reply Reply Quote 0
                  • A
                    abunuha
                    last edited by

                    @damur:

                    This are the queues from my config:

                    
	 <shaper><queue><interface>lan</interface>
			<name>lan</name>
			<scheduler>HFSC</scheduler>
			<bandwidth>1</bandwidth>
			<bandwidthtype>Gb</bandwidthtype>
			 <queue><name>proxy</name>
				<interface>lan</interface>
				<priority>3</priority>
				<bandwidth>950</bandwidth>
				<bandwidthtype>Mb</bandwidthtype>
				<enabled>on</enabled></queue> 
			 <queue><name>internet</name>
				<interface>lan</interface>
				<priority>3</priority>
				<bandwidth>50</bandwidth>
				<bandwidthtype>Mb</bandwidthtype>
				<enabled>on</enabled>
				 <queue><name>ack</name>
					<interface>lan</interface>
					<priority>6</priority>
					<bandwidth>2</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled></queue> 
				 <queue><name>real_time</name>
					<interface>lan</interface>
					<priority>7</priority>
					<bandwidth>23</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					<realtime3>8%</realtime3>
					<realtime>on</realtime></queue> 
				 <queue><name>non_real_time</name>
					<interface>lan</interface>
					<priority>3</priority>
					<bandwidth>75</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					 <queue><name>p2p</name>
						<interface>lan</interface>
						<priority>1</priority>
						<bandwidth>10</bandwidth>
						<bandwidthtype>%</bandwidthtype>
						<enabled>on</enabled>
						<ecn>ecn</ecn>
						<rio>rio</rio></queue> 
					 <queue><name>others</name>
						<interface>lan</interface>
						<priority>3</priority>
						<bandwidth>40</bandwidth>
						<bandwidthtype>%</bandwidthtype>
						<enabled>on</enabled>
						<default>default</default>
						<ecn>ecn</ecn>
						<rio>rio</rio></queue> 
					 <queue><name>http</name>
						<interface>lan</interface>
						<priority>4</priority>
						<bandwidth>50</bandwidth>
						<bandwidthtype>%</bandwidthtype>
						<enabled>on</enabled>
						<rio>rio</rio>
						<ecn>ecn</ecn></queue> 
					<ecn>ecn</ecn>
					<rio>rio</rio></queue> 
				<upperlimit3>50Mb</upperlimit3>
				<upperlimit>on</upperlimit></queue> 
			<enabled>on</enabled></queue> 
		 <queue><interface>wan</interface>
			<name>wan</name>
			<scheduler>HFSC</scheduler>
			<bandwidth>10</bandwidth>
			<bandwidthtype>Mb</bandwidthtype>
			 <queue><name>non_real_time</name>
				<interface>wan</interface>
				<priority>3</priority>
				<bandwidth>50</bandwidth>
				<bandwidthtype>%</bandwidthtype>
				<enabled>on</enabled>
				 <queue><name>p2p</name>
					<interface>wan</interface>
					<priority>1</priority>
					<bandwidth>10</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					<red>red</red>
					<ecn>ecn</ecn>
					<rio>rio</rio></queue> 
				 <queue><name>others</name>
					<interface>wan</interface>
					<priority>3</priority>
					<bandwidth>40</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					<default>default</default>
					<red>red</red>
					<ecn>ecn</ecn>
					<rio>rio</rio></queue> 
				 <queue><name>http</name>
					<interface>wan</interface>
					<priority>4</priority>
					<bandwidth>50</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					<red>red</red>
					<rio>rio</rio>
					<ecn>ecn</ecn></queue> 
				<red>red</red>
				<rio>rio</rio>
				<ecn>ecn</ecn></queue> 
			 <queue><name>ack</name>
				<interface>wan</interface>
				<priority>6</priority>
				<bandwidth>25</bandwidth>
				<bandwidthtype>%</bandwidthtype>
				<enabled>on</enabled></queue> 
			 <queue><name>real_time</name>
				<interface>wan</interface>
				<priority>7</priority>
				<bandwidth>25</bandwidth>
				<bandwidthtype>%</bandwidthtype>
				<enabled>on</enabled>
				<realtime3>25%</realtime3>
				<realtime>on</realtime></queue> 
			<enabled>on</enabled></queue></shaper>

                    Thank you for your sharing

                    1 Reply Last reply Reply Quote 0
                    • C
                      cheonne
                      last edited by

                      @damur:

                      This are the queues from my config:

                      
	 <shaper><queue><interface>lan</interface>
			<name>lan</name>
			<scheduler>HFSC</scheduler>
			<bandwidth>1</bandwidth>
			<bandwidthtype>Gb</bandwidthtype>
			 <queue><name>proxy</name>
				<interface>lan</interface>
				<priority>3</priority>
				<bandwidth>950</bandwidth>
				<bandwidthtype>Mb</bandwidthtype>
				<enabled>on</enabled></queue> 
			 <queue><name>internet</name>
				<interface>lan</interface>
				<priority>3</priority>
				<bandwidth>50</bandwidth>
				<bandwidthtype>Mb</bandwidthtype>
				<enabled>on</enabled>
				 <queue><name>ack</name>
					<interface>lan</interface>
					<priority>6</priority>
					<bandwidth>2</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled></queue> 
				 <queue><name>real_time</name>
					<interface>lan</interface>
					<priority>7</priority>
					<bandwidth>23</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					<realtime3>8%</realtime3>
					<realtime>on</realtime></queue> 
				 <queue><name>non_real_time</name>
					<interface>lan</interface>
					<priority>3</priority>
					<bandwidth>75</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					 <queue><name>p2p</name>
						<interface>lan</interface>
						<priority>1</priority>
						<bandwidth>10</bandwidth>
						<bandwidthtype>%</bandwidthtype>
						<enabled>on</enabled>
						<ecn>ecn</ecn>
						<rio>rio</rio></queue> 
					 <queue><name>others</name>
						<interface>lan</interface>
						<priority>3</priority>
						<bandwidth>40</bandwidth>
						<bandwidthtype>%</bandwidthtype>
						<enabled>on</enabled>
						<default>default</default>
						<ecn>ecn</ecn>
						<rio>rio</rio></queue> 
					 <queue><name>http</name>
						<interface>lan</interface>
						<priority>4</priority>
						<bandwidth>50</bandwidth>
						<bandwidthtype>%</bandwidthtype>
						<enabled>on</enabled>
						<rio>rio</rio>
						<ecn>ecn</ecn></queue> 
					<ecn>ecn</ecn>
					<rio>rio</rio></queue> 
				<upperlimit3>50Mb</upperlimit3>
				<upperlimit>on</upperlimit></queue> 
			<enabled>on</enabled></queue> 
		 <queue><interface>wan</interface>
			<name>wan</name>
			<scheduler>HFSC</scheduler>
			<bandwidth>10</bandwidth>
			<bandwidthtype>Mb</bandwidthtype>
			 <queue><name>non_real_time</name>
				<interface>wan</interface>
				<priority>3</priority>
				<bandwidth>50</bandwidth>
				<bandwidthtype>%</bandwidthtype>
				<enabled>on</enabled>
				 <queue><name>p2p</name>
					<interface>wan</interface>
					<priority>1</priority>
					<bandwidth>10</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					<red>red</red>
					<ecn>ecn</ecn>
					<rio>rio</rio></queue> 
				 <queue><name>others</name>
					<interface>wan</interface>
					<priority>3</priority>
					<bandwidth>40</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					<default>default</default>
					<red>red</red>
					<ecn>ecn</ecn>
					<rio>rio</rio></queue> 
				 <queue><name>http</name>
					<interface>wan</interface>
					<priority>4</priority>
					<bandwidth>50</bandwidth>
					<bandwidthtype>%</bandwidthtype>
					<enabled>on</enabled>
					<red>red</red>
					<rio>rio</rio>
					<ecn>ecn</ecn></queue> 
				<red>red</red>
				<rio>rio</rio>
				<ecn>ecn</ecn></queue> 
			 <queue><name>ack</name>
				<interface>wan</interface>
				<priority>6</priority>
				<bandwidth>25</bandwidth>
				<bandwidthtype>%</bandwidthtype>
				<enabled>on</enabled></queue> 
			 <queue><name>real_time</name>
				<interface>wan</interface>
				<priority>7</priority>
				<bandwidth>25</bandwidth>
				<bandwidthtype>%</bandwidthtype>
				<enabled>on</enabled>
				<realtime3>25%</realtime3>
				<realtime>on</realtime></queue> 
			<enabled>on</enabled></queue></shaper>

                      how to apply or upload this in pfsense? sorry for my noob question

                      1 Reply Last reply Reply Quote 0
                      • A
                        abunuha
                        last edited by

                        I had made a queue named 'qProxy' as your setting and then setting up firewall rules as you recommended. But why there are no activity in in queue status for 'qProxy'. I also had inserted your code in custom field for proxy server settings.

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.