HOWTO: Traffic Shaping with Squid Cache



  • I haven't found information on how to handle cache hits in the traffic shaper and because I'm very satisfied with my setup I want to share:

    Since you usually don't want to shape traffic that comes from your proxy cache you need to distinguish between cache hits and misses. But in the filter rules you can only set the queue per state. That means, all traffic of a connection goes to the same queue. What you need is a per packet filtering.
    We have per packet filtering on ACKs, because you usually use an extra ACK queue. So what i did, is marking proxy cache hits as ACKs by passing the following options to squid2:

    zph_mode tos;
    zph_local 0x10;
    

    Now catch all your http traffic with a floating rule and put it into the queue you want, but set the ACK-queue to your proxy cache hit queue with a lot of bandwith. On a typical setup with more downstream than upstream, ACKs are only a small amount of the incoming traffic, so it shouldn't be a problem that ACKs and chache hits get merged.

    I hope I didn't forget something, because it's quite some time ago since i made this setup. As far as I know this only works with squid2. Squid3 needs a diffent parameter to be set and additionally needs the kernel compiled with some options which aren't set in pfsense by default.

    Below some overview of my setup:






  • In your rules, why do you specify the ACK queue for non-TCP traffic?
    There are no ACK packets for UDP & ICMP.



  • @netsysadmin:

    In your rules, why do you specify the ACK queue for non-TCP traffic?
    There are no ACK packets for UDP & ICMP.

    You are right, that isn't necessary. Thanks



  • Thank you in advance for your share mr. damur

    Can you explain in more detail about the above screenshot …
    And where to put the settings

    zph_mode tos;
    zph_local 0x10;
    

    in the squid2

    I beg for enlightenment …
    Sorry for my bad english



  • If you have installed the squid package, go to Services -> Proxy Server and paste the options into the "Custom Options" field.

    The screenshot shows my filter rules. The most important is the first one, because it catches all the traffic going through your proxy. If you look at your queues under Status -> Queues, you will see traffic in your ack-queue, whenever you get a cache hit. Now you can limit your LAN interface in the traffic shaper to e.g. 100 MBit/s and separate it into two queues: one with your actual inbound bandwidth (e.g. 6 MBit/s) and one with the rest (94 MBit/s). Use the 94 MBit/s queue only for cache hits and add the rest of the queues as childs of the 6 MBit/s queue.



  • While I guess, from reading some other posts that basically claim you can't run shaper and squid on the same box, some folks want to shape the cache hits(?!), this (shape the actual traffic, and DON'T shape the cache hits) is certainly the behavior I want on my box. As usual, documentation is sparse, scattered and half the time written by people who want to shape cache hits, it seems.

    I run a moderate (my standards - disk is cheap, and bandwidth is expensive) huge (other people's standards - they only cache tiny files?) disk cache of 340 GB or so, 8GB RAM cache, with a max cached object size of 4GB to try and handle most of the piggish system updates from cache. Hit percentage is often low, but with a large cache even a low percentage is a significant amount of data that gets re-served locally without having to load up the outside line. Fired up my second pfsense box finally and gave the shaper a whirl, (1 wan, 1 lan at present - set up with the multi-wan wizard and setting wan to one - start off the choice of wizards with having to check the documentation to find which one to use for the most common case since it's not there…)

    "Naturally" the shaper promptly killed all the advantage of the cache by limiting the LAN to WAN traffic speed (the WAN is a 100Mb connection to a 10Mb/1Mb cable modem - the LAN is gigabit - the LAN was effectively 10Mb with the shaper on.) I removed it this morning and cache served performance improved again; trying to sort out what's actually going on with the method described here (I feel like there's a step missing in the description - or I haven't seen something yet that's in front of my nose) is the first step in bringing the shaper back online, if that is going to happen. This certainly seems to be the best bet I can find searching the forum for answers other than "you can't do that."

    pfsense 2.0.3 and squid 2.7.9

    <edit-add>This http://forum.pfsense.org/index.php/topic,50337.msg275275.html#msg275275 also sheds some dim light on the subject.</edit-add>



  • @Ecnerwal:

    […], this (shape the actual traffic, and DON'T shape the cache hits) is certainly the behavior I want on my box. […]

    Just to clarify: The cache hits do get shaped since they pass the shaper. The goal is to filter them and put them into a queue where they are not getting throttled.

    I don't know where you are stuck, so I will post the steps for a minimalistic setup:

    • add the zph_* settings to squid

    • create a traffic shaper (dont use the wizard). In your case 1Mb WAN and 1Gb LAN

    • create 2 queues on yout LAN: one (A) limited to 10 Mb. The other one (B) gets the remaining 990Mb

    • create one queue (A) for your WAN: 1Mb

    • now create a filter rule that catches all TCP traffic and set the queuing to B/A

    • Always reset your states in Diagnostic->States before testing

    Now you should see all traffic going to queue A by default. When you get a cache hit (test it on a large picture) you can see the traffic in B.

    Hope that helps!



  • Can you give the detail setup for your traffic shaping?



  • This are the queues from my config:

    
    	 <shaper><queue><interface>lan</interface>
    			<name>lan</name>
    			<scheduler>HFSC</scheduler>
    			<bandwidth>1</bandwidth>
    			<bandwidthtype>Gb</bandwidthtype>
    			 <queue><name>proxy</name>
    				<interface>lan</interface>
    				<priority>3</priority>
    				<bandwidth>950</bandwidth>
    				<bandwidthtype>Mb</bandwidthtype>
    				<enabled>on</enabled></queue> 
    			 <queue><name>internet</name>
    				<interface>lan</interface>
    				<priority>3</priority>
    				<bandwidth>50</bandwidth>
    				<bandwidthtype>Mb</bandwidthtype>
    				<enabled>on</enabled>
    				 <queue><name>ack</name>
    					<interface>lan</interface>
    					<priority>6</priority>
    					<bandwidth>2</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled></queue> 
    				 <queue><name>real_time</name>
    					<interface>lan</interface>
    					<priority>7</priority>
    					<bandwidth>23</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					<realtime3>8%</realtime3>
    					<realtime>on</realtime></queue> 
    				 <queue><name>non_real_time</name>
    					<interface>lan</interface>
    					<priority>3</priority>
    					<bandwidth>75</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					 <queue><name>p2p</name>
    						<interface>lan</interface>
    						<priority>1</priority>
    						<bandwidth>10</bandwidth>
    						<bandwidthtype>%</bandwidthtype>
    						<enabled>on</enabled>
    						<ecn>ecn</ecn>
    						<rio>rio</rio></queue> 
    					 <queue><name>others</name>
    						<interface>lan</interface>
    						<priority>3</priority>
    						<bandwidth>40</bandwidth>
    						<bandwidthtype>%</bandwidthtype>
    						<enabled>on</enabled>
    						<default>default</default>
    						<ecn>ecn</ecn>
    						<rio>rio</rio></queue> 
    					 <queue><name>http</name>
    						<interface>lan</interface>
    						<priority>4</priority>
    						<bandwidth>50</bandwidth>
    						<bandwidthtype>%</bandwidthtype>
    						<enabled>on</enabled>
    						<rio>rio</rio>
    						<ecn>ecn</ecn></queue> 
    					<ecn>ecn</ecn>
    					<rio>rio</rio></queue> 
    				<upperlimit3>50Mb</upperlimit3>
    				<upperlimit>on</upperlimit></queue> 
    			<enabled>on</enabled></queue> 
    		 <queue><interface>wan</interface>
    			<name>wan</name>
    			<scheduler>HFSC</scheduler>
    			<bandwidth>10</bandwidth>
    			<bandwidthtype>Mb</bandwidthtype>
    			 <queue><name>non_real_time</name>
    				<interface>wan</interface>
    				<priority>3</priority>
    				<bandwidth>50</bandwidth>
    				<bandwidthtype>%</bandwidthtype>
    				<enabled>on</enabled>
    				 <queue><name>p2p</name>
    					<interface>wan</interface>
    					<priority>1</priority>
    					<bandwidth>10</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					<red>red</red>
    					<ecn>ecn</ecn>
    					<rio>rio</rio></queue> 
    				 <queue><name>others</name>
    					<interface>wan</interface>
    					<priority>3</priority>
    					<bandwidth>40</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					<default>default</default>
    					<red>red</red>
    					<ecn>ecn</ecn>
    					<rio>rio</rio></queue> 
    				 <queue><name>http</name>
    					<interface>wan</interface>
    					<priority>4</priority>
    					<bandwidth>50</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					<red>red</red>
    					<rio>rio</rio>
    					<ecn>ecn</ecn></queue> 
    				<red>red</red>
    				<rio>rio</rio>
    				<ecn>ecn</ecn></queue> 
    			 <queue><name>ack</name>
    				<interface>wan</interface>
    				<priority>6</priority>
    				<bandwidth>25</bandwidth>
    				<bandwidthtype>%</bandwidthtype>
    				<enabled>on</enabled></queue> 
    			 <queue><name>real_time</name>
    				<interface>wan</interface>
    				<priority>7</priority>
    				<bandwidth>25</bandwidth>
    				<bandwidthtype>%</bandwidthtype>
    				<enabled>on</enabled>
    				<realtime3>25%</realtime3>
    				<realtime>on</realtime></queue> 
    			<enabled>on</enabled></queue></shaper> 
    
    


  • @damur:

    This are the queues from my config:

    
    	 <shaper><queue><interface>lan</interface>
    			<name>lan</name>
    			<scheduler>HFSC</scheduler>
    			<bandwidth>1</bandwidth>
    			<bandwidthtype>Gb</bandwidthtype>
    			 <queue><name>proxy</name>
    				<interface>lan</interface>
    				<priority>3</priority>
    				<bandwidth>950</bandwidth>
    				<bandwidthtype>Mb</bandwidthtype>
    				<enabled>on</enabled></queue> 
    			 <queue><name>internet</name>
    				<interface>lan</interface>
    				<priority>3</priority>
    				<bandwidth>50</bandwidth>
    				<bandwidthtype>Mb</bandwidthtype>
    				<enabled>on</enabled>
    				 <queue><name>ack</name>
    					<interface>lan</interface>
    					<priority>6</priority>
    					<bandwidth>2</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled></queue> 
    				 <queue><name>real_time</name>
    					<interface>lan</interface>
    					<priority>7</priority>
    					<bandwidth>23</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					<realtime3>8%</realtime3>
    					<realtime>on</realtime></queue> 
    				 <queue><name>non_real_time</name>
    					<interface>lan</interface>
    					<priority>3</priority>
    					<bandwidth>75</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					 <queue><name>p2p</name>
    						<interface>lan</interface>
    						<priority>1</priority>
    						<bandwidth>10</bandwidth>
    						<bandwidthtype>%</bandwidthtype>
    						<enabled>on</enabled>
    						<ecn>ecn</ecn>
    						<rio>rio</rio></queue> 
    					 <queue><name>others</name>
    						<interface>lan</interface>
    						<priority>3</priority>
    						<bandwidth>40</bandwidth>
    						<bandwidthtype>%</bandwidthtype>
    						<enabled>on</enabled>
    						<default>default</default>
    						<ecn>ecn</ecn>
    						<rio>rio</rio></queue> 
    					 <queue><name>http</name>
    						<interface>lan</interface>
    						<priority>4</priority>
    						<bandwidth>50</bandwidth>
    						<bandwidthtype>%</bandwidthtype>
    						<enabled>on</enabled>
    						<rio>rio</rio>
    						<ecn>ecn</ecn></queue> 
    					<ecn>ecn</ecn>
    					<rio>rio</rio></queue> 
    				<upperlimit3>50Mb</upperlimit3>
    				<upperlimit>on</upperlimit></queue> 
    			<enabled>on</enabled></queue> 
    		 <queue><interface>wan</interface>
    			<name>wan</name>
    			<scheduler>HFSC</scheduler>
    			<bandwidth>10</bandwidth>
    			<bandwidthtype>Mb</bandwidthtype>
    			 <queue><name>non_real_time</name>
    				<interface>wan</interface>
    				<priority>3</priority>
    				<bandwidth>50</bandwidth>
    				<bandwidthtype>%</bandwidthtype>
    				<enabled>on</enabled>
    				 <queue><name>p2p</name>
    					<interface>wan</interface>
    					<priority>1</priority>
    					<bandwidth>10</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					<red>red</red>
    					<ecn>ecn</ecn>
    					<rio>rio</rio></queue> 
    				 <queue><name>others</name>
    					<interface>wan</interface>
    					<priority>3</priority>
    					<bandwidth>40</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					<default>default</default>
    					<red>red</red>
    					<ecn>ecn</ecn>
    					<rio>rio</rio></queue> 
    				 <queue><name>http</name>
    					<interface>wan</interface>
    					<priority>4</priority>
    					<bandwidth>50</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					<red>red</red>
    					<rio>rio</rio>
    					<ecn>ecn</ecn></queue> 
    				<red>red</red>
    				<rio>rio</rio>
    				<ecn>ecn</ecn></queue> 
    			 <queue><name>ack</name>
    				<interface>wan</interface>
    				<priority>6</priority>
    				<bandwidth>25</bandwidth>
    				<bandwidthtype>%</bandwidthtype>
    				<enabled>on</enabled></queue> 
    			 <queue><name>real_time</name>
    				<interface>wan</interface>
    				<priority>7</priority>
    				<bandwidth>25</bandwidth>
    				<bandwidthtype>%</bandwidthtype>
    				<enabled>on</enabled>
    				<realtime3>25%</realtime3>
    				<realtime>on</realtime></queue> 
    			<enabled>on</enabled></queue></shaper> 
    
    

    Thank you for your sharing



  • @damur:

    This are the queues from my config:

    
    	 <shaper><queue><interface>lan</interface>
    			<name>lan</name>
    			<scheduler>HFSC</scheduler>
    			<bandwidth>1</bandwidth>
    			<bandwidthtype>Gb</bandwidthtype>
    			 <queue><name>proxy</name>
    				<interface>lan</interface>
    				<priority>3</priority>
    				<bandwidth>950</bandwidth>
    				<bandwidthtype>Mb</bandwidthtype>
    				<enabled>on</enabled></queue> 
    			 <queue><name>internet</name>
    				<interface>lan</interface>
    				<priority>3</priority>
    				<bandwidth>50</bandwidth>
    				<bandwidthtype>Mb</bandwidthtype>
    				<enabled>on</enabled>
    				 <queue><name>ack</name>
    					<interface>lan</interface>
    					<priority>6</priority>
    					<bandwidth>2</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled></queue> 
    				 <queue><name>real_time</name>
    					<interface>lan</interface>
    					<priority>7</priority>
    					<bandwidth>23</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					<realtime3>8%</realtime3>
    					<realtime>on</realtime></queue> 
    				 <queue><name>non_real_time</name>
    					<interface>lan</interface>
    					<priority>3</priority>
    					<bandwidth>75</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					 <queue><name>p2p</name>
    						<interface>lan</interface>
    						<priority>1</priority>
    						<bandwidth>10</bandwidth>
    						<bandwidthtype>%</bandwidthtype>
    						<enabled>on</enabled>
    						<ecn>ecn</ecn>
    						<rio>rio</rio></queue> 
    					 <queue><name>others</name>
    						<interface>lan</interface>
    						<priority>3</priority>
    						<bandwidth>40</bandwidth>
    						<bandwidthtype>%</bandwidthtype>
    						<enabled>on</enabled>
    						<default>default</default>
    						<ecn>ecn</ecn>
    						<rio>rio</rio></queue> 
    					 <queue><name>http</name>
    						<interface>lan</interface>
    						<priority>4</priority>
    						<bandwidth>50</bandwidth>
    						<bandwidthtype>%</bandwidthtype>
    						<enabled>on</enabled>
    						<rio>rio</rio>
    						<ecn>ecn</ecn></queue> 
    					<ecn>ecn</ecn>
    					<rio>rio</rio></queue> 
    				<upperlimit3>50Mb</upperlimit3>
    				<upperlimit>on</upperlimit></queue> 
    			<enabled>on</enabled></queue> 
    		 <queue><interface>wan</interface>
    			<name>wan</name>
    			<scheduler>HFSC</scheduler>
    			<bandwidth>10</bandwidth>
    			<bandwidthtype>Mb</bandwidthtype>
    			 <queue><name>non_real_time</name>
    				<interface>wan</interface>
    				<priority>3</priority>
    				<bandwidth>50</bandwidth>
    				<bandwidthtype>%</bandwidthtype>
    				<enabled>on</enabled>
    				 <queue><name>p2p</name>
    					<interface>wan</interface>
    					<priority>1</priority>
    					<bandwidth>10</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					<red>red</red>
    					<ecn>ecn</ecn>
    					<rio>rio</rio></queue> 
    				 <queue><name>others</name>
    					<interface>wan</interface>
    					<priority>3</priority>
    					<bandwidth>40</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					<default>default</default>
    					<red>red</red>
    					<ecn>ecn</ecn>
    					<rio>rio</rio></queue> 
    				 <queue><name>http</name>
    					<interface>wan</interface>
    					<priority>4</priority>
    					<bandwidth>50</bandwidth>
    					<bandwidthtype>%</bandwidthtype>
    					<enabled>on</enabled>
    					<red>red</red>
    					<rio>rio</rio>
    					<ecn>ecn</ecn></queue> 
    				<red>red</red>
    				<rio>rio</rio>
    				<ecn>ecn</ecn></queue> 
    			 <queue><name>ack</name>
    				<interface>wan</interface>
    				<priority>6</priority>
    				<bandwidth>25</bandwidth>
    				<bandwidthtype>%</bandwidthtype>
    				<enabled>on</enabled></queue> 
    			 <queue><name>real_time</name>
    				<interface>wan</interface>
    				<priority>7</priority>
    				<bandwidth>25</bandwidth>
    				<bandwidthtype>%</bandwidthtype>
    				<enabled>on</enabled>
    				<realtime3>25%</realtime3>
    				<realtime>on</realtime></queue> 
    			<enabled>on</enabled></queue></shaper> 
    
    

    how to apply or upload this in pfsense? sorry for my noob question



  • I had made a queue named 'qProxy' as your setting and then setting up firewall rules as you recommended. But why there are no activity in in queue status for 'qProxy'. I also had inserted your code in custom field for proxy server settings.