Traffic shaping with transparent squid proxy
-
I would like to be able to put the cache hits from squid in a different queue than the regular traffic. I am using the HFSC traffic shaper, which works great btw, and a transparent squid proxy. The issue is that the traffic originating from the squid proxy (the cache hits) are also shaped. This is of course not what I want, I want to have cache hits sent at (near) LAN speeds.
I've read a great deal of forum posts on this subject and concluded that it seems impossible as of now. Can someone confirm or deny this? Another option for me would be to use a separate machine as a squid cache to avoid problems with the traffic shaper. I am using a virtualized environment for pfSense, so I could make another VM for the squid proxy.
Can anyone help me? In short: I want to shape my traffic, but exclude cache hits from squid.
Thanks!
-
Squid does not mark the traffic in anyway to indicate that it is a hit. Plus, it may resuse the same connection/state for multiple requests, some of which are probably not hits, and you can't change queues once set.
-
Is it possible to setup a squid cache on a separate machine to work around these issues? In other words, can I make a squid cache (transparent proxy) on another machine and route traffic through that cache without having these traffic shaping issues?
-
I wonder if you can exclude Squid traffic from shaping on the LAN and use limiters to shape incoming HTTP traffic on the WAN.
-
If squid runs on a separate machine, I can use a floating firewall rule to single out all traffic originating from that machine, simply by filtering that IP. Therefore, I could direct it to a certain queue that is not limited by the traffic shaping rules. However, I have no idea to setup such a thing.
Thanks for the help.
-
I am also interested in this. It would be nice to provide cache hits at close to wire speed while enforcing limits on the WAN connection. This was the desired functionality when I first set this up and I, like the OP, have been searching for clues. I know from my previous experience with traffic shaping and PF that you can only shape the egress on an interface. My goal is to maintain low latency for VoIP and gaming mainly in situations where a large download may be taking place. I've found that limiting the ingress to around 90% of the link's capacity seems to maintain low latency, but I wish that all could be configured on the WAN interface (27mbit/sec down, 5200kbit/sec up for me) while the LAN interface could better utilize the gigabit connection to my network. I may decide to simply prioritize the traffic and sacrifice the low latency enforcement. I'm very curious to see if this ever gets solved.
-
Hi
Please take a look at this topic I opened today:
https://forum.pfsense.org/index.php?topic=125646.0In fact IT IS possible to mark Squid HITs with specific value.
In Pfsense you can use DSCP value to build desired firewall rule or you that value in your traffic shaping.
https://www.tucny.com/Home/dscp-tos
Here are corresponding TOS values (2nd column in HEX) and DSCP value (last column)So in my example I am using qos_flows local-hit=0x30 directive in squid.conf to mark them and it seems to work (run tcpdump to check)
And in this example DSCP corresponding value will be 12 (in pfsense firewall advanced options it is AF12) -
I have a 25/10 DSL connection and for well over a year I've been able to setup queues successfully for regular internet traffic (qInternet), VoIP traffic (qVoIP), and other traffic such as LAN to OPT1 and OPT1 to LAN transfers as well as a Squid Transparent Proxy (qOther).
The squid traffic was easily matched using a floating rule for any connection who's destination port was 3128. This has worked for both transparent and non-transparent configurations.
The problem I am seeing now is that traffic from the firewall/squid is not being matched to qOther. Instead it gets matched only with the default qInternet. LAN to OPT1 transfers enter qOther properly though. The problem seems to be related to traffic originating at the firewall.
To confirm, I placed a 1GB.zip file in /usr/local/www and then set a floating rule to match traffic connecting to this firewall itself on any port from any source IP/port for qOther.
Upon download, the packets still ended up in qInternet instead of the intended qOther.
FYI my latest version of the squid package for pfsense is 0.4.36_2.
-
use tcpdump to look for packets that are coming from squid and see if they are marked properly or not. Investigate why..
-
Never had to use tagged packets for squid data to be shuttled to the proper queue. As I said in my earlier reply, a single match rule for connections made to port 3128 has worked for almost two years in various versions of pfsense (2.2.6 to 2.3.2).
Something has changed in the way squid packets leave the interface. The connections in the states table appear no different though than they have in the past.
-
Probably with latest pfsense update it stopped properly handling connections to transparent proxy.
Could you set proxy address manually at one of your workstations, generate some http traffic volume and see if it falls under correct queue?
-
If your QoS goal is simply bufferbloat on the web end of things, you could try limiters on wan, and nothing on lan.
-
Is it possible to setup a squid cache on a separate machine to work around these issues? In other words, can I make a squid cache (transparent proxy) on another machine and route traffic through that cache without having these traffic shaping issues?
This is the recommended way of doing it. It works REALLY well. I may change to this system myself.
-
What's the point of having a squid package if you can't use it properly?
Sure, I could run squid on another box. I could also user another box for a dhcp server. Oh, and maybe I'll use a third box just to manage my let's encrypt certificates. For good measure, let's not waste any more time and add a fourth box so that my log files don't overload my main pfsense router…
-
@thehammer86:
Never had to use tagged packets for squid data to be shuttled to the proper queue. As I said in my earlier reply, a single match rule for connections made to port 3128 has worked for almost two years in various versions of pfsense (2.2.6 to 2.3.2).
Something has changed in the way squid packets leave the interface. The connections in the states table appear no different though than they have in the past.
Can you tell me if you already fix this? im having the same issue, i cant shape my iOs download because it's go always to the default queeu and cant control the banwith allowed.
-
I ended up switching to a PRIQ setup instead. I limit the upload on my WAN slightly so I don't saturate my uplink. I then set my LAN bandwidth to 980 Mb/s and squid then flows at almost full interface speed. It would be nice to be able to depriortize squid but for now at least it works.