Outgoing OpenVPN Traffic Not Respecting WAN queue Bandwidth Limit
I'm running pfSense 2.0-BETA4 Dec 21 but I've also seen this behaviour on recent builds of BETA5.
Let me start off by saying that I am NOT trying to shape traffic inside the site-to-site OpenVPN tunnel and I am NOT (yet) trying to shape the entire tunnel with a rule on port 1194. In other words, my OpenVPN traffic should be going into the default queue.
I have created a lab with two routers and a fake gateway machine between them to simulate the internet. The fake gateway does NOT limit the speed of traffic at all. Without any shaping on either router, traffic moves at maximum wire speed. That makes it easy to see the effect of the bandwidth limit on the LAN and WAN queues when you apply them.
I have a web server on router A's LAN and forward a port on router A to it. I have a web client running wget on router B's LAN.
The bandwidth limit on router A's WAN queue is set to 675 Kbps.
When the web client on B's LAN downloads a file from the web server on A's LAN via the forwarded port (i.e. NOT via the VPN), qDefault on WAN on router B shows bandwidth usage of about 675 Kbps on the Queue Status page. This is what I would expect. In other words, router A's WAN queue bandwidth limit of 675 Kbps is being respected. Note that wget on the web client shows the speed to be around 80 KBps or 640 Kbps which is very close to 675 Kbps. The small difference of 35 Kbps is probably overhead.
Now here's the weird part. When I repeat the download from the web server but instead have the web client download the file over the site-to-site OpenVPN (i.e. using the LAN address of the web server and NOT the port forwarded WAN address), the WAN queue bandwidth limit of 675 Kbps is NOT respected. qDefault still shows usage of about 675 Kbps, which leads you to believe that the download is being shaped. HOWEVER, this time wget shows the speed to be around 500 KBps or 4000 Kbps, which is WAY higher than the limit of 675 Kbps. And, naturally, the download finishes much quicker so wget is not lying.
So it appears to me that OpenVPN packets leaving the WAN interface are somehow bypassing the WAN queue's bandwidth limit. Why and can I do anything about it? I'm worried about OpenVPN traffic drowning out higher priority traffic.
My bad. My very, very bad. We've got two reasonably smart (or so I thought) guys over here looking at this for two days. We thought the problem was something to do with the packets not hitting the rules to get into the queues. Boy were we wrong. Like they said in the movie
contact, "the simplest explanation is usually the correct one".
The test file we were transferring was 50 GB of zeros made with dd and /dev/zero. It's HIGHLY compressible. Stupid dolts that we are, we had somehow forgotten that we had enabled compression on the OpenVPN tunnel (i.e. we checked the checkbox "Compress tunnel packets using the LZO algorithm").
After turning off compression (temporarily, of course) we got the expected results. That is, the file downloaded at about 675 Kbps over the VPN (same as non-VPN test).
Please don't flame me for being so stupid. :)