Shaping Google Drive uploads
-
Google Drive uploader quickly saturates my upload bandwidth here, so I spent a day getting a solid working shaper rule and thought I'd post it here.
Interestingly, unlike rsync, dropbox and other comparable solutions, Google Drive uploader doesn't currently utilize deltas, so even minor changes to large files can hammer upload bandwidth. There also doesn't seem to be much in the way of throttling and there certainly aren't any user facing throttling controls.
1. Add alias named googledrive_hosts or similar: (new alias of type host)
Hosts to add in this alias:
upload.drive.google.com large-uploads.l.google.com2. Add alias named googledrive_ports or similar (new alias of type port)
Ports to add in this alias:
80 443The reason I recommend doing this instead of using an existing "web ports" alias (if you already have one) is probably clear: in case Google adds other ports in future you can modify this alias instead of the web ports alias.
3. New floating firewall rule (assuming you already have shaping queues set up):
A new rule to queue based on:
Quick match Interface: LAN TCP/IP: v4 Protocol: TCP Destination - Type: single host or alias Desintation - Address: googledrive_hosts Destination port range - From: (other) googledrive_ports Destination port range - To: (other) googledrive_ports Ackqueue/Queue: qACK/qOthersLow (or whatever low priority queue you have set up)IMPORTANTLY: the qOthersLow that I'm using is not "straight out of the wizard". I added an Upperlimit/m2 value of 50% to ensure that I only use a maximum of 50% of my available bandwidth for the qOthersLow queue, otherwise latency ensues when this queue fills up.
Keep in mind that if you are using a 80/443 shaper rule to prioritize http/https traffic, google drive traffic will be getting caught by that. I realize that you can set this rule to quick match and locate it prior to the catch all http/https prioritization rule, though I haven't had luck getting that to work yet so right now my normal (non google drive uploader) http/https traffic is just going into the default queue
I'm still a shaper noob in most respects so let me know if anyone thinks these recommendations are off in any way. I'm not actually certain that using the qACK queue is right in this case, in particular, but regardless it's all working well.
-
Nice one, using google drive myself, don't have very big files in there, so haven't noticed this behavior.
One thing I would do, is move it to a floating rule, instead of LAN.
-
Nice one, using google drive myself, don't have very big files in there, so haven't noticed this behavior.
Until recently I hadn't noticed it either. It only showed up when I dumped a couple of isos into it. I'm hoping they implement at least deltas, if not throttling.
One thing I would do, is move it to a floating rule, instead of LAN.
I did make the rule floating but it's not clear in that post; I'll add a note to clarify. I have the floating rule assigned to LAN since I didn't care about queuing incoming on those connections (I'm assuming those are the upload-only connections judging by the domains). I did consider making a different rule (or altering this one) to handle incoming downloads, but I haven't had a chance to sort out which domains/IPs I need to filter against and test that yet.