How to shape Amazon S3



  • Amazon S3 uses a CIDR in my case of 72.21.192.0/19 and 205.251.192.0/18 which is 16,896 possible IPs. thru port 443
    I have an alias "backup" with both of those ip ranges entered.

    the problem I'm having is that amazon S3 backups keep getting shoved into the qWEB vs qBACKUP queues and I cant figure out why

    5 queues exist on the WAN, qACK, qWEB, qBACKUP, qEMAIL, qDEFAULT

    Below is how the rules are setup, the action on all of these are set to "Queue"

    it seems that for some reason traffic to port 443 to amazon is skipping the backup rules and going right into the catch all port 80/443 qWEB, I've tried quick enabled and disabled w/o a change.

    I'm wondering if the ip range in the alias is simply too big. i even calculated all the ips and copied them in as a new alias to no avail. If I remember correctly, pfsense processes rules top down like ipchains/tables vs normal pf on bsd where the last rule to match gets triggered. So the backup rules should be in the right order sitting on top.

    Any ideas are greatly appreciated!



  • Floating rules are last matching. This is different from WAN and LAN rule that are first matching.
    Try putting those new rules at the bottom.



  • no change unfortunately.

    here's a tcpdump -v -n -i em1 src or dst port 443 or port 80
    which is what the qWEB is looking for. The 72.21.194.31 in here is AMZN which is listed in the backup alias and should get sent to the backup queue.

    it's weird bc all the other queues work as intended during normal traffic aside from S3 traffic



  • In your floating rule, take out the destination port of 443 and see if that helps.



  • Still doesn't work. qWEB gobbles it up. Pretty interesting.

    Here's what does work, I'm uploading to 3 different buckets. Two of them are caught by the floating rule properly and queued to qBACKUP. sending to the other bucket results in traffic going into qWEB even though tcpdump shows the address used is well w/in the subnet specified in the alias.



  • alright here's the issue, when you send data to a bucket the ip changes every time you connect to it and the range of ips is huge (which we know)

    the current backup (which will take 40+ hrs) initially referenced an ip w/in
    72.21.192.0/19 and hit the qBACKUP no problem

    duplicity chops up volumes into smaller sections, each section when sent changes the S3 ip it connects to. Now its connecting to
    207.171.163.195 which is w/in 207.171.160.0/19

    previously i only had:
    205.251.192.0/18
    72.21.192.0/19

    in the alias, 207.171.160.0/19 needs to be added. i'm not sure how many ip blocks AMZN has but i'll keep updating this post as i find them out

    AMZN S3 aliases needed for North America:
    72.21.192.0/19
    205.251.192.0/18
    207.171.160.0/19



  • I get very slow uploads to european s3-buckets using amanda.

    Do I have to add rules in pfsense for this??

    If yes, pls specify … thanks!

    Stefan



  • You'll only need to add a rule if you need some type of QoS to ensure someone else can't hog all the bandwidth on your network while transfering.

    I haven't used Amanda before but as long as the software works properly you should be able to transfer to S3 as fast as your network connection will permit.

    I'm not sure what the EU ip blocks are for S3, you'd have to monitor your traffic as mentioned in above. You'll want to focus on destination port 443 and record the ips, and who a whois on them to get the CIDR



  • Thanks, I will retry asap.

    EDIT:

    did a cross check while I was using another ALIX with ipfire on it … there the backup went through fine and much quicker.
    I don't have a clue what to look for in the iptables there ...

    Just as a reference.
    I will re-check things when I plugged in the pfsense-box again.


Locked