Can I hope too improve on my 2 Mb/s download with pfSense traffic shaping



  • Hello. I am not (yet) a pfSense user and consider to be, but I feel humbled by the average skill level out here. I apologise in advance for the unsatisfactory aspect of the description below.

    In view of my slow bandwidth at my home (about 2 Mb/S down, 0.7 Mb/s up), I want to explore the possibility of throttling the bandwidth of most bandwidth users in my home, because from time to time, my computers get sick at not receiving the info they need from servers to survive (and they get unable to conduct a screen sharing session with Apple Assistance). I wonder if using a pfSense router (like the little red SG-1000) and use on it some traffic shaping tricks would help.

    Or maybe I am being insane.

    The main issues clogging my macs that I have identified (at home, we have 2 macs, 2 iPhones, one iPad and many Dropbox shared folders) are:

    1 - Automatic wifi updates of my wife's iPhone pressuring my ADSL line just when the Apple servers delivering the updater are overloaded with tons of demands from other luckier connected iPhones (Sadly, I cannot suggest her out of that "no-hassle" habit if I want to preserve my happy marriage);

    2 - iTunes automatic downloads of stuff I buy from any of my machines (iPad or iPhone) or of podcasts or other stuff;

    3 - iTunes manual updates for iPhone & iPad Apps, and for iPhone and iPad iOS updates;

    4 - App store automatic downloads of just about everything with automatic install, for mac OS security updates, mac OS updates and mac Apps;

    5 - iCloud sync of iCloud data, which total volume I reduced to trickle (Data volume of less that 0.5 GB in the Apple servers);

    6 - Dropbox (data volume totaling about 11 GB), which I use a lot and is installed on two accounts each on my MacBook (Abeille webmaster and me) and iMac (my wife and me), where bandwith is adjustable and which I limited on each account to : Down: 10 kb/s; Up 5 kb/s;

    7 - Mail IMAP accounts (I still receive from crazy friendly friends who live close to their ADSL provider massive 30 MB dumps of photos which block everything when they come to pass);

    8 - Others, undoubtedly.

    My questions to pfSense wise and unwise people are:

    Can pfSense traffic shaping help me to overcome the above issues ?

    If I purchase the SG-1000 device, red, with its one year subscription to whatever training there is to it from Netgate and keep on sustaining my vow of never using linux terminal commands (too hard for me), can a newbie like me cope with the effort ?

    TIA.


  • Banned

    In short, no.

    You will be able to shape your traffic and limit the bandwidth in various ways on your network but ultimately it isn't going to change the fact that 2Mbps is very slow.

    It takes your connection 4 seconds to move one Megabyte of data. The average ios app is 23MB or a minute and a half to download. 16 seconds for a song, 2 minutes for a 5 minute video clip. You get the picture. Those numbers are at full line speed. You simply can't traffic shape 2Mbps into being "fast" in 2017.

    All of that having been said, pfSense will shape your traffic and improve some of your problems. It's just that there's so little to work with that the relative improvement may not even be noticeable.



  • Thanks pfBasic, for giving me the hard reply ("NO") to my question 1 above ("Can pfSense traffic shaping help me to overcome the above issues ?"), and the reasons for it.

    As I read your post, I feel like my question #1 should have been more clear and I will attempt to do that hoping that I may have a chance to improve on my problems with a pfSense router. I will also restate my insecurity as stated in my #2 question above.

    My problem is not that I *need faster downloads (or uploads). I have lived the beginning of the internet with an Apple djinn modem (4.8 kb/s), then a Global village platinum modem (33 kb/s), then a 56 kb/s modem, then a Numeris instant 2 X 64 kb/s connection, then an ADSL 0.5 Mb/s connection, and so on, with slow increments… I am not bandwith-hungry and know how to be patient.

    My technical explanations may be widely off-mark, but I will give it a try:

    1 - Modem connection lost. Example: Yesterday night, I restarted the computer, the modem was happy and "ON" and there was no traffic that I could see in my home. I triggered through iTunes the iOS 10.3.3 update (a 1.87 GB download) for my iPhone. During the night, the connection stalled and the modem restarted at about 1:00 am. This massive download was lost and I had to do it again from zero this morning. It succeeded (lucky me). This occurs far to often for my being satisfied about it. An example of crude solution I found to that class of problem was with Dropbox, where I share photo collections with groups of friends (bicycle events, marriages, and so on), where I am seldom the uploader of the photos, and some of them, uploaded by friends, can be as huge as a 400 MB collection of 100 pics weighing 4 MB each (uncompressed). I throttled the Dropbox bandwidth to 10 kb/s down and 5 kb/s up and I am no longer bothered by Dropbox, which syncs nevertheless, slowly but surely. I have seeked the same kind of settings on iTunes or the mac App Store, and not found them (they do not exist). I believe I need a substitute to that, which may be nearly available in the pfSense settings.

    2 - Computer to become a dumb, or frozen, computer. I have read on a pfSense wiki entry (<https://doc.pfsense.org/index.php/Traffic_Shaping_Guide>): "When data is downloaded, a computer needs to send (upload) ACK packets. These are basically saying "yep, I got that part of the download OK". If the computer being downloaded from detects that an ACK has not been received, it assumes that the data was not received and sends it again. The rate at which ACKs are sent back is also used to help determine the maximum speed at which data may be downloaded, so it is important that ACKs get sent as soon as possible and don't get dropped in order to keep downloads flowing fast. Also, repeatedly dropped ACKs can result in dropped connections, web page time-outs etc.". THAT may be one of the possible causes of my computer becoming dumb and unresponsive while the modem has not stalled (yet).

    I have explored these issues with Apple: no luck, and my ISP's technician told me that the problem I indicated was, as seen by him, far more serious on Windows 10 computers with their automatic updaters "ON" than on macs, lucky me.

    I have explored these issues with my ISP, the technician was at my home two days ago and told me my ADSL line was as good as it could. No hope for improvement there. He also stated that the issues would tend to fade away at year's end when Orange, my ISP, will set-up an access point in my village, which DSL improvement will be from 2 Mb/s to 10 Mb/s at worst, 20 Mb/s at best (depending on distance from the village's access point to my home). I prepare for year's end but cannot be satisfied with "tomorrow will be better…". I seek an improvement now.

    Since you (pfBasic) concluded with : "pfSense will shape your traffic and improve some of your problems. It's just that there's so little to work with that the relative improvement may not even be noticeable", maybe it *could be noticeable (?). If, due to some clever settings in a pfSense modem, I do not lose my modem connection and if my computer does not become a dumb animal when traffic demand is too big, THAT improvement will be wonders to me.

    Any hope on that front ?

    Last, do you believe a newbie with no inclinations towards terminal commands (Linux or else) but taking whatever time is needed to learn, could actually succeed to set-up a red SG-1000 pfSense router ?

    TIA.


  • Banned

    Well pfSense is routing software, not a modem. So any issues you are experiencing with your modems hardware will remain, pfSense would go in between the modem and the computer(s) or Wireless access point.

    It might be possible that pfSense could improve your network but I'm not confident that it will.

    One package that I could see helping in your case would be squid, but that really depends on how you use your network. I also have very limited experience using squid so hopefully someone will chime in here if I'm wrong.

    Normally squid is not useful for small networks.
    However, if you were to significantly increase the size of cached objects (into the multi gigabyte range) I believe that squid would cache your large downloads as you were downloading them. The benefit of this would be, if your modem fails in the middle of a big download and you have to restart the download, squid should already have everything you had downloaded before the modem failed which would allow your client to re-download all of that data from the pfSense router directly (very fast, not using internet). Then you would not have to re-download the whole file. Also, if another client on your network needed the same download, they would be able to download it very fast locally instead of using the internet.
    You would have to setup squid to MiTM your network in order to intercept HTTP/S traffic or this wouldn't be very effective, but there are many guides on here for that.

    Again, I'm not sure that this will work as I've described it. I do not use squid. Hopefully someone else that knows will confirm or deny.
    If squid does work the way I think it does and it would help your network then the SG-1000 would not be solution for you, it has neither  enough of or the right type of storage to utilize squid in this way. In that case I would recommend re-using an old computer or building your own out of used parts from eBay.

    You won't need any previous command line knowledge to use pfsense.

    In your case, I would strongly recommend that you install pfSense to spare computer or a borrowed computer and test this out before you purchase anything. If that isn't an option then I would recommend you contact Netgate sales directly before you buy anything and explain your situation. Ask them what their return policies are. Basically, I have a low confidence that pfSense is capable of helping you in any significant way so it would be a shame to spend money only to find out that your problem is not solved.

    Out of curiosity, what part of the world are you in?



  • pfBasic wrote: "Well pfSense is routing software, not a modem. So any issues you are experiencing with your modems hardware will remain, pfSense would go in between the modem and the computer(s) or Wireless access point."

    Sorry, my mistake, I meant "router". I recently moved my modem-router from the attic to the basement and moved with it my Apple Extreme router-wifi to the basement as a second router (I do double NAT). I want to remove the Apple Extreme from the basement where its wifi features are wasted to the ground floor where its is expected as a bridge wifi access point. So I need a router-firewall in the basement to replace the Apple Extreme router. No additional modem, no wifi, no additional switch is needed in the basement.

    pfBasic wrote: "One package that I could see helping in your case would be squid, but that really depends on how you use your network. I also have very limited experience using squid so hopefully someone will chime in here if I'm wrong."

    Thank you for your reply. I do not anticipate to install and implement packages. I prefer FTP (its ability to resume downloads where they stopped) to iTunes but need to live with iTunes. Squid objective to cache downloads router-side is quite appealing. I just found an iTunes setting which was "ON" : "Allow multiple downloads". I ticked it "OFF". Now iTunes downloads the iOS App updaters one at a time. Yesterday night the iOS updater was being downloaded in parallel with two iOS App updaters, all of which ended up not being downloaded either. This morning I made sure the iOS updater was alone and not parallel with other iTunes downloads, and I downloaded the 6 iOS Apps after, one by one. Everything worked. Maybe prohibiting multiple downloads will improve the situation.

    I am convinced now, thanks to you: Traffic Shaping can be a plus but must not be for me a prime consideration.

    I need the new router to replace my Apple Extreme in its current router configuration (main network separate from guests), in the basement. It would initially simply do VLAN (two separate LANs: Main and guests as presently configured on my Apple Extreme), with the intent of changing it later on to provide three separate LANs rather than two: Main, Guests and connected objects [IoT]). The rest, including optimising the house firewall, can wait: double NAT already provides comfort.

    Thank you also for attempting to reply to my last question. I may follow your suggestion to use a retired PC to play with and learn pfSense. I am not sure, though, I want to add to my home the complexity of dealing with an old Windows machine. I will ask Netgate advice as you suggest.

    pfBasic wrote: "Out of curiosity, what part of the world are you in?"

    France, west of Paris, hence my initial concern about EU plugs for powering the SG-1000 (I have US transformers, but using one of them is not very smart).

    Thanks for the help.



  • @Michel-angelo:

    Hello. I am not (yet) a pfSense user and consider to be, but I feel humbled by the average skill level out here. I apologise in advance for the unsatisfactory aspect of the description below.

    In view of my slow bandwidth at my home (about 2 Mb/S down, 0.7 Mb/s up), I want to explore the possibility of throttling the bandwidth of most bandwidth users in my home, because from time to time, my computers get sick at not receiving the info they need from servers to survive (and they get unable to conduct a screen sharing session with Apple Assistance). I wonder if using a pfSense router (like the little red SG-1000) and use on it some traffic shaping tricks would help.

    Or maybe I am being insane.

    The main issues clogging my macs that I have identified (at home, we have 2 macs, 2 iPhones, one iPad and many Dropbox shared folders) are:

    1 - Automatic wifi updates of my wife's iPhone pressuring my ADSL line just when the Apple servers delivering the updater are overloaded with tons of demands from other luckier connected iPhones (Sadly, I cannot suggest her out of that "no-hassle" habit if I want to preserve my happy marriage);

    2 - iTunes automatic downloads of stuff I buy from any of my machines (iPad or iPhone) or of podcasts or other stuff;

    3 - iTunes manual updates for iPhone & iPad Apps, and for iPhone and iPad iOS updates;

    4 - App store automatic downloads of just about everything with automatic install, for mac OS security updates, mac OS updates and mac Apps;

    5 - iCloud sync of iCloud data, which total volume I reduced to trickle (Data volume of less that 0.5 GB in the Apple servers);

    6 - Dropbox (data volume totaling about 11 GB), which I use a lot and is installed on two accounts each on my MacBook (Abeille webmaster and me) and iMac (my wife and me), where bandwith is adjustable and which I limited on each account to : Down: 10 kb/s; Up 5 kb/s;

    7 - Mail IMAP accounts (I still receive from crazy friendly friends who live close to their ADSL provider massive 30 MB dumps of photos which block everything when they come to pass);

    8 - Others, undoubtedly.

    My questions to pfSense wise and unwise people are:

    Can pfSense traffic shaping help me to overcome the above issues ?

    If I purchase the SG-1000 device, red, with its one year subscription to whatever training there is to it from Netgate and keep on sustaining my vow of never using linux terminal commands (too hard for me), can a newbie like me cope with the effort ?

    TIA.

    Traffic-shaping can help with a lot of your problems but there are a numerous traffic-shaping methods you need to choose from. The easiest is probably this, which proportionally shares upload & download bandwidth among active local IPs: https://forum.pfsense.org/index.php?topic=63531.msg364520#msg364520

    A good intro to the fundamentals of traffic-shaping can be found here: http://www.linksysinfo.org/index.php?threads/qos-tutorial.68795/
    Most (all?) of the info in that tutorial is OS-agnostic so it applies to pfSense.


  • Banned

    @Michel-angelo:

    Thank you also for attempting to reply to my last question. I may follow your suggestion to use a retired PC to play with and learn pfSense. I am not sure, though, I want to add to my home the complexity of dealing with an old Windows machine. I will ask Netgate advice as you suggest.

    If you have something on hand you can install pfSense to it won't function as an old Windows machine anymore. pfSense becomes the operating system, when you boot it up it will only boot pfSense. Installing pfSense to an old computer effectively makes that machine pfSense only, Windows or Mac or whatever OS you had on there before will be completely gone unless you decide to remove pfSense and reinstall the old OS someday.

    It's looking like you'd have some noteworthy improvements on your network with pfSense so I'd still recommend installing it on an old computer if it's available to you. That way you know for sure it's what you want before you order the SG-1000 and don't have to potentially deal with returns.
    If you don't have a computer readily accessible, then by all means see if Netgate will let your return it if you don't like it. I'm not familiar with their return policy but I would imagine you'd have some sort of return window.



  • Thank you, Nullity and pfBasic.

    Nullity wrote: "Traffic-shaping can help with a lot of your problems but there are a numerous traffic-shaping methods you need to choose from. The easiest is probably this, which proportionally shares upload & download bandwidth among active local IPs: <https://forum.pfsense.org/index.php?topic=63531.msg364520#msg364520>"

    "… among active local IPs", like: My wife's iPhone downloading an iOS update; or my wife doing internet on the iMac or, if I instruct it to do so, the iMac's downloading of a mac OS or App update; or my iPad downloading automatically a podcast; or the old Apple TV 1st generation downloading a movie to its hard drive; or my MacBook Pro doing many things alone or according to my instructions. Yes, this suggestion makes sense, as a first easy configuration.

    Nullity also wrote: "A good intro to the fundamentals of traffic-shaping can be found here: <http://www.linksysinfo.org/index.php?threads/qos-tutorial.68795/>
    Most (all?) of the info in that tutorial is OS-agnostic so it applies to pfSense.
    "

    Great ! That is a great great read, thanks. Indeed, reading this is most useful to begin. I am presently half way reading it. His favourite example is a line capable of 2 Mb/s download and 0.5 Mb/s upload, just what I have at home and complain about ! It also helps me to understand that the problems I have been facing during the past years could be simply due to my traffic (my non-optimising it), rather than to Orange poor line (which I tried to make more compliant to my needs, and failed) or the lack of download limits on the App Store or iTunes (which I believed were the root causes to my problems while such limitations is simply a crude and suboptimal instance of traffic shaping). Clearly, a "must read".

    pfBasic, I will check with Netgate but am not worried anymore. As long as, as the author of the above tutorial indicates, traffic shaping using a software like pfSense is as good as my understanding of the problems I face and my understanding of the solutions I want to apply to them, it seems to me understanding is key. Thus learning is key. I just need to allocate time to that learning job.

    Thanks for the immense help. I will summarize on mac-forums and the Orange forum what I learned here to conclude there on [still] local open questions. I will come back to this helpful forum from time to time, as soon as I have a pfSense device, which should be soon.


  • Banned

    Great, let us know how it works out!



  • @Michel-angelo:


    it seems to me understanding is key. Thus learning is key. I just need to allocate time to that learning job.
    ...

    I love your attitude.

    While I'm strangely obsessed with traffic-shaping, ever since learning that ACK prioritization could have fixed my many years of problems with 56k where any upload caused a substantial drop in download bitrate, I must partly agree with pfBasic; there is a point of diminishing returns for your educational investment. Traffic-shaping is complex.

    Good luck! :)
    I am happy to help, especially after you've put your new-found knowledge to the test (and failed…), as are many others around here.



  • The single best thing you can do to make your internet feel faster and be more reactive to to keep latency low. Codel should help with this. In your situation, a single single default queue plus ACK queue using HFSC should probably work just fine. Possibly adding one or two extra queues for specific bandwidth requirements.