Whenever I saturate my ISP's upload speed (by uploading multiple photos to Amazon Photos), the Rtd of my WAN gets up to 1s and my gateway is marked as "offline". But then I can still browse the Internet although it is very spotty.
What does pfsense do internally when the main ISP gateway is marked as offline? Does it remove the gateway from being used in the routing table or what? I don't believe this is the case because I can still browse the Internet but I just wanted to understand this deeper.
Thanks for the help.
If you only have one gateway then it doesn't do anything other than log it as an issue and trigger a load of scripts unnecessarily.
You can edit the gateway and either set the latency alarm value much higher for your gateway or check 'Disable Gateway Monitoring Action'.
Or you could tackle the issue from the other side and try to shape your traffic to prevent the latency.
Sorry for the super late reply. If I disable gateway monitoring though, will I still encounter the spotty Internet experience while my upload speed is being saturated? I guess I just want to understand why am I experiencing spottiness only when the upload speed is saturated but not when the download speed is saturated.
It's usually much easier to saturate the upload. Also your ISP may be traffic shaping what leaves their end to give a better experience but they can't control what packets arrive from your end.
Some traffic shaping on the upload traffic would probably help. Try just enabling a traffic shaping queue on the WAN with the bandwidth set to slightly less than your actual upload speed and set the scheduler to codelq.
I see. Why don't they do the same on the download then?
Yeah, I'll probably try implementing upload traffic shaping to fix this.
Traffic shaping really only works well from the sending end. That's the only place you can have any real control.
If you try to shape the download traffic in pfSense for example, the best you can do is put a queue on the LAN and start dropping packets there. If whatever is saturating the WAN download is TCP then the dropped packets will cause the TCP window to scale back and you can achieve some throttling. If it's not TCP then it makes no difference, the other end will keep sending at full speed no matter what you do at the receiving end.
Thus the opposite is also true. The ISP can, potentially, apply shaping to traffic leaving their network that you see as download but can do nothing for upload traffic.
In this sort of situation you can usually get a good improvement with a basic shaper on WAN.
Thanks for the reply. That completely makes sense. I'll experiment on upload traffic shaping to see if this solves my issue.