Thanks for the thoughts. I'm gonna work to get traffic shaping setup for my needs. I have something basic rolling now that brings the performance back to something more sane while the backup is rolling. I have Ooma (third party VOIP) some gaming systems, host some personal websites, have this epic backup going, and of course want the internet to be usable for surfing and watching stuff online. I just gotta do it anyway.
For now at least, I'm calling this issue closed. I am 99% confident my pfSense performance issues were due to that forgotten backup job without using traffic shaping and I was comparing to my old firewall that had traffic shaping all configured on it. Why it knocked me down so much is a bit of a mystery. I'm assuming the uncontrolled backup traffic was causing contention on the way the speedtest.net site works.