@stilez:
I had stuff here that was causing similar issues a couple of years ago, with pfsense 2.0.x. The advice above matches what I found in the end. Some more things to try:
Check the system RRD graphs, especially quality. A big issue for me was that dropped packets rose from 0.2% to 35-40% under heavy load, if the config didn't allow enough resources. Worth checking if that's part of your issue.
I got frustrated with this and ended up turning the esxi box off (and pfsense along with it). I set it up about a month ago because I had an assignment for uni where I need to build a test domain environment.
Anyway I got pfsense running again with clients all using pfsense. I still had the torrenting issue. But I noticed the ram usage was high, even though I gave it I think 4GB of RAM. I decided to turn RRD graphs off.
Problem solved! For whatever reason, the RRD graphs were killing my browsing for clients, as well as killing the reverse proxy (squid would just stop, service would NOT restart).
Hopefully this might help people in the future!