BaNZ, have you spread the channels that your wireless APs use? Also are you getting any interference from other wireless nearby? You should also make sure that that the APs are as far apart as you can manage. Best thing would be to try with just one on then see what happens as you add the others.
Yes, I've checked my neighbour channel and changed to a different one. I've changed my channel to 3 and I tried 4 / 5. I found out it's not interference. Its something to do with my cheap AP and pfsense. If I connect my AP to pfsense all the clients on the AP gets around 50-70k/s and if I turn on a p2p application I get 20-40k/s. This is really strange because if I plug a router directly to pfsense it is perfectly fine and I get my max speed. So I took my AP and plug it in directly to WAN instead of going through AP, this works fine too I get my max speed. This proves that my AP is working. When I check my AP in the dhcp page on pfsense it shows its offline. But after pinging the AP from pfsense it shows online and brings the speed up to around 100k/s. But if I reset the AP it will again show its offline and the speed drops down again.
I tried to plug another wifi router and I get my max speed fine. So it is definitely something wrong with the AP with pfsense. I tried fiddling around the threshold in my AP and it improves it slightly but not much. I probably gonna have to get a new wifi router. Or maybe use pfsense->router->ap since I have a spare router lying somewhere and I can save some money.
update: bought new wifi router and plugged it in.
It is now fine. Had a problem earlier when I'm transferring large files, basically it kills all other clients connection when there is a huge amount of traffic. I changed the channel and it is fine.