When I managed to figure out what the problem was, and it's not pfsense 8)
My nokia N810 uses wifi power saving which it set to maximum by default. There is a know issue with connecting to Atheros based wifi cards. Setting the power saving to intermediate solved the problem.
I also have a Fonera, which has the same Atheros chipset as my wifi card in my pfsense firewall, and it too has the same problems.