Trouble with pfSense – no problems (almost) with ClarkConnect



  • We're experiencing some major pain in the arse with our pfSense installation.

    We have a WLAN spanning 4 villages where broadband is not possible, about 8 km of total distance, currently about 15 computers are connected, our endpoint is a 10 Mbit ADSL connection.

    After a testing period with a Linksys AP that also acted as a router (poorly) we decided to install an old P4 1.7, 384MB RAM, 30GB and pfSense. The aforementioned AP is now connected to the LAN interface of our P4 gateway.

    The problems we are experiencing with our pfSense installation are the following:

    Traffic shaping is enabled and we say, for example, limit P2P to 1024 kbit. A user (doesn't matter which one) on the network starts downloading torrents (or files via http) with the full allowed speed (1024 in this case). No one else, and I can't stress this enough, no one else is able to normally access the internet during this time. This user, while downloading, can surf the web at top speed, but everyone else is completely unable to do so. With all the other users, web pages start to load and after a few seconds loading stops (only the title of the pages display, for example). Someone starts to download a file from some FTP, it downloads a few kB (slooow) and it stops. In the meantime the torrenting user can access the net with no problems at all. And when he's done, everything goes back to normal (until someone else decides to download something large).

    If we limit the traffic shaper higher, same story. At the time it seemed like if a shaper queue is full, no one else could do anything.

    But then we tried to disable the shaper, to see if things would improve. They didn't. Same story, except this time there was no upper bandwidth limit. As soon as someone was constantly downloading something at, say only 2 or 3 Mbits or whatever, the others weren't able to do anything. And even if the user downloading was occupying all or almost all of the bandwidth (DLing at over 1 MB/s) he was still able to access the web at very fast speeds, while all the others could not. It's the same thing if the downloader uses just 10% of the bandwidth or 90%, the others are completely left in the cold during this time.

    In the middle of one such „episode“ I tried to log on to the pfSense gateway and download something from the console. There were no problems, I could download at very fast speeds from there. I could also upload the file upstream from the gateway, over WLAN, to our FTP server with 1,5 MB/s. And in the meantime, our „downloader“ was downloading and all the others could do nothing but sit and wait (and curse).

    Our „antenna guy“ suggested there were not enough „sessions“ so we eventually decided to try some other software router. So we installed ClarkConnect. And guess what? Our problems were gone! Even when someone is now constantly downloading and using a lot of bandwidth, the others are able to normally (with minimal drawbacks) use the web. Web pages load, files download, harmony is restored.

    I prefer FreeBSD but in this situation I can't pinpoint the problem, I don't remotely see where the trouble lies. I have no idea why this is happening. Does anyone have any suggestions? What is going on here? Any help or any pointers would be very welcome!

    P.S. One more thing, about a week ago, our WAN interface was down. Couldn't get it up manually. The error message was: „mpd: MESG: insufficient resources available to authenticate user“. A reboot fixed things. IIRC RAM usage was never that high as that time (about 45% (fourty-five)). I almost can't believe some guys saying they have pfSense with uptimes of hundreds of days.

    P.S.S. The web GUI is sometimes sooo slow. Sometimes it even stops loading. With monowall or ClarkConnect it's so much more responsive.

    P.S.S.S. Ok, truth is our problems were not completely solved by this Linux router, by under the same circumstances it works much better than pfSense…



  • How does the CPU load look like while someone is downloading?



  • @GruensFroeschli:

    How does the CPU load look like while someone is downloading?

    When someone is downloading at full speed, the CPU load is always relatively low, 0.15-0.20. It seldom goes higher.

    But we're talking about a P4 here, if we look at the raw processing power, it should cope with such loads with no problems (CPU load is always low).



  • It could have been a interrupt problem
    –> even little traffic can lead to high CPU load.

    But since that isnt the case: what packages do you have installed?



  • @GruensFroeschli:

    It could have been a interrupt problem
    –> even little traffic can lead to high CPU load.

    Oh, right, I see what you mean.

    @GruensFroeschli:

    But since that isnt the case: what packages do you have installed?

    With the first installation we wanted to install ntop, but there were some problems, it just wouldn't install (probably some problems with our CompactFlash card, very slow), so we had zero packages installed. With the second installation we only had ntop (hard disk this time).



  • @Strawin:

    I almost can't believe some guys saying they have pfSense with uptimes of hundreds of days.

    Sorry, couldn't resist. Granted it's a fairly simple setup, but it's running the main office for a nfp with Exchange, Terminal server behind it on recycled hardware. (P233/256)




  • Are all clients connected wirelessly behind the AP? Is the AP direcctly connected to the LAN interface? Do you see interfaceerrors at status>interfaces? This sounds like some kind of collisionproblem to me. Maybe some autonegotiationerror on the lan interface?



  • Well, to me it sounds like misconfigured even pfSense or radio link for the 8km.

    BTW, the shaper after the wizard setup needs some tweaking to behave correctly for these installement.



  • @hoba:

    Are all clients connected wirelessly behind the AP? Is the AP direcctly connected to the LAN interface?

    Yes, all clients are wirelessly connected to the AP, which is in turn connected to the LAN interface of the router.

    @hoba:

    Do you see interfaceerrors at status>interfaces? This sounds like some kind of collisionproblem to me. Maybe some autonegotiationerror on the lan interface?

    I don't recall seeing any errors on the interfaces…

    @ermal:

    Well, to me it sounds like misconfigured even pfSense or radio link for the 8km.

    Today we've done some more extensive testing and more and more things appear to be pointing to our wireless connection and not the router itself. We'll try to connect a few laptops directly to the AP switch connected to the router, do some heavy downloading + surfing and see what comes out of it.

    Believe me, if this turns out to be a WLAN problem and not a pfSense problem, I'll actually be very glad, because that'll mean we'll install pfSense again (I don't like ClarkConnect.)

    Thanks for everyone's pointers at this time. I'll be sure to post again if we find anything of interest.



  • Keep in mind the air is a shared media. The heavier the traffic gets the more likely collisions will happen and packages have to be resent or come in as garbage. If you want to simulate this without the wlan just drop in a switch and throw some of the clients connected on cablenics on that switch. Then retest. I bet it's the wlan and not pfsense.


Locked