LAN (!) performance problem after Squid update

  • Last friday I've updated Squid on a pfSense system running under ESXi 5.5. Everything went fine, the config worked instantly. Over time I've noticed massive slowdowns on the LAN interface though, which made, for instance, RDP over OpenVPN nearly unusable. I can't understand why a Squid update can have such an impact, and hope someone can help pinpoint (and maybe solve?) the problem. For the moment I've rolled back to a snapshot created just before the Squid update, and everything is back to "normal" again.

    My system:

    • ESXi 5.5. running on an Intel S1200RP Server (1 x Xeon E3-1230 v3, 32 GB RAM)
    • pfSense 2.2.4 x64 (2 vCPU, 4 GB RAM)
    • 2 x WAN (Gateway Group, 2 x FRITZ!Box routers handing out a LAN IP-address each to the pfSense WAN interfaces, IPs of pfSense WAN interfaces setup as Exposed Host)
    • Squid 0.2.9 Transparent, SquidGuard 1.9.14, Captive Portal, Multiple NICs with different VLANs (VLANs configured in ESXi, so pfSense sees only standard E1000 NICs - planning on switching the NICs to vmxnet3, which wasn't available when I've installed the VM)

    What happened:

    • Updated Squid to and SquidGuard to 1.9.15. Tested the system after the update, everything worked and was running smoothly
    • Two or three days after the update performance was starting to deteriorate. Noticable effects included randomly disconnecting RDP sessions on the LAN (!) and slow internet access
    • Finally, 5 days after the update, latency in my Admin RDP over OpenVPN connection (which worked perfectly fine for nearly a year) began getting higher and higher, until every mouse click took up to 10 seconds to register
    • Ping from the pfSense machine to the WAN interface(s) showed high latency in irregular intervals
    • Internet access from LAN extremely slow, low throughput and high latency
    • Rebooting pfSense, ESXi host, Switches, Routers, etc. had no effect
    • "Solved" by switching back to a snapshot right before the Squid update

    I didn't have time to check log files thoroughly because my service time windows are very small, but I've kept a snapshot of the "problematic state" and can check the logs if the info is needed. I'd also be happy to provide additional info. The system in its "pre-update-config" was running perfecly fine for at least 6 months, troubles started right after the update.