[SOLVED] After pfSense upgrade from 2.4.3_1 to 2.4.4_3, neither Squid reverse proxy nor HAProxy work anymore
after upgrading my pfSense firewall (versions in the title), the squid reverse proxy simply does not work anymore.
When I try to access a server behind the firewall from outside, the connection times out. Before the pfSense upgrade, it worked well. To make sure that the firewall actually received the packets, I made a packet capture for all incoming packets from the IP of the device used, and indeed the packets were there. The squid reverse proxy configuration also seems to be correct in the web interface.
I can't find any errors in the logs (squid logs, system logs). Also tried uninstalling and reinstalling the package, completely removing the package and all settings as described here, then copying back my configuration, to no avail.
Checked wether there were any compatibility breaking changes in squid-3.5.27_3, but couldn't find any. The commands
squid -k check squid -k parse
didn't show any errors in the configuration file either. What else should I try or check to troubleshoot this?
Thank you for any help.
Does Squid start correctly?
Is it actually listening on the ports configured?
Is anything logged to show it's 'seeing' traffic?
@stephenw10 thanks for your answer. I'll try to give you the necessary information:
Can't see any errors related to squid not starting correctly. Where should I look?
This is the log on the old installation:
$ tail -f /var/squid/logs/cache.log 2019/05/31 15:02:23 kid1| Shutdown: Negotiate authentication. 2019/05/31 15:02:23 kid1| Shutdown: Digest authentication. 2019/05/31 15:02:23 kid1| Shutdown: Basic authentication. CPU Usage: 500.471 seconds = 368.727 user + 131.744 sys Maximum Resident Size: 214288 KB Page faults with physical i/o: 4 2019/05/31 15:09:08 kid1| Starting Squid Cache version 3.5.27 for amd64-portbld-freebsd11.1... 2019/05/31 15:09:08 kid1| Service Name: squid 2019/05/31 15:11:45| Squid is already running! Process ID 93887 2019/06/04 07:03:23| Squid is already running! Process ID 93887
vs. the new one:
$ tail -f /var/squid/logs/cache.log 2019/06/04 08:10:45 kid1| Service Name: squid 2019/06/04 08:10:51 kid1| Shutdown: NTLM authentication. 2019/06/04 08:10:51 kid1| Shutdown: Negotiate authentication. 2019/06/04 08:10:51 kid1| Shutdown: Digest authentication. 2019/06/04 08:10:51 kid1| Shutdown: Basic authentication. CPU Usage: 0.098 seconds = 0.082 user + 0.016 sys Maximum Resident Size: 166336 KB Page faults with physical i/o: 0 2019/06/04 08:10:59 kid1| Starting Squid Cache version 3.5.27 for amd64-portbld-freebsd11.2... 2019/06/04 08:10:59 kid1| Service Name: squid
The squid processes are also running:
$ ps -auxww | grep squid squid 19331 0.0 1.0 106084 42632 - S 08:58 0:00.13 (squid-1) -f /usr/local/etc/squid/squid.conf (squid) squid 19650 0.0 0.1 9824 4888 - I 08:58 0:00.00 (unlinkd) (unlinkd) root 21276 0.0 0.5 28280 21336 - Is 08:52 0:00.00 /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf root 12378 0.0 0.1 6564 2460 0 S+ 09:02 0:00.00 grep squid
Listening on ports
On the old installation it looks like this:
$ sockstat -l | grep squid squid squid 93887 6 udp46 *:51383 *:* squid squid 93887 7 udp4 *:22505 *:* squid squid 93887 9 stream /var/run/php-fpm.socket squid squid 93887 18 tcp4 XXX.XXX.XXX.XXX:3128 *:* squid squid 93887 19 tcp4 XXX.XXX.XXX.XXX:80 *:* squid squid 93887 20 tcp4 XXX.XXX.XXX.XXX:443 *:* root squid 93563 9 stream /var/run/php-fpm.socket
on the new one:
$ sockstat -l | grep squid squid squid 21697 6 udp46 *:58599 *:* squid squid 21697 7 udp4 *:37077 *:* squid squid 21697 12 stream /var/run/php-fpm.socket squid squid 21697 13 stream /var/run/php-fpm.socket squid squid 21697 19 tcp4 XXX.XXX.XXX.XXX:3128 *:* squid squid 21697 20 tcp4 XXX.XXX.XXX.XXX:80 *:* squid squid 21697 21 tcp4 XXX.XXX.XXX.XXX:443 *:* root squid 21276 12 stream /var/run/php-fpm.socket root squid 21276 13 stream /var/run/php-fpm.socket
With XXX.XXX.XXX.XXX being the external IP of the firewall. Squid seems to be listening on the correct ports.
Looking at the access.log on the old installation, I can clearly see "movement" (new entries) when accessing from the outside. On the new installation, no entries are written to the log file.
Do you need more information? Where else can I find some info?
By the way, the packet capture still works. So the firewall is getting the packets, squid is listening on the correct ports, but it isn't receiving them?
Do you see open firewall states for the incoming traffic?
Reverse squid is not supported so it's not something we test. But many people do run it and I'm not aware of any changes in that might have affected it. The jump from 2.4.3 is quite large though.
@stephenw10 Had a look at Diagnostics > States and saw the following entry after trying to connect to one of my reverse proxy peers from the outside:
Interface Protocol Source -> Destination State Packets Bytes WAN tcp <IP:Port of outside device> -> <IP:Port of firewall> SYN_SENT:ESTABLISHED 3 / 3 180 B / 180 B
It works for both HTTP(80) and HTTPS(443) connections as there are two rules which open up the ports. Looks like that would've been a faster way than doing a packet capture...
Even though it isn't officially supported it's hard to imagine that I'm the only one upgrading to 2.4.4 using squid reverse, right? :-)
Yeah, there's almost no way you are!
So 3 packets each way. Looks like squid is listening and talking back. Do you see any open states to the server?
The packet capture did not show any reply packets?
I might have expected it to return an error though if it can't see the server or is failing for some other reason. And log something for that matter.
@stephenw10 Yes, there are two open states (the firewall is connected via two virtual interfaces) to the server, so that's not the problem either.
By the way, sometimes it's not 3 packets each way as you said, but the number is slowly rising from 3/3 to 3/6 as if there were difficulties sending them out or something? With slowly I mean after refreshing the page in short intervals of ~1sec I could see the number rising. Isn't that strange? It should work in an instant, I'd say.
The packet capture only shows three incoming packets for every request from outside. So no reply. On the working installation with 2.4.3 there's a load of packets going both in and out when a page is requested, obviously.
I think it looks more like squid is not talking back at all.
Here's an example for what an incoming request looks like on the new installation:
06:54:22.599432 AF IPv4 (2), length 64: (tos 0x0, ttl 55, id 38541, offset 0, flags [DF], proto TCP (6), length 60) 188.8.131.52.39106 > XXX.XXX.XXX.XXX.443: Flags [S], cksum 0x2734 (correct), seq 4116721917, win 29200, options [mss 1452,sackOK,TS val 412198512 ecr 0,nop,wscale 7], length 0 06:54:23.597925 AF IPv4 (2), length 64: (tos 0x0, ttl 55, id 38542, offset 0, flags [DF], proto TCP (6), length 60) 184.108.40.206.39106 > XXX.XXX.XXX.XXX.443: Flags [S], cksum 0x263a (correct), seq 4116721917, win 29200, options [mss 1452,sackOK,TS val 412198762 ecr 0,nop,wscale 7], length 0 06:54:25.599832 AF IPv4 (2), length 64: (tos 0x0, ttl 55, id 38543, offset 0, flags [DF], proto TCP (6), length 60) 220.127.116.11.39106 > XXX.XXX.XXX.XXX.443: Flags [S], cksum 0x2445 (correct), seq 4116721917, win 29200, options [mss 1452,sackOK,TS val 412199263 ecr 0,nop,wscale 7], length 0
(XXX.XXX.XXX.XXX is the external IP of the firewall)
Also had a close look at the logs again but couldn't see anything suspicious. I can even see that the requests are being allowed by my rules in the Status > System Logs > Firewall view.
I just set up the same reverse proxy configuration using HAProxy, tested the same things as before (like checking states, doing a packet capture, checking if haproxy listens on the configured ports), and had exactly the same results. haproxy runs, listens on the ports, I can see states in the state table, three incoming packets in the packet capture when connecting from outside, but no packets sent back. Does that perhaps sound familiar? :-) It doesn't seem to be an issue with squid after all then, does it?
Haproxy has a more elaborate stats page which shows that no sessions have been created on frontend and backend whatsoever. But if the firewall gets a request and allows it because of the rule, why doesn't it pass it on from there? I don't know about the internals, but if the packets evidently passed the "rule filtering stage", what way do they go before they're handed to a process listening on a port (squid or haproxy here)?
Ok, well I know HAProxy works fine in 2.4.4pX, I've tested that myself.
I'm thinking maybe Squid is replying via the wrong gateway/interface. The default gateway setup did change quite a lot between 2.4.3 and 2.4.4. Make sure you have a default gateway set in System > Routing > Gateways.
@stephenw10 The gateway was the very thing that was wrong. It was set to the LAN gateway, which is obviously wrong, it has to be the WAN gateway. That also explains why HAProxy didn't work.
Thank you very much for your time and effort, and Kudos to your whole team at Netgate who's building quality open source software.
Ah, nice. So reverse Squid back to full function?
@stephenw10 Yes, the Squid reverse proxy is fully functional again.