[SOLVED] After pfSense upgrade from 2.4.3_1 to 2.4.4_3, neither Squid reverse proxy nor HAProxy work anymore

[VPfarr]

Hello everyone,

after upgrading my pfSense firewall (versions in the title), the squid reverse proxy simply does not work anymore.

When I try to access a server behind the firewall from outside, the connection times out. Before the pfSense upgrade, it worked well. To make sure that the firewall actually received the packets, I made a packet capture for all incoming packets from the IP of the device used, and indeed the packets were there. The squid reverse proxy configuration also seems to be correct in the web interface.

I can't find any errors in the logs (squid logs, system logs). Also tried uninstalling and reinstalling the package, completely removing the package and all settings as described here, then copying back my configuration, to no avail.

Checked wether there were any compatibility breaking changes in squid-3.5.27_3, but couldn't find any. The commands

squid -k check
squid -k parse

didn't show any errors in the configuration file either. What else should I try or check to troubleshoot this?

Thank you for any help.

[stephenw10]

Does Squid start correctly?

Is it actually listening on the ports configured?

Is anything logged to show it's 'seeing' traffic?

Steve

[VPfarr]

Hello,

@stephenw10 thanks for your answer. I'll try to give you the necessary information:

Startup
Can't see any errors related to squid not starting correctly. Where should I look?
This is the log on the old installation:

$ tail -f /var/squid/logs/cache.log
2019/05/31 15:02:23 kid1| Shutdown: Negotiate authentication.
2019/05/31 15:02:23 kid1| Shutdown: Digest authentication.
2019/05/31 15:02:23 kid1| Shutdown: Basic authentication.
CPU Usage: 500.471 seconds = 368.727 user + 131.744 sys
Maximum Resident Size: 214288 KB
Page faults with physical i/o: 4
2019/05/31 15:09:08 kid1| Starting Squid Cache version 3.5.27 for amd64-portbld-freebsd11.1...
2019/05/31 15:09:08 kid1| Service Name: squid
2019/05/31 15:11:45| Squid is already running!  Process ID 93887
2019/06/04 07:03:23| Squid is already running!  Process ID 93887

vs. the new one:

$ tail -f /var/squid/logs/cache.log
2019/06/04 08:10:45 kid1| Service Name: squid
2019/06/04 08:10:51 kid1| Shutdown: NTLM authentication.
2019/06/04 08:10:51 kid1| Shutdown: Negotiate authentication.
2019/06/04 08:10:51 kid1| Shutdown: Digest authentication.
2019/06/04 08:10:51 kid1| Shutdown: Basic authentication.
CPU Usage: 0.098 seconds = 0.082 user + 0.016 sys
Maximum Resident Size: 166336 KB
Page faults with physical i/o: 0
2019/06/04 08:10:59 kid1| Starting Squid Cache version 3.5.27 for amd64-portbld-freebsd11.2...
2019/06/04 08:10:59 kid1| Service Name: squid

The squid processes are also running:

$ ps -auxww | grep squid
squid   19331    0.0  1.0 106084  42632  -  S    08:58     0:00.13 (squid-1) -f /usr/local/etc/squid/squid.conf (squid)
squid   19650    0.0  0.1   9824   4888  -  I    08:58     0:00.00 (unlinkd) (unlinkd)
root    21276    0.0  0.5  28280  21336  -  Is   08:52     0:00.00 /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf
root    12378    0.0  0.1   6564   2460  0  S+   09:02     0:00.00 grep squid

Listening on ports
On the old installation it looks like this:

$ sockstat -l | grep squid
squid    squid      93887 6  udp46  *:51383               *:*
squid    squid      93887 7  udp4   *:22505               *:*
squid    squid      93887 9  stream /var/run/php-fpm.socket
squid    squid      93887 18 tcp4   XXX.XXX.XXX.XXX:3128   *:*
squid    squid      93887 19 tcp4   XXX.XXX.XXX.XXX:80     *:*
squid    squid      93887 20 tcp4   XXX.XXX.XXX.XXX:443    *:*
root     squid      93563 9  stream /var/run/php-fpm.socket

on the new one:

$ sockstat -l | grep squid
squid    squid      21697 6  udp46  *:58599               *:*
squid    squid      21697 7  udp4   *:37077               *:*
squid    squid      21697 12 stream /var/run/php-fpm.socket
squid    squid      21697 13 stream /var/run/php-fpm.socket
squid    squid      21697 19 tcp4   XXX.XXX.XXX.XXX:3128   *:*
squid    squid      21697 20 tcp4   XXX.XXX.XXX.XXX:80     *:*
squid    squid      21697 21 tcp4   XXX.XXX.XXX.XXX:443    *:*
root     squid      21276 12 stream /var/run/php-fpm.socket
root     squid      21276 13 stream /var/run/php-fpm.socket

With XXX.XXX.XXX.XXX being the external IP of the firewall. Squid seems to be listening on the correct ports.

Traffic
Looking at the access.log on the old installation, I can clearly see "movement" (new entries) when accessing from the outside. On the new installation, no entries are written to the log file.

Do you need more information? Where else can I find some info?

Cheers

EDIT
By the way, the packet capture still works. So the firewall is getting the packets, squid is listening on the correct ports, but it isn't receiving them?

[stephenw10]

Do you see open firewall states for the incoming traffic?

Reverse squid is not supported so it's not something we test. But many people do run it and I'm not aware of any changes in that might have affected it. The jump from 2.4.3 is quite large though.

Steve

[VPfarr]

@stephenw10 Had a look at Diagnostics > States and saw the following entry after trying to connect to one of my reverse proxy peers from the outside:

Interface   Protocol  Source -> Destination                                  State                      Packets        Bytes
WAN	    tcp       <IP:Port of outside device> -> <IP:Port of firewall>   SYN_SENT:ESTABLISHED	3 / 3	180 B / 180 B

It works for both HTTP(80) and HTTPS(443) connections as there are two rules which open up the ports. Looks like that would've been a faster way than doing a packet capture...

Even though it isn't officially supported it's hard to imagine that I'm the only one upgrading to 2.4.4 using squid reverse, right? :-)

[stephenw10]

Yeah, there's almost no way you are!

So 3 packets each way. Looks like squid is listening and talking back. Do you see any open states to the server?

The packet capture did not show any reply packets?

I might have expected it to return an error though if it can't see the server or is failing for some other reason. And log something for that matter.

Steve

[VPfarr]

@stephenw10 Yes, there are two open states (the firewall is connected via two virtual interfaces) to the server, so that's not the problem either.

By the way, sometimes it's not 3 packets each way as you said, but the number is slowly rising from 3/3 to 3/6 as if there were difficulties sending them out or something? With slowly I mean after refreshing the page in short intervals of ~1sec I could see the number rising. Isn't that strange? It should work in an instant, I'd say.

The packet capture only shows three incoming packets for every request from outside. So no reply. On the working installation with 2.4.3 there's a load of packets going both in and out when a page is requested, obviously.
I think it looks more like squid is not talking back at all.
Here's an example for what an incoming request looks like on the new installation:

06:54:22.599432 AF IPv4 (2), length 64: (tos 0x0, ttl 55, id 38541, offset 0, flags [DF], proto TCP (6), length 60)
    85.17.24.66.39106 > XXX.XXX.XXX.XXX.443: Flags [S], cksum 0x2734 (correct), seq 4116721917, win 29200, options [mss 1452,sackOK,TS val 412198512 ecr 0,nop,wscale 7], length 0
06:54:23.597925 AF IPv4 (2), length 64: (tos 0x0, ttl 55, id 38542, offset 0, flags [DF], proto TCP (6), length 60)
    85.17.24.66.39106 > XXX.XXX.XXX.XXX.443: Flags [S], cksum 0x263a (correct), seq 4116721917, win 29200, options [mss 1452,sackOK,TS val 412198762 ecr 0,nop,wscale 7], length 0
06:54:25.599832 AF IPv4 (2), length 64: (tos 0x0, ttl 55, id 38543, offset 0, flags [DF], proto TCP (6), length 60)
    85.17.24.66.39106 > XXX.XXX.XXX.XXX.443: Flags [S], cksum 0x2445 (correct), seq 4116721917, win 29200, options [mss 1452,sackOK,TS val 412199263 ecr 0,nop,wscale 7], length 0

(XXX.XXX.XXX.XXX is the external IP of the firewall)

Also had a close look at the logs again but couldn't see anything suspicious. I can even see that the requests are being allowed by my rules in the Status > System Logs > Firewall view.

Cheers

EDIT
I just set up the same reverse proxy configuration using HAProxy, tested the same things as before (like checking states, doing a packet capture, checking if haproxy listens on the configured ports), and had exactly the same results. haproxy runs, listens on the ports, I can see states in the state table, three incoming packets in the packet capture when connecting from outside, but no packets sent back. Does that perhaps sound familiar? :-) It doesn't seem to be an issue with squid after all then, does it?

EDIT 2
Haproxy has a more elaborate stats page which shows that no sessions have been created on frontend and backend whatsoever. But if the firewall gets a request and allows it because of the rule, why doesn't it pass it on from there? I don't know about the internals, but if the packets evidently passed the "rule filtering stage", what way do they go before they're handed to a process listening on a port (squid or haproxy here)?

[stephenw10]

Ok, well I know HAProxy works fine in 2.4.4pX, I've tested that myself.

I'm thinking maybe Squid is replying via the wrong gateway/interface. The default gateway setup did change quite a lot between 2.4.3 and 2.4.4. Make sure you have a default gateway set in System > Routing > Gateways.

Steve

[VPfarr]

@stephenw10 The gateway was the very thing that was wrong. It was set to the LAN gateway, which is obviously wrong, it has to be the WAN gateway. That also explains why HAProxy didn't work.

Thank you very much for your time and effort, and Kudos to your whole team at Netgate who's building quality open source software.

Cheers

[stephenw10]

Ah, nice. So reverse Squid back to full function?

Steve

[VPfarr]

@stephenw10 Yes, the Squid reverse proxy is fully functional again.