Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    [SOLVED] After pfSense upgrade from 2.4.3_1 to 2.4.4_3, neither Squid reverse proxy nor HAProxy work anymore

    Scheduled Pinned Locked Moved Cache/Proxy
    11 Posts 2 Posters 995 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • V
      VPfarr
      last edited by VPfarr

      Hello everyone,

      after upgrading my pfSense firewall (versions in the title), the squid reverse proxy simply does not work anymore.

      When I try to access a server behind the firewall from outside, the connection times out. Before the pfSense upgrade, it worked well. To make sure that the firewall actually received the packets, I made a packet capture for all incoming packets from the IP of the device used, and indeed the packets were there. The squid reverse proxy configuration also seems to be correct in the web interface.

      I can't find any errors in the logs (squid logs, system logs). Also tried uninstalling and reinstalling the package, completely removing the package and all settings as described here, then copying back my configuration, to no avail.

      Checked wether there were any compatibility breaking changes in squid-3.5.27_3, but couldn't find any. The commands

      squid -k check
squid -k parse

      didn't show any errors in the configuration file either. What else should I try or check to troubleshoot this?

      Thank you for any help.

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Does Squid start correctly?

        Is it actually listening on the ports configured?

        Is anything logged to show it's 'seeing' traffic?

        Steve

        1 Reply Last reply Reply Quote 1
        • V
          VPfarr
          last edited by VPfarr

          Hello,

          @stephenw10 thanks for your answer. I'll try to give you the necessary information:

          Startup
          Can't see any errors related to squid not starting correctly. Where should I look?
          This is the log on the old installation:

          $ tail -f /var/squid/logs/cache.log
2019/05/31 15:02:23 kid1| Shutdown: Negotiate authentication.
2019/05/31 15:02:23 kid1| Shutdown: Digest authentication.
2019/05/31 15:02:23 kid1| Shutdown: Basic authentication.
CPU Usage: 500.471 seconds = 368.727 user + 131.744 sys
Maximum Resident Size: 214288 KB
Page faults with physical i/o: 4
2019/05/31 15:09:08 kid1| Starting Squid Cache version 3.5.27 for amd64-portbld-freebsd11.1...
2019/05/31 15:09:08 kid1| Service Name: squid
2019/05/31 15:11:45| Squid is already running!  Process ID 93887
2019/06/04 07:03:23| Squid is already running!  Process ID 93887

          vs. the new one:

          $ tail -f /var/squid/logs/cache.log
2019/06/04 08:10:45 kid1| Service Name: squid
2019/06/04 08:10:51 kid1| Shutdown: NTLM authentication.
2019/06/04 08:10:51 kid1| Shutdown: Negotiate authentication.
2019/06/04 08:10:51 kid1| Shutdown: Digest authentication.
2019/06/04 08:10:51 kid1| Shutdown: Basic authentication.
CPU Usage: 0.098 seconds = 0.082 user + 0.016 sys
Maximum Resident Size: 166336 KB
Page faults with physical i/o: 0
2019/06/04 08:10:59 kid1| Starting Squid Cache version 3.5.27 for amd64-portbld-freebsd11.2...
2019/06/04 08:10:59 kid1| Service Name: squid

          The squid processes are also running:

          $ ps -auxww | grep squid
squid   19331    0.0  1.0 106084  42632  -  S    08:58     0:00.13 (squid-1) -f /usr/local/etc/squid/squid.conf (squid)
squid   19650    0.0  0.1   9824   4888  -  I    08:58     0:00.00 (unlinkd) (unlinkd)
root    21276    0.0  0.5  28280  21336  -  Is   08:52     0:00.00 /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf
root    12378    0.0  0.1   6564   2460  0  S+   09:02     0:00.00 grep squid

          Listening on ports
          On the old installation it looks like this:

          $ sockstat -l | grep squid
squid    squid      93887 6  udp46  *:51383               *:*
squid    squid      93887 7  udp4   *:22505               *:*
squid    squid      93887 9  stream /var/run/php-fpm.socket
squid    squid      93887 18 tcp4   XXX.XXX.XXX.XXX:3128   *:*
squid    squid      93887 19 tcp4   XXX.XXX.XXX.XXX:80     *:*
squid    squid      93887 20 tcp4   XXX.XXX.XXX.XXX:443    *:*
root     squid      93563 9  stream /var/run/php-fpm.socket

          on the new one:

          $ sockstat -l | grep squid
squid    squid      21697 6  udp46  *:58599               *:*
squid    squid      21697 7  udp4   *:37077               *:*
squid    squid      21697 12 stream /var/run/php-fpm.socket
squid    squid      21697 13 stream /var/run/php-fpm.socket
squid    squid      21697 19 tcp4   XXX.XXX.XXX.XXX:3128   *:*
squid    squid      21697 20 tcp4   XXX.XXX.XXX.XXX:80     *:*
squid    squid      21697 21 tcp4   XXX.XXX.XXX.XXX:443    *:*
root     squid      21276 12 stream /var/run/php-fpm.socket
root     squid      21276 13 stream /var/run/php-fpm.socket

          With XXX.XXX.XXX.XXX being the external IP of the firewall. Squid seems to be listening on the correct ports.

          Traffic
          Looking at the access.log on the old installation, I can clearly see "movement" (new entries) when accessing from the outside. On the new installation, no entries are written to the log file.

          Do you need more information? Where else can I find some info?

          Cheers

          EDIT
          By the way, the packet capture still works. So the firewall is getting the packets, squid is listening on the correct ports, but it isn't receiving them?

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Do you see open firewall states for the incoming traffic?

            Reverse squid is not supported so it's not something we test. But many people do run it and I'm not aware of any changes in that might have affected it. The jump from 2.4.3 is quite large though.

            Steve

            1 Reply Last reply Reply Quote 0
            • V
              VPfarr
              last edited by VPfarr

              @stephenw10 Had a look at Diagnostics > States and saw the following entry after trying to connect to one of my reverse proxy peers from the outside:

              Interface   Protocol  Source -> Destination                                  State                      Packets        Bytes
WAN	    tcp       <IP:Port of outside device> -> <IP:Port of firewall>   SYN_SENT:ESTABLISHED	3 / 3	180 B / 180 B

              It works for both HTTP(80) and HTTPS(443) connections as there are two rules which open up the ports. Looks like that would've been a faster way than doing a packet capture...

              Even though it isn't officially supported it's hard to imagine that I'm the only one upgrading to 2.4.4 using squid reverse, right? :-)

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                Yeah, there's almost no way you are!

                So 3 packets each way. Looks like squid is listening and talking back. Do you see any open states to the server?

                The packet capture did not show any reply packets?

                I might have expected it to return an error though if it can't see the server or is failing for some other reason. And log something for that matter.

                Steve

                1 Reply Last reply Reply Quote 1
                • V
                  VPfarr
                  last edited by VPfarr

                  @stephenw10 Yes, there are two open states (the firewall is connected via two virtual interfaces) to the server, so that's not the problem either.

                  By the way, sometimes it's not 3 packets each way as you said, but the number is slowly rising from 3/3 to 3/6 as if there were difficulties sending them out or something? With slowly I mean after refreshing the page in short intervals of ~1sec I could see the number rising. Isn't that strange? It should work in an instant, I'd say.

                  The packet capture only shows three incoming packets for every request from outside. So no reply. On the working installation with 2.4.3 there's a load of packets going both in and out when a page is requested, obviously.
                  I think it looks more like squid is not talking back at all.
                  Here's an example for what an incoming request looks like on the new installation:

                  06:54:22.599432 AF IPv4 (2), length 64: (tos 0x0, ttl 55, id 38541, offset 0, flags [DF], proto TCP (6), length 60)
    85.17.24.66.39106 > XXX.XXX.XXX.XXX.443: Flags [S], cksum 0x2734 (correct), seq 4116721917, win 29200, options [mss 1452,sackOK,TS val 412198512 ecr 0,nop,wscale 7], length 0
06:54:23.597925 AF IPv4 (2), length 64: (tos 0x0, ttl 55, id 38542, offset 0, flags [DF], proto TCP (6), length 60)
    85.17.24.66.39106 > XXX.XXX.XXX.XXX.443: Flags [S], cksum 0x263a (correct), seq 4116721917, win 29200, options [mss 1452,sackOK,TS val 412198762 ecr 0,nop,wscale 7], length 0
06:54:25.599832 AF IPv4 (2), length 64: (tos 0x0, ttl 55, id 38543, offset 0, flags [DF], proto TCP (6), length 60)
    85.17.24.66.39106 > XXX.XXX.XXX.XXX.443: Flags [S], cksum 0x2445 (correct), seq 4116721917, win 29200, options [mss 1452,sackOK,TS val 412199263 ecr 0,nop,wscale 7], length 0

                  (XXX.XXX.XXX.XXX is the external IP of the firewall)

                  Also had a close look at the logs again but couldn't see anything suspicious. I can even see that the requests are being allowed by my rules in the Status > System Logs > Firewall view.

                  Cheers

                  EDIT
                  I just set up the same reverse proxy configuration using HAProxy, tested the same things as before (like checking states, doing a packet capture, checking if haproxy listens on the configured ports), and had exactly the same results. haproxy runs, listens on the ports, I can see states in the state table, three incoming packets in the packet capture when connecting from outside, but no packets sent back. Does that perhaps sound familiar? :-) It doesn't seem to be an issue with squid after all then, does it?

                  EDIT 2
                  Haproxy has a more elaborate stats page which shows that no sessions have been created on frontend and backend whatsoever. But if the firewall gets a request and allows it because of the rule, why doesn't it pass it on from there? I don't know about the internals, but if the packets evidently passed the "rule filtering stage", what way do they go before they're handed to a process listening on a port (squid or haproxy here)?

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Ok, well I know HAProxy works fine in 2.4.4pX, I've tested that myself.

                    I'm thinking maybe Squid is replying via the wrong gateway/interface. The default gateway setup did change quite a lot between 2.4.3 and 2.4.4. Make sure you have a default gateway set in System > Routing > Gateways.

                    Steve

                    1 Reply Last reply Reply Quote 1
                    • V
                      VPfarr
                      last edited by

                      @stephenw10 The gateway was the very thing that was wrong. It was set to the LAN gateway, which is obviously wrong, it has to be the WAN gateway. That also explains why HAProxy didn't work.

                      Thank you very much for your time and effort, and Kudos to your whole team at Netgate who's building quality open source software.

                      Cheers

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        Ah, nice. So reverse Squid back to full function?

                        Steve

                        V 1 Reply Last reply Reply Quote 0
                        • V
                          VPfarr @stephenw10
                          last edited by VPfarr

                          @stephenw10 Yes, the Squid reverse proxy is fully functional again.

                          1 Reply Last reply Reply Quote 1
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.