• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Squid transparent proxy + SquidGuard error "transaction-end-before-headers"

Scheduled Pinned Locked Moved Cache/Proxy
23 Posts 10 Posters 34.5k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • X
    xalex1977
    last edited by xalex1977 Jun 4, 2020, 4:03 PM Jun 4, 2020, 4:02 PM

    Hi,
    i have version 2.4.5 of PfSense, i have installed squid in transparent mode (SSL/MITM Mode Splice All) + squidguard:
    3966ee7c-f74a-47e1-9b66-55baa1cc4284-image.png

    Squid proxy SSL/MITM Mode Splice All and SquidGuard Default access ALL DENY and only a few domains allowed;

    in the squid log , from all clients I find many of these:

    NONE/000 error:transaction-end-before-headers
    NONE/503 http:443
    NONE/200 http:443

    2a6c2633-827c-43b4-9918-f8d434414330-image.png

    Why?
    is it a problem?
    ☹

    X K 2 Replies Last reply Jun 5, 2020, 7:58 AM Reply Quote 0
    • X
      xalex1977 @xalex1977
      last edited by Jun 5, 2020, 7:58 AM

      reading here:
      https://www.freshports.org/www/squid
      http://www.squid-cache.org/Versions/v4/squid-4.11-RELEASENOTES.html#ss3.2
      08830325-988a-4253-ace1-004fd1206640-image.png

      that with version 4.11 they solved the problem?
      how can I update my current version to 4.11?

      Thanks

      X 1 Reply Last reply Jun 19, 2020, 7:21 AM Reply Quote 0
      • X
        xalex1977
        last edited by xalex1977 Jun 12, 2020, 7:33 AM Jun 12, 2020, 7:32 AM

        I installed the latest updates 0.4.44_26 but the Squid version is always 4.10 ☹

        9b3e64fc-012d-4ceb-939f-fef0691ffab9-image.png

        1 Reply Last reply Reply Quote 0
        • C
          CaliPilot
          last edited by Jun 12, 2020, 8:05 AM

          MITM is always some pain in the back and it took me a while to make it work. Without to know your current config & ACLs it is impossible to point you in the right direction to find a solution but I can tell you that it works, if the setup is done right.

          Chris

          X 1 Reply Last reply Jun 12, 2020, 8:27 AM Reply Quote 0
          • X
            xalex1977 @CaliPilot
            last edited by xalex1977 Jun 12, 2020, 8:30 AM Jun 12, 2020, 8:27 AM

            @CaliPilot
            in Squid there are no particular ACLs, the whole network passes through the transparent proxy,

            in SquidGuard Common ACL Target Categories "Default access [all]" is DENY and "Lark" is ALLOW
            in SquidGuard "Target categories" (Lark) witch Domain list
            869cbeca-ca1a-4ada-8854-23d66e5268da-image.png

            the "error: transaction-end-before-headers" problem be fixed on version 4.11 or not?
            how do i update it?

            C 1 Reply Last reply Jun 12, 2020, 9:59 AM Reply Quote 0
            • C
              CaliPilot @xalex1977
              last edited by Jun 12, 2020, 9:59 AM

              @xalex1977 You should post your squid.conf because (with no offense) I guess there are other issues. With splice all it should work more or less out of the box if the setup is right. You can also check this post ...

              https://forum.netgate.com/topic/153933/solved-squid-0-4-44_25-assertion-failed-http-cc-1533-comm-monitorsread-serverconnection-fd/2

              ... where I answered on our own issues and how we solved them.

              Chris

              1 Reply Last reply Reply Quote 0
              • X
                xalex1977
                last edited by Jun 14, 2020, 6:00 PM

                7339c9df-4441-4827-b75a-4d2d6009cd51-image.png
                f1cefc15-d28b-499f-a24e-c3ebfd09c7d5-image.png

                squid.conf:

                # This file is automatically generated by pfSense
                # Do not edit manually !
                
                http_port 192.168.1.253:3128 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE
                
                http_port 127.0.0.1:3128 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE
                
                https_port 127.0.0.1:3129 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=10MB cert=/usr/local/etc/squid/serverkey.pem capath=/usr/local/share/certs/ cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS tls-dh=prime256v1:/etc/dh-parameters.2048 options=NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE
                
                icp_port 0
                digest_generation off
                dns_v4_first on
                pid_filename /var/run/squid/squid.pid
                cache_effective_user squid
                cache_effective_group proxy
                error_default_language en
                icon_directory /usr/local/etc/squid/icons
                visible_hostname localhost
                cache_mgr admin@localhost
                access_log /var/squid/logs/access.log
                cache_log /var/squid/logs/cache.log
                cache_store_log none
                netdb_filename /var/squid/logs/netdb.state
                pinger_enable on
                pinger_program /usr/local/libexec/squid/pinger
                sslcrtd_program /usr/local/libexec/squid/security_file_certgen -s /var/squid/lib/ssl_db -M 4MB -b 2048
                tls_outgoing_options capath=/usr/local/share/certs/
                tls_outgoing_options options=NO_SSLv3,NO_TLSv1,SINGLE_DH_USE,SINGLE_ECDH_USE
                tls_outgoing_options cipher=EECDH+ECDSA+AESGCM:EECDH+aRSA+AESGCM:EECDH+ECDSA+SHA384:EECDH+ECDSA+SHA256:EECDH+aRSA+SHA384:EECDH+aRSA+SHA256:EECDH+aRSA+RC4:EECDH:EDH+aRSA:!RC4:!aNULL:!eNULL:!LOW:!3DES:!SHA1:!MD5:!EXP:!PSK:!SRP:!DSS
                tls_outgoing_options flags=DONT_VERIFY_PEER
                sslcrtd_children 5
                sslproxy_cert_error allow all
                
                logfile_rotate 7
                debug_options rotate=7
                shutdown_lifetime 3 seconds
                # Allow local network(s) on interface(s)
                acl localnet src  192.168.1.0/24
                forwarded_for on
                uri_whitespace strip
                
                acl dynamic urlpath_regex cgi-bin \?
                cache deny dynamic
                
                cache_mem 64 MB
                maximum_object_size_in_memory 256 KB
                memory_replacement_policy heap GDSF
                cache_replacement_policy heap LFUDA
                minimum_object_size 0 KB
                maximum_object_size 4 MB
                cache_dir ufs /var/squid/cache 100 16 256
                offline_mode off
                cache_swap_low 90
                cache_swap_high 95
                cache allow all
                # Add any of your own refresh_pattern entries above these.
                refresh_pattern ^ftp:    1440  20%  10080
                refresh_pattern ^gopher:  1440  0%  1440
                refresh_pattern -i (/cgi-bin/|\?) 0  0%  0
                refresh_pattern .    0  20%  4320
                
                
                #Remote proxies
                
                
                # Setup some default acls
                # ACLs all, manager, localhost, and to_localhost are predefined.
                acl allsrc src all
                acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 8080 3128 3129 1025-65535 
                acl sslports port 443 563 8080 
                
                acl purge method PURGE
                acl connect method CONNECT
                
                # Define protocols used for redirects
                acl HTTP proto HTTP
                acl HTTPS proto HTTPS
                
                # SslBump Peek and Splice
                # http://wiki.squid-cache.org/Features/SslPeekAndSplice
                # http://wiki.squid-cache.org/ConfigExamples/Intercept/SslBumpExplicit
                # Match against the current step during ssl_bump evaluation [fast]
                # Never matches and should not be used outside the ssl_bump context.
                #
                # At each SslBump step, Squid evaluates ssl_bump directives to find
                # the next bumping action (e.g., peek or splice). Valid SslBump step
                # values and the corresponding ssl_bump evaluation moments are:
                #   SslBump1: After getting TCP-level and HTTP CONNECT info.
                #   SslBump2: After getting TLS Client Hello info.
                #   SslBump3: After getting TLS Server Hello info.
                # These ACLs exist even when 'SSL/MITM Mode' is set to 'Custom' so that
                # they can be used there for custom configuration.
                acl step1 at_step SslBump1
                acl step2 at_step SslBump2
                acl step3 at_step SslBump3
                acl whitelist dstdom_regex -i "/var/squid/acl/whitelist.acl"
                http_access allow manager localhost
                
                http_access deny manager
                http_access allow purge localhost
                http_access deny purge
                http_access deny !safeports
                http_access deny CONNECT !sslports
                
                # Always allow localhost connections
                http_access allow localhost
                
                request_body_max_size 0 KB
                delay_pools 1
                delay_class 1 2
                delay_parameters 1 -1/-1 -1/-1
                delay_initial_bucket_level 100
                delay_access 1 allow allsrc
                
                # Reverse Proxy settings
                
                
                # Package Integration
                url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
                url_rewrite_bypass off
                url_rewrite_children 16 startup=8 idle=4 concurrency=0
                
                # Custom options before auth
                
                
                # Always allow access to whitelist domains
                http_access allow whitelist
                ssl_bump peek step1
                ssl_bump splice all
                # Setup allowed ACLs
                # Allow local network(s) on interface(s)
                http_access allow localnet
                # Default block all to be sure
                http_access deny allsrc
                
                
                

                squidGuard.conf:

                # ============================================================
                # SquidGuard configuration file
                # This file generated automaticly with SquidGuard configurator
                # (C)2006 Serg Dvoriancev
                # email: dv_serg@mail.ru
                # ============================================================
                
                logdir /var/squidGuard/log
                dbhome /var/db/squidGuard
                
                # 
                dest Lark {
                	domainlist Lark/domains
                	log block.log
                }
                
                # 
                rew safesearch {
                	s@(google\..*/search?.*q=.*)@\1\&safe=active@i
                	s@(google\..*/images.*q=.*)@\1\&safe=active@i
                	s@(google\..*/groups.*q=.*)@\1\&safe=active@i
                	s@(google\..*/news.*q=.*)@\1\&safe=active@i
                	s@(yandex\..*/yandsearch?.*text=.*)@\1\&fyandex=1@i
                	s@(search\.yahoo\..*/search.*p=.*)@\1\&vm=r&v=1@i
                	s@(search\.live\..*/.*q=.*)@\1\&adlt=strict@i
                	s@(search\.msn\..*/.*q=.*)@\1\&adlt=strict@i
                	s@(\.bing\..*/.*q=.*)@\1\&adlt=strict@i
                	s@(duckduckgo\..*/?.*q=.*)@\1\&kp=1@i
                	s@(rambler\..*/?.*query=.*)@\1\&adult=family@i
                	s@(qwant\..*/?.*q=.*)@\1\&s=2@i
                	log block.log
                }
                
                # 
                acl  {
                	# 
                	default  {
                		pass !in-addr Lark none
                		redirect http://192.168.1.253:8080/sgerror.php?url=403%20&a=%a&n=%n&i=%i&s=%s&t=%t&u=%u
                		log block.log
                	}
                }
                

                I also attach
                squidguard_conf.xml

                C 1 Reply Last reply Jul 15, 2020, 5:46 PM Reply Quote 0
                • X
                  xalex1977 @xalex1977
                  last edited by Jun 19, 2020, 7:21 AM

                  @xalex1977 said in Squid transparent proxy + SquidGuard error "transaction-end-before-headers":

                  reading here:
                  https://www.freshports.org/www/squid
                  http://www.squid-cache.org/Versions/v4/squid-4.11-RELEASENOTES.html#ss3.2
                  08830325-988a-4253-ace1-004fd1206640-image.png

                  that with version 4.11 they solved the problem?
                  how can I update my current version to 4.11?

                  Thanks

                  when will squid-4.11 be installed from packages?

                  X 1 Reply Last reply Jul 1, 2020, 12:28 PM Reply Quote 0
                  • X
                    xalex1977 @xalex1977
                    last edited by Jul 1, 2020, 12:28 PM

                    can anyone tell me when squid-4.11 will be available to solve the problem?
                    thank you

                    1 Reply Last reply Reply Quote 0
                    • S
                      srlek
                      last edited by srlek Jul 3, 2020, 11:10 AM Jul 3, 2020, 10:59 AM

                      Have sam issue here, after update to 4.10, users started to complain and have same errors in logs, squid 4.11 is not available for now in package manager

                      X 1 Reply Last reply Jul 3, 2020, 11:21 AM Reply Quote 0
                      • X
                        xalex1977 @srlek
                        last edited by Jul 3, 2020, 11:21 AM

                        @srlek have you encountered the same problem? how can we solve? 😕

                        1 Reply Last reply Reply Quote 0
                        • viktor_gV
                          viktor_g Netgate
                          last edited by Jul 10, 2020, 9:09 AM

                          Redmine issue created: https://redmine.pfsense.org/issues/10608

                          1 Reply Last reply Reply Quote 0
                          • L
                            ltripoli
                            last edited by Jul 13, 2020, 1:00 PM

                            Same Error..... too bad .....

                            1 Reply Last reply Reply Quote 0
                            • T
                              tchadrack
                              last edited by tchadrack Jul 15, 2020, 5:07 PM Jul 15, 2020, 5:01 PM

                              Same error here, squid-4.10

                              I found some release notes about squid version 4.11, here:

                              http://ftp.meisei-u.ac.jp/mirror/squid/squid-4.11-RELEASENOTES.html#toc3.2

                              Section 3.2 Changes to existing tags:

                              Unused connections received in http_port or https_port or transactions terminated before reading[parsing] request headers are logged with URI error:transaction-end-before-headers.

                              1 Reply Last reply Reply Quote 0
                              • C
                                CaliPilot @xalex1977
                                last edited by CaliPilot Jul 15, 2020, 5:47 PM Jul 15, 2020, 5:46 PM

                                @xalex1977 larksuite.com is running on a CDN. We faced the same issues like you, with other and more dramatic consequences (squid crashes and so on). We pinned it down to DNS. We changed from Google-DNS to the local ones from our ISPs, we made DNS-Resolver work (and played with minimum TTLs), fixed our Windows-DNS Servers and disabled the name caches (which doesn't work, we clean them now manually every 5 Minutes via a script) and we disabled name-caching on our clients. Key is that your DNS between Pfsense, internal DNS and your clients is always sync. Squid is a b*tch if one of your boxes is responding with the wrong host or ips.

                                We made for critical sites (like SAP C4C) static host overrides because they drive crazy with changing their IP every few minutes. All in all i'm very happy right now with or setup!

                                Another trick we learned that you can use an Alias in Squid to Bypass traffic like for services hosted by Apple (iTunes, App-Store and so on). Key is that you add all known cnames from that cdn-hosted site to your Alias and add that Alias to Squid in the "bypass proxy for these destination IPs".

                                aa591f58-c505-454d-8c89-31663061aa2c-image.png

                                f2eaac79-8a45-477e-84c0-300bcc99375e-image.png

                                7abdf363-36ba-4bc4-ad93-b607a97419a9-image.png

                                I'm not a hardcore linux or network specialist but if you check your config, fix your dns and have everything in place it works pretty well. It is some work in the beginning (GotoMeeting, Teams and Apple stuff) and sometimes it's challenging but it will work!

                                Chris

                                X D 2 Replies Last reply Jul 16, 2020, 2:40 PM Reply Quote 1
                                • X
                                  xalex1977 @CaliPilot
                                  last edited by Jul 16, 2020, 2:40 PM

                                  @CaliPilot hi, unfortunately larksuite has a lot of third level domains and it is impossible for me to create aliases;
                                  we use the windows active directory server dns, I would not want disabled Windows-DNS Servers name caches and name-caching on our clients, it is too uncomfortable.

                                  Will this problem not be corrected with a new version of Squid?

                                  Thanks

                                  1 Reply Last reply Reply Quote 0
                                  • C
                                    coffeelover
                                    last edited by Jul 27, 2020, 8:13 AM

                                    The error in your log seems to be a CONNECT issue.
                                    The Browser opens a CONNECT session to the target site and will only accept a socket address, not a URL.
                                    The Rewrite URL from squidguard https://site.com/sgerror.php is parsed as a socket address like host:port

                                    We have squid with SSL MITM, ClamAV and Squidguard with correct url redirect working with the following setting:

                                    squid mitm: splice whitelist, bump otherwise

                                    additional advanced options:
                                    url_rewrite_access deny CONNECT
                                    url_rewrite_access allow all

                                    This will deny CONNECT sessions for non-whitelisted sites and will let the redirect work.

                                    As redirect function in squidguard you need to set "ext url move", not redirect.

                                    K 1 Reply Last reply Jul 7, 2021, 8:50 PM Reply Quote 1
                                    • K
                                      kasalencar @xalex1977
                                      last edited by Jul 7, 2021, 8:38 PM

                                      @xalex1977 também tenho esse mesmo problema! Estou com pfsnse 2.5.0 + Proxy Transparente + interceptação SSL. Notei que esses erros está diretamente relacionado ao dispositivo que acessam via wifi. Não sei se tem algo relacionado ou só consciência.

                                      X 1 Reply Last reply Jul 7, 2021, 10:03 PM Reply Quote 0
                                      • K
                                        kasalencar @coffeelover
                                        last edited by Jul 7, 2021, 8:50 PM

                                        Amigo onde você insere essa informação adicional?
                                        Seria no campo "Opções personalizadas (SSL / MITM)" ???

                                        1 Reply Last reply Reply Quote 0
                                        • X
                                          xalex1977 @kasalencar
                                          last edited by xalex1977 Jul 7, 2021, 10:05 PM Jul 7, 2021, 10:03 PM

                                          @kasalencar i have version 2.4.5-RELEASE-p1 + squid in transparent mode (SSL/MITM Mode Splice All) + squidguard and i still have the problem 😢

                                          K 1 Reply Last reply Jul 8, 2021, 9:15 PM Reply Quote 0
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received