TLS pr_end_of_file_error (non proxy related)

joeschmoe · Apr 10, 2022, 6:53 AM

Im getting 2 types of responses from the pfsense cli for the same website. problem is that when i hit the write:errno=54, the pr_end_of_file_error message in squid appears. This is not a squid cert error since im not running mitm/transparent. The testing below is directly from the pfsense cli.

Reading wireshark difference, if the client hello is shows tls1, the error is encountered. Weird is that its from the same site.

With ssl error(browser pr_end_of_file_error response):
[[2.5.2-RELEASE][whoami@oo.ooo.oooo]/: openssl s_client -connect radiopaedia.org:443
CONNECTED(00000003)
write:errno=54

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 319 bytes
Verification: OK

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---]

Without ssl error on browser:
[2.5.2-RELEASE][whoami@oo.ooo.oooo]/: openssl s_client -connect radiopaedia.org:443
CONNECTED(00000003)
depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify return:1
depth=1 C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = sni.cloudflaressl.com
verify return:1

Certificate chain
0 s:C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = sni.cloudflaressl.com
i:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
1 s:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
i:C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root

Server certificate
-----BEGIN CERTIFICATE-----
MIIFOzCCBOGgAwIBAgIQBBWphhBLhdqiMXLlVIVxHDAKBggqhkjOPQQDAjBKMQsw
CQYDVQQGEwJVUzEZMBcGA1UEChMQQ2xvdWRmbGFyZSwgSW5jLjEgMB4GA1UEAxMX
Q2xvdWRmbGFyZSBJbmMgRUNDIENBLTMwHhcNMjEwNzE2MDAwMDAwWhcNMjIwNzE1
MjM1OTU5WjB1MQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQG
A1UEBxMNU2FuIEZyYW5jaXNjbzEZMBcGA1UEChMQQ2xvdWRmbGFyZSwgSW5jLjEe
MBwGA1UEAxMVc25pLmNsb3VkZmxhcmVzc2wuY29tMFkwEwYHKoZIzj0CAQYIKoZI
zj0DAQcDQgAExwKT2LHQQCz78ykhm7Tetk5oAUnKCtR26kMroBEjNFNwN4lNHjMe
nF3y6ko18vNnUFYDEBQ6pAOMJkbF75NfEqOCA3wwggN4MB8GA1UdIwQYMBaAFKXO
N+rrsHUOlGeItEX62SQQh5YfMB0GA1UdDgQWBBQXkNbm9qfWFez+i37TnR5sScC1
3jBEBgNVHREEPTA7ghVzbmkuY2xvdWRmbGFyZXNzbC5jb22CESoucmFkaW9wYWVk
aWEub3Jngg9yYWRpb3BhZWRpYS5vcmcwDgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQW
MBQGCCsGAQUFBwMBBggrBgEFBQcDAjB7BgNVHR8EdDByMDegNaAzhjFodHRwOi8v
Y3JsMy5kaWdpY2VydC5jb20vQ2xvdWRmbGFyZUluY0VDQ0NBLTMuY3JsMDegNaAz
hjFodHRwOi8vY3JsNC5kaWdpY2VydC5jb20vQ2xvdWRmbGFyZUluY0VDQ0NBLTMu
Y3JsMD4GA1UdIAQ3MDUwMwYGZ4EMAQICMCkwJwYIKwYBBQUHAgEWG2h0dHA6Ly93
d3cuZGlnaWNlcnQuY29tL0NQUzB2BggrBgEFBQcBAQRqMGgwJAYIKwYBBQUHMAGG
GGh0dHA6Ly9vY3NwLmRpZ2ljZXJ0LmNvbTBABggrBgEFBQcwAoY0aHR0cDovL2Nh
Y2VydHMuZGlnaWNlcnQuY29tL0Nsb3VkZmxhcmVJbmNFQ0NDQS0zLmNydDAMBgNV
HRMBAf8EAjAAMIIBfAYKKwYBBAHWeQIEAgSCAWwEggFoAWYAdgBGpVXrdfqRIDC1
oolp9PN9ESxBdL79SbiFq/L8cP5tRwAAAXqv4nBNAAAEAwBHMEUCIDeIF02r7EMA
OE1LH5PhS1ttNN4w3DbxOThH1miCXz3NAiEAogaiG5eTgoDUgs28qhCxUqDq0AJJ
L9q7f2YUbEJQFP4AdQBRo7D1/QF5nFZtuDd4jwykeswbJ8v3nohCmg3+1IsF5QAA
AXqv4nA6AAAEAwBGMEQCICh9ZzfAmgWg3Nqj265ukh8RGtb/7y9fgOeQSqBO5For
AiBpAUcLp1uV7TgFqHQxdmhrZBI9aBfzraQq/wxCsOgYEQB1AEHIyrHfIkZKEMah
OglCh15OMYsbA+vrS8do8JBilgb2AAABeq/ib+oAAAQDAEYwRAIgaBwC/NlXMU22
ljza1hK3+5qReb1GNeqiirk9t0TLUdUCIBo6fV/YLCGOFZBpUZ1HjsVvlNqgqtuZ
o1NhcJzSj601MAoGCCqGSM49BAMCA0gAMEUCIQCElb21g6QRXZFn1yLMnB2kpgQl
aLlOe2YUbDtCGQDBggIgJO7PVSz2rXalPTpHdDoICNIdRzAFgkG3D5+3sn1O6sU=
-----END CERTIFICATE-----
subject=C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = sni.cloudflaressl.com

issuer=C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3

No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits

SSL handshake has read 2633 bytes and written 399 bytes
Verification: OK

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

read:errno=54

joeschmoe · Apr 10, 2022, 7:03 AM

showing curl response difference as well:

OK Response
2.5.2-RELEASE][whoami@oo.ooo.oooo]/: curl https://radiopaedia.org/ -vkI

Trying 172.67.72.247:443...
Connected to radiopaedia.org (172.67.72.247) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /usr/local/share/certs/ca-root-nss.crt
CApath: none
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
TLSv1.3 (IN), TLS handshake, Certificate (11):
TLSv1.3 (IN), TLS handshake, CERT verify (15):
TLSv1.3 (IN), TLS handshake, Finished (20):
TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.3 (OUT), TLS handshake, Finished (20):
SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
ALPN, server accepted to use h2
Server certificate:
subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
start date: Jul 16 00:00:00 2021 GMT
expire date: Jul 15 23:59:59 2022 GMT
issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
SSL certificate verify ok.
Using HTTP2, server supports multi-use
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
Using Stream ID: 1 (easy handle 0x801435800)

HEAD / HTTP/2
Host: radiopaedia.org
user-agent: curl/7.76.1
accept: /

TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
old SSL session ID is stale, removing
Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200
HTTP/2 200
< date: Fri, 08 Apr 2022 06:03:32 GMT
date: Fri, 08 Apr 2022 06:03:32 GMT
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-download-options: noopen
x-download-options: noopen
< x-permitted-cross-domain-policies: none
x-permitted-cross-domain-policies: none
< referrer-policy: strict-origin-when-cross-origin
referrer-policy: strict-origin-when-cross-origin
< vary: X-UA-Device
vary: X-UA-Device
< cache-control: max-age=0, private, must-revalidate
cache-control: max-age=0, private, must-revalidate
< set-cookie: _radiopaedia_session=VFo1RWVtOTB5OVRtVjg1RlRNSmVlYmRHMldwdFRNWGlNNDV1eEE3V0NXV08vM2pEdW5hUUtTeUw3WmFRVXMxaUpENXllTGp3SGw5cm5QbWpFNlBmWElqcXZSK29rZmhaR0lJRlFFQVkyUGplelovOFA0SCtJQ3B2OTNRWmI3OU1vckRzMVpDclV5Q29YMytuNitRUWlhQXVhb3FxUEpkdUc1NFZPTVJ2MkRZUVRLZVM1RFVqR3lkdWk5SGRLSmxTZTRuV1VUaXQ0TFpsbEQyelZzMmU5U2trZVhQbmJ1NHRPdVpEdG9EQm0yMG5UcU8wRytucW5GM0NCaGRsQWNyRnNRMmNjN1hLR3M2eUlQWXV6b3ZYSE01V1dMdWU2d0IvaDI5YmFZZ3ZwNmVSbGVkVy9adWhiUVNsNFNOSFVVQVNNR2ZRQmVjelFWenhaeFQ1dGhiMlJnPT0tLWpBbDNOcjg4MHZncmRCSkVrZUd0R1E9PQ%3D%3D--cae10cc899a0a04c2d9a48fd366ef013014f182d; path=/; secure; HttpOnly; SameSite=Lax
set-cookie: _radiopaedia_session=VFo1RWVtOTB5OVRtVjg1RlRNSmVlYmRHMldwdFRNWGlNNDV1eEE3V0NXV08vM2pEdW5hUUtTeUw3WmFRVXMxaUpENXllTGp3SGw5cm5QbWpFNlBmWElqcXZSK29rZmhaR0lJRlFFQVkyUGplelovOFA0SCtJQ3B2OTNRWmI3OU1vckRzMVpDclV5Q29YMytuNitRUWlhQXVhb3FxUEpkdUc1NFZPTVJ2MkRZUVRLZVM1RFVqR3lkdWk5SGRLSmxTZTRuV1VUaXQ0TFpsbEQyelZzMmU5U2trZVhQbmJ1NHRPdVpEdG9EQm0yMG5UcU8wRytucW5GM0NCaGRsQWNyRnNRMmNjN1hLR3M2eUlQWXV6b3ZYSE01V1dMdWU2d0IvaDI5YmFZZ3ZwNmVSbGVkVy9adWhiUVNsNFNOSFVVQVNNR2ZRQmVjelFWenhaeFQ1dGhiMlJnPT0tLWpBbDNOcjg4MHZncmRCSkVrZUd0R1E9PQ%3D%3D--cae10cc899a0a04c2d9a48fd366ef013014f182d; path=/; secure; HttpOnly; SameSite=Lax
< x-request-id: 7f78f573-23ad-4123-8caf-11f2af75e4e7
x-request-id: 7f78f573-23ad-4123-8caf-11f2af75e4e7
< x-runtime: 0.379540
x-runtime: 0.379540
< x-server-hostname-ssl: ip-10-10-4-150
x-server-hostname-ssl: ip-10-10-4-150
< x-server-hostname: ip-10-10-4-150
x-server-hostname: ip-10-10-4-150
< cf-cache-status: DYNAMIC
cf-cache-status: DYNAMIC
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v3?s=VTmmQB1piq9gxsnQTyAbxiuKaGbe%2Ff4WawDIO70CsHntMwil5ySdoDiAHfE0zV3yetJz5wn%2BPrP2syjgXhX3PBQ%2Bi1OF7Mf2iYPEGVJ%2FkfdmGXlZGPUIAermD7xwxUbtGg%3D%3D"}],"group":"cf-nel","max_age":604800}
report-to: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v3?s=VTmmQB1piq9gxsnQTyAbxiuKaGbe%2Ff4WawDIO70CsHntMwil5ySdoDiAHfE0zV3yetJz5wn%2BPrP2syjgXhX3PBQ%2Bi1OF7Mf2iYPEGVJ%2FkfdmGXlZGPUIAermD7xwxUbtGg%3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
server: cloudflare
< cf-ray: 6f88cce05cf33d60-HKG
cf-ray: 6f88cce05cf33d60-HKG
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

<

Connection #0 to host radiopaedia.org left intact
NOT OK Response ===========
[2.5.2-RELEASE][whoami@oo.ooo.oooo]/usr/local/etc/squid: curl https://radiopaedia.org/ -vkI
Trying 104.26.9.61:443...
Connected to radiopaedia.org (104.26.9.61) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /usr/local/share/certs/ca-root-nss.crt
CApath: none
TLSv1.3 (OUT), TLS handshake, Client hello (1):
OpenSSL SSL_connect: Connection reset by peer in connection to radiopaedia.org:443
Closing connection 0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to radiopaedia.org:443

stephenw10 · Apr 11, 2022, 2:23 PM

What states do you see when you run those tests? I'm not sure why Squid would be involved at all.

The curl responses make it look like the remote side is just doing something different. Like maybe it has load-balanced servers and they are not identical. Do you see this on other sites?

Steve

joeschmoe · Apr 29, 2022, 2:23 AM

@stephenw10 same behaviuor on some sites as well.

[2.5.2-RELEASE][whoami@oo.ooo.oooo]/etc/ssl/certs: curl https://globemybusiness.zendesk.com -kvl
*   Trying 104.16.51.111:443...
* Connected to globemybusiness.zendesk.com (104.16.51.111) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /usr/local/share/certs/ca-root-nss.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to globemybusiness.zendesk.com:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to globemybusiness.zendesk.com:443
[2.5.2-RELEASE][whoami@oo.ooo.oooo]/etc/ssl/certs:

stephenw10 · Apr 29, 2022, 1:10 PM

Looks like some things may have been missing from that post.

But same question; what states do you see when you run these tests?

@joeschmoe said in TLS pr_end_of_file_error (non proxy related):

pr_end_of_file_error message in squid appears

Where are you actually seeing that? In the Squid logs?

Connections from the pfSense CLI should not go via Squid. If they are being redirected somehow that would show in the states created.

Do you still see this issue if you disable Squid?

Steve

TLS pr_end_of_file_error (non proxy related)

With ssl error(browser pr_end_of_file_error response): [[2.5.2-RELEASE][whoami@oo.ooo.oooo]/: openssl s_client -connect radiopaedia.org:443 CONNECTED(00000003) write:errno=54

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 319 bytes Verification: OK

No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: ECDSA Server Temp Key: X25519, 253 bits

SSL handshake has read 2633 bytes and written 399 bytes Verification: OK

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 Server public key is 256 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated Early data was not sent Verify return code: 0 (ok)

With ssl error(browser pr_end_of_file_error response):
[[2.5.2-RELEASE][whoami@oo.ooo.oooo]/: openssl s_client -connect radiopaedia.org:443
CONNECTED(00000003)
write:errno=54

SSL handshake has read 0 bytes and written 319 bytes
Verification: OK

No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits

SSL handshake has read 2633 bytes and written 399 bytes
Verification: OK

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)