HAproxy with tcp - problems on 2.8.1

tsmalmbe

I have a serious issue with haproxy + tcp to an ldap server.

I have two pfsenses and two ldap-servers - both pfsenses have a haproxy config that uses the same two ldap-servers. Scenario is as follows, with SSL offloading an re-encrypting.

pfsense 1 - haproxy setup that can reach ipa-03 and ipa-04 - healthchecks green
pfsense 1 - ldapsearch from pfsense verified to reach pfsense 1 vip, ipa-03 636 and ipa-04 636 but FAILS on pfsense 2 vip
pfsense 2 - haproxy setup that can reach ipa-03 and ipa-04 - healthchecks green
pfsense 2 - ldapsearch from pfsense verified to reach pfsense 1 vip, ipa-03 and ipa-04 but fails on pfsense 2 vip

Both pfsense are 2.8.1 and latest haproxy. They are very similar configurationwise - difference being one is running in Vultr, the other in Proxmox. Ldap server firewalld's are not an issue as the connectivity above tells us (I have of course also tried turning them off). Both pfsenses are using the same frontend certificates (entrust in this case), and the backend servers are using internal certicates (but as I have turned off verification, this is not relevant).

Also, I am running http backends on pfsense 2 with no problems whatsoever with SSL offloading.

This is the config of pfsense 2 - the problematic one. Everything irrelevant has bit by bit essentially been stripped trying to eliminate the issue.

frontend ldap.mintsecurity.fi
bind 10.97.10.1:636 name 10.97.10.1:636 ssl crt-list /var/etc/haproxy/ldap.mintsecurity.fi.crt_list
mode tcp
log global
timeout client 30000
default_backend ldap.mintsecurity.fi_ipvANY

backend ldap.mintsecurity.fi_ipvANY
mode tcp
id 106
log global
timeout connect 30000
timeout server 30000
retries 3
load-server-state-from-file global
server ipa-03.ipa.mintsecurity.fi 10.97.0.11:636 id 107 ssl verify none

The ldapsearch command from pfsenses I use to test connectivity is this:
env LDAPTLS_REQCERT=never ldapsearch -H ldaps://10.97.10.1:636 -x -s base -b "" "(objectClass=*)"

for pfsense 2 vip, it times out silently after 30000ms.

While testing the above, on the ldap server I see this
20:07:21.117199 IP 10.97.0.3.61846 > 10.97.0.11.ldaps: Flags [S], seq 427452033, win 65228, options [mss 1410,nop,wscale 7,sackOK,TS val 796984471 ecr 0], length 0
20:07:21.117347 IP 10.97.0.11.ldaps > 10.97.0.3.61846: Flags [S.], seq 2244739247, ack 427452034, win 64860, options [mss 1410,nop,nop,sackOK,nop,wscale 7], length 0
20:07:21.117710 IP 10.97.0.3.61846 > 10.97.0.11.ldaps: Flags [.], ack 1, win 518, length 0
20:07:21.117921 IP 10.97.0.3.61846 > 10.97.0.11.ldaps: Flags [P.], seq 1:15, ack 1, win 518, length 14
20:07:21.117941 IP 10.97.0.11.ldaps > 10.97.0.3.61846: Flags [.], ack 15, win 507, length 0
20:07:51.128082 IP 10.97.0.3.61846 > 10.97.0.11.ldaps: Flags [F.], seq 15, ack 1, win 518, length 0
20:07:51.128082 IP 10.97.0.3.61846 > 10.97.0.11.ldaps: Flags [R.], seq 16, ack 1, win 0, length 0

where

10.97.0.3 is the pfsense 2 interface ip and
10.97.0.11 is ipa-03 (inet 10.97.0.11/16 brd 10.97.255.255 scope global noprefixroute ens9)
also 10.97.10.1 is the vip - the interface has a /16 subnet defined

Interestingly if I choose to NOT to terminate TLS on the frontend, the backend works perfectly, echoing he internal certificate to the ldapsearch (this is verified with ssl_client).

Everything is pingable.

Also interestingly, this setup USED to work - at least on 2.7.0 but also on 2.8.0. Now with 2.8.1 i run into these problems.

tsmalmbe

Adding some more info on this one.

[2.8.1-RELEASE][admin@cloudwall.mintsecurity.fi]/root: tcpdump -i vtnet3 -n -s0 -X host 10.97.0.11 and port 636 -c 10
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vtnet3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:59:06.535803 IP 10.97.0.3.58640 > 10.97.0.11.636: Flags [S], seq 1242701899, win 65228, options [mss 1410,nop,wscale 7,sackOK,TS val 1442932259 ecr 0], length 0
0x0000: 4500 003c 0000 0000 4006 65ed 0a61 0003 E..<....@.e..a..
0x0010: 0a61 000b e510 027c 4a12 204b 0000 0000 .a.....|J..K....
0x0020: a002 fecc 2687 0000 0204 0582 0103 0307 ....&...........
0x0030: 0402 080a 5601 6623 0000 0000 ....V.f#....
21:59:06.544238 IP 10.97.0.11.636 > 10.97.0.3.58640: Flags [S.], seq 3880004700, ack 1242701900, win 64860, options [mss 1410,nop,nop,sackOK,nop,wscale 7], length 0
0x0000: 4500 0034 0000 4000 4006 25f5 0a61 000b E..4..@.@.%..a..
0x0010: 0a61 0003 027c e510 e744 2c5c 4a12 204c .a...|...D,\J..L
0x0020: 8012 fd5c f77a 0000 0204 0582 0101 0402 ....z..........
0x0030: 0103 0307 ....
21:59:06.544257 IP 10.97.0.3.58640 > 10.97.0.11.636: Flags [.], ack 1, win 518, length 0
0x0000: 4500 0028 0000 0000 4006 6601 0a61 0003 E..(....@.f..a..
0x0010: 0a61 000b e510 027c 4a12 204c e744 2c5d .a.....|J..L.D,]
0x0020: 5010 0206 3372 0000 P...3r..
21:59:06.544280 IP 10.97.0.3.58640 > 10.97.0.11.636: Flags [P.], seq 1:15, ack 1, win 518, length 14
0x0000: 4500 0036 0000 0000 4006 65f3 0a61 0003 E..6....@.e..a..
0x0010: 0a61 000b e510 027c 4a12 204c e744 2c5d .a.....|J..L.D,]
0x0020: 5018 0206 73e9 0000 300c 0201 0160 0702 P...s...0....`..
0x0030: 0103 0400 8000 ......
21:59:06.545520 IP 10.97.0.11.636 > 10.97.0.3.58640: Flags [.], ack 15, win 507, length 0
0x0000: 4500 0028 4efb 4000 4006 d705 0a61 000b E..(N.@.@....a..
0x0010: 0a61 0003 027c e510 e744 2c5d 4a12 205a .a...|...D,]J..Z
0x0020: 5010 01fb 336f 0000 P...3o..
21:59:36.539028 IP 10.97.0.3.58640 > 10.97.0.11.636: Flags [F.], seq 15, ack 1, win 518, length 0
0x0000: 4500 0028 0000 0000 4006 6601 0a61 0003 E..(....@.f..a..
0x0010: 0a61 000b e510 027c 4a12 205a e744 2c5d .a.....|J..Z.D,]
0x0020: 5011 0206 3363 0000 P...3c..
21:59:36.539960 IP 10.97.0.11.636 > 10.97.0.3.58640: Flags [F.], seq 1, ack 16, win 507, length 0
0x0000: 4500 0028 4efc 4000 4006 d704 0a61 000b E..(N.@.@....a..
0x0010: 0a61 0003 027c e510 e744 2c5d 4a12 205b .a...|...D,]J..[
0x0020: 5011 01fb 336d 0000 P...3m..
21:59:36.539996 IP 10.97.0.3.58640 > 10.97.0.11.636: Flags [.], ack 2, win 518, length 0
0x0000: 4500 0028 0000 0000 4006 6601 0a61 0003 E..(....@.f..a..
0x0010: 0a61 000b e510 027c 4a12 205b e744 2c5e .a.....|J..[.D,^
0x0020: 5010 0206 3362 0000 P...3b..
22:03:05.203261 IP 10.97.0.3.55264 > 10.97.0.11.636: Flags [S], seq 1798037791, win 65228, options [mss 1410,nop,wscale 7,sackOK,TS val 10418126 ecr 0], length 0
0x0000: 4500 003c 0000 0000 4006 65ed 0a61 0003 E..<....@.e..a..
0x0010: 0a61 000b d7e0 027c 6b2b e11f 0000 0000 .a.....|k+......
0x0020: a002 fecc 1581 0000 0204 0582 0103 0307 ................
0x0030: 0402 080a 009e f7ce 0000 0000 ............
22:03:05.204341 IP 10.97.0.11.636 > 10.97.0.3.55264: Flags [S.], seq 2429908886, ack 1798037792, win 64860, options [mss 1410,nop,nop,sackOK,nop,wscale 7], length 0
0x0000: 4500 0034 0000 4000 4006 25f5 0a61 000b E..4..@.@.%..a..
0x0010: 0a61 0003 027c d7e0 90d5 7796 6b2b e120 .a...|....w.k+..
0x0020: 8012 fd5c 2df2 0000 0204 0582 0101 0402 ...-...........
0x0030: 0103 0307 ....

Now i analyzed this with ChatGPT:
Look at packet 4:
21:59:06.544280 IP 10.97.0.3.58640 > 10.97.0.11.636: Flags [P.], length 14
0000: 30 0c 02 01 01 60 07 02 01 03 04 00 80 00
That is not a TLS ClientHello.
That is a cleartext LDAP BindRequest (BER-encoded, 14 bytes).

I don't really know where to go next with this.

tsmalmbe

I have tested this with a simple unencrpted echoserver (ncat) and using telnet.

HAProxy simply DOES NOT work with tcp.

Where should I be loking? I have tcpdumped the hell ouf of everything, but when there is no life, there is nothing to dump.