HAProxy backend only resolves IPv6 only

[lindhe]

I have a backend configured in HAProxy that points to a Cloudflare worker. But the backend is detected as DOWN. When I inspect why it says "Layer4 connection problem: General socket error (No route to host)". Turns out it has resolved the domain name to 2606:4700:3031::6815:3f67. That is not wrong per se, that address do belong to my worker. The issue lies in the fact that I do not have a public IPv6 address yet, so routing to that address will be tricky.

I've rarely or never ran into this kind of problem before. I assume that all my other clients and programs disregard addresses that does not work and tries one of the other ones until it finds an IPv4 address (or something else that makes them work always). But HAProxy seems to be hung up on this IPv6 address.

How do I fix this?

Info

The Cloudflare worker's address i matrix-well-known.andreas-lindh.workers.dev and currently resolves like this:

$ nslookup matrix-well-known.andreas-lindh.workers.dev 10.0.10.1
Server:		10.0.10.1
Address:	10.0.10.1#53

Non-authoritative answer:
Name:	matrix-well-known.andreas-lindh.workers.dev
Address: 172.67.170.141
Name:	matrix-well-known.andreas-lindh.workers.dev
Address: 104.21.63.103
Name:	matrix-well-known.andreas-lindh.workers.dev
Address: 2606:4700:3031::6815:3f67
Name:	matrix-well-known.andreas-lindh.workers.dev
Address: 2606:4700:3034::ac43:aa8d

This is my backend config:

These are my DNS settings for HAProxy: