Haproxy - Mobile Networks in UK completely broken
-
You saw retransmissions from the remote test client coming into your WAN?
That implies haproxy did not reply to them. Or the replies never arrived.
-
@stephenw10 I've had a few friends test on both Celluar and DSL. The capture shows a lot of Re-transmissions going out and In I believe. It is worst on Celluar than DSL though but i guess that makes sense.
217.45 is one of my Static IPs, 82.13 is a friends Virgin Internet connection, You can see there is a couple of Re-Transmissions, now I dunno if this is causing a problem or not.
Regards
-
Ok the duplicated ACKs there imply the remote host is not seeing the packets pfSense is sending, and re-sending. For some reason.
I assume that remote host is seeing problems connecting the site hosted at that static IP.
-
@stephenw10 The client seems to connect to Nextcloud okay, and download files, one thing i have noticed is there are poor speeds out of the WAN too, I have a 18meg up speed but the Client is only seeing 1mbp down when downloading a file.
What would you suggest to do next? because i am complexed by this.
Regards
-
I would test at the client end to confirm that when it's sending ACKs like that it isn't seeing the packets we are sending.
Then I would probably try setting a much lower MTU/MSS on something and see if that makes any difference.
-
@stephenw10 Yeah MTU is set to 1492 and MSS is to 1407 these are what BT Business recommends. I mean you think adjusting will help?
Regards
-
I have seen networks failing to pass packets that large and also failing to send the required ICMP-packet-too-large replies. So yes I would try 1300 to be sure.
You can see the packets pfSense is sending at 1411 but I believe that's the frame size. That's neither 1492 or 1407 though.
But pcap at the remote side first to be sure they are not reaching it.
-
@stephenw10 1411 is odd frame size for a 1500 mtu.. Wireshark would normally show those as 1514.. If it was a full frame
So 1411 is either not a full frame or the mtu is not 1500.
-
Exactly. And it's not anything I'd expected from setting MSS to 1407 either. Implying something in the route the is running a low MTU and sending back ICMP notices. But maybe not low enough.
But also 1407 seems like a weird value for that. I'd double check it.
-
@stephenw10 Changed to 1300, but still the same problem. Not sure where to go from here now. I did check on Client too each side shows both in Wireshark.
Regards
-
So it does show the retransmissions from pfSense arriving at the client?
-
@stephenw10 Good evening, I’ve given up with it for now.
I believe this problem is something to do with Haproxy. All services that is routed through Haproxy keeps dropping out even internally.
So I am not sure whether if it’s the latest version of Haproxy or 2.7.2.
Regards
-
Good afternoon,
Would like to update you, the problem seems to be resolved. I believe the problem was due to Public DNS Servers on Mobile Networks.
Regards
-
Ah, interesting. The first rule of troublshooting: it's always DNS!
Though I wouldn't have guessed that here fro what you were seeing.Glad it was resolved.
-
@stephenw10 I dropped the MTU, MSS on WAN1, but the DNS lookup seems to be fixed with Vodaphone, Three. I do plan on moving Name Servers to my own VMs so i won't have to rely on someone else.
All errors in Wireshark seems to be resolved too so those errors could be related to the ISP. I will keep an eye on it though
Regards
-
@stephenw10 Hi folks, thought i'd chime in again. There is a problem somewhere and i am beginning to wonder if its more down to iOS devices. When connected to 4G and connecting to services which runs through Haproxy over WAN is where the problem is but it doesn't happen all of the time,
Services that are Public facing and accessing through 4G connecting via WAN is the problem here, it works perfectly fine when connected to WIFI internally or WIFI at work, families internet etc.
As dumb as it sounds i am beginning to wonder if this is something to do with Apples Webkit and the way the devices handles Encryption.
Regards
-
Hmm, bizarre. Though it's hard to imagine how that would vary when the phone is connected to WiFi.
Have you ever seen the issue on anything other than an iOS device connected over 4G?
-
@stephenw10 not that I am a aware of, I have family and friends that uses Element / Matrix-Synapse and it’s the iPhone users that are reporting this issue. SSL Errors, I’ve tested on my iPhone SE 2020 and also suffers the same problem, tested on Android and no issues at all. It’s a strange one.
Regards
-
Just want to update you folks. After a few hours of scouring the Internet. I have found that iOS 17.4 is broken for certain devices which seems to affect only certain iOS devices.
I have been messing around here what I’ve found so far. Using EDGE instead of 4G no problems, no SSL errors or time outs but of course very slow, using 4G is where the problem arises on the device itself.
Tethering using 4G to either a Desktop/Laptop or Android phone no problems at all. Connecting to WI-FI internally or externally no problems at all.
Remove SIM from the iPhone installed it into an Android and did all the same tests no errors at all. APIs and Browsers work perfectly fine.
This is very interesting however iOS 17.5 has been released so I will update and do all the tests. But it seems a lot of APIs and websites are broken with Apples Native Webkit.
Regards
-
@stephenw10 Good evening,
Just want to update you on this situation, decided to move DNS Providers from DigitalOcean to our own Name Servers we built from scratch, this problem no longer happens very strange. Since moving we no longer get issues with DNS Probe Errors neither inbound emails being very delayed.
Not sure if this was a issue due to Geolocation or maybe a DNS Resolver problem but running ns1 on Premise and ns2 off-premise all of these errors are resolved on both iOS devices and Android devices.
I am aware that DigitalOcean uses Cloudflare as there providers so possibly a issue with Cloudflare who knows/
Regards
Jack
