Haproxy - Mobile Networks in UK completely broken

[johnpoz]

@VioletDragon I use the Hurricane electric network tools app on my iphone - lots of stuff it can do, one of which is query any dns you want.

https://networktools.he.net/

This suite of network tools implements most of the network diagnostics that you need as a Network Engineer or System Administrator. This is the first version of our mobile app, please help us improve it by giving us feedback.

Interface Information: Get information regarding your device's current network state.
ARP / NDP: Lookup local devices found on your network using either ARP (IPv4) or NDP (IPv6).
DNS lookup: Search for a server's DNS zone records such as SOA, NS, A, AAAA, MX, TXT, and rDNS.
IP Calculator: Calculate the Netmask, Wildcard, Network, Host range, and Broadcast for a given subnet.
iperf (v2 and v3): Run TCP and UDP performance tests.
One Time Password: Manage time (TOTP) and counter (HOTP) based OTP passwords. Uses iCloud Keychain sync (if enabled) on iOS devices.
Ping: Send ICMP packets to a single IPv4/v6 address.
Ping Sweep: Send ICMP packets to an entire subnet range.
Traceroute: Find the route packets take to reach your destination.
Progressive Traceroute: Get detailed statistics for a given route.
Port Scan: Scan a list of custom or pre-defined TCP ports on any server.
SSL/TLS Information: Check used protocol, ciphers and certificate information.
Whois: Get information regarding domain registration.
NO ADS!

[VioletDragon]

@stephenw10 when testing over Hotspot, websites loads just a blank white screen nothing else.

Interestingly enough, websites randomly stops working too. Network connection was lost in Safari. Now I don’t know if this is a problem with the iPhone or not. I will test on android to see. It is very strange.

I switched from Three network because the Network was so congested you couldn’t load any websites.

I am beginning to wonder if this is more down to network congestion ?

Regards

[VioletDragon]

@johnpoz I have been looking for a app like this for awhile. All resolves in the app under DNS lookup.

Regards

[stephenw10]

At some point you were seeing DNS_PROBE_FINISHED_NXDOMAIN though?

[VioletDragon]

@stephenw10 Yeah on odd occasions though. It is very strange.

[VioletDragon]

@stephenw10 Just out of interest. Could it be the way the block of 8 IPs are being routed? I configured them as VIPs. Then on Haproxy it’s configured on just WAN?

Regards

[stephenw10]

That should be fine. If it was broken it would be broken for everything.

[VioletDragon]

@stephenw10 yeah thought so. Just tested it on an android phone. It all seems to be working apart from one of the services I host but no one else is having the issue.

Problem I have right now, is that no that I know can reproduce the problem there end. So I am not sure where to go from here now.

[stephenw10]

I would send some traffic and see if it arrives at your WAN. That will tell you for sure if the mobile network is filtering it.

[VioletDragon]

@stephenw10 What would you use for that ? Wire shark ?

Regards

[stephenw10]

Run a pcap in pfSense initially just to see if anything arrives. Filter it by the mobile IP you're testing from. That's easier if you get a real IP. I know 3UK still give you a real address.

[VioletDragon]

@stephenw10 Tested with pcap and used Wireshark to read the packet capture, I am seeing a lot of re-transmissions on Public facing WAN, I tested internally to see if i get the re-transmissions and I can say it's only on the Public facing WAN side.

Regards

[stephenw10]

You saw retransmissions from the remote test client coming into your WAN?

That implies haproxy did not reply to them. Or the replies never arrived.

[VioletDragon]

@stephenw10 I've had a few friends test on both Celluar and DSL. The capture shows a lot of Re-transmissions going out and In I believe. It is worst on Celluar than DSL though but i guess that makes sense.

217.45 is one of my Static IPs, 82.13 is a friends Virgin Internet connection, You can see there is a couple of Re-Transmissions, now I dunno if this is causing a problem or not.

Regards

[stephenw10]

Ok the duplicated ACKs there imply the remote host is not seeing the packets pfSense is sending, and re-sending. For some reason.

I assume that remote host is seeing problems connecting the site hosted at that static IP.

[VioletDragon]

@stephenw10 The client seems to connect to Nextcloud okay, and download files, one thing i have noticed is there are poor speeds out of the WAN too, I have a 18meg up speed but the Client is only seeing 1mbp down when downloading a file.

What would you suggest to do next? because i am complexed by this.

Regards

[stephenw10]

I would test at the client end to confirm that when it's sending ACKs like that it isn't seeing the packets we are sending.

Then I would probably try setting a much lower MTU/MSS on something and see if that makes any difference.

[VioletDragon]

@stephenw10 Yeah MTU is set to 1492 and MSS is to 1407 these are what BT Business recommends. I mean you think adjusting will help?

Regards

[stephenw10]

I have seen networks failing to pass packets that large and also failing to send the required ICMP-packet-too-large replies. So yes I would try 1300 to be sure.

You can see the packets pfSense is sending at 1411 but I believe that's the frame size. That's neither 1492 or 1407 though.

But pcap at the remote side first to be sure they are not reaching it.

[johnpoz]

@stephenw10 1411 is odd frame size for a 1500 mtu.. Wireshark would normally show those as 1514.. If it was a full frame

So 1411 is either not a full frame or the mtu is not 1500.