Weird DNS issue with Android Wifi Calling

britcowboy · Mar 26, 2016, 4:43 PM

Weird one, it's probably not strictly a pfsense problem, but maybe you guys can point me in the right direction.

There is some oddity with android wifi calling and my network - I realised this by connecting to a quick network I setup that bypasses pfsense etc and it worked fine. After investigating it seems there's a weird conflict between Android Wifi Calling and my Microsoft DNS (my GF's iPhone works fine on Wifi Calling on my network)

Before I had my phone connecting via DHCP to my pfsense box which then gave it DNS servers on my network, those DNS Servers end up forwarding to Google's DNS servers. For some reason, despite my whole network working with this, and my android phone generally working with this, it seems that this destroys wifi calling. If I connect to the wifi network with a static IP and use Google's DNS servers directly, it works.

Has anyone got any ideas? This is a very strange one.

Durandaul · Mar 28, 2016, 4:39 PM

That's very odd but it sounds like you'll learn something new and interesting through resolving this! :)

From some slight perusing, the only thing I was able to find that seem relevant was this T-Mobile issued statement on WiFi-Calling issues:

https://support.t-mobile.com/docs/DOC-4950

The REG99: Unable to Connect error may appear when Wi-Fi Calling is active and connected to certain routers or devices and is usually caused by the following:

Device is not sending the request out due to a software error.
It may be an issue with the DNS response from the DNS server when the device tries to resolve the domain name configured in the IMS client.
Router is blocking the register request from the device before it is sent to the ISP.
This could be a firewall or other router related issue. Sometimes, this will happen with public networks because their firewall is configured to block this type of traffic.
ISP is not forwarding the request to the T-Mobile network. The request may not reach the T-Mobile network due to an internal issue. This indicates the device is not able to connect to Wi-Fi Calling.
Device can make calls over the network, but not via Wi-Fi Calling

From the PFSense forums, this looked to be a similar topic that was resolved by other users: https://forum.pfsense.org/index.php?topic=93506.msg566074#msg566074
Have you had a chance to take a look there?

I was hoping to find more information on WiFi calling at a mid-technical tier that had information on both the current usage and expected results but a lot of it was Android programmer Q&A

Are you getting an error message or will the calls just fail with no other information? Also what is your carrier?

britcowboy · Mar 28, 2016, 11:33 PM

This is the problem, there are no error logs or messages, it either works or it doesn't.

The only change that seems to matter is which DNS it uses, if I point it to my windows DNS servers, it won't work, if I point it to Googles it works fine. This is why I'm thinking it's a DNS issue.

I've done a lot of reading over the past few days, and my understanding is that the EE wifi calling does a ipsec vpn tunnel using ipv6. My home network is completely ipv4, and my ISP is ipv4 only. The apn is:
<apn carrier="EE IMS" <br="">mcc="234"
mnc="30"
apn="ims"
type="ims"
protocol="IPV6"
roaming_protocol="IPV6"
/>

This is where I get fuzzy, I'm not sure how this ipv6 stuff works on an ipv4 network, and my DNS server doesn't have am ipv6 ip. Could this be the source of my troubles?

My carrier is EE (Everything Everywhere) in the UK</apn>

Derelict · Mar 28, 2016, 11:51 PM

I would packet capture and look at the DNS queries and answers.

britcowboy · Mar 29, 2016, 12:19 AM

Hi,

I've just done this, and I think I've found the culprit.

For some reason my DNS isn't responding to this DNS query, you can see it trying my backup DNS, whereas Google responds first time. Anyone have any idea why?! All other DNS queries look fine.

Derelict · Mar 29, 2016, 12:30 AM

$ dig epdg.epc.mnc030.mcc234.pub.3gppnetwork.org a

; <<>> DiG 9.8.3-P1 <<>> epdg.epc.mnc030.mcc234.pub.3gppnetwork.org a
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64656
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 6, ADDITIONAL: 0

;; QUESTION SECTION:
;epdg.epc.mnc030.mcc234.pub.3gppnetwork.org. IN A

;; ANSWER SECTION:
epdg.epc.mnc030.mcc234.pub.3gppnetwork.org. 300 IN A 109.249.188.56
epdg.epc.mnc030.mcc234.pub.3gppnetwork.org. 300 IN A 109.249.180.0
epdg.epc.mnc030.mcc234.pub.3gppnetwork.org. 300 IN A 109.249.186.72

;; AUTHORITY SECTION:
mnc030.mcc234.pub.3gppnetwork.org. 86400 IN NS j4.nstld.com.
mnc030.mcc234.pub.3gppnetwork.org. 86400 IN NS k4.nstld.com.
mnc030.mcc234.pub.3gppnetwork.org. 86400 IN NS g4.nstld.com.
mnc030.mcc234.pub.3gppnetwork.org. 86400 IN NS l4.nstld.com.
mnc030.mcc234.pub.3gppnetwork.org. 86400 IN NS f4.nstld.com.
mnc030.mcc234.pub.3gppnetwork.org. 86400 IN NS a4.nstld.com.

;; Query time: 282 msec
;; SERVER: 192.168.223.1#53(192.168.223.1)
;; WHEN: Mon Mar 28 17:29:26 2016
;; MSG SIZE rcvd: 219

Resolves here.

What do you get? You'll want to be sure you're pointing drill at the resolver these clients are using if doing it from pfSense.

eg. $ drill @127.0.0.1 epdg.epc.mnc030.mcc234.pub.3gppnetwork.org a

britcowboy · Mar 29, 2016, 12:48 AM

Thats the weird thing. If I do a NSLOOKUP on one of my machines it resolves fine - it's simply the request from the android device that isn't being responded to. I setup a log on my DNS Server to show it's received and sent packets, and the DNS service doesn't seem to be getting these DNS queries. I don't understand, there's no firewall on the DNS server :/

britcowboy · Mar 29, 2016, 1:02 AM

I found a tool that did a NSLOOKUP style thing on Android, and that got a response from my DNS server so I analysed the differences in packets. I've found it.

The MAC address seems to be wrong on the DNS request.

The ones that work have this:

Dst: Universa_08:67:34 (WINDOWS MAC ADDRESS)

The ones that get no response have this:

Dst: Microsof_01:f4:01 (00:15:5d:01:f4:01) - which appears to be Google DNS's mac address.

Why is the MAC address wrong?!

britcowboy · Mar 29, 2016, 1:17 AM

Sorry to post (again), but I changed my DNS to OpenDNS to test, and it still sends it the wrong MAC address, but it responds anyway. Seems to be an android bug, any way to setup my windows server to ignore checking the MAC address and accept the packet?

cmb · Mar 29, 2016, 2:12 AM

MAC addresses are only locally relevant, only visible on the same broadcast domain. That's not Google DNS's MAC, it's your default gateway's MAC. Sounds like you have an IP conflict for your gateway IP. Only one MAC should ever answer for a given IP.

britcowboy · Mar 29, 2016, 2:27 AM

@cmb:

MAC addresses are only locally relevant, only visible on the same broadcast domain. That's not Google DNS's MAC, it's your default gateway's MAC. Sounds like you have an IP conflict for your gateway IP. Only one MAC should ever answer for a given IP.

Thanks for the response, thanks for the correction on MAC addresses. I'm not saying you're incorrect, but I'm just confused as to why only my android device (and only on that one DNS request) is using my gateway (pfsense) mac while talking to my internal DNS server. Im 99% positive my pfsense server and DNS server are the only computers using their ip addresses. I suppose the easy way to fix this (can't test currently as now in bed) is to set google DNS as my secondary DNS, as I see it always checks that after not getting a response from the primary, but it'd be nice to fix the problem properly.

What do you think I should try

Derelict · Mar 29, 2016, 2:47 AM

Look at android. What are its exact, configured name servers when connected to this network? The different MAC addresses could also also be queries to two different local hosts.

johnpoz · Mar 29, 2016, 3:58 AM

"I'm not saying you're incorrect"

Your kind of trying to hint at it ;) Sorry to inform you but Derelict is correct here.. (as always) MAC are local, so unless your in the DC where one of googles namerservers are connected to the same L2 you are that is not googles mac address ;)

I never get why anyone would set a 2nd name server that do not have the same data.. That fine if you want to point to say google and opendns out on the public internet. But if you want too resolve your local stuff you have to point at a name server that resolves your local stuff. So it is pointless to point at pfsense, and then 2nd point to google..

If you need failover then you should be pointing to 2 local nameservers that both resolve your local stuff, and then either forward and resolve stuff that is not local. I would concur with Derelict as well to the assumption that you have a duplicate IP issue. You should never get 2 different mac back for the same IP.. So clearly you got something wrong on your network..

Derelict · Mar 29, 2016, 4:14 AM

It's right up there in the windns.PNG file. First it asks 10.0.1.30 then 10.0.1.22. Neither of which return answers. Look at your arp table and see if the two MACs match those two IP addresses. The question is why those do not return answers.

A better question is what local DNS servers did the person who designed this network intend to be used by local hosts?

Read the post above again. All configured name servers need to return the same answers to the same queries. At least for local records. For internet results it's possible they could get and cache different answers.

cmb · Mar 29, 2016, 5:24 AM

I assumed you were using Google's DNS given the comment re: the MAC. Source of the problem still sounds the same, just on a diff IP. If the DNS server in question is a local IP, then it's that local IP that has the IP conflict.

britcowboy · Mar 29, 2016, 5:53 AM

I feel like I need to explain myself a bit better.

My gateway is pfsense on 10.0.1.19
I have two local DNS servers, both Windows DNS on an active domain which are synced. These are 10.0.1.30 and 10.0.1.22.

These DNS servers both forward to Google DNS, if it can't resolve it.

This has been my setup for months, all my machines work fine, always resolving websites and local services.

The only issue I have noticed is this wifi calling DNS request made by android. While monitoring it with wireshark all over DNS queries were made to 10.0.1.30 with the correct mac address. I downloaded a nslookup tool for Android and when requesting the same host through that the packet was correct and went to the correct DNS server with the correct mac address.

When I said I'm not saying you're incorrect but earlier, I wasn't disputing the MACs only working locally, just the duplicate ips.

I honestly think that this is a bug in Androids implementation of Wifi calling, because my girlfriend iPhone has always consistently worked (with my local DNS servers) with EE Wifi Calling. It's only been my android phone that's had any issues. It seems there's a bug where this one call gets the ip of the DNS server and the mac of the gateway server mixed up and uses the wrong combination. I think this one call is special because it's a core built in service.

I'm really happy to be wrong and to find a solution though. But that's my thinking at the moment. This bug could be easily missed as 90% of home users won't run a local DNS so the DNS MAC will match the gateway MAC.

Here is my ARP cache in pfsense. https://imgur.com/6X6LU5g

Honestly I want to be wrong because an Android bug is going to be a PITA to get fixed. Anyone think I've misunderstood or am wrong?

Cheers

(also sorry for bad spelling, on mobile)

Derelict · Mar 29, 2016, 6:46 AM

Well if your Windows DNS Servers forward to google for global resolution I don't see where pfSense fits into the picture.

MAC addresses really have nothing to do with your problem unless, as has been said, you have an IP address conflict. Just forget about MAC addresses.

It really sounds like you need to realize a DNS strategy, get it working, then make sure all your devices are configured to use it.

Is the working iPhone configured to use the same DNS servers as the broken android phone?

britcowboy · Mar 29, 2016, 12:07 PM

@Derelict:

Well if your Windows DNS Servers forward to google for global resolution I don't see where pfSense fits into the picture.

MAC addresses really have nothing to do with your problem unless, as has been said, you have an IP address conflict. Just forget about MAC addresses.

It really sounds like you need to realize a DNS strategy, get it working, then make sure all your devices are configured to use it.

Is the working iPhone configured to use the same DNS servers as the broken android phone?

Yes, the iPhone has 10.0.1.30 and 10.0.1.22 as DNS servers and works perfectly, as does every other computer on the network. The only reason I bring up mac addresses is because the broken DNS packet as seen via wireshark has the wrong mac address so the DNS server isn't recieving the packet (i'm right in the assumption that no matter what the ip address is, NIC will ignore packets sent to it with a different mac address aren't I?)

johnpoz · Mar 29, 2016, 12:11 PM

"NIC will ignore packets sent to it with a different mac address aren't I?)"

What?? Post up this sniff showing what you think is a wrong mac..

In your sniff there was NO answer to your queries.. No shit if your dns does not answer you can not look up stuff..

britcowboy · Mar 29, 2016, 12:38 PM

@johnpoz:

"NIC will ignore packets sent to it with a different mac address aren't I?)"

What?? Post up this sniff showing what you think is a wrong mac..

In your sniff there was NO answer to your queries.. No shit if your dns does not answer you can not look up stuff..

Ive attached the traces

In the Onlythatdnsrequestbeingignored.png you can see that the DNS server is happily responding to every other DNS request coming from the android phone.

The workingdns image shows a packet being sent to 10.0.1.30 with its correct mac address, so the dns server responds.

The brokendns image shows the only packet thats broken. The only difference is the MAC address (which is my gateway not the dns server).

I put Microsoft DNS in debugging mode, which logs all requests, and it never even sees the broken DNS packet (because, i assume, the NIC ignores it as it has the incorrect MAC address)

See what I mean? This seems like unusual behaviour from my android device.