Amazon Echo no longer working



  • About 4 days ago all of my Amazon Echo devices quit working (I have 4) - could no longer register with Amazon. I tunnel all of my DNS traffic via PIA (VPN tunnel is terminated on my PFSENSE firewall) - and I use PIA provided DNS servers (209.222.18.222 and 218). If I swapped out my PFSENSE firewall with a generic home router-built-in WIFI - problem solved. So that pointed me to an issue with my PFSENSE firewall. I'll cut to what I found - took me 3 days doing sniffer traces: Amazon Echo - upon bootup - does several things before it declares sucess - one of the things that it does is a DNS query for www.example.com asking for a AAAA record (an ipv6 address). If it does not get a sucessful answer back it CANNOT register with Amazon - and what the user will hear from Echo is some message that it cannot communicate to the internet. I changed my DNS configuration in PFSENSE to use google (8.8.8.8)...all 4 Echo devices immediately was able to register. I don't know if Amazon upgraded code on their Echos 4 days ago - or the behavior of PIA's provided DNS servers changed when doing a AAAA DNS query - but it was one or the other. This was not a Pfsense issue - but I wanted to at least let other folk know if they have a similar setup....The bitmap attached (hopefully attached) shows a bad wiresharp trace on the left (line 38) and on the right a good (sucessful Amazon registration upon bootup) trace. 0_1530368057432_Amazon-Echo-DNS-Issue.png


  • Netgate Administrator

    Interesting. Hard to believe Echo cannot work without IPv6. Still a good portion of the world in that category. Unfortunately.

    Did you try adding a host override for www.example.net/com?

    Steve



  • Host over ride? Assume you mean a manual local entry? I'm sure that would have worked but I never tried it. I reported this issue to Amazon - I'll be curious if I hear anything (serious) back from the technical side. And I don't care - I just wanted to at least let some folks know about this. It would have been a lot easier to trouble shoot this problem if I had a way to capture the traffic when it was working (i.e. going thru a regular home router/WIFI) - getting a capture of when it was broken (going thru pfsense) was easy. It wasn't until I found out that there was a tcpdump abiltiy via CLI in my UNIFI AP that gave me that ability that I was able to generate a "good capture" - and compare the 2 traces....Before I could that CLI tcpdump on my Unifi AP I was getting closer to dragging out an old Cisco switch and putting it in-between the home router and my ISP box - and port mirror the traffic to another port so I could sniff it. I hated doing that since you no longer see your internal sources - only the local NAT - but that was my final approach if all else failed. Major PIA. I use Echo for a lot of home automation - music - talk - the spouse was not happy during those 3 days...



  • I noticed something like this that just started happening. As my main dns servers in pfsense are setup PIA dns servers and has been working for over a year this way until the last couple weeks when I could not longer reach amazon or Netflix. I switched out the dns servers with goggle dns and it all started working again.



  • Somewhat related comment. I just found out that that D-Link DGS-1100 managed switches (which someone in this forum recommend) support port mirroring. That capability would have really helped if I had been aware of it. I just made the assumption that you needed an enterprise capable switch for that feature.



  • @4romany said in Amazon Echo no longer working:

    About 4 days ago all of my Amazon Echo devices quit working (I have 4) - could no longer register with Amazon. I tunnel all of my DNS traffic via PIA (VPN tunnel is terminated on my PFSENSE firewall) - and I use PIA provided DNS servers (209.222.18.222 and 218). If I swapped out my PFSENSE firewall with a generic home router-built-in WIFI - problem solved. So that pointed me to an issue with my PFSENSE firewall. I'll cut to what I found - took me 3 days doing sniffer traces: Amazon Echo - upon bootup - does several things before it declares sucess - one of the things that it does is a DNS query for www.example.com asking for a AAAA record (an ipv6 address). If it does not get a sucessful answer back it CANNOT register with Amazon - and what the user will hear from Echo is some message that it cannot communicate to the internet. I changed my DNS configuration in PFSENSE to use google (8.8.8.8)...all 4 Echo devices immediately was able to register. I don't know if Amazon upgraded code on their Echos 4 days ago - or the behavior of PIA's provided DNS servers changed when doing a AAAA DNS query - but it was one or the other. This was not a Pfsense issue - but I wanted to at least let other folk know if they have a similar setup....The bitmap attached (hopefully attached) shows a bad wiresharp trace on the left (line 38) and on the right a good (sucessful Amazon registration upon bootup) trace. 0_1530368057432_Amazon-Echo-DNS-Issue.png

    Since you are bypassing PIAs DNS servers and setting it to use google are you not worried now about all your data leaking? Seems counterproductive.


  • Netgate Administrator

    Pretty sure he did that as a test but even if not if depends what you're using the VPN for really.

    Steve



  • I also experienced an issue with my Amazon Echo not being able to connect to the WiFi Network. It had worked okay for may months. But after a power outage, I was not able to reconnect it. I use an Asus AC68U in AP mode as the Access Point. I was able to see the MAC address of the Amazon Echo on the Access Point. On pfSense itself, I could see the Amazon echo was assigned the static IP I had assigned it. But it could never finish connecting to the WiFi network and result in an error message about a failure to register device. I was able to connect it to my backup Asus AC88U router with no issue.

    I have one WAN interface and three OpenVPN interface and selectively route devices and traffic thru each tunnel. I checked all of the log files. No domains were being blocked per the pfBlockerNG reports and the firewall logs showed no blocks either.

    The solution was to remove the Amazon Echo from the TorGuard OpenVPN client interface and assign it to the WAN interface. Doing this step allowed the Amazon Echo to connect to the WiFi network. After completing the connection, I reassigned the Amazon Echo to the TorGuard OpenVPN Client Interface.

    Hope this helps others who may have a similar issue. I spent many hours on the problem.



  • Re: Amazon Echo no longer working

    I did some more testing and this is my current status:

    • The Amazon Echo must be assigned to the WAN interface to connecd to the WiFi network.

    • I can them assign the Echo to the VPN network and it will work - for awhile. Eventually, usually within a 12 to 24 hour window, the Amazon Echo will reach a state where it can no longer phone home and will show up as "Offline" on the Amazon Alexa app. Reassigning the Echo to the WAN interface reconnects the Echo to the WiFi network.

    • The Amazon Echo appears to only work consistently when assigned to the WAN interface

    For next steps, I will perform similar analysis conducted by the OP. I don't want to use Google DNS as the fix though. I currently use Cloudflare DoT for WAN and VPN interfaces.

    What appears to be a common theme is Echo not working when connected to an OpenVPN Client interface.



  • @xentrk said in Amazon Echo no longer working:

    can no longer phone home

    You wiresharked to find out what it does when its phoning home ?
    Can you do so yourself : resolving - connecting to - etc by hand at that moment ?
    Or, said differently : DNS is ok at that moment ? VPN (client ?) is restarting at that moment ? WAN is up ?



  • @gertjan

    I don't see any traffic from the Amazon Echo when using Wireshark (this is very strange) with one caveat. It was in a failure mode. I fired up Wireshark to start debugging. I first filtered on the source IP address (ip.src == 192.168.1.162). I saw some records from the Amazon Echo that it is using MDNS protocol. A web search led me to these resources:

    https://docs.netgate.com/pfsense/en/latest/packages/avahi-package.html
    https://www.lawrencesystems.com/pfsense-and-rules-for-iot-devices-with-mdns/

    Avahi is a system which facilitates service discovery on a local network. This means that a laptop or computer may be connected into a network and instantly be able to view other people to chat with, find printers to print to or find files being shared.

    I installed Avahi and placed the Echo back in the VPN tunnel. Later on in the day, it stopped working again about 12 hours later. The Echo only appears to work consistently when assigned to the WAN iface. This morning, I assigned the Amazon Echo back to the VPN iface and will monitor some more. Based on my last experiment, I expect it to fail sometime within the next 12 hours.


Log in to reply