MDNS struggles
-
@johnpoz Hi John, thanks. I can see the SSDP from the Denon but from googling around I'm still pretty sure that Airplay uses mdns, and I can see that from the answer in the MDNS query when my phone is on the MAIN/11 network.
the sniff on the 11 interface while the phone sends queries on the 12 network is uploaded as main.pcap and I cannot see pfSense passing on the mdns query, which makes me think Avahi is not working correctly then.
-
@nazuro said in MDNS struggles:
and I can see that from the answer in the MDNS query when my phone is on the MAIN/11 network.
No your not - has NO IP in what your thinking is an answer - so how would you talk to it?
If your not seeing avahi pass on the traffic like I showed - then avahi is not running, or your firewall rules do not allow avahi to see the traffic to be able to pass it on.
you posted your avahi settings - where are the firewall rules that allow pfsense/avahi to see that broadcast so it can send it on? That floating rule looks to be outbound? See the little double green arrow icon. Oh that is for quick.. What direction is the rule set for?
If your not seeing pfsense pass on the traffic - then no it would not be possible to discover this via mdns..
What are you interface rules? As mentioned before - even if you discover it, if you do not have the firewall rules to allow the traffic (after discovered) then no your not going to be able to talk to it either. But what you posted, had no IP in it.. And that was coming from your phone.. As an announcement maybe.. For mdns to work - you should see a specific response from the device with its IP in it..
edit: Ah!! Always helps to go to the RFCS ;)
https://datatracker.ietf.org/doc/html/rfc6762#section-7.1
When a Multicast DNS querier sends a query to which it already knows
some answers, it populates the Answer Section of the DNS query
message with those answersThis explains what your seeing in that frame from your phone.
edit2:
If your phone already knows the IP, then possible your problem is firewall not letting it talk to that IP on whatever that is - but if that was the case you would see traffic to that IP in your sniff on the 12 network trying to go to the 11. address? Which I didn't see.. -
@johnpoz on my floating rule the direction is set to any.
By the way, same problem the other way round, I have an Apple HomePod (192.168.12.145) on IOT and I am not able to "discover" it with my phone on MAIN. Pcap does not show any SSDP from the HomePod just MDNS.
I've created a test floating rule to allow all from MAIN to IOT. However, I do already have explicit default deny rules on all interfaces with logging enabled and cannot see any blocked firewall traffic from the Denon (192.168.11.20). Still no luck.
-
If you do not see the query passed on, then no it would not discover. What version of pfsense are you running, what is the version of the package?
Did you try actually selecting the interface in avahi - vs not selecting any? And hoping it listening on all?
Your screenshot is so zoom in, maybe you have some other interface selected.. Select the 2 interfaces your wanting to use directly.
You do not have avahi set to filter service.. Any other settings?
-
@johnpoz pfSense 2.5.2 - selected MAIN and IOT interfaces only for Avahi.
Not changed anything else in the Avahi settings
-
@nazuro why are publishing pfsense - it has no services you would use via mdns.. those are not default..
This is how it should be set.. Set it like this - and set your firewall rule to 224.0.0.251
I show current package as 2.2 and its running
If you do not see it passing on the traffic via sniff - with a response - then no its not going to work, and something is wrong.
-
@johnpoz Ok, I've adjusted my settings to match yours including the floating FW rule. I too am on v 2.2. Not sure what else could be the issue unless something funny going on with the ubiquiti AP or Netgear switch :(
-
@nazuro said in MDNS struggles:
going on with the ubiquiti AP or Netgear switch :(
Those have little to do with if you sniff on interface A of pfsense and see the query, it should be passed on out interface B coming from pfsense IP address on interface B.
I specifically showed exactly how to validate and troubleshoot this... If pfsense never sees query to 224.0.0.251 from the device looking.. It will not pass it on, if it passes it on but doesn't get a response then no the device is not going to find what its looking for..
In your sniff I see from 192.168.11.128 to 224.0.0.251 a query - DO YOU SEE it sent on on interface B from the pfsense IP??
-
@johnpoz In my sniffs I can't seem to see query FROM pfsense IP, even when iPhone is connected to MAIN network and I am able to see/connect to Denon AVR. Whether I am incorrectly doing the tcpdump or not I'm not sure. Should I sniff on the parent interface with something like
tcpdump -i igb1 -nn -e vlanOR, separate sniffing on the virtual interface like
tcpdump -I igb1.11 -nn
tcpdump -I igb1.12 -nn -
Why would you have to do anything other than the gui to do the package capture.. You shouldn't nave to see the vlan tags in the sniff.. Since pfsense has an IP in the 11 interface, and it has an IP in the 12 interface does it not? If it didn't how would ever see the traffic to pass on the traffic.. Be it tagged or not..
Just do the package capture in the gui on those interfaces - if its not seeing the traffic to 224.0.0.251 then your firewall rules are not correct. Or you have something blocking that traffic before pfsense could see it? Like sure an AP or switch.. You clearly posted up a sniff of 11.128 sending to 244.0.0.251 - where did you do that sniff at.. Was that not done on pfsense?
If your not sniffing on pfsense when you posted up that traffic - then yeah pfsense maybe not ever seeing the traffic to pass it on. You need to sniff on pfsense!! To validate it sees the traffic, and passes it on!
I assumed that was were you were sniffing..
You should just sniff on whatever iot vlan or main interface in the gui..
Example - my psk is a vlan interface - And just sniff like this..
And here I see it - I could download it sure into wireshark - but can see that some mdns traffic was sent from IP of my phone to 224.0.0.251, and should then see such traffic leaving the IP of pfsense to 224.0.0.251 on other interface..
If your not seeing it via sniffing on pfsense - then no its not going to work because pfsense is not seeing the traffic to send on, either its really not seeing it - or your firewall rule is not allowing avahi to see it, if you see it in the sniff on pfsense.
Its possible your AP is not passing on the multicast traffic for pfsense to see it from your phone.. But I assumed you were sniffing on pfsense when you posted up 11.128 sending to 224.0.0.251
edit: You need to be sniffing here
If your not seeing it there then pfsense is not seeing it, if not seeing it - can not pass it on.. If it passes on and no response.. Then maybe your device is not seeing it because of AP or switch settings on that network... But you need to sniff on pfsense, and see the traffic if ever going to work.
-
@johnpoz Hi John, yes all my sniffs have been from pfSense
This is sniff from IOT. I can see mdns traffic leaving phone (192.168.12.7) 192.168.12.105 is another IOT device.
However, on the MAIN interface I do not see anything going FROM pfsense. In this sniff, 192.168.11.20 is my denon AVR, .119 is a laptop.
It is very odd because as I mentioned, all of my interfaces have explicit deny rules with logging enabled, and I do not see any traffic deny to 224.0.0.251.
BUT, as I said before, even if I start a sniff on MAIN interface and switch my phone to the MAIN network (and thus am able to discover the Denon AVR), I still do not see any traffic FROM pfSense IP address to 224.0.0.251
-
@nazuro said in MDNS struggles:
FROM pfSense IP address to 224.0.0.251
Why would you think that pfsense would send any traffic from itself to 224.0.0.251 unless it was forwarding traffic via avahi.
If you see traffic to 224.0.0.251 on your iot interface, but do not see it going out the main.. Then your firewall rules are not allowing it. Or avahi is not working, not actually running..
You do understand you have to sniff on main, while you actively searching for something via your phone on iot right.. Or atleast other mdns traffic being broadcast on the iot.
I am running 21.05.1 - but I fired up my 2.5.2 pfsense vm to try and duplicate this. But from this last post, My guess would be avahi services isn't actually running. Or your firewall rules are not allowing access to the avahi for it to forward it..
So I have 2 lan side interface on pfsense lan (192.168.9.33 pfsense IP) and then another interface in my wlan 192.168.2/24 network. OPT on this vm, pfsense IP address 192.168.2.252
I installed the avahi package.. Set it up as you see. Notice I have the 2 interfaces I want to avahi to use highlighted. When I left it blank avahi did not want to start.. See services where it shows avahi actually running.
Created my floating rule.. My lan and opt rules.. And connected my phone to the 192.168.2 network and you can see via sniff phone at 192.168.2.199 sending traffic to 224.0.0.251 (mdns)..
If I sniff on the lan interface in my pfsense vm, you can see traffic being sent from pfsense IP 192.168.9.33 to 224.0.0.251... There is no responses since I have nothing in this network doing anything with mdns.. Its a chatty protocol that has really little value.. I prevent it and disable it on anything I can disable it on..
Here is details of the floating rule and sniffs
But this works just fine.. Tested both on 21.05.1 and now 2.5.2 and unable to duplicate any sort of problem with avahi and how it is suppose to work. I see no bugs in redmine related, and I would think if it was broke - then there would be lots of people screaming about it ;)
So its something unique to your setup/environment where we seem to be missing part of the puzzle.
I would actually validate the avahi service is running if your not seeing any mdns traffic passed on. And I would actually select the 2 interfaces in the avahi setup you want to pass traffic between.
-
@johnpoz said in MDNS struggles:
Why would you think that pfsense would send any traffic from itself to 224.0.0.251 unless it was forwarding traffic via avahi
Exactly, I am looking for it forwarding traffic via Avahi, like you demonstrated in your pcap from earlier on.
You do understand you have to sniff on main, while you actively searching for something via your phone on iot right.. Or atleast other mdns traffic being broadcast on the iot.
Yes! I do.
Notice I have the 2 interfaces I want to avahi to use highlighted
I do have them highlighted, looks a bit odd on my Safari but you can just about see it.
Also Avahi service is always running when I check it in Status.
Nothing in the logs either of interest. Apart from when it restarts (this is after I reselected MAIN and IOT in the Avahi GUI and saved the config:
Looks like my configs all align with what you have so must be something odd in my environment.
-
@nazuro do you have any IPS installed, snort or suricata? Or pimd maybe?
Trying to think of what could be preventing avahi from repeating..
edit: Maybe this?
https://redmine.pfsense.org/issues/10253Do you have pfblocker with vip setup? Or any other vips setup on your interfaces? From your log it looks correct with 12.1 and 11.1 - but I just can not duplicate anything close to the problem your having.
-
@johnpoz
I have suricata running on WAN and LAN but it is not in blocking mode. I will try disabling it.No pfblocker / VIP set up.
Well, I do really appreciate your help you have taught me a few things. Real head scratcher this
-
@johnpoz I've noticed that if I restart the avahi service, I seem to be able to discover across VLAN for a short period of time (not tested exactly but about 24hrs maybe). Odd because the Avahi service seems to be running fine otherwise. After 24hr-ish I then can no longer discover it and I need to restart Avahi again
-
@nazuro well avahi not actually running would put a hamper on discovery.. You see nothing in the log showing it stopped?
-
@johnpoz Oh wow, think I found it. So, I think we must have fixed it (thanks again for your great help) about half way up the thread, but then I didn't realise it automatically broke at 00:30.
Oct 13 00:30:21 pfSense php[45999]: [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Oct 13 00:30:24 pfSense php[45999]: [Suricata] Emerging Threats Open rules file update downloaded successfully.
Oct 13 00:30:24 pfSense php[45999]: [Suricata] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz...
Oct 13 00:30:25 pfSense php[45999]: [Suricata] Snort GPLv2 Community Rules file update downloaded successfully.
Oct 13 00:30:28 pfSense php[45999]: [Suricata] Updating rules configuration for: WAN ...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Enabling any flowbit-required rules for: WAN...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Building new sid-msg.map file for WAN...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Suricata STOP for WAN_IDS(igb0)...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Suricata START for WAN_IDS(igb0)...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Suricata has restarted with your new set of rules for WAN...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Updating rules configuration for: LAN ...
Oct 13 00:30:46 pfSense php[45999]: [Suricata] Building new sid-msg.map file for LAN...
Oct 13 00:30:46 pfSense php[45999]: [Suricata] Suricata STOP for LAN(igb1)...
Oct 13 00:30:47 pfSense kernel: igb1: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.11: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.12: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.14: link state changed to DOWN
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.11
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.12
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.14
Oct 13 00:30:48 pfSense php-fpm[339]: /rc.linkup: Hotplug event detected for LAN(lan) static IP (192.168.10.1 )
Oct 13 00:30:48 pfSense check_reload_status[378]: Reloading filter
Oct 13 00:30:48 pfSense php-fpm[340]: /rc.linkup: Hotplug event detected for MAIN(opt2) static IP (192.168.11.1 )
Oct 13 00:30:48 pfSense php-fpm[339]: /rc.linkup: Hotplug event detected for IOT(opt3) static IP (192.168.12.1 )
Oct 13 00:30:48 pfSense check_reload_status[378]: Reloading filter
Oct 13 00:30:48 pfSense php-fpm[66890]: /rc.linkup: Hotplug event detected for GUEST(opt6) static IP (192.168.14.1 )
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.11.IPv6 with address fe80::21b:21ff:fe33:9ee9.
Oct 13 00:30:49 pfSense kernel: igb1: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.11: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.12: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.14: link state changed to UP
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.11.IPv4 with address 192.168.11.1.
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.12.IPv6 with address fe80::21b:21ff:fe33:9ee9.
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.12.IPv4 with address 192.168.12.1. -
@nazuro said in MDNS struggles:
@johnpoz Oh wow, think I found it. So, I think we must have fixed it (thanks again for your great help) about half way up the thread, but then I didn't realise it automatically broke at 00:30.
Oct 13 00:30:21 pfSense php[45999]: [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Oct 13 00:30:24 pfSense php[45999]: [Suricata] Emerging Threats Open rules file update downloaded successfully.
Oct 13 00:30:24 pfSense php[45999]: [Suricata] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz...
Oct 13 00:30:25 pfSense php[45999]: [Suricata] Snort GPLv2 Community Rules file update downloaded successfully.
Oct 13 00:30:28 pfSense php[45999]: [Suricata] Updating rules configuration for: WAN ...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Enabling any flowbit-required rules for: WAN...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Building new sid-msg.map file for WAN...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Suricata STOP for WAN_IDS(igb0)...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Suricata START for WAN_IDS(igb0)...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Suricata has restarted with your new set of rules for WAN...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Updating rules configuration for: LAN ...
Oct 13 00:30:46 pfSense php[45999]: [Suricata] Building new sid-msg.map file for LAN...
Oct 13 00:30:46 pfSense php[45999]: [Suricata] Suricata STOP for LAN(igb1)...
Oct 13 00:30:47 pfSense kernel: igb1: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.11: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.12: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.14: link state changed to DOWN
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.11
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.12
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.14
Oct 13 00:30:48 pfSense php-fpm[339]: /rc.linkup: Hotplug event detected for LAN(lan) static IP (192.168.10.1 )
Oct 13 00:30:48 pfSense check_reload_status[378]: Reloading filter
Oct 13 00:30:48 pfSense php-fpm[340]: /rc.linkup: Hotplug event detected for MAIN(opt2) static IP (192.168.11.1 )
Oct 13 00:30:48 pfSense php-fpm[339]: /rc.linkup: Hotplug event detected for IOT(opt3) static IP (192.168.12.1 )
Oct 13 00:30:48 pfSense check_reload_status[378]: Reloading filter
Oct 13 00:30:48 pfSense php-fpm[66890]: /rc.linkup: Hotplug event detected for GUEST(opt6) static IP (192.168.14.1 )
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.11.IPv6 with address fe80::21b:21ff:fe33:9ee9.
Oct 13 00:30:49 pfSense kernel: igb1: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.11: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.12: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.14: link state changed to UP
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.11.IPv4 with address 192.168.11.1.
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.12.IPv6 with address fe80::21b:21ff:fe33:9ee9.
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.12.IPv4 with address 192.168.12.1.I see you are using Suricata. Be aware that if using Suricata with Inline IPS Mode (which uses the netmap kernel device), each time Suricata restarts it will physically cycle the interface it is running on. That means the interface will behave the same as if you manually did an "ifdn" followed by an "ifup" command sequence. Many network applications do not like the interface they are running on going offline and then coming back online. They will either crash, or automatically shut themselves down in that scenario.
To prevent Suricata from cycling the interface when updating rules, you can go to the GLOBAL SETTINGS tab and enable the option to "Live Update" the rules. That option is unchecked by default. The only real downside of enabling that option is RAM usage will briefly increase during the rules update job because Suricata will keep two complete copies of the rules in RAM as it updates. When the update is finished, it will discard the old copy of the rules.
-
@bmeeks Thank you so much - that seems to have solved the issue! And thanks to John as well
