MDNS struggles

nazuro

@johnpoz Hi John, yes all my sniffs have been from pfSense

This is sniff from IOT. I can see mdns traffic leaving phone (192.168.12.7) 192.168.12.105 is another IOT device.
Screenshot 2021-10-06 at 09.51.04.png

However, on the MAIN interface I do not see anything going FROM pfsense. In this sniff, 192.168.11.20 is my denon AVR, .119 is a laptop.
Screenshot 2021-10-06 at 09.54.17.png

It is very odd because as I mentioned, all of my interfaces have explicit deny rules with logging enabled, and I do not see any traffic deny to 224.0.0.251.

BUT, as I said before, even if I start a sniff on MAIN interface and switch my phone to the MAIN network (and thus am able to discover the Denon AVR), I still do not see any traffic FROM pfSense IP address to 224.0.0.251

johnpoz

@nazuro said in MDNS struggles:

FROM pfSense IP address to 224.0.0.251

Why would you think that pfsense would send any traffic from itself to 224.0.0.251 unless it was forwarding traffic via avahi.

If you see traffic to 224.0.0.251 on your iot interface, but do not see it going out the main.. Then your firewall rules are not allowing it. Or avahi is not working, not actually running..

You do understand you have to sniff on main, while you actively searching for something via your phone on iot right.. Or atleast other mdns traffic being broadcast on the iot.

I am running 21.05.1 - but I fired up my 2.5.2 pfsense vm to try and duplicate this. But from this last post, My guess would be avahi services isn't actually running. Or your firewall rules are not allowing access to the avahi for it to forward it..

So I have 2 lan side interface on pfsense lan (192.168.9.33 pfsense IP) and then another interface in my wlan 192.168.2/24 network. OPT on this vm, pfsense IP address 192.168.2.252

I installed the avahi package.. Set it up as you see. Notice I have the 2 interfaces I want to avahi to use highlighted. When I left it blank avahi did not want to start.. See services where it shows avahi actually running.

Created my floating rule.. My lan and opt rules.. And connected my phone to the 192.168.2 network and you can see via sniff phone at 192.168.2.199 sending traffic to 224.0.0.251 (mdns)..

If I sniff on the lan interface in my pfsense vm, you can see traffic being sent from pfsense IP 192.168.9.33 to 224.0.0.251... There is no responses since I have nothing in this network doing anything with mdns.. Its a chatty protocol that has really little value.. I prevent it and disable it on anything I can disable it on..

Here is details of the floating rule and sniffs

But this works just fine.. Tested both on 21.05.1 and now 2.5.2 and unable to duplicate any sort of problem with avahi and how it is suppose to work. I see no bugs in redmine related, and I would think if it was broke - then there would be lots of people screaming about it ;)

So its something unique to your setup/environment where we seem to be missing part of the puzzle.

I would actually validate the avahi service is running if your not seeing any mdns traffic passed on. And I would actually select the 2 interfaces in the avahi setup you want to pass traffic between.

nazuro

@johnpoz said in MDNS struggles:

Why would you think that pfsense would send any traffic from itself to 224.0.0.251 unless it was forwarding traffic via avahi

Exactly, I am looking for it forwarding traffic via Avahi, like you demonstrated in your pcap from earlier on.

You do understand you have to sniff on main, while you actively searching for something via your phone on iot right.. Or atleast other mdns traffic being broadcast on the iot.

Yes! I do.

Notice I have the 2 interfaces I want to avahi to use highlighted
I do have them highlighted, looks a bit odd on my Safari but you can just about see it.

Also Avahi service is always running when I check it in Status.
Screenshot 2021-10-06 at 21.02.58.png

Nothing in the logs either of interest. Apart from when it restarts (this is after I reselected MAIN and IOT in the Avahi GUI and saved the config:
Screenshot 2021-10-06 at 20.51.20.png

Looks like my configs all align with what you have so must be something odd in my environment.

johnpoz

@nazuro do you have any IPS installed, snort or suricata? Or pimd maybe?

Trying to think of what could be preventing avahi from repeating..

edit: Maybe this?
https://redmine.pfsense.org/issues/10253

Do you have pfblocker with vip setup? Or any other vips setup on your interfaces? From your log it looks correct with 12.1 and 11.1 - but I just can not duplicate anything close to the problem your having.

nazuro

@johnpoz
I have suricata running on WAN and LAN but it is not in blocking mode. I will try disabling it.

No pfblocker / VIP set up.

Well, I do really appreciate your help you have taught me a few things. Real head scratcher this

nazuro

@johnpoz I've noticed that if I restart the avahi service, I seem to be able to discover across VLAN for a short period of time (not tested exactly but about 24hrs maybe). Odd because the Avahi service seems to be running fine otherwise. After 24hr-ish I then can no longer discover it and I need to restart Avahi again

johnpoz

@nazuro well avahi not actually running would put a hamper on discovery.. You see nothing in the log showing it stopped?

nazuro

@johnpoz Oh wow, think I found it. So, I think we must have fixed it (thanks again for your great help) about half way up the thread, but then I didn't realise it automatically broke at 00:30.

Oct 13 00:30:21 pfSense php[45999]: [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Oct 13 00:30:24 pfSense php[45999]: [Suricata] Emerging Threats Open rules file update downloaded successfully.
Oct 13 00:30:24 pfSense php[45999]: [Suricata] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz...
Oct 13 00:30:25 pfSense php[45999]: [Suricata] Snort GPLv2 Community Rules file update downloaded successfully.
Oct 13 00:30:28 pfSense php[45999]: [Suricata] Updating rules configuration for: WAN ...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Enabling any flowbit-required rules for: WAN...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Building new sid-msg.map file for WAN...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Suricata STOP for WAN_IDS(igb0)...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Suricata START for WAN_IDS(igb0)...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Suricata has restarted with your new set of rules for WAN...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Updating rules configuration for: LAN ...
Oct 13 00:30:46 pfSense php[45999]: [Suricata] Building new sid-msg.map file for LAN...
Oct 13 00:30:46 pfSense php[45999]: [Suricata] Suricata STOP for LAN(igb1)...
Oct 13 00:30:47 pfSense kernel: igb1: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.11: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.12: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.14: link state changed to DOWN
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.11
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.12
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.14
Oct 13 00:30:48 pfSense php-fpm[339]: /rc.linkup: Hotplug event detected for LAN(lan) static IP (192.168.10.1 )
Oct 13 00:30:48 pfSense check_reload_status[378]: Reloading filter
Oct 13 00:30:48 pfSense php-fpm[340]: /rc.linkup: Hotplug event detected for MAIN(opt2) static IP (192.168.11.1 )
Oct 13 00:30:48 pfSense php-fpm[339]: /rc.linkup: Hotplug event detected for IOT(opt3) static IP (192.168.12.1 )
Oct 13 00:30:48 pfSense check_reload_status[378]: Reloading filter
Oct 13 00:30:48 pfSense php-fpm[66890]: /rc.linkup: Hotplug event detected for GUEST(opt6) static IP (192.168.14.1 )
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.11.IPv6 with address fe80::21b:21ff:fe33:9ee9.
Oct 13 00:30:49 pfSense kernel: igb1: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.11: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.12: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.14: link state changed to UP
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.11.IPv4 with address 192.168.11.1.
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.12.IPv6 with address fe80::21b:21ff:fe33:9ee9.
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.12.IPv4 with address 192.168.12.1.

bmeeks

@nazuro said in MDNS struggles:

@johnpoz Oh wow, think I found it. So, I think we must have fixed it (thanks again for your great help) about half way up the thread, but then I didn't realise it automatically broke at 00:30.

Oct 13 00:30:21 pfSense php[45999]: [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Oct 13 00:30:24 pfSense php[45999]: [Suricata] Emerging Threats Open rules file update downloaded successfully.
Oct 13 00:30:24 pfSense php[45999]: [Suricata] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz...
Oct 13 00:30:25 pfSense php[45999]: [Suricata] Snort GPLv2 Community Rules file update downloaded successfully.
Oct 13 00:30:28 pfSense php[45999]: [Suricata] Updating rules configuration for: WAN ...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Enabling any flowbit-required rules for: WAN...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Building new sid-msg.map file for WAN...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Suricata STOP for WAN_IDS(igb0)...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Suricata START for WAN_IDS(igb0)...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Suricata has restarted with your new set of rules for WAN...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Updating rules configuration for: LAN ...
Oct 13 00:30:46 pfSense php[45999]: [Suricata] Building new sid-msg.map file for LAN...
Oct 13 00:30:46 pfSense php[45999]: [Suricata] Suricata STOP for LAN(igb1)...
Oct 13 00:30:47 pfSense kernel: igb1: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.11: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.12: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.14: link state changed to DOWN
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.11
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.12
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.14
Oct 13 00:30:48 pfSense php-fpm[339]: /rc.linkup: Hotplug event detected for LAN(lan) static IP (192.168.10.1 )
Oct 13 00:30:48 pfSense check_reload_status[378]: Reloading filter
Oct 13 00:30:48 pfSense php-fpm[340]: /rc.linkup: Hotplug event detected for MAIN(opt2) static IP (192.168.11.1 )
Oct 13 00:30:48 pfSense php-fpm[339]: /rc.linkup: Hotplug event detected for IOT(opt3) static IP (192.168.12.1 )
Oct 13 00:30:48 pfSense check_reload_status[378]: Reloading filter
Oct 13 00:30:48 pfSense php-fpm[66890]: /rc.linkup: Hotplug event detected for GUEST(opt6) static IP (192.168.14.1 )
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.11.IPv6 with address fe80::21b:21ff:fe33:9ee9.
Oct 13 00:30:49 pfSense kernel: igb1: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.11: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.12: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.14: link state changed to UP
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.11.IPv4 with address 192.168.11.1.
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.12.IPv6 with address fe80::21b:21ff:fe33:9ee9.
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.12.IPv4 with address 192.168.12.1.

I see you are using Suricata. Be aware that if using Suricata with Inline IPS Mode (which uses the netmap kernel device), each time Suricata restarts it will physically cycle the interface it is running on. That means the interface will behave the same as if you manually did an "ifdn" followed by an "ifup" command sequence. Many network applications do not like the interface they are running on going offline and then coming back online. They will either crash, or automatically shut themselves down in that scenario.

To prevent Suricata from cycling the interface when updating rules, you can go to the GLOBAL SETTINGS tab and enable the option to "Live Update" the rules. That option is unchecked by default. The only real downside of enabling that option is RAM usage will briefly increase during the rules update job because Suricata will keep two complete copies of the rules in RAM as it updates. When the update is finished, it will discard the old copy of the rules.

nazuro

@bmeeks Thank you so much - that seems to have solved the issue! And thanks to John as well

bmeeks

@nazuro said in MDNS struggles:

@bmeeks Thank you so much - that seems to have solved the issue! And thanks to John as well

Glad that helped you.

What really happens is when Suricata stops and then restarts, it makes a call to the netmap device code to first "close" the open netmap interface (when stopping); and then it makes a call to "open" the netmap interface (when starting). The netmap kernel device performs a literal "down" and then "up" cycle as it switches the interface from and to netmap-mode (netmap-mode unhooks the interface from its normal kernel connections). You can see this in your log snippet:

Oct 13 00:30:47 pfSense kernel: igb1: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.11: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.12: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.14: link state changed to DOWN

followed later by:

Oct 13 00:30:49 pfSense kernel: igb1: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.11: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.12: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.14: link state changed to UP

Using the "Live Update" mode for rules updates means Suricata never actually stops and restarts. It keeps running during the update cycle and does the in-memory rules duplication/update thing I described earlier. Since it never restarts, it does not call the netmap code to close and then open the interface.