MDNS struggles

nazuro

@johnpoz Ok, I've adjusted my settings to match yours including the floating FW rule. I too am on v 2.2. Not sure what else could be the issue unless something funny going on with the ubiquiti AP or Netgear switch :(

johnpoz

@nazuro said in MDNS struggles:

going on with the ubiquiti AP or Netgear switch :(

Those have little to do with if you sniff on interface A of pfsense and see the query, it should be passed on out interface B coming from pfsense IP address on interface B.

I specifically showed exactly how to validate and troubleshoot this... If pfsense never sees query to 224.0.0.251 from the device looking.. It will not pass it on, if it passes it on but doesn't get a response then no the device is not going to find what its looking for..

In your sniff I see from 192.168.11.128 to 224.0.0.251 a query - DO YOU SEE it sent on on interface B from the pfsense IP??

nazuro

@johnpoz In my sniffs I can't seem to see query FROM pfsense IP, even when iPhone is connected to MAIN network and I am able to see/connect to Denon AVR. Whether I am incorrectly doing the tcpdump or not I'm not sure. Should I sniff on the parent interface with something like
tcpdump -i igb1 -nn -e vlan

OR, separate sniffing on the virtual interface like
tcpdump -I igb1.11 -nn
tcpdump -I igb1.12 -nn

johnpoz

Why would you have to do anything other than the gui to do the package capture.. You shouldn't nave to see the vlan tags in the sniff.. Since pfsense has an IP in the 11 interface, and it has an IP in the 12 interface does it not? If it didn't how would ever see the traffic to pass on the traffic.. Be it tagged or not..

Just do the package capture in the gui on those interfaces - if its not seeing the traffic to 224.0.0.251 then your firewall rules are not correct. Or you have something blocking that traffic before pfsense could see it? Like sure an AP or switch.. You clearly posted up a sniff of 11.128 sending to 244.0.0.251 - where did you do that sniff at.. Was that not done on pfsense?

If your not sniffing on pfsense when you posted up that traffic - then yeah pfsense maybe not ever seeing the traffic to pass it on. You need to sniff on pfsense!! To validate it sees the traffic, and passes it on!

I assumed that was were you were sniffing..

You should just sniff on whatever iot vlan or main interface in the gui..

Example - my psk is a vlan interface - And just sniff like this..

And here I see it - I could download it sure into wireshark - but can see that some mdns traffic was sent from IP of my phone to 224.0.0.251, and should then see such traffic leaving the IP of pfsense to 224.0.0.251 on other interface..

If your not seeing it via sniffing on pfsense - then no its not going to work because pfsense is not seeing the traffic to send on, either its really not seeing it - or your firewall rule is not allowing avahi to see it, if you see it in the sniff on pfsense.

Its possible your AP is not passing on the multicast traffic for pfsense to see it from your phone.. But I assumed you were sniffing on pfsense when you posted up 11.128 sending to 224.0.0.251

edit: You need to be sniffing here

If your not seeing it there then pfsense is not seeing it, if not seeing it - can not pass it on.. If it passes on and no response.. Then maybe your device is not seeing it because of AP or switch settings on that network... But you need to sniff on pfsense, and see the traffic if ever going to work.

nazuro

@johnpoz Hi John, yes all my sniffs have been from pfSense

This is sniff from IOT. I can see mdns traffic leaving phone (192.168.12.7) 192.168.12.105 is another IOT device.
Screenshot 2021-10-06 at 09.51.04.png

However, on the MAIN interface I do not see anything going FROM pfsense. In this sniff, 192.168.11.20 is my denon AVR, .119 is a laptop.
Screenshot 2021-10-06 at 09.54.17.png

It is very odd because as I mentioned, all of my interfaces have explicit deny rules with logging enabled, and I do not see any traffic deny to 224.0.0.251.

BUT, as I said before, even if I start a sniff on MAIN interface and switch my phone to the MAIN network (and thus am able to discover the Denon AVR), I still do not see any traffic FROM pfSense IP address to 224.0.0.251

johnpoz

@nazuro said in MDNS struggles:

FROM pfSense IP address to 224.0.0.251

Why would you think that pfsense would send any traffic from itself to 224.0.0.251 unless it was forwarding traffic via avahi.

If you see traffic to 224.0.0.251 on your iot interface, but do not see it going out the main.. Then your firewall rules are not allowing it. Or avahi is not working, not actually running..

You do understand you have to sniff on main, while you actively searching for something via your phone on iot right.. Or atleast other mdns traffic being broadcast on the iot.

I am running 21.05.1 - but I fired up my 2.5.2 pfsense vm to try and duplicate this. But from this last post, My guess would be avahi services isn't actually running. Or your firewall rules are not allowing access to the avahi for it to forward it..

So I have 2 lan side interface on pfsense lan (192.168.9.33 pfsense IP) and then another interface in my wlan 192.168.2/24 network. OPT on this vm, pfsense IP address 192.168.2.252

I installed the avahi package.. Set it up as you see. Notice I have the 2 interfaces I want to avahi to use highlighted. When I left it blank avahi did not want to start.. See services where it shows avahi actually running.

Created my floating rule.. My lan and opt rules.. And connected my phone to the 192.168.2 network and you can see via sniff phone at 192.168.2.199 sending traffic to 224.0.0.251 (mdns)..

If I sniff on the lan interface in my pfsense vm, you can see traffic being sent from pfsense IP 192.168.9.33 to 224.0.0.251... There is no responses since I have nothing in this network doing anything with mdns.. Its a chatty protocol that has really little value.. I prevent it and disable it on anything I can disable it on..

Here is details of the floating rule and sniffs

But this works just fine.. Tested both on 21.05.1 and now 2.5.2 and unable to duplicate any sort of problem with avahi and how it is suppose to work. I see no bugs in redmine related, and I would think if it was broke - then there would be lots of people screaming about it ;)

So its something unique to your setup/environment where we seem to be missing part of the puzzle.

I would actually validate the avahi service is running if your not seeing any mdns traffic passed on. And I would actually select the 2 interfaces in the avahi setup you want to pass traffic between.

nazuro

@johnpoz said in MDNS struggles:

Why would you think that pfsense would send any traffic from itself to 224.0.0.251 unless it was forwarding traffic via avahi

Exactly, I am looking for it forwarding traffic via Avahi, like you demonstrated in your pcap from earlier on.

You do understand you have to sniff on main, while you actively searching for something via your phone on iot right.. Or atleast other mdns traffic being broadcast on the iot.

Yes! I do.

Notice I have the 2 interfaces I want to avahi to use highlighted
I do have them highlighted, looks a bit odd on my Safari but you can just about see it.

Also Avahi service is always running when I check it in Status.
Screenshot 2021-10-06 at 21.02.58.png

Nothing in the logs either of interest. Apart from when it restarts (this is after I reselected MAIN and IOT in the Avahi GUI and saved the config:
Screenshot 2021-10-06 at 20.51.20.png

Looks like my configs all align with what you have so must be something odd in my environment.

johnpoz

@nazuro do you have any IPS installed, snort or suricata? Or pimd maybe?

Trying to think of what could be preventing avahi from repeating..

edit: Maybe this?
https://redmine.pfsense.org/issues/10253

Do you have pfblocker with vip setup? Or any other vips setup on your interfaces? From your log it looks correct with 12.1 and 11.1 - but I just can not duplicate anything close to the problem your having.

nazuro

@johnpoz
I have suricata running on WAN and LAN but it is not in blocking mode. I will try disabling it.

No pfblocker / VIP set up.

Well, I do really appreciate your help you have taught me a few things. Real head scratcher this

nazuro

@johnpoz I've noticed that if I restart the avahi service, I seem to be able to discover across VLAN for a short period of time (not tested exactly but about 24hrs maybe). Odd because the Avahi service seems to be running fine otherwise. After 24hr-ish I then can no longer discover it and I need to restart Avahi again

johnpoz

@nazuro well avahi not actually running would put a hamper on discovery.. You see nothing in the log showing it stopped?

nazuro

@johnpoz Oh wow, think I found it. So, I think we must have fixed it (thanks again for your great help) about half way up the thread, but then I didn't realise it automatically broke at 00:30.

Oct 13 00:30:21 pfSense php[45999]: [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Oct 13 00:30:24 pfSense php[45999]: [Suricata] Emerging Threats Open rules file update downloaded successfully.
Oct 13 00:30:24 pfSense php[45999]: [Suricata] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz...
Oct 13 00:30:25 pfSense php[45999]: [Suricata] Snort GPLv2 Community Rules file update downloaded successfully.
Oct 13 00:30:28 pfSense php[45999]: [Suricata] Updating rules configuration for: WAN ...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Enabling any flowbit-required rules for: WAN...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Building new sid-msg.map file for WAN...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Suricata STOP for WAN_IDS(igb0)...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Suricata START for WAN_IDS(igb0)...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Suricata has restarted with your new set of rules for WAN...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Updating rules configuration for: LAN ...
Oct 13 00:30:46 pfSense php[45999]: [Suricata] Building new sid-msg.map file for LAN...
Oct 13 00:30:46 pfSense php[45999]: [Suricata] Suricata STOP for LAN(igb1)...
Oct 13 00:30:47 pfSense kernel: igb1: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.11: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.12: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.14: link state changed to DOWN
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.11
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.12
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.14
Oct 13 00:30:48 pfSense php-fpm[339]: /rc.linkup: Hotplug event detected for LAN(lan) static IP (192.168.10.1 )
Oct 13 00:30:48 pfSense check_reload_status[378]: Reloading filter
Oct 13 00:30:48 pfSense php-fpm[340]: /rc.linkup: Hotplug event detected for MAIN(opt2) static IP (192.168.11.1 )
Oct 13 00:30:48 pfSense php-fpm[339]: /rc.linkup: Hotplug event detected for IOT(opt3) static IP (192.168.12.1 )
Oct 13 00:30:48 pfSense check_reload_status[378]: Reloading filter
Oct 13 00:30:48 pfSense php-fpm[66890]: /rc.linkup: Hotplug event detected for GUEST(opt6) static IP (192.168.14.1 )
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.11.IPv6 with address fe80::21b:21ff:fe33:9ee9.
Oct 13 00:30:49 pfSense kernel: igb1: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.11: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.12: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.14: link state changed to UP
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.11.IPv4 with address 192.168.11.1.
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.12.IPv6 with address fe80::21b:21ff:fe33:9ee9.
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.12.IPv4 with address 192.168.12.1.

bmeeks

@nazuro said in MDNS struggles:

@johnpoz Oh wow, think I found it. So, I think we must have fixed it (thanks again for your great help) about half way up the thread, but then I didn't realise it automatically broke at 00:30.

Oct 13 00:30:21 pfSense php[45999]: [Suricata] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Oct 13 00:30:24 pfSense php[45999]: [Suricata] Emerging Threats Open rules file update downloaded successfully.
Oct 13 00:30:24 pfSense php[45999]: [Suricata] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz...
Oct 13 00:30:25 pfSense php[45999]: [Suricata] Snort GPLv2 Community Rules file update downloaded successfully.
Oct 13 00:30:28 pfSense php[45999]: [Suricata] Updating rules configuration for: WAN ...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Enabling any flowbit-required rules for: WAN...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Building new sid-msg.map file for WAN...
Oct 13 00:30:30 pfSense php[45999]: [Suricata] Suricata STOP for WAN_IDS(igb0)...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Suricata START for WAN_IDS(igb0)...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Suricata has restarted with your new set of rules for WAN...
Oct 13 00:30:45 pfSense php[45999]: [Suricata] Updating rules configuration for: LAN ...
Oct 13 00:30:46 pfSense php[45999]: [Suricata] Building new sid-msg.map file for LAN...
Oct 13 00:30:46 pfSense php[45999]: [Suricata] Suricata STOP for LAN(igb1)...
Oct 13 00:30:47 pfSense kernel: igb1: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.11: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.12: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.14: link state changed to DOWN
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.11
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.12
Oct 13 00:30:47 pfSense check_reload_status[378]: Linkup starting igb1.14
Oct 13 00:30:48 pfSense php-fpm[339]: /rc.linkup: Hotplug event detected for LAN(lan) static IP (192.168.10.1 )
Oct 13 00:30:48 pfSense check_reload_status[378]: Reloading filter
Oct 13 00:30:48 pfSense php-fpm[340]: /rc.linkup: Hotplug event detected for MAIN(opt2) static IP (192.168.11.1 )
Oct 13 00:30:48 pfSense php-fpm[339]: /rc.linkup: Hotplug event detected for IOT(opt3) static IP (192.168.12.1 )
Oct 13 00:30:48 pfSense check_reload_status[378]: Reloading filter
Oct 13 00:30:48 pfSense php-fpm[66890]: /rc.linkup: Hotplug event detected for GUEST(opt6) static IP (192.168.14.1 )
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.11.IPv6 with address fe80::21b:21ff:fe33:9ee9.
Oct 13 00:30:49 pfSense kernel: igb1: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.11: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.12: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.14: link state changed to UP
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.11.IPv4 with address 192.168.11.1.
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.12.IPv6 with address fe80::21b:21ff:fe33:9ee9.
Oct 13 00:30:49 pfSense avahi-daemon[91829]: Leaving mDNS multicast group on interface igb1.12.IPv4 with address 192.168.12.1.

I see you are using Suricata. Be aware that if using Suricata with Inline IPS Mode (which uses the netmap kernel device), each time Suricata restarts it will physically cycle the interface it is running on. That means the interface will behave the same as if you manually did an "ifdn" followed by an "ifup" command sequence. Many network applications do not like the interface they are running on going offline and then coming back online. They will either crash, or automatically shut themselves down in that scenario.

To prevent Suricata from cycling the interface when updating rules, you can go to the GLOBAL SETTINGS tab and enable the option to "Live Update" the rules. That option is unchecked by default. The only real downside of enabling that option is RAM usage will briefly increase during the rules update job because Suricata will keep two complete copies of the rules in RAM as it updates. When the update is finished, it will discard the old copy of the rules.

nazuro

@bmeeks Thank you so much - that seems to have solved the issue! And thanks to John as well

bmeeks

@nazuro said in MDNS struggles:

@bmeeks Thank you so much - that seems to have solved the issue! And thanks to John as well

Glad that helped you.

What really happens is when Suricata stops and then restarts, it makes a call to the netmap device code to first "close" the open netmap interface (when stopping); and then it makes a call to "open" the netmap interface (when starting). The netmap kernel device performs a literal "down" and then "up" cycle as it switches the interface from and to netmap-mode (netmap-mode unhooks the interface from its normal kernel connections). You can see this in your log snippet:

Oct 13 00:30:47 pfSense kernel: igb1: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.11: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.12: link state changed to DOWN
Oct 13 00:30:47 pfSense kernel: igb1.14: link state changed to DOWN

followed later by:

Oct 13 00:30:49 pfSense kernel: igb1: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.11: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.12: link state changed to UP
Oct 13 00:30:49 pfSense kernel: igb1.14: link state changed to UP

Using the "Live Update" mode for rules updates means Suricata never actually stops and restarts. It keeps running during the update cycle and does the in-memory rules duplication/update thing I described earlier. Since it never restarts, it does not call the netmap code to close and then open the interface.