New Avahi package
-
@xpxp2002 Hmm... with your configuration, Avahi should not emit any packets when it is restarted, so I'm not sure how restarting it would cause the device to start working again. The only thing Avahi does in that configuration is replay (reflect) mDNS packets received from one interface to the other interface. If you leave it alone for an extended period, does the device ever start working again on it's own?
One thing is to check the firewall log to see if there are any point to point packets being blocked between the devices on the different subnets.
Another thing to try would be disabling IPv6. I know, sounds weird. I haven't experienced this myself, but I've seen a few references to iDevices having issues that were resolved by disabling (or sometime enabling) IPv6.
Failing that, we're down to looking at packet dumps to figure out what is going on.
-
@dennypage I've looked at firewall logs, and didn't see anything that indicated an issue. It is worth noting that I recently noticed the same behavior with a printer that is mDNS-discoverable, as well.
I took some pcaps that I'd be happy to share with you privately. I don't expect that they'll have any sensitive information in them, other than say MAC addresses and whatnot. This is the strangest part of all. In the captures, I can see the iDevice switch continuing to send out its unsolicited mDNS query answers indefinitely. But when I look with Bonjour browser on the Mac or on the iPhone, the switch only appears in the hap.local domain while it is visible in the Home app for the first 5-10 minutes after I restart Avahi. Even though the mDNS multicasts continue to occur on the subnet, the device disappears from the mDNS browser at the same time that it disappears from the Home app. That's the part I don't understand. The only step left that I can think to do is look at the mDNS answers before and after it disappears from the mDNS browser and see if anything is changing in the packet that might cause it to be disappearing.
I don't see the device acquiring an IPv6 address, but in the meantime since it's quick and simple I'll try turning off IPv6 in Avahi.
-
@dennypage Thanks for your help troubleshooting this. In an interesting development tonight, the issue may not be related to the reflector (as you may have already suspected). I installed an updated firmware that just became available to my APs and switches this evening, and so far I've gone about an hour without the issue presenting. I'm going to keep an eye on it over the next 24 hours and see if I can confirm that it is no longer a problem.
-
The Avahi package also appears to have issues with GRE tunnel. I have a IPSEC site to site tunnel using a VTI interface. At both sites the ipsec tunnels are up and the gre tunnels are up as well. When Avahi starts it does not register the service on the GRE tunnel and the VTI tunnels even though both have been selected in the interface list.
-
@xpxp2002 Cool.
-
@sammybernard This would generally be a limitation of Avahi itself, rather than the pfSense package.
Not sure I can be of much help here, but some questions that come to mind:
-
By "does not register the service" do you mean that it doesn't bind to the interface? Or something else?
-
Do you have Avahi reflection on both ends?
-
Have you restarted Avahi (both ends) after the tunnels are up?
-
Do any of the interfaces have the POINTOPOINT flag set?
-
Have you separately confirmed multicast connectivity through the tunnel?
-
Have you done any packet sniffing to determine which hop is not working?
-
-
@dennypage
—— By "does not register the service" do you mean that it doesn't bind to the interface? Or something else?I have selected 4 interfaces for a Avahi bind to, LAN1, LAN2, GRETunnel and IPSEcVTI Interface. Avahi only binds to LAN1 and 2 at both end points of IPSec tunnels before the interface enumeration message appears and does. It bind to any further interfaces. Minimal log info to see why it failed to bind to the GREUnnel or IPSEcVTI interfaces.
Do you have Avahi reflection on both ends?
Yes Reflection is enabled at. Othbend points.
Have you restarted Avahi (both ends) after the tunnels are up?
Yes have tested restarting Avahi and entire routers as well.
Do any of the interfaces have the POINTOPOINT flag set?
No interface has any poittopoint flag set.
Have you separately confirmed multicast connectivity through the tunnel?
Prior to AVAHi 2.0.0 update it worked well with the same tunnel setup but noting after 2.0.0.0 update.
Have you done any packet sniffing to determine which hop is not working?
Yes, packet capture only shows the keep alive pings going across gretunnel but no other Avahi packets. It this is. It surprising since Avahi does not seem to be binding to the greinterface.
-
@sammybernard said in New Avahi package:
I have selected 4 interfaces for a Avahi bind to, LAN1, LAN2, GRETunnel and IPSEcVTI Interface. Avahi only binds to LAN1 and 2 at both end points of IPSec tunnels before the interface enumeration message appears and does. It bind to any further interfaces. Minimal log info to see why it failed to bind to the GREUnnel or IPSEcVTI interfaces.
Avahi doesn't actually bind to interfaces. It binds to wildcards.
Please post the following:
- Name and description of each of the interfaces involved
- Contents of /usr/local/etc/avahi/avahi-daemon.conf
- System log messages for process "avahi"
- Output of ifconfig for each of the interfaces involved
- Output of ifconfig -a -n | grep 5353
-
This post is deleted! -
@dennypage
So after some digging into thing it appears that IPSEC endpoints and GRE endpoints would be considered point to point links and hence why avahi is failing. The default behavior is for POINT-to-POINT links be ignored and the new package gui does not seem to allow for this "allow-point-to-point=yes" flag to be set. If I manually edit the /usr/local/etc/avahi/avahi-daemon.conf to add this entry it seems avahi then latches on to the GRE and IPSEC interfaces and now we can have multicast messages across the IPSec tunnels between two different sites. This might be an important GUI option since a lot of folks might want to use avahi reflector functioning to enable multicast over an IPSec tunnel and GRE would be the only way to do it which would be a point-to-point link. -
@sammybernard Yea, I thought that the interfaces might be point-to-point.
Addressed in 2.0.0_2. Pull request pending.
-
@dennypage I spoke too soon on my issue. It began occurring again on Thursday, even with the new switch and AP firmware. I think that may have been a coincidence, though I'm not sure why it was functional without intervention for several days.
That being said, I did some reading on mDNS and took some more pcaps. I did confirm that IGMP snooping is not interfering. I'm trying several changes, one at a time, to see if any make a difference. Right now, I'm keeping an eye on the possibility that having a VIP on one of the reflection interfaces might be causing the problem.
I have my pfBlockerNG DNSVL VIP on the LAN interface (hn1), and I noticed that Avahi listens on that interface with both IPs. I've turned off the DNSBL feature for now, which removes the VIP, and so far I've gone about 2 hours without a problem. I'll keep an eye on it for up to a week, and let you know if the issue seems to be permanently resolved with the VIP removed.
-
@xpxp2002 Thank you for the update. Just out of curiosity, how did you confirm the IGMP snooping functionality on the wifi devices?
-
@dennypage My pcaps show that Avahi is sending a pair of IGMPv3 joins for 224.0.0.251 when it starts, then no additional control packets from Avahi are showing up on the wire afterward, even on a capture that I let run 20 minutes. Perhaps my understanding of how Avahi works is wrong, but shouldn't I see periodic IGMP membership reports from Avahi for this mcast group?
The switches and AP have separate IGMP snooping settings, of which I've tried every combination of snooping on and off for the VLANs connected to each interface (hn1 and hn3) and each combination of having it enabled on all hardware, disabled on all hardware, only enabled on switches, and only enabled on APs. No matter what combination I use, snooping does not seem to affect the behavior at all.
When the switch snooping is enabled, the switches will act as and elect a querier by default. I've read that in some environments, people have had success keeping mDNS working by using an IGMP querier (though I'd expect Avahi to fulfill that role if a querier on a switch is not present) because snooping will aggressive prune ports that don't generate queries or reports to indicate that they want to remain joined to the group. Given that I'm not seeing any membership reports over 20 minutes, even with snooping completely off, could this be causing the device to stop participating in mDNS?
And for what it's worth, we're going on about 4 hours now with the pfBlockerNG VIP removed from hn1 and still haven't seen the culprit device drop from mDNS yet. Again, not sure if this is coincidence and whether the issue lies with a pfSense behavior or IGMP, but it's a promising result so far.
-
@xpxp2002 said in New Avahi package:
@dennypage My pcaps show that Avahi is sending a pair of IGMPv3 joins for 224.0.0.251 when it starts, then no additional control packets from Avahi are showing up on the wire afterward, even on a capture that I let run 20 minutes. Perhaps my understanding of how Avahi works is wrong, but shouldn't I see periodic IGMP membership reports from Avahi for this mcast group?
Just for clarity, applications such as Avahi do not send IGMP packets, operating systems do.
Simplified version... when the first application on a host joins a multicast group on an interface, the OS sends an initial IGMP message indicating that it has joined the group. When the last application on the host leaves the group, the OS will send a IGMP message indicating that it is leaving the group. In between the join and leave, It will not send another membership message for the group unless it receives a specific request from a router or a switch to confirm that it is still a member of the group.
Debugging IGMP multicast issues can be a serious pain. Unless you have a great deal of confidence in the implementations, I would simply disable IGMP snooping. Particularly for wifi. Unless you have a large amount of multicast in your network that you need to prune, it isn't going to have a negative impact. Just make sure that multicast forwarding is enabled (if such a setting exists in your gear).
-
Thanks. I have edited the avahi.inc file for the moment while waiting the package update. Just want to make sure if the package got updated via GUI then the avahi-deamon.conf gets updated with the point to point flag.
-
@sammybernard said in New Avahi package:
Thanks. I have edited the avahi.inc file for the moment while waiting the package update. Just want to make sure if the package got updated via GUI then the avahi-deamon.conf gets updated with the point to point flag.
Not sure if you are asking about the behavior with the current version 2.0.0_1 or the coming 2.0.0_2...
With 2.0.0_1, any change to the configuration will cause avahi-daemon.conf to be overwritten, and the manual edit for allow-point-to-point flag will be lost.
With 2.0.0_2 and beyond, the config file will always be written with the allow-point-to-point flag set. Given that the interface list is positive selection only, there really isn’t a reason to make the point-to-point setting configurable via the GUI.
I expect the new version will be available shortly after folk return from holiday next week. You can follow the PR using the link above.
-
Same package, another question :
In the good old days, I could see what avahi "sees" doing by running :avahi-browse -a -vOr, now :
avahi-browse -a -v Failed to create client object: Daemon not runningWith the help of Google and the left mouse button, I found out that in the file
/usr/local/etc/avahi/avahi-daemon.conf
This lineenable-dbus=nois hard coded to "no".
Making itenable-dbus=yesrewriting the config and .....
avahi-browse -a -v Server version: avahi 0.7; Host name: pfsense.local E Ifce Prot Name Type Domain + fxp0 IPv6 pfsense [00:12:3f:b3:58:75] _workstation._tcp local + fxp0 IPv4 pfsense [00:12:3f:b3:58:75] _workstation._tcp local + sis0 IPv6 pfsense [00:0f:b5:fe:4e:e7] _workstation._tcp local + sis0 IPv4 pfsense [00:0f:b5:fe:4e:e7] _workstation._tcp local + fxp0 IPv6 pfsense _ssh._tcp local + fxp0 IPv4 pfsense _ssh._tcp local + sis0 IPv6 pfsense _ssh._tcp local ......Thoughts ?
-
@gertjan Yes, this is intentional. There are no local mDNS browse clients for pfSense, so there isn't much use for dbus support on the firewall itself. Further dbus was the cause of a couple of significant issues, one being the minimum 5 second startup delay, and the other being a sporadic failure of Avahi to start at boot for many users.
If you want to see what is in the network, I would recommend doing this from a general workstation or laptop in the network. This will also give you a better view into the overall functionality of reflection. There are several tools that support this. If you are a Mac user, then there is a free application called "Discovery" that is pretty nice. For a Unix based system, you can use avahi-discover (GUI) or avahi-browse (command line). I haven't used Windows in many years, but I'm sure there are some decent tools there as well.
-
@dennypage said in New Avahi package:
@gertjan Yes, this is intentional. There are no local mDNS browse clients for pfSense, so there isn't much use for dbus support on the firewall itself. Further dbus was the cause of a couple of significant issues, one being the minimum 5 second startup delay, and the other being a sporadic failure of Avahi to start at boot for many users
Ok, get it - the only browser that exist on the firewall was ..... avahi-browser ^^ (this one needs avahi - logic, and mbus I guess).
I have the Discovery app my Mac (iPhone) : great tool !
Thanks for the detailed explanation.
