Avahi not restarting at boot - Resolved (as of latest 23.09 dev build)

dennypage

Have you tried re-saving the config in the gui?

dennypage

@RobbieTT said in Avahi not restarting at boot:

I'm I missing a setting somewhere?

Disable IPv6 in the Avahi config.

RobbieTT

@stephenw10 said in Avahi not restarting at boot:

Is the ix1 NIC linked at boot when Avahi tries to use it? I assume Avahi starts fine manually after boot?
I could imagine between those times that ix1 might have changed status.

The ix1 is linked (SFP+ DAC) to my first switch and carries the LAN & VLAN. I am not sure of the relative timing of Avahi vs the interface coming up - I had presumed the interfaces would come up before the GUI itself started.

@dennypage said in Avahi not restarting at boot:

Have you tried re-saving the config in the gui?

Yes but I guess everything is fine by that point anyway. I have also removed and reinstalled the package plus numerous reboots and updates.

@dennypage said in Avahi not restarting at boot:

Disable IPv6 in the Avahi config.

I could try that for diagnostics but would prefer to have it enabled for production use.

️

RobbieTT

@dennypage @stephenw10

It worked first time with IPv6 disabled in Avahi.

Presumably either an interface race-condition or that Avahi tries to start too early in the sequence, before IPv6 is ready?

I didn't expect it to be a bug; borking it myself seemed more likely.

️

RobbieTT

@stephenw10 @dennypage

The log for the Avahi start at reboot without IPv6 enabled is attached below.

Even with just IPv4 enabled it seems to require a number of attempts to launch over a 90 second period:

Sep  7 19:03:21 Router-8 php-fpm[990]: /rc.start_packages: Restarting/Starting all packages.
Sep  7 19:03:21 Router-8 php-fpm[990]: /rc.start_packages: Starting service avahi
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: Successfully dropped root privileges.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: avahi-daemon 0.8 starting up.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: No service file found in /usr/local/etc/avahi/services.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: Joining mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: New relevant interface ix1.1003.IPv4 for mDNS.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: Joining mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: New relevant interface ix1.IPv4 for mDNS.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: Network interface enumeration completed.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: Server startup complete. Host name is Router-8.local. Local service cookie is 2998446032.
Sep  7 19:03:32 Router-8 root[12001]: Bootup complete
Sep  7 19:03:33 Router-8 php-fpm[991]: /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 9x.xx.xxx.x5 ->  9x.xx.xxx.x5 - Restarting packages.
Sep  7 19:03:33 Router-8 check_reload_status[1028]: Starting packages
Sep  7 19:03:34 Router-8 php-fpm[93828]: /rc.start_packages: Restarting/Starting all packages.
Sep  7 19:03:34 Router-8 php-fpm[93828]: /rc.start_packages: Stopping service avahi
Sep  7 19:03:34 Router-8 avahi-daemon[58191]: Got SIGTERM, quitting.
Sep  7 19:03:34 Router-8 avahi-daemon[58191]: Leaving mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:03:34 Router-8 avahi-daemon[58191]: Leaving mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:03:34 Router-8 avahi-daemon[58191]: avahi-daemon 0.8 exiting.
Sep  7 19:03:34 Router-8 php-fpm[93828]: /rc.start_packages: Starting service avahi
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: Successfully dropped root privileges.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: avahi-daemon 0.8 starting up.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: No service file found in /usr/local/etc/avahi/services.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: Joining mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: New relevant interface ix1.1003.IPv4 for mDNS.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: Joining mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: New relevant interface ix1.IPv4 for mDNS.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: Network interface enumeration completed.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: Server startup complete. Host name is Router-8.local. Local service cookie is 393772984.
Sep  7 19:04:49 Router-8 php-fpm[990]: /avahi_settings.php: Configuration Change: redacted@2xxx:xxx:xxxe:1:d440:c78d:b2f0:9fe0 (Local Database): Updated Avahi settings
Sep  7 19:04:49 Router-8 php-fpm[990]: /avahi_settings.php: Stopping service avahi
Sep  7 19:04:49 Router-8 avahi-daemon[62110]: Got SIGTERM, quitting.
Sep  7 19:04:49 Router-8 avahi-daemon[62110]: Leaving mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:04:49 Router-8 avahi-daemon[62110]: Leaving mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:04:49 Router-8 avahi-daemon[62110]: avahi-daemon 0.8 exiting.
Sep  7 19:04:49 Router-8 php-fpm[990]: /avahi_settings.php: Starting service avahi
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Successfully dropped root privileges.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: avahi-daemon 0.8 starting up.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: No service file found in /usr/local/etc/avahi/services.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Joining mDNS multicast group on interface ix1.1003.IPv6 with address 2xxx:xxx:xxxe:2:92ec:77ff:fe1b:70a9.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: New relevant interface ix1.1003.IPv6 for mDNS.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Joining mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: New relevant interface ix1.1003.IPv4 for mDNS.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Joining mDNS multicast group on interface ix1.IPv6 with address 2xxx:xxx:xxxe:1:92ec:77ff:fe1b:70a9.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: New relevant interface ix1.IPv6 for mDNS.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Joining mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: New relevant interface ix1.IPv4 for mDNS.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Network interface enumeration completed.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Server startup complete. Host name is Router-8.local. Local service cookie is 1284148945.
Sep  7 19:04:54 Router-8 php-fpm[991]: /avahi_settings.php: Configuration Change: redacted@2xxx:xxx:xxxe:1:d440:c78d:b2f0:9fe0 (Local Database): Updated Avahi settings
Sep  7 19:04:54 Router-8 php-fpm[991]: /avahi_settings.php: Stopping service avahi
Sep  7 19:04:54 Router-8 avahi-daemon[5251]: Got SIGTERM, quitting.
Sep  7 19:04:54 Router-8 avahi-daemon[5251]: Leaving mDNS multicast group on interface ix1.1003.IPv6 with address 2xxx:xxx:xxxe:2:92ec:77ff:fe1b:70a9.
Sep  7 19:04:54 Router-8 avahi-daemon[5251]: Leaving mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:04:54 Router-8 avahi-daemon[5251]: Leaving mDNS multicast group on interface ix1.IPv6 with address 2xxx:xxx:xxxe:1:92ec:77ff:fe1b:70a9.
Sep  7 19:04:54 Router-8 avahi-daemon[5251]: Leaving mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:04:54 Router-8 avahi-daemon[5251]: avahi-daemon 0.8 exiting.
Sep  7 19:04:54 Router-8 php-fpm[991]: /avahi_settings.php: Starting service avahi
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Successfully dropped root privileges.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: avahi-daemon 0.8 starting up.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: No service file found in /usr/local/etc/avahi/services.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Joining mDNS multicast group on interface ix1.1003.IPv6 with address 2xxx:xxx:xxxe:2:92ec:77ff:fe1b:70a9.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: New relevant interface ix1.1003.IPv6 for mDNS.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Joining mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: New relevant interface ix1.1003.IPv4 for mDNS.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Joining mDNS multicast group on interface ix1.IPv6 with address 2xxx:xxx:xxxe:1:92ec:77ff:fe1b:70a9.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: New relevant interface ix1.IPv6 for mDNS.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Joining mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: New relevant interface ix1.IPv4 for mDNS.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Network interface enumeration completed.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Server startup complete. Host name is Router-8.local. Local service cookie is 83068962.

️

dennypage

@RobbieTT said in Avahi not restarting at boot:

It worked first time with IPv6 disabled in Avahi.

Thought so.

@RobbieTT said in Avahi not restarting at boot:

Even with just IPv4 enabled it seems to require a number of attempts to launch over a 90 second period

What you are seeing there is restarts based upon interface configuration (IP) changes. This is normal behavior.

I expect that you are using DHCP to acquire an IPv6 delegation from the WAN, and then using it to configure internal interfaces via Track Interface.

Unless you are using fully static IPv6, I would recommend that you do not use IPv6 in Avahi. You don't loose anything by having IPv6 disabled in Avahi, and many devices that depend upon mDNS won't use IPv6 even if it is offered. IPv4 works just fine.

RobbieTT

@dennypage

Unfortunately I am somewhat reliant on IPv6 mDNS (HomePod & HomeKit Post iOS 15 onwards). Without it some internal device-to-device functions do not work correctly; IPv6 mDNS repeating gets me around that hurdle.

Ideally, either the Avahi start-up sequence could be changed or simply delayed when the user has the IPv6 flag enabled. IPv6 via DHCP is probably the norm for most IPv6 users.

Thanks for the explanation on the expected IPv4 behaviour and isolating the IPv6 issue.

️

dennypage

@RobbieTT said in Avahi not restarting at boot:

Unfortunately I am somewhat reliant on IPv6 mDNS (HomePod & HomeKit Post iOS 15 onwards). Without it some internal device-to-device functions do not work correctly; IPv6 mDNS repeating gets me around that hurdle.

Can you provide some detail about what internal device-to-device functions are not working?

I've done a bit of mDNS work (I wrote the pfSense Avahi package), but I am not aware of anything that Apple has that is IPv6 dependent with regard to mDNS. I have a fair bit of iOS, tvOS, MacOS and HomeKit entities, with a very rather large amount of mDNS entities across several LAN segments. I have IPv6 disabled in Avahi with no issues, so I am rather curious as to what you are seeing.

Ideally, either the Avahi start-up sequence could be changed or simply delayed when the user has the IPv6 flag enabled.

I'm afraid that a startup delay would likely introduce other problems for folk, and probably would not completely address the issue anyway. The problem would likely happen again whenever the IPv6 delegation changes. A proper fix would have to be implemented in Avahi itself.

RobbieTT

@dennypage said in Avahi not restarting at boot:

@RobbieTT said in Avahi not restarting at boot:
Can you provide some detail about what internal device-to-device functions are not working?

Sure, elements like device hand-off or the intercom either degrade or stop and accessing an iTunes server will just fail; specifically:

_daap._tcp. (Digital Audio Access Protocol (iTunes)) (port 3689)
_dacp._tcp. (Digital Audio Control Protocol (iTunes)) (port 49162)
_home-sharing._tcp. (iTunes Home Sharing) (port 3689)

I also had to change my RA mode to finally unlock it all agin post iOS15 et al - a suggestion by @stephenw10 to help with a different IPv6 issue but it added the final ingredient to get it all running perfectly.

There is probably more that I could add but it is getting late here. I'll add more detail where needed; I may even have the original pcaps.

️

dennypage

@RobbieTT said in Avahi not restarting at boot:

Sure, elements like device hand-off or the intercom either degrade or stop and accessing an iTunes server will just fail

Hmm... I'm thinking that there is another issue as Handoff works in my environment with IPv6 disabled in Avahi.

All Apple OSs function in environments where only IPv4 is available, so I don't see a circumstance in which they will not work without IPv6 mDNS. My initial guess would that you may have had a mDNS caching entity in your network that interfered.

FWIW, I looked in the other thread... seems like you had a lot of IPv6 problems. FWIW, I don't think @stephenw10 actually said that you should use RA Assisted, I believe he said that you should use RA Assisted unless you had a reason not to. And he followed up by saying that it isn't something he had spent much time looking at.

I run my networks RA Managed. I reserve/assign addresses for all hosts, and I have had no issues with any of the Apple clients.

RobbieTT

@dennypage

It's probably a bit harsh on Netgate to say I have had a lot of problems with IPv6 on Netgate hardware. Yes, I have stumbled into some bugs. especially with the aggravating and interconnecting PPPoE issues; but they have worked with me, are keeping a good issues log and have issued bug fixes for me.

It is unfortunate that 1 bug fix just unmasked another bug hidden underneath but that is how it goes sometimes. It is awkward or inconvenient at times and I have had to make plans to take pfSense out of production should I be away for a protracted period but I am far from throwing my 6100 out and giving up.

Crashes and reboots are clearly undesirable but Netgate are clearly engaged with their efforts to fix the mixed PPPoE / IPv6 / Interface problems. For my part I would have probably never noticed the IPv6 issue with pfSense Avahi if it wasn't for the frequent reboots and crashes caused by different problems. I am also cognisant that the mixed issues I have would simply not apply (for example) to a US user of Netgate devices.

So in sum, the experience is not great but it is getting better and the issues I have only apply to a subset of Netgate / pfSense users.

️

NogBadTheBad

@RobbieTT To be honest my experience with AVAHI wasn't very good.

I kept seeing multiple devices on my lan that were actually the same device with an _2, _3, etc .... in finder.

I gave up and added a multicast rule on each interface that was needed:-

Screenshot 2023-09-08 at 11.49.15.png

I also added the devices into the shared services for good measure on my APs:-

Screenshot 2023-09-08 at 11.51.20.png

It just works.

RobbieTT

@NogBadTheBad

I guess our individual setups make a difference with mDNS, especially with multiple APs and switches to traverse as well as direct point-to-point. My home has 6 APs broadcasting 2 subnets for mDNS to work with. It also has multiple hubs and gateway devices for Thread / HomeKit et al.

Aside from the Avahi IPv6 flag preventing robust restarts it does its job and everything can talk to each other internally and reach externally as required. I also have quite a few mDNS devices - this (redacted) snapshot is during a relatively quiet period:

‎Discovery png.‎001.png

Plenty going on.

️

dennypage

I kept seeing multiple devices on my lan that were actually the same device with an _2, _3, etc .... in finder.

Yep. What this means is that there are name collisions. How does this happen?

When a system initially connects to a network (such as boot) and sends out a name resolution query for the name it is configured for. So if you name the system "myhost", when it comes on the network it will send a mDNS resolution request for "myhost". If it receives a response saying that "myhost" can be reached at address X (it doesn't matter if it's the same address the host is on, or even if it's a different address family such as IPv4 vs IPv6), the host will append a sequential counter to the name and try again. "myhost-1", "myhost-2", and so on. It will continue until the resolution fails, and then it will settle on that name. This process ensures that no two hosts will have the same local name. [NB: this is the local mDNS name which is unrelated to a DHCP assigned name]

Assuming that you did not intentionally assign the same name to multiple hosts, for this collision to happen means that you have some form of mDNS caching in your network. Having a caching mDNS server was once thought to be a really great idea. It's not.

Avahi is able to operate as a mDNS cache. There was a very old version of the pfSense Avahi package that had this as an option, and it wreaked a lot of havoc. When I wrote the new (current) Avahi package, I hardcoded the configuration to disable caching.

It can be hard to locate the caching entity. There are several places to look:

The most common is a Linux system running Avahi with a legacy configuration. Check the avahi-daemon.conf file. If the configuration does not have a line that says "cache-entries-max=0", add it and restart the service.
Many WiFi APs have the ability to run a caching service. There are some vlan configurations where this is appropriate, but none in which Avahi is already present in both networks.
Embedded systems. These are generally systems that use old Linux versions as the base OS, and have Avahi installed with caching enabled. If this is the case, you have to reach out to the manufacturer to get them to fix it. FWIW, I've encountered this with a couple of AirPlay audio systems, and in both cases the manufacturer was responsive and happy to address the issue once it was explained to them.

If you are not able to identify the network cache, then you have to resort to sniffing on the host or use a SPAN port on your switch.

@NogBadTheBad, In your case, the issue I see is that you have caching enabled in your APs, and also have Avahi spanning networks with reflection. You should have one or the other, but not both.

NogBadTheBad

@dennypage I don’t actually have AVAHI running at all now and I’m also using a different brand AP.

I don’t have any issues now.

It been set up as per my previous post for a few years.

terryzb

@dennypage I don't want to hijack the thread but am wondering if you can recommend a novice guide or resources for setting up HomeKit / Thread / etc., ideally with a Netgate router? I have a simple network with a 2100 running Avahi for the AppleTV, iOS devices and printer, but have been considering looking at migrating away from X-10 home automation and wondering what that might entail.

dennypage

@terryzb said in Avahi not restarting at boot:

setting up HomeKit / Thread / etc.

Quite different technologies there, and as a general rule, unrelated to something like pfSense.

One thing I will say is that if you are looking to migrate from X-10, you have a big decision ahead of you, which is what core technology do you want to invest in? Z-Wave, Zigbee, Thread or Wifi. All have some advantages and disadvantages. I recommend that you seek guidance in a home automation forum regarding the relative benefits of each.

RobbieTT

@dennypage said in Avahi not restarting at boot:

I expect that you are using DHCP to acquire an IPv6 delegation from the WAN, and then using it to configure internal interfaces via Track Interface.

Unless you are using fully static IPv6, I would recommend that you do not use IPv6 in Avahi. You don't loose anything by having IPv6 disabled in Avahi, and many devices that depend upon mDNS won't use IPv6 even if it is offered. IPv4 works just fine.

On pfSense 23.09d-20230921-1219 Avahi does now restart at boot with IPv6 selected.

Hurrah!

️