Avahi not restarting at boot - Resolved (as of latest 23.09 dev build)

RobbieTT

I don't think it has ever restarted for me (v23 onwards)

[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: ls /usr/local/etc/avahi/services
sftp-ssh.service.disabled ssh.service.disabled

Those 2 files mentioned are there, no idea of the significance though.

stephenw10

I'm failing to replicate this. How do you have Avahi configured?

RobbieTT

@stephenw10

I only set it via the GUI:

2023-09-07 at 15.00.44.png

These feels like a user error, perhaps I have something that conflicts with it?

Is there any particular files I should be looking at rather than the GUI?

️

RobbieTT

@stephenw10
Perhaps this is more helpful:

[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root: cat /usr/local/etc/avahi/avahi-daemon.conf 
# This file is generated by the pfSense Avahi package.
# Do not edit this file, it will be overwritten automatically.

[server]
allow-interfaces=ix1.1003,ix1
allow-point-to-point=yes
use-ipv4=yes
use-ipv6=yes
enable-dbus=no
cache-entries-max=0

[wide-area]
enable-wide-area=no

[publish]
disable-publishing=yes
publish-addresses=no
publish-hinfo=no
publish-workstation=no
publish-domain=no
publish-aaaa-on-ipv4=no
publish-a-on-ipv6=no
disable-user-service-publishing=yes

[reflector]
enable-reflector=yes
[23.09-DEVELOPMENT][admin@Router-8.redacted.me]/root:

️

stephenw10

Hmm, interesting that it shows 2 interfaces in the conf but 4 n the gui. Are two of those not available when it starts perhaps?

RobbieTT

@stephenw10
The GUI shows 2 selected out of the 4 displayed (VLAN and LAN). These are the only subnets that require mDNS traversal.

The lack of clarity or contrast between selected and unselected list items in the pfSense GUI is annoying for Safari users but it is what it is.

️

stephenw10

Hmm, my config is identical in 23.09 and starts every time. The only thing that might be different is the interface status there.

RobbieTT

@stephenw10
If I use a Chromium browser (Brave, in this example) my GUI looks like this:

2023-09-07 at 15.57.35.png

[ix1 = LAN and ix1.1003 = VLAN]

Is that more akin to what you are expecting to see?

️

stephenw10

Yeah, the contrast was low but I see it. That's how my test devices are configured except no VLANs.

Is the ix1 NIC linked at boot when Avahi tries to use it? I assume Avahi starts fine manually after boot?
I could imagine between those times that ix1 might have changed status.

dennypage

Have you tried re-saving the config in the gui?

dennypage

@RobbieTT said in Avahi not restarting at boot:

I'm I missing a setting somewhere?

Disable IPv6 in the Avahi config.

RobbieTT

@stephenw10 said in Avahi not restarting at boot:

Is the ix1 NIC linked at boot when Avahi tries to use it? I assume Avahi starts fine manually after boot?
I could imagine between those times that ix1 might have changed status.

The ix1 is linked (SFP+ DAC) to my first switch and carries the LAN & VLAN. I am not sure of the relative timing of Avahi vs the interface coming up - I had presumed the interfaces would come up before the GUI itself started.

@dennypage said in Avahi not restarting at boot:

Have you tried re-saving the config in the gui?

Yes but I guess everything is fine by that point anyway. I have also removed and reinstalled the package plus numerous reboots and updates.

@dennypage said in Avahi not restarting at boot:

Disable IPv6 in the Avahi config.

I could try that for diagnostics but would prefer to have it enabled for production use.

️

RobbieTT

@dennypage @stephenw10

It worked first time with IPv6 disabled in Avahi.

Presumably either an interface race-condition or that Avahi tries to start too early in the sequence, before IPv6 is ready?

I didn't expect it to be a bug; borking it myself seemed more likely.

️

RobbieTT

@stephenw10 @dennypage

The log for the Avahi start at reboot without IPv6 enabled is attached below.

Even with just IPv4 enabled it seems to require a number of attempts to launch over a 90 second period:

Sep  7 19:03:21 Router-8 php-fpm[990]: /rc.start_packages: Restarting/Starting all packages.
Sep  7 19:03:21 Router-8 php-fpm[990]: /rc.start_packages: Starting service avahi
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: Successfully dropped root privileges.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: avahi-daemon 0.8 starting up.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: No service file found in /usr/local/etc/avahi/services.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: Joining mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: New relevant interface ix1.1003.IPv4 for mDNS.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: Joining mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: New relevant interface ix1.IPv4 for mDNS.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: Network interface enumeration completed.
Sep  7 19:03:21 Router-8 avahi-daemon[58191]: Server startup complete. Host name is Router-8.local. Local service cookie is 2998446032.
Sep  7 19:03:32 Router-8 root[12001]: Bootup complete
Sep  7 19:03:33 Router-8 php-fpm[991]: /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 9x.xx.xxx.x5 ->  9x.xx.xxx.x5 - Restarting packages.
Sep  7 19:03:33 Router-8 check_reload_status[1028]: Starting packages
Sep  7 19:03:34 Router-8 php-fpm[93828]: /rc.start_packages: Restarting/Starting all packages.
Sep  7 19:03:34 Router-8 php-fpm[93828]: /rc.start_packages: Stopping service avahi
Sep  7 19:03:34 Router-8 avahi-daemon[58191]: Got SIGTERM, quitting.
Sep  7 19:03:34 Router-8 avahi-daemon[58191]: Leaving mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:03:34 Router-8 avahi-daemon[58191]: Leaving mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:03:34 Router-8 avahi-daemon[58191]: avahi-daemon 0.8 exiting.
Sep  7 19:03:34 Router-8 php-fpm[93828]: /rc.start_packages: Starting service avahi
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: Successfully dropped root privileges.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: avahi-daemon 0.8 starting up.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: No service file found in /usr/local/etc/avahi/services.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: Joining mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: New relevant interface ix1.1003.IPv4 for mDNS.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: Joining mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: New relevant interface ix1.IPv4 for mDNS.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: Network interface enumeration completed.
Sep  7 19:03:34 Router-8 avahi-daemon[62110]: Server startup complete. Host name is Router-8.local. Local service cookie is 393772984.
Sep  7 19:04:49 Router-8 php-fpm[990]: /avahi_settings.php: Configuration Change: redacted@2xxx:xxx:xxxe:1:d440:c78d:b2f0:9fe0 (Local Database): Updated Avahi settings
Sep  7 19:04:49 Router-8 php-fpm[990]: /avahi_settings.php: Stopping service avahi
Sep  7 19:04:49 Router-8 avahi-daemon[62110]: Got SIGTERM, quitting.
Sep  7 19:04:49 Router-8 avahi-daemon[62110]: Leaving mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:04:49 Router-8 avahi-daemon[62110]: Leaving mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:04:49 Router-8 avahi-daemon[62110]: avahi-daemon 0.8 exiting.
Sep  7 19:04:49 Router-8 php-fpm[990]: /avahi_settings.php: Starting service avahi
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Successfully dropped root privileges.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: avahi-daemon 0.8 starting up.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: No service file found in /usr/local/etc/avahi/services.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Joining mDNS multicast group on interface ix1.1003.IPv6 with address 2xxx:xxx:xxxe:2:92ec:77ff:fe1b:70a9.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: New relevant interface ix1.1003.IPv6 for mDNS.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Joining mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: New relevant interface ix1.1003.IPv4 for mDNS.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Joining mDNS multicast group on interface ix1.IPv6 with address 2xxx:xxx:xxxe:1:92ec:77ff:fe1b:70a9.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: New relevant interface ix1.IPv6 for mDNS.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Joining mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: New relevant interface ix1.IPv4 for mDNS.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Network interface enumeration completed.
Sep  7 19:04:49 Router-8 avahi-daemon[5251]: Server startup complete. Host name is Router-8.local. Local service cookie is 1284148945.
Sep  7 19:04:54 Router-8 php-fpm[991]: /avahi_settings.php: Configuration Change: redacted@2xxx:xxx:xxxe:1:d440:c78d:b2f0:9fe0 (Local Database): Updated Avahi settings
Sep  7 19:04:54 Router-8 php-fpm[991]: /avahi_settings.php: Stopping service avahi
Sep  7 19:04:54 Router-8 avahi-daemon[5251]: Got SIGTERM, quitting.
Sep  7 19:04:54 Router-8 avahi-daemon[5251]: Leaving mDNS multicast group on interface ix1.1003.IPv6 with address 2xxx:xxx:xxxe:2:92ec:77ff:fe1b:70a9.
Sep  7 19:04:54 Router-8 avahi-daemon[5251]: Leaving mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:04:54 Router-8 avahi-daemon[5251]: Leaving mDNS multicast group on interface ix1.IPv6 with address 2xxx:xxx:xxxe:1:92ec:77ff:fe1b:70a9.
Sep  7 19:04:54 Router-8 avahi-daemon[5251]: Leaving mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:04:54 Router-8 avahi-daemon[5251]: avahi-daemon 0.8 exiting.
Sep  7 19:04:54 Router-8 php-fpm[991]: /avahi_settings.php: Starting service avahi
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Found user 'avahi' (UID 558) and group 'avahi' (GID 558).
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Successfully dropped root privileges.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: avahi-daemon 0.8 starting up.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: No service file found in /usr/local/etc/avahi/services.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Joining mDNS multicast group on interface ix1.1003.IPv6 with address 2xxx:xxx:xxxe:2:92ec:77ff:fe1b:70a9.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: New relevant interface ix1.1003.IPv6 for mDNS.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Joining mDNS multicast group on interface ix1.1003.IPv4 with address 172.16.1.1.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: New relevant interface ix1.1003.IPv4 for mDNS.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Joining mDNS multicast group on interface ix1.IPv6 with address 2xxx:xxx:xxxe:1:92ec:77ff:fe1b:70a9.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: New relevant interface ix1.IPv6 for mDNS.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Joining mDNS multicast group on interface ix1.IPv4 with address 10.0.1.1.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: New relevant interface ix1.IPv4 for mDNS.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Network interface enumeration completed.
Sep  7 19:04:54 Router-8 avahi-daemon[15744]: Server startup complete. Host name is Router-8.local. Local service cookie is 83068962.

️

dennypage

@RobbieTT said in Avahi not restarting at boot:

It worked first time with IPv6 disabled in Avahi.

Thought so.

@RobbieTT said in Avahi not restarting at boot:

Even with just IPv4 enabled it seems to require a number of attempts to launch over a 90 second period

What you are seeing there is restarts based upon interface configuration (IP) changes. This is normal behavior.

I expect that you are using DHCP to acquire an IPv6 delegation from the WAN, and then using it to configure internal interfaces via Track Interface.

Unless you are using fully static IPv6, I would recommend that you do not use IPv6 in Avahi. You don't loose anything by having IPv6 disabled in Avahi, and many devices that depend upon mDNS won't use IPv6 even if it is offered. IPv4 works just fine.

RobbieTT

@dennypage

Unfortunately I am somewhat reliant on IPv6 mDNS (HomePod & HomeKit Post iOS 15 onwards). Without it some internal device-to-device functions do not work correctly; IPv6 mDNS repeating gets me around that hurdle.

Ideally, either the Avahi start-up sequence could be changed or simply delayed when the user has the IPv6 flag enabled. IPv6 via DHCP is probably the norm for most IPv6 users.

Thanks for the explanation on the expected IPv4 behaviour and isolating the IPv6 issue.

️

dennypage

@RobbieTT said in Avahi not restarting at boot:

Unfortunately I am somewhat reliant on IPv6 mDNS (HomePod & HomeKit Post iOS 15 onwards). Without it some internal device-to-device functions do not work correctly; IPv6 mDNS repeating gets me around that hurdle.

Can you provide some detail about what internal device-to-device functions are not working?

I've done a bit of mDNS work (I wrote the pfSense Avahi package), but I am not aware of anything that Apple has that is IPv6 dependent with regard to mDNS. I have a fair bit of iOS, tvOS, MacOS and HomeKit entities, with a very rather large amount of mDNS entities across several LAN segments. I have IPv6 disabled in Avahi with no issues, so I am rather curious as to what you are seeing.

Ideally, either the Avahi start-up sequence could be changed or simply delayed when the user has the IPv6 flag enabled.

I'm afraid that a startup delay would likely introduce other problems for folk, and probably would not completely address the issue anyway. The problem would likely happen again whenever the IPv6 delegation changes. A proper fix would have to be implemented in Avahi itself.

RobbieTT

@dennypage said in Avahi not restarting at boot:

@RobbieTT said in Avahi not restarting at boot:
Can you provide some detail about what internal device-to-device functions are not working?

Sure, elements like device hand-off or the intercom either degrade or stop and accessing an iTunes server will just fail; specifically:

_daap._tcp. (Digital Audio Access Protocol (iTunes)) (port 3689)
_dacp._tcp. (Digital Audio Control Protocol (iTunes)) (port 49162)
_home-sharing._tcp. (iTunes Home Sharing) (port 3689)

I also had to change my RA mode to finally unlock it all agin post iOS15 et al - a suggestion by @stephenw10 to help with a different IPv6 issue but it added the final ingredient to get it all running perfectly.

There is probably more that I could add but it is getting late here. I'll add more detail where needed; I may even have the original pcaps.

️

dennypage

@RobbieTT said in Avahi not restarting at boot:

Sure, elements like device hand-off or the intercom either degrade or stop and accessing an iTunes server will just fail

Hmm... I'm thinking that there is another issue as Handoff works in my environment with IPv6 disabled in Avahi.

All Apple OSs function in environments where only IPv4 is available, so I don't see a circumstance in which they will not work without IPv6 mDNS. My initial guess would that you may have had a mDNS caching entity in your network that interfered.

FWIW, I looked in the other thread... seems like you had a lot of IPv6 problems. FWIW, I don't think @stephenw10 actually said that you should use RA Assisted, I believe he said that you should use RA Assisted unless you had a reason not to. And he followed up by saying that it isn't something he had spent much time looking at.

I run my networks RA Managed. I reserve/assign addresses for all hosts, and I have had no issues with any of the Apple clients.

RobbieTT

@dennypage

It's probably a bit harsh on Netgate to say I have had a lot of problems with IPv6 on Netgate hardware. Yes, I have stumbled into some bugs. especially with the aggravating and interconnecting PPPoE issues; but they have worked with me, are keeping a good issues log and have issued bug fixes for me.

It is unfortunate that 1 bug fix just unmasked another bug hidden underneath but that is how it goes sometimes. It is awkward or inconvenient at times and I have had to make plans to take pfSense out of production should I be away for a protracted period but I am far from throwing my 6100 out and giving up.

Crashes and reboots are clearly undesirable but Netgate are clearly engaged with their efforts to fix the mixed PPPoE / IPv6 / Interface problems. For my part I would have probably never noticed the IPv6 issue with pfSense Avahi if it wasn't for the frequent reboots and crashes caused by different problems. I am also cognisant that the mixed issues I have would simply not apply (for example) to a US user of Netgate devices.

So in sum, the experience is not great but it is getting better and the issues I have only apply to a subset of Netgate / pfSense users.

️