PIMD loosing multicast sources
-
Hello,
I'm using PIMD package to route multicast traffic.
When starting PIMD, after a few seconds it works as it should, seeing multicast sources and routing it if needed.
But after about 3 minutes, PIMD is "loosing" multicast sources even if pfSense still receive this multicast traffic (packet capures, and network traffic). PIMD does not "receive" multicast source anymore. Restarting PIMD makes it see again multicast sources until it looses it again after about 3 minutes.I opened a github issue on PIMD project but I am not sure if this is a pfsense/FreeBSD problem or a problem with this PIMD program.
I'm running pfSense CE 2.8.0.
Here is my pimd.conf generated file :##################### DO NOT EDIT THIS FILE! ###################### ################################################################### # This file was created by an automatic configuration generator. # # The contents of this file will be overwritten without warning! # ################################################################### phyint lagg0.3 enable igmpv2 phyint lagg0.99 enable igmpv2 phyint lagg0.50 enable igmpv2 phyint tun_wg1 enable phyint tun_wg2 disable bsr-candidate lagg0.3 rp-candidate lagg0.3 group-prefix 239.1.1.0/24 group-prefix 224.2.127.254/32What I tried :
- running official PIMD package (with PIMD software v2.3.2)
- compiling and runing PIMD 3.0beta1 (from github sources) because official PIMD 2.3.2 is very old (2016) : I have the same problem
- running a fresh new VM on another port on my switch, installing pfSense CE 2.8.0 and running official package PIMD (2.3.2) or compiled 3.0 : the same problem
I also noticed running netstat -gs while PIMD is running I have a lot of "multicast forwarding cache misses" and "datagrams with no route for origin" and "upcall queue overflows"
netstat -gs IPv4 multicast forwarding: 2277892556 multicast forwarding cache lookups 2254777122 multicast forwarding cache misses 265402 upcalls to multicast routing daemon 2253971473 upcall queue overflows 0 upcalls dropped due to full socket buffer 262429 cache cleanups 2254777122 datagrams with no route for origin 0 datagrams arrived with bad tunneling 0 datagrams could not be tunneled 132033 datagrams arrived on wrong interface 0 datagrams selectively dropped 0 datagrams dropped due to queue overflow 0 datagrams dropped for being too large IPv6 multicast forwarding: 0 multicast forwarding cache lookups 0 multicast forwarding cache misses 0 upcalls to multicast routing daemon 0 upcall queue overflows 0 upcalls dropped due to full socket buffer 0 cache cleanups 0 datagrams with no route for origin 0 datagrams arrived with bad tunneling 0 datagrams could not be tunneled 0 datagrams arrived on wrong interface 0 datagrams selectively dropped 0 datagrams dropped due to queue overflow 0 datagrams dropped for being too largeWhat does this means ?
Does someone manage to make PIMD work on pfSense ?
Is there a pfSense / FreeBSD configuration to make it work ?
Thanks a lot
-
Is it always about 3mins? Excatly?
That sounds like something expiring and not being renewed.
What exactly are you using it for?
It's always worked fine for me when I've tested it but I don't think I've ever tried to use it with wireguard.
-
The problem is weird : after launching PIMD, it starts seeing multicast source groups after about 1 minutes.
At this moment PIMD sees 30 groups (stream I have on my LAN, on the same subnet as my LAN on pfsense LAGG0.3).After 3min and 20 sec, PIMD "looses" some multicast groups, only seeing 8 groups. Those 8 groups are always the same : 239.1.1.11 // 239.1.1.12 // 239.1.1.14 // 239.1.1.15 // 239.1.1.16 // 239.1.1.30 // 239.1.1.78 and 2 other one.
After some time it looses every groups.I'm also thinking something is expiring but I don't understand what. Pfsense is still receiving the multicast traffic when the problem happen : I can see it with packet captures and with netstat -g :
netstat -g IPv4 Virtual Interface Table Vif Thresh Local-Address Remote-Address Pkts-In Pkts-Out 0 1 192.168.3.1 0 0 1 1 192.168.3.1 0 0 2 1 192.168.50.1 0 0 3 1 192.168.99.1 0 0 4 1 192.168.27.1 0 0 IPv4 Multicast Forwarding Table Origin Group Packets In-Vif Out-Vifs:Ttls 192.168.3.152 239.1.1.16 0 65535 192.168.3.152 239.1.1.17 0 65535 192.168.3.152 239.1.1.1 0 65535 192.168.3.31 224.2.127.254 0 65535 192.168.3.152 239.1.1.2 0 65535 192.168.3.152 239.1.1.18 0 65535 192.168.3.152 239.1.1.19 0 65535 192.168.3.152 239.1.1.3 0 65535 192.168.3.152 239.1.1.4 0 65535 192.168.3.152 239.1.1.20 0 65535 192.168.3.152 239.1.1.5 0 65535 192.168.3.152 239.1.1.21 0 65535 192.168.3.152 239.1.1.22 0 65535 192.168.3.152 239.1.1.6 0 65535 192.168.3.152 239.1.1.7 0 65535 192.168.3.152 239.1.1.23 0 65535 192.168.3.152 239.1.1.8 0 65535 192.168.3.152 239.1.1.24 0 65535 192.168.3.152 239.1.1.9 0 65535 192.168.3.152 239.1.1.25 0 65535 192.168.3.152 239.1.1.26 0 65535 192.168.3.152 239.1.1.10 0 65535 192.168.3.152 239.1.1.11 0 65535 192.168.3.152 239.1.1.12 0 65535 192.168.3.152 239.1.1.77 0 65535 192.168.3.152 239.1.1.13 0 65535 192.168.3.152 239.1.1.78 0 65535 192.168.3.152 239.1.1.14 0 65535 192.168.3.152 239.1.1.30 0 65535 192.168.3.152 239.1.1.15 0 65535 IPv6 Multicast Interface Table is empty IPv6 Multicast Forwarding Table is empty -
@stephenw10 said in PIMD loosing multicast sources:
What exactly are you using it for?
I'm streaming video multicast traffic from my network to other networks.
I have to say during the time PIMD "sees" sources, it works and I can join multicast groups on other networks, effectively routing this multicast group between router.
But when PIMD "looses" sources, it doesn't work anymore until I restart PIMD -
Nothing logged?
Are you actually using the wireguard interfaces?
-
@stephenw10
With official PIMD v2.3.2 and with manually build 3.0beta (from sources) with the config I showed above, in the logs I don't have a lot of info :pimd version 3.0-beta1 starting. and exitningand
Timeout waiting for reply from routing socket for 169.254.0.1I tried manually running PIMD from the CLI, with run in foreground and every debug options, and I see a lot of multicast forwarding cache misses, as expected with the results of netstats -gs above. But I don't really understand if this is normal
If you are running PIMD without problem, do you have the same results as me in netstats -gs ?
@stephenw10 said in PIMD loosing multicast sources:
Are you actually using the wireguard interfaces?
Yes I'm using it, without any problem, with OSPF on FRR on it.
-
So you see that timeout for the APIPA address at the 3min mark when it stops forwarding?
Can you test it without the WG interfaces in play?
I don't actually have PIMD running on anything currently. It just always has worked when I've tested it. I'll try to get something setup again.
-
@stephenw10 said in PIMD loosing multicast sources:
So you see that timeout for the APIPA address at the 3min mark when it stops forwarding?
No I get the "Timeout waiting for reply from routing socket for 169.254.0.1" at the same time the PIMD program is lauching.
When PIMD looses multicast groups, there is nothing on the logs. -
@stephenw10 said in PIMD loosing multicast sources:
Can you test it without the WG interfaces in play?
I tried removing WG interfaces from PIMD interfaces : I only have 3 VLANs interfaces on PIMD, with the same results
-
Here are the logs of manually running PIMD in foreground in CLI with this command : log.txt
The problem happen at 13:34:47.
I do not see any errors at this moment, but you can see before 13:34:47 there are reports of all multicast groups and after, only 8 of them.
I can see a lot of "cache miss" entries, but I don't know if this is normalif someone can help, I tried a lot of things but have no idea why I have this problem. Thanks
-
I don't think the cache misses are a problem. Certainly shouldn't prevent connectivity.
Did you have this working in 2.7.2?
-
@stephenw10 said in PIMD loosing multicast sources:
Did you have this working in 2.7.2?
Unfortunately I have an intel X710 NIC which had a bug in pfSense 2.7.2 which prevented it from using multicast
I'm going to reinstall a VM with pfsense 2.7.2 on another server (without intel X710) to see if the problem still occurs
-
@stephenw10
Ok. I just reinstalled pfSense 2.7.2 on a proxmox VM, giving access to my LAN.
I installed official package PIMD 2.3.2 on it.
I disabled pf (with pfctl disable).I have the exact same behavior !
Could it be a problem on my network ? On my switch ?
But if so, I would not receive multicast packet on tcpdump on pfSense (for groups PIMD is not seeing anymore after 3min20 sec, but still receiving with packet captures).I have a Ubiquiti EdgeSwitch 16XG. IGMP snooping is enabled on interfaces, and enable on corresponding VLANs.
Also I can see through CLI on this switch that when starting PIMD, pfSens's port is enable as a multicast router port, and I can see every multicast groups forwarded to this port.
I can also see on my switch when starting PIMD that pfSense is the multicast querier.Weird
-
I built PIMD from sources (to have the latest version) on a Debian VM : it looks like the problem is not present on Debian. So it looks like it is a pfSense / FreeBSD specific problem
-
Do you see any blocked traffic when this happens? The timing involved could be a state timeout. And I imagine you didn't have a firewall running in Debian.
-
@stephenw10 No I do not see any traffic blocked.
I don't have a firewall on this test Debian VM but, I also tried disabling pf on pfSense with pfctl -d while runing PIMD, without any change in this problem.
Also my multicast sources are on my LAN, with "Allow all" rule. -
Just to add information while I have the bug with PIMD not seeing sources, pfSense seem to still see it : in the state summary those IP are still here.
-
I built PIMD from sources on a FreeBSD 15 VM, and following this guide I launched PIMD :
On FreeBSD I have the exact same problem so it must be either a FreeBSD kernel bug or a bug with the way PIMD works on FreeBSD.If someone can confirm he has a working PIMD config (receiving sources during more than a few minutes) on pfSense 2.7.2 or 2.8.0 it would help me (I don't see a lot of people using PIMD on pfSense here). Thanks.
-
I am normally using PIMD to access my media server from multiple vlan's.
To that I am using a long ago compiled version of the beta just like you. However at this moment it is not working.
For unkown reasons, but recently I changed very much related to my pfSense system, To note a few:
- significant hardware changes
- running the pfSense plus beta now, having of course a much newer version of FreebSD
- largely changes to my firewall rules
However I surely like to see it running again. I need it to access my media server!
However .... I am busy at the moment and I have guests. So bad moment to recompile the beta (I never do that, so have to find out a lot again) and to do testing.
However keep me/us updated !!
-
Some more info.
I still had pimd installed however dispite started it was not running. So I wondered what whould happen if I installed the packages as compiled long ago again.
So:
- I accessed pf sense via SSH and copied pimd-3.0.b1.txz and pfSense-pkg-pimd-3.0.1.txz to the root its home directory
- removed the old / same packages
pkg remove pfSense-pkg-pimd-3.0.1
pkg remove pimd-3.0.b1 - installed them again from the roots home
pkg install pfSense-pkg-pimd-3.0.1
pkg install pimd-3.0.b1 - installed pimd from the pfsense package manager
I did not remove my pimd config so that config is still there - start pimd from the pimd service menu now present option pimd
- pimd starts and status shows info but the output is not showing pimd clients as expected. (see picture below)
- looking in the system logs System LogsSystemGeneral
I see that pimd has problems with the config
Jun 13 08:01:49 pfSense pimd[3561]: /var/etc/pimd/pimd.conf:14 - Invalid phyint address 'mlxen0.26'
Jun 13 08:01:49 pfSense pimd[3561]: /var/etc/pimd/pimd.conf:13 - Invalid phyint address 'lagg0.130'
Jun 13 08:01:49 pfSense pimd[3561]: /var/etc/pimd/pimd.conf:12 - Invalid phyint address 'bridge0'
Jun 13 08:01:49 pfSense pimd[3561]: Recommended querier timeout = Robustness x query-interval + response-time / 2 = 3 x 12 + 10 / 2 = 41
Jun 13 08:01:45 pfSense check_reload_status[678]: Syncing firewall
Jun 13 08:01:44 pfSense php-fpm[68271]: /pkg_edit.php: Configuration Change:
Jun 13 08:01:37 pfSense pimd[90461]: /var/etc/pimd/pimd.conf:14 - Invalid phyint address 'mlxen0.26'
Jun 13 08:01:37 pfSense pimd[90461]: /var/etc/pimd/pimd.conf:13 - Invalid phyint address 'lagg0.130'
Jun 13 08:01:37 pfSense pimd[90461]: /var/etc/pimd/pimd.conf:12 - Invalid phyint address 'bridge0'
Jun 13 08:01:37 pfSense pimd[90461]: Recommended querier timeout = Robustness x query-interval + response-time / 2 = 3 x 12 + 10 / 2 = 41
So on my machine PIMD (as said the version as compiled long ago) is running but not the way it should. I still have to find out why.
