23.01.b.20230106.0600 IGMP proxy stops TV stream

DBMandrake

@stephenw10 Sorry for the noise in the thread, you were right about the checksum offloading.

I temporarily disabled IGMP snooping on the switch (to stop it filtering/handling the requests itself) and was able to observe the IGMP reports from the upstream side of IGMP proxy at another device arriving safely across the network with correct checksums. So checksums is not my issue so I never bothered to try turning off hardware checksuming.

However something really weird is going on - when the switch which is normally the IGMP querier (10.0.2.1) is the querier, it is somehow suppressing IGMP membership reports from the upstream interface of IGMP proxy on PFSense, (10.0.1.254) but when I make a different switch the querier (10.0.2.3 - the switch PFsense is connected to) suddenly IGMP proxy starts sending the upstream IGMP membership reports that it should be and it starts to work - at least partially, as I was able to find and play an SSDP advertised multicast stream between VLAN's.

A lot more debugging required on my part to figure out what's going on. One problem I have is a large mix of different switch models, with some of the older ones not necessarily playing nicely - I have a suspicion there is at least one other switch on the network which is not following the gentlemans rule of "don't be IGMP querier when a lower IP address device is already sending IGMP queries" as I have seen intermittent changes in the multicast router destination pointing towards these suspect downstream switches even though they are explicitly configured to never be an IGMP querier!

Beerman

Unfortunately, I have the same problem here. Just updated from 22.05 to

23.01-RC (amd64)
built on Wed Feb 08 14:19:05 UTC 2023

I can confirm that the IGMP proxy does not work and IPTV with this version unfortunately then also not.

Then the bug probably also applies to "CE 2.7.0", right?

stephenw10

I would expect it to be the same in 2.7 currently, yes.

thebear

@stephenw10 @jimp

Is threre any way the fix will make it into 23.01? That would be very appreciated.

https://redmine.pfsense.org/issues/13929

stephenw10

Not at this point, at least not in the release build. There is no fix yet as I understand it and the release is imminent. If it's something that can be patched at runtime it can be added to the system patches package later.

Beerman

This is not good news...
Then I have to skip this update. :(

stephenw10

I agree it's unfortunate but we couldn't hold the release for this. As soon as we pin down the issue we'll know more.

tohil

@stephenw10 is this issue affecting all IGMP proxy setups? Not just particullar driver or configuration. Combinations?

stephenw10

I haven't been able to replicate it here yet so I can't say for sure. However I would expect it to be something that affects all NICs/drivers. Other multicast traffic, like CARP, does not appear affected so it looks like something in igmpproxy or the libraries it uses.

Steve

mxt3rs

I faced the same issue when upgraded to 23.01, Disable hardware checksum offload didnt help, bad checksum was in the packet capture
after reverting to BE 22.05 no bad checksum and ip tv works,
my interface is I226-V

jrueger

It would be nice if this issue was included in the release notes for 23.01. I read through all the documentation for 23.01 and then proceeded with the upgrade only to find that igmp-proxy did not work. Several hours of troubleshooting lead me to this forum post where it appears that the issue was known for some time before the release of 23.01. This is not the introduction to Netgate and pfSense that I wanted, having just migrated to Netgate products from Ubiquiti Unifi products specifically for the igmp-proxy functionality. Oh well, now I'll get an early introduction to the process for rolling back upgrades.

thebear

@jrueger said in 23.01.b.20230106.0600 IGMP proxy stops TV stream:

Oh well, now I'll get an early introduction to the process for rolling back upgrades.

Well that's not new to UniFi users right ;-) but yes I think the dev's underestimate the impact of this bug for home deployments. The list get longer and longer when users hit the update button.

At least it could be documented under know issues.

jrueger

@thebear - no kidding with the Unifi gateways ... I finally abandoned Ubiquiti because of the opaqueness of their processes and capabilities. They did finally introduce igmp-proxy functionality in the latest release but with no documentation ... only a checkbox in the GUI and no explanation of just what exactly was enabled when checking the box. So far I am enjoying pfSense immensely so I can live with the igmp-proxy issue since I get to tinker more with things. My wife wasn't happy that her television show was interrupted!

stephenw10

Yes, I apologise, it should have been in the notes. The issue was undefined, and still is to some extent. Hopefully we can pin it down shortly.

Steve

mxt3rs

@stephenw10
issue seems to be from upstream FreeBSD-14 , I was able reproduce checksum errors on FreeBSD-14.0-CURRENT-amd64-20230216-2894c8c96b9b-260969 while freeBSD 13.1-Release doesn't have the issue, pcap attached
13.pcap
14.pcap

mxt3rs

I think i found the issue updated bugtracker, I am attaching patch + binary , it is working for me now, if someone else can confirm that would be great

igmpproxy.tar.gz

thebear

@mxt3rs said in 23.01.b.20230106.0600 IGMP proxy stops TV stream:

I think i found the issue updated bugtracker, I am attaching patch + binary , it is working for me now, if someone else can confirm that would be great

igmpproxy.tar.gz

Sure happy to test, TV is currently in use so need to wait until tomorrow. File is loaded and ready for testing.

thebear

@mxt3rs very good work! IGMP if flowing and the TV stream also.

Every membership query is answered:

cmcdonald

@thebear Thanks for the report! I've got the patch staged up and ready to bring into the tree. Until then, for anyone who wants a pkg that can be installed, I have bundled up the patched igmpproxy and attached it to the redmine issue:

https://redmine.pfsense.org/issues/13929#note-14

jrueger

And then feeling brave and daring I went and managed to install the pkg on my 2100, not paying attention to the note that the pkg was for amd64 ... and now igmpproxy is borked and won't start (nice red x under Status - Services). Such is the life of a newbie to pfSense (if only I actually had some sense ).