Deploy disk images with inter-VLANs mulicast?

lucasg

Hello,

We use FOG software ( https://fogproject.org/ ) to deploy disk images on multiples PCs with multicast. (FOG use udpcast / udp-sender / udp-receiver sofware for multicast).

We have several VLANs. Our FOG server is in an another VLAN than the PCs. On our actual router, HP HSR6602-XG with Comware 7, we use "pim sm" on the network interface of the FOG server and we use "pim sm" AND "igmp enable" on the network interfaces of the PCs. It works from years.

We will replace our HP 6600 with Netgate XG-1541 ( https://www.netgate.com/solutions/pfsense/xg-1541.html ) with PFSense 2.4.5-RELEASE-p1 + FreeBSD 11.3-STABLE. Inter-VLAN multicast does not work. We enabled IGMP proxy. We disabled PF (pfctl -d). We installed and enabled the software package pimd ( https://forum.netgate.com/topic/149909/new-package-pimd ). We haven't configured pimd. On first time, we have tried igmp proxy without additional configuration. After, we have configured upstream = WAN (see the reason below) + downstream = PCs networks interfaces. After, we have configured multicast rendezvous address on FOG server (because we read it on FOG forum). All this has failed.

When the PCs try to join the multicast group, IGMP proxy log theses lines for each PCs (10.4.251.11 is one of theses PCs) :
"No interfaces found for source 10.4.251.11
RECV V3 member report from 10.4.251.11 to 224.0.0.22"

Questions :

Is "igmp enable" (without configuration) + "pim sm" (without configuration) on HP 6600 enable the same features than IGMP proxy (without configuration) + pimd on PFSense (without configuration) ?
- In particular, PFSense allow only one network interface. We have two FOG servers in two separate VLANs. Our HP routeur does the job, so I really think "igmp enable" (HP) != igmp proxy. Does anyone have an opinion?
IGMP proxy doesn't start when WAN interface isn't configured as upstream. But, our FOG server is located in another VLAN, not WAN! I think that why we have the error message "No interfaces found for source". All our network interfaces on PFSense are VLANs interfaces on the same aggregation. Configure the default IP gateway (default route) on an another interface don't allow to use it as upstream interface in IGMP proxy (igmp proxy still fail at startup). Anyone can confirm that we use an other interface than WAN as upstream interface?
I'm not sure: for our usage (deploy disk images with udpcast/FOG), do we need IGMP proxy AND pimd OR only IGMP proxy OR only pimd? As I wrote, on our actual HP router, we have IGMP AND pimd.
Anyone have a working PFSense + FOG + inter-VLANs multicast setup? Can you give me your parameters?

Regards.

stephenw10

Generally pimd replaces igmp proxy in the applications I have seen it used.

I would install and configure that and disable igmpproxy and see what happens.

You should certainly be able to select multiple interfaces there.

I have not seen that specific software previously though.

Steve

amassi

Hi,

We make a lot of try. You have to know that the configuration is clustered (carp). And also that there are 50 vlans and 40 virtuals ips... may be a bit more.
So we try to start pimd with Default Bind to all and to Defaut to none. In Bind to none we put all interfaces we have exept one in Interface Binding in nerver binding. We never started pimd service. I find this information because we have the same comportement. https://github.com/troglobit/pimd/issues/39. Do you know if the compilation options in question in this thread are effective in a pfsense 2.4.5-RELEASE-p1 (amd64)?
About IGMP proxy it s look worse. Igmpproxy start but is not effective on some interface but not the vlan we realy need. Impossible to determine the logic which makes that the service starts on certain vlan (3 or 4) and none of the 40 others.
the only criteria would be: too much virtual interface on the vlan and then too high interface number.
Best regards

stephenw10

I don't see that set here: https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pimd/Makefile

amassi

@stephenw10 can we add this option ?

stephenw10

Maybe. Better to open it as a feature request here:
https://redmine.pfsense.org/projects/pfsense-packages

Steve

amassi

Hi Steve, I ask some help to a friend who is good about this subject.

As indicated, MAXVIFS must be identical to the one in the kernel.

And the FreeBSD kernel equals 32.
(type:
grep '#define.*MAXVIFS' /usr/src/sys/netinet/ip_mroute.h
#define MAXVIFS 32
)

So prior to pimd's MAXVIFS evolution, the kernel has to be patched, modifying the value.

This is why it has never been stated as important to enable its configuration through the Makefile of the port, because it remains independent from the kernel value.

For information purpose, where does this 32 limit in the kernel comes from (Linux, BSDs, ...)?
Consult page https://flylib.com/books/en/3.475.1.87/1/
Picture 17-11: (CPU 32 bits heritage)

The FreeBSD code is as follows:
74 /*
75 * Types and macros for handling bitmaps with one bit per virtual interface.
76 /
77 #define MAXVIFS 32
78 typedef u_long vifbitmap_t;
79 typedef u_short vifi_t; / type of a vif index */
80 #define ALL_VIFS (vifi_t)-1

=> To know wether the VIFS are configured or not, there is a board (vifbitmap_t) in which each of its bits indicates if the corresponding VIF is created or not.
Historically, on a 32bits machine, the u_long provides by default a 32bits variable, restricting then the MAXVIFS to 32.
With today's 64bits machines, the default MAXVIFS value could be set to 64 (but not more!)

So can we make pfsense manage it totaly ? Is it possible for you ?
I think this not the only issue there is to review about pimd.
the second point is to try to understand why the option Default Bind to Bind to none make pimd already scan every interface and ip in the system.

amassi

@stephenw10 done for the feature request in redmine

lucasg

Hello,

After months of work with @amassi, here is our feedback.

Multicast accross VLANs works with igmpproxy on pfSense <= 21.02.2-RELEASE but there are several cumulative constraints:

Only one upstream interface so only one VLAN can send multicast at once.
- In theory, pimd (available in additionnal package) permits several upstream interfaces but it's totally buggy (when we start it, it tries to bind() on each network interface so it exceeds MAXVIFS kernel value - 32 - so it crashes. Obviously, it ignores its configuration file in which we have disabled unwanted network interfaces and it still tries to bind() on all interfaces).

Only 32 VLANs with multicast enabled at the same time (upstream + downstreams). Cause: MAXVIFS = 32 in FreeBSD kernel.
- When we add CARP on each VLAN, the limit becomes 16 multicast-VLANs activable in igmpproxy. Cause: igmpproxy sees each VIP as a network interface so it tries to bind() on it and reaches MAXVIFS.
- The more VIP we add on multicast-enabled interfaces, the less number of multicast-available interfaces we have. Same cause.

A multicast-enabled interface can't have more than six VIPs on it. Otherwise igmpproxy refuses to start.

A multicast-enabled VLAN must be in the XX first VLANs listed in Interface > Assignments > VLANs (all our VLANs are configured on lagg0). Otherwise igmpproxy don't bind() on it (no log message "adding VIF, Ix XX Fl 0x0 IP 0xXXXXXXXX lagg0.XXX).
- On fresh install XX = 22. With CARP (for routing purpose) on all of our VLANs, XX = 21 (obviously, only VIPs on multicast-enabled VLANs are counted). With CARP for routing and destination NAT, XX = 20. If we add additionnal VIPs on these multicast-enabled VLANs, XX = 19.
- We have moved our VLANs with a lot of VIPs at the end of the list => they are not counted.
- We have added "parking" VLANs (unused VLAN IDs) in 17 th, 18 th, 19 th position in Interface > Assignments > VLANs. If we need to add VIPs on multicast-enabled VLANs, we will delete them. If we need to enable multicast on new VLAN, we will replace one ununsed VLAN ID by the new one and so VLAN will be in the 20 first multicast-activable VLANs.

In addition to these limits, we had an unknown problem with our FOG setup. We have installed a new storage node (in FOG terminology) and attached it to our existing FOG server => multicast works. New storage node has same OS and same FOG version (1.5.7) than the old one. For multicast, FOG uses the udpcast tool. sha256sum of updcast binaries are equal. So no idea of the root cause, but we now have a working inter-VLANs multicast FOG server with pfSense.

Finally, our XG-1541 reboot when we plug DAC cable in Chelsio's port and igmpproxy is enabled. At reboot, web gui prints core dump. Disable igmpproxy before plug DAC = no crash.

In summary: if you want to use inter-VLANs multicast with pfSense, you need to use igmpproxy + take previously-listed limits into account + maybe reinstalle your FOG storage node.

Bye