IGMP issues causing ISP to perform multicast DOS on my pfSense
-
Hi guys
I am using my pfSense FW as a WAN router/firewall and also as an IGMP proxy server for controlling IPTV multicast streams from my ISP.
I have set up my pfSense firewall as the IGMP proxy, but I think I have screwed up the configuration somewhere. I've managed to get the IGMP messages flowing to a certain degree. I can switch channels on my IPTV tuner and get the multicast streams streaming - but for some reason I suspect the "IGMP leave reports" aren't returned properly back to my ISP. The result, as you can imagine, is that my ISP is bombarding me with X multicast streams causing the load of my pfSense to hit the roof and become unstable. This usually happens when 9-10 streams are active simultanously and the multicast bandwidth usage from my ISP to the pfsense FW exceeds 90MBit/s.
The only way I work around this issue at the moment is to disconnect power from the fiber/ethernet converter between my pfSense and the ISP.
My IPTV tuner is on a separate LAN named TVLAN and directly connected to my pfSense FW. My other computers are behind another router which WAN port is connected to one of the ports in the pfSense FW. I have added a diagram of my network topology to clarify any misunderstandings.
I am positive that this is not related to the firewall rules since I can verify with tcpdump that there are indeed IGMP messages going on both vr0 (WAN) and vr1 (TVLAN). No IGMP packages are sent to vr2 (LAN).
Here are my igmpproxy settings:
##------------------------------------------------------ ## Enable Quickleave mode (Sends Leave instantly) ##------------------------------------------------------ quickleave phyint vr0 upstream ratelimit 0 threshold 2 altnet 148.122.7.125/24 altnet 93.91.111.0/24 phyint vr1 downstream ratelimit 0 threshold 2 altnet 10.10.0.0/24 phyint vr2 disabledTo further clarify, my WAN interface is vr0, my TVLAN interface is vr1 and LAN is vr2.
Here is example output from tcpdump showing IGMP packets on vr0 (the WAN interface). As seen, IGMP packets are sent from pfSense to my ISP.
[2.1-RELEASE][root@morannon.me.lan]/root(36): tcpdump -ivr0 igmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vr0, link-type EN10MB (Ethernet), capture size 96 bytes 18:42:27.891721 IP 148.122.7.125 > all-systems.mcast.net: igmp query v2 18:43:15.315588 IP ti0140a400-1922.bb.online.no > igmp.mcast.net: igmp v3 report, 1 group record(s) 18:43:15.316639 IP ti0140a400-1922.bb.online.no > 233.171.129.255: igmp v2 report 233.171.129.255 18:43:19.418903 IP ti0140a400-1922.bb.online.no > igmp.mcast.net: igmp v3 report, 1 group record(s) 18:43:19.419725 IP ti0140a400-1922.bb.online.no > 233.171.129.217: igmp v2 report 233.171.129.217 18:43:48.795802 IP ti0140a400-1922.bb.online.no > 224.0.39.17: igmp v2 report 224.0.39.17 18:43:48.797620 IP ti0140a400-1922.bb.online.no > igmp.mcast.net: igmp v3 report, 1 group record(s) 18:43:53.638593 IP ti0140a400-1922.bb.online.no > igmp.mcast.net: igmp v3 report, 1 group record(s) 18:43:53.639964 IP ti0140a400-1922.bb.online.no > 224.0.39.17: igmp v2 report 224.0.39.17 ^C 9 packets captured 196448 packets received by filter 0 packets dropped by kernelAnd here are the IGMP packets from vr1 (TVLAN) in the same time period as the dump above. The IPTV tuner sends the packets to the igmpproxy server running on the pfSense FW. I have changed the channel on the IPTV tuner during the dump and also returned to the original channel.
[2.1-RELEASE][root@morannon.me.lan]/root(1): tcpdump -ivr1 igmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vr1, link-type EN10MB (Ethernet), capture size 96 bytes 18:43:14.912876 IP morannon.me.lan > all-systems.mcast.net: igmp query v2 18:43:14.915432 IP morannon.me.lan > all-systems.mcast.net: igmp query v3 18:43:15.311056 IP 10.10.0.200 > 233.171.129.255: igmp v2 report 233.171.129.255 18:43:16.962510 IP morannon.me.lan > all-routers.mcast.net: igmp v2 report all-routers.mcast.net 18:43:19.415822 IP 10.10.0.200 > 233.171.129.217: igmp v2 report 233.171.129.217 18:43:48.618457 IP 10.10.0.200 > all-routers.mcast.net: igmp leave 233.171.129.217 18:43:48.620029 IP morannon.me.lan > 233.171.129.217: igmp query v2 [gaddr 233.171.129.217] 18:43:48.621219 IP morannon.me.lan > all-systems.mcast.net: igmp query v3 [gaddr 233.171.129.217] 18:43:48.622795 IP morannon.me.lan > 233.171.129.217: igmp query v2 [gaddr 233.171.129.217] 18:43:48.623429 IP morannon.me.lan > all-systems.mcast.net: igmp query v3 [gaddr 233.171.129.217] 18:43:48.793386 IP 10.10.0.200 > 224.0.39.17: igmp v2 report 224.0.39.17 18:43:53.634198 IP 10.10.0.200 > 224.0.39.17: igmp v2 report 224.0.39.17 ^C 12 packets captured 69587 packets received by filter 0 packets dropped by kernelOne thing I notice is that when the IPTV tuner sends IGMP leave message to the igmpproxy at the pfSense FW, the igmpproxy doesn't just forward the IGMP leave message. Instead it sends it as an "igmp v3 report" with 1 group record. Could it be that my ISP does not support this?
Here is an example:
TUNER to FW - 18:43:48.618457 IP 10.10.0.200 > all-routers.mcast.net: igmp leave 233.171.129.217 FW to ISP - 18:43:48.797620 IP ti0140a400-1922.bb.online.no > igmp.mcast.net: igmp v3 report, 1 group record(s)Thanks for reading!
edit: some further clarifications and fix of spelling errors.
-
Update.
Spent a couple of hours this weekend trying to forge IGMPv2 packets using nmap/nping and wrote the following script that generates an IGMPv2 leave report. Using nmap, I sent the packet from my FW towards my ISP. By the way, 233.171.129.255 is the destination address of one of the multicast streams that I was trying to stop.
local nmap = require "nmap" local stdnse = require "stdnse" local io = require "io" local bin = require "bin" local packet = require "packet" local ipOps = require "ipOps" prerule = function() if nmap.address_family() ~= 'inet' then stdnse.print_verbose("s is IPv3 only.", SCRIPT_NAME) return false end if ( not(nmap.is_privileged()) ) then stdnse.print_verbose("%s not running due to lack of privileges.", SCRIPT_NAME) return false end return true end local igmpLeave; --- Sends an IGMP leave report igmpLeave = function(interface, version, ipAddress) local igmpPacket = bin.pack(">C", 0x17) -- Leave group query stdnse.print_debug("%s: Creating IGMP leave packet...", SCRIPT_NAME) igmpPacket = igmpPacket .. bin.pack(">C", 0x00) -- Max response time: 0 seconds igmpPacket = igmpPacket .. bin.pack(">S", 0x00) -- Checksum. Calculated later. --igmpPacket = igmpPacket .. bin.pack(">I", 0xe1010104) -- 225.1.1.4 igmpPacket = igmpPacket .. bin.pack(">I", ipOps.todword(ipAddress)) -- 224.0.39.84 3758106452 E0002754 -- 233.171.129.255 3920331263 E9AB81FF -- 233.171.129.75 3920331083 E9AB814B igmpPacket = igmpPacket:sub(1,2) .. bin.pack(">S", packet.in_cksum(igmpPacket)) .. igmpPacket:sub(5) stdnse.print_debug("%s: Packet created!", SCRIPT_NAME) return igmpPacket end action = function(host, port) --- START stdnse.print_debug("--- START ---") local version = stdnse.get_script_args(SCRIPT_NAME .. ".version") or 2 local leaveMe = stdnse.get_script_args(SCRIPT_NAME .. ".ip") or "233.171.129.255" local interface = nmap.get_interface() local responses, results = {}, {} local result if interface==nil then return ("ERROR: Failed to retrieve interface information.") end interface = nmap.get_interface_info(interface) local srcip = interface.address local dstip = "224.0.0.2" --local dstip = "233.171.129.255" stdnse.print_debug("%s: Will use %s interface.", SCRIPT_NAME, interface) stdnse.print_debug("%s: Will use IGMP version %s.", SCRIPT_NAME, version) -- if we want to listen for results: -- local co = stdnse.new_thread(igmpListener, interface, timeout, responses) local igmp_leave = igmpLeave(interface, version, leaveMe) local ip_raw = bin.pack("H", "45c00040ed780000010218bc0a00c8750a00c86b") .. igmp_leave local igmp_packet = packet.Packet:new(ip_raw, ip_raw:len()) igmp_packet:ip_set_bin_src(ipOps.ip_to_str(srcip)) igmp_packet:ip_set_bin_dst(ipOps.ip_to_str(dstip)) igmp_packet:ip_set_len(#igmp_packet.buf) igmp_packet:ip_count_checksum() local sock = nmap.new_dnet() sock:ethernet_open(interface.device) local eth_hdr = bin.pack("HAH", "01 00 5e 00 00 01", interface.mac, "08 00") sock:ethernet_send(eth_hdr .. igmp_packet.buf) sock:ethernet_close() stdnse.print_debug("%s: Packet sent!.", SCRIPT_NAME) stdnse.print_debug("--- STOP ---") --- STOP endTo send the packets I installed nmap on my pfSense FW and ran the following command:
nmap -e vr0 -d --script igmpLeave.nseIf I listen on the vr0 interface while I run this command, the following entry appears:
[2.1-RELEASE][root@morannon.me.lan]/root(1): tcpdump -ivr0 igmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vr0, link-type EN10MB (Ethernet), capture size 96 bytes 08:05:05.058830 IP ti0140a400-1922.bb.online.no > all-routers.mcast.net: igmp leave 233.171.129.255As the above output confirms, the packet was sent on the interface, but this did not close the multicast stream either..! I think I need to study the IGMP protocol more to fully understand this..
-
v3gard, where you able to resolve this issue? I currently have the same issue with the same ISP (Telenor).
-
v3gard, where you able to resolve this issue?
Yes I did, but not with pfSense unfortunately. I ended up using a MikroTik router that used IGMPv2 natively, with the same configuration as I posted earlier for my old pfSense unit. I am certain that you can get this to work with pfSense also once you figure out the IGMPv2/IGMPv3 issues, but I had to make priorities as my time was required elsewhere.
-
Sorry for digging this out again, but right now I have the same issue as mentioned by v3gard with pfSense2.2
@v3gard: Are you still using MikroTik or have you tried it with pfSense again? Let me guess, you've asked on the ISP's forums already, because I saw someone posting there about this problem (after reading a guide of tuxone ;))
Is someone else in the community having the same issue or maybe a possible solution? Could this be a bug in pfSense itself?
-
@v3gard: Are you still using MikroTik or have you tried it with pfSense again?
Still using MikroTik. Works like a charm!
@wenga:Let me guess, you've asked on the ISP's forums already, because I saw someone posting there about this problem (after reading a guide of tuxone ;))
Nope, not me (I think).I don't think this is an ISP issue as my ISP has standardized on IGMPv2. The problem is that I couldn't get IGMPv3 (which was installed on pfSense) to work seamless with IGMPv2. My main problem was that the active streams never disconnected.
@wenga:Could this be a bug in pfSense itself?
I assume you can solve the problem if you manage to compile a version of the igmp tools where you force IGMPv2 - but I haven't tried it myself.
-
Nope, not me (I think).I don't think this is an ISP issue as my ISP has standardized on IGMPv2. The problem is that I couldn't get IGMPv3 (which was installed on pfSense) to work seamless with IGMPv2. My main problem was that the active streams never disconnected.
Oh, I thought you were that user, since you also switched to MikroTik. But based on your tcpdumps you are on a different ISP. I'm currently on Swisscom.
I assume you can solve the problem if you manage to compile a version of the igmp tools where you force IGMPv2 - but I haven't tried it myself.
I'm not sure if this is going to help. I configured FreeBSD now that it should use only v2 as default. Additionally I set the force_igmp_version in sysctl to "2". I have read somewhere, that igmpproxy uses v2 as default for downstreams. For upstreams it uses the version configured in the kernel igmp-module of the FreeBSD.
This didn't really help for now. But based on your description, my problem might be a different one. At least the symptoms are different. As soon as I configure an upstream IGMP proxy to my ISP, my ISP router is restarting itself. This happens over and over again, as long as the upstream is active.
Looking at the tcpdump didn't help at all. I just see that there are two IGMPv2 and IGMPv3 packets, both seem to be okay. After that the ISP router is gone (about 10 seconds after).
Does anyone have any idea how I can solve this problem?