After a lot of head scratching I finally managed to get it to work. I'll post the settings I used, in case someone reads this post later and possibly find them useful.
Services>Rules: WAN
Proto: UDP
Source: *
Source Port: *
Destination: 224.0.0.0/4
Destination Port: *
Services>Rules: LAN
Modify the "Default allow LAN to any rule" and enable the option "This allows packets with IP options to pass. Otherwise they are blocked by default. This is usually only seen with multicast traffic.". It is found under "Advanced Options" near the bottom of the page, when editing this rule.
(The next rule is specific for my service provider and can be skipped if your service provider doesn't have a similar service. In my case it provides access to the Warner Films channel.)
Services>NAT: Port Forwarding
Proto: UDP
Src. addr: *
Src ports: *
Dest. addr: WAN address
Dest. ports: 8254
NAT IP: <ip of="" iptv-box="">NAT Ports: 8254
Services>IGMP Proxy
Here I made two rules. One for upstream and one for downstream traffic.
Upstream
Name: WAN
Type: upstream
Threshold: <empty>Values: 224.0.0.0/4, 195.238.200.0/24 (*the second address mask specific to my service provider. See note below.)
Downstream
Name: LAN
Type: downstream
Threshold: <empty>Values: 192.168.0.0/24 (This is the subnet mask of my internal network. Change it to what you use.)
Notes
The second subnet mask I specified for the upstream is the net from where the data stream is coming from, in my case. Without this I get nothing but a black screen.
To debug what was going on, I ssh'd to my pfSense box and ran igmpproxy from the command line:
igmpproxy -d -c /tmp/igmpproxy.conf
After switching to an IPTV channel I noticed the following warning coming from igmpproxy:
Warn: The source address 195.238.200.2 for group 239.192.105.14, is not in any valid net for upstream VIF.
I checked the IP and turns out my service provider own the 195.238.200.0/24 network. After adding it to my upstream settings, I was able to watch tv.
For reference, I am using "Elisa Viihde", a Finnish IPTV solution.
pfSense version that I'm using is 2.0-BETA4 (i386) built on Fri Dec 10 20:38:22 EST 2010 running on a PC Engines ALIX 2D3 board.
Posts I found useful
http://forum.pfsense.org/index.php/topic,16943.msg113914.html#msg113914
http://forum.lissyara.su/viewtopic.php?f=3&t=17080#p158230 (in Russian)</empty></empty></ip>