How to get Bell Fibe in Quebec/Ontario (Internet and IPTV) working with pfSense

[stephenw10]

@patian
If you are seeing a partial reduction like 750Mbps down from 940Mbps it probably is the PPPoE limitation you're hitting. What hardware are you running on there?
Have you tried the suggested sysctl tweak?
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

Steve

[techanalyst]

@patian Getting every drop out of that line.....make sure the device you test with have all the power saving crap disabled on the nic, use speedtest cli to run your performance tests (they have it for every os). Also check your egress and make sure you have it set, for years I usually use 925-930Mbit, when i lived in Guelph this was enough to give me perfect uploads.

[stephenw10]

When testing throughput be sure to do so from a client behind the firewall. speedtest-cli can be run on pfSense but it uses significant CPU cycles to do so. You will always see a lower result on the firewall itself.

Steve

[techanalyst]

@stephenw10 This is true, its like installing iperf on the pfsense box, you get an idea but connect a device as the next hop and you can adjust for far better performance vs taking the pfsense boxes word for it

[Patian]

@techanalyst
Hi ,
Thank you for all the inputs. I have reported the issue with bell, they will send me new equipment to replace. I will keep everyone inform on the outcome in a few days after I receive the new equipment. It is because i am getting slow speed (better say inconsistent speed) on the Bell modem as well.

I have a Netgate SG-5100. I think using the netgate device will not have hardware compatibility issue with pfsense.

My test setup is very simple. There is only one laptop ethernet connected to the LAN1 port of Bell modem or igb 1 LAN port of SG-5100 and nothing else. Running Firefox browser with speedtest.net as a testing medium. I just swap the Bell modem with SG-5100 for direct comparison on the speedtest.

ISP only guarantee speed at their modem LAN port.

Should I stop the pfbockerNG and Suricata for the speed test on SG-5100?

All firmware and OS are up to date.

[Patian]

@techanalyst
I received the new bell modem today and used the build-in speedtest within the modem portal to do the test. At the modem interface, I constantly got 927-932M Download and 700M upload.

With iMac alone connected to the bell modem, the speed test was 932M Download and 654-700M upload.

With iMac connected to SG-5100, the speed test was 757-790M Download and 613-668M Upload.
The PPPoE seems to be the issue on the speed. There is no solution on it.

Should I keep the bell modem in the connection, use Double NAT/DMZ, SG-5100 in DHCP?

Any recommendation?

[josh256]

@patian If you don't need voice and your network supports VLANs I would create a dedicated interface on PfSense [and VLAN on the LAN] to contain the internal multicast (unless you have Cisco and/or Meraki who's IGMP snooping actually works in which case a single internal interface would suffice).

Edit: I am recommending you ditch the HH!

So, in total four interfaces:

Interfaces-->Assignments:
WAN_PPPoE [tagged VLAN 35, PPPoE]
WAN_IPTV [tagged VLAN 36, DHCP]
LAN [192.168.X/24]
LAN_IPTV. [192.168.2/24]

System-->RoutingStatic-->Routes
10.0.0.0/8 to WAN_IPTV_DHCP

Firewall-->Rules-->LAN_IPTV
permit any any etc... expand Advanced Options: Enable IP Options

Firewall-->Rules-->WAN_IPTV
permit any any etc... expand Advanced Options: Enable IP Options

Services-->IGMP Proxy:
WAN_IPTV upstream 224.0.0.0/4,10.0.0.0/8,192.168.1.0/24,192.168.2.0/24
LAN_IPTV downstream 192.168.2.0/24

Services-->DNS Forwarder: Custom Options
rebind-domain-ok=bell.ca
rebind-domain-ok=bell.com
server=/bell.ca/10.2.127.228
server=/bell.ca/10.2.127.196
server=/bell.com/10.2.127.228
server=/bell.com/10.2.127.196

[Patian]

Hi Josh256,
Thank you for your suggestion.

I already have wan_PPP0e(tagged VLAN 35, PPPoE) setup. It is working fine.

I do not use Bell Fibre TV, I watch most of the program on streaming. I do not have setup for IPTV. The phone line is branched out from the ONT.

My network setup is simple, Mostly internet traffic, streaming, using VLAN for security camera, IOT and guest network.

The issue I have is, I am not able to get the full internet bandwidth from the netgate device using PPPoE.

I subscribe to a 1GB plan (Max 940down, 750up), the best is 780down and 690 with netgate device.

If I use the Bell modem , i can get a full internet speed using the modem build-in speed test portal.

I am a bit disappointed with the money invested into the netgate device. Apart from that, everything seems to be working well.

I wonder if I should use double NAT and put back the Bell modem into the WAN interface and using DMZ on Bell modem for Netgate.

[josh256]

@patian

I'm on 1G Fibe FTTH vs. the 1.5G and am 830+ down, 720+ up -- I'm running PfSense virtualized (ESXi) on an old Core i3 (Gigabyte mini-itx Z77-WIFI)...

[Patian]

@josh256

You also do not get the full internet bandwidth using pfsense with PPPoE.

[josh256]

@patian PPPoE overhead is mainly irrelevant (8 bytes overhead on a 1500byte frame) Also not really even a PfSense constraint so much as a PHY+MAC constraint (NIC and SFP) --> certain Broadcom adaptors support >1Gbps by way of a firmware reflash and you should be able to get PfSense running 1.4Gbps+ per TX/RX directions with a the Bell ONT/GPON in one of those NICs..

[Patian]

@josh256
Hmm, Before the pfsense, I was getting close to 900down and 700up. I do not really mean to get full internet bandwidth from the Bell advertised speed.

I have a Netgate SG-5100 with the latest 21.02.2 version of pfsense installed. Currently, I get 750down and 620up. It is a significant decrease in speed.

With reference from another post, I was told it was due to how FreeBSD implement PPPoE. I guess most pfsense user's WAN setup is auto IP.

If someone can get a close to max internet speed from fibre Bell, I am very interested to know it is done.

I wonder if I let Bell modem does the PPPoE, with DMZ setting, auto DHCP to Netgate device, double NAT, auto IP on pfsense will get better speed.

[stephenw10]

Did you add the values recommended here yet?
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

Steve

[Patian]

@stephenw10
Thank you for your advise.
I am not aware of such setting. I will study it and give it a try.

[techanalyst]

@patian The link that stephenw10 gave has most of the things I followed when I used pfsense but I switched to untangle, mostly because its linux, and I use BBR, the speed differences for folks was significant, any speed test across the contry on bell I get almost my max speeds or better. login-to-view

[stephenw10]

Nice, I can only dream of a connection like that.

Yeah, if you have a PPPoE connection that fast you're unlikely to fill it with anything FreeBSD based while that single thread restriction exists. You would need something very fast for single thread use. Up to 1Gbps is possible on reasonable hardware though.

Steve

[techanalyst]

@stephenw10 said in How to get Bell Fibe in Quebec/Ontario (Internet and IPTV) working with pfSense:

Yeah, if you have a PPPoE connection that fast you're unlikely to fill it with anything FreeBSD based while that single thread restriction exists. You would need something very fast for single thread use. Up to 1Gbps is possible on reasonable hardware though.

Don't get me wrong, I love pfsense, theres so much it does out of box that just works and just works well. FreeBSD is a minor limiter here, nothing to do with pfsense. The machine I'm running this on is a dual 2699v4 :) I was able to tweak to ALMOST match, download actually wasn't any different, upload is where I saw my speed limits but were talking getting 890 vs 1000, keeping in mind sure I have a good upload but we don't see the drops on the ONT.

Linux just seems more flexible, BBR alone made all the world a difference for me and hosting my plex server, people can now stream 140Mbit remux blurays off me, with reno/cubic they can't.

[stephenw10]

Interesting. BBR should make no significant difference on a firewall/router as the TCP connections are not terminated there. It would all be on the forwarded hosts.

[techanalyst]

@stephenw10 Thats what you'd think and thats the theory, but I retested new reno/cubic on linux, folks max out at 35-55mbit from me out of the local province, within province they're fine, outside, the most stable bitrate is 20mbit, with BBR on the firewall, the absolutely only change I've made, they can pull original at 140 no issue, also did FTP/file share tests, same speed limit. So I agree with you, but the results tell us something different. We can saturate my pipe with BBR, anything else we can't and the only change is BBR, nothing on clients etc (cant change the dummy clients).

[techanalyst]

@stephenw10 Here's an example, this guy is located in Florida, he was never able to do more than a 10Mbit stream on me (Pfsense, reno or cubic, linux anything reno or cubic), with BBR, he's been doing these streams, starts as he says "instantly", never buffers and never pauses, totally not doable otherwise. He's also doing it over a wireless ISP and his apple tv is wireless....so its impressive.

login-to-view