How to get Bell Fibe in Quebec/Ontario (Internet and IPTV) working with pfSense

[Patian]

@techanalyst
Hi ,
Thank you for all the inputs. I have reported the issue with bell, they will send me new equipment to replace. I will keep everyone inform on the outcome in a few days after I receive the new equipment. It is because i am getting slow speed (better say inconsistent speed) on the Bell modem as well.

I have a Netgate SG-5100. I think using the netgate device will not have hardware compatibility issue with pfsense.

My test setup is very simple. There is only one laptop ethernet connected to the LAN1 port of Bell modem or igb 1 LAN port of SG-5100 and nothing else. Running Firefox browser with speedtest.net as a testing medium. I just swap the Bell modem with SG-5100 for direct comparison on the speedtest.

ISP only guarantee speed at their modem LAN port.

Should I stop the pfbockerNG and Suricata for the speed test on SG-5100?

All firmware and OS are up to date.

[Patian]

@techanalyst
I received the new bell modem today and used the build-in speedtest within the modem portal to do the test. At the modem interface, I constantly got 927-932M Download and 700M upload.

With iMac alone connected to the bell modem, the speed test was 932M Download and 654-700M upload.

With iMac connected to SG-5100, the speed test was 757-790M Download and 613-668M Upload.
The PPPoE seems to be the issue on the speed. There is no solution on it.

Should I keep the bell modem in the connection, use Double NAT/DMZ, SG-5100 in DHCP?

Any recommendation?

[josh256]

@patian If you don't need voice and your network supports VLANs I would create a dedicated interface on PfSense [and VLAN on the LAN] to contain the internal multicast (unless you have Cisco and/or Meraki who's IGMP snooping actually works in which case a single internal interface would suffice).

Edit: I am recommending you ditch the HH!

So, in total four interfaces:

Interfaces-->Assignments:
WAN_PPPoE [tagged VLAN 35, PPPoE]
WAN_IPTV [tagged VLAN 36, DHCP]
LAN [192.168.X/24]
LAN_IPTV. [192.168.2/24]

System-->RoutingStatic-->Routes
10.0.0.0/8 to WAN_IPTV_DHCP

Firewall-->Rules-->LAN_IPTV
permit any any etc... expand Advanced Options: Enable IP Options

Firewall-->Rules-->WAN_IPTV
permit any any etc... expand Advanced Options: Enable IP Options

Services-->IGMP Proxy:
WAN_IPTV upstream 224.0.0.0/4,10.0.0.0/8,192.168.1.0/24,192.168.2.0/24
LAN_IPTV downstream 192.168.2.0/24

Services-->DNS Forwarder: Custom Options
rebind-domain-ok=bell.ca
rebind-domain-ok=bell.com
server=/bell.ca/10.2.127.228
server=/bell.ca/10.2.127.196
server=/bell.com/10.2.127.228
server=/bell.com/10.2.127.196

[Patian]

Hi Josh256,
Thank you for your suggestion.

I already have wan_PPP0e(tagged VLAN 35, PPPoE) setup. It is working fine.

I do not use Bell Fibre TV, I watch most of the program on streaming. I do not have setup for IPTV. The phone line is branched out from the ONT.

My network setup is simple, Mostly internet traffic, streaming, using VLAN for security camera, IOT and guest network.

The issue I have is, I am not able to get the full internet bandwidth from the netgate device using PPPoE.

I subscribe to a 1GB plan (Max 940down, 750up), the best is 780down and 690 with netgate device.

If I use the Bell modem , i can get a full internet speed using the modem build-in speed test portal.

I am a bit disappointed with the money invested into the netgate device. Apart from that, everything seems to be working well.

I wonder if I should use double NAT and put back the Bell modem into the WAN interface and using DMZ on Bell modem for Netgate.

[josh256]

@patian

I'm on 1G Fibe FTTH vs. the 1.5G and am 830+ down, 720+ up -- I'm running PfSense virtualized (ESXi) on an old Core i3 (Gigabyte mini-itx Z77-WIFI)...

[Patian]

@josh256

You also do not get the full internet bandwidth using pfsense with PPPoE.

[josh256]

@patian PPPoE overhead is mainly irrelevant (8 bytes overhead on a 1500byte frame) Also not really even a PfSense constraint so much as a PHY+MAC constraint (NIC and SFP) --> certain Broadcom adaptors support >1Gbps by way of a firmware reflash and you should be able to get PfSense running 1.4Gbps+ per TX/RX directions with a the Bell ONT/GPON in one of those NICs..

[Patian]

@josh256
Hmm, Before the pfsense, I was getting close to 900down and 700up. I do not really mean to get full internet bandwidth from the Bell advertised speed.

I have a Netgate SG-5100 with the latest 21.02.2 version of pfsense installed. Currently, I get 750down and 620up. It is a significant decrease in speed.

With reference from another post, I was told it was due to how FreeBSD implement PPPoE. I guess most pfsense user's WAN setup is auto IP.

If someone can get a close to max internet speed from fibre Bell, I am very interested to know it is done.

I wonder if I let Bell modem does the PPPoE, with DMZ setting, auto DHCP to Netgate device, double NAT, auto IP on pfsense will get better speed.

[stephenw10]

Did you add the values recommended here yet?
https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#pppoe-with-multi-queue-nics

Steve

[Patian]

@stephenw10
Thank you for your advise.
I am not aware of such setting. I will study it and give it a try.

[techanalyst]

@patian The link that stephenw10 gave has most of the things I followed when I used pfsense but I switched to untangle, mostly because its linux, and I use BBR, the speed differences for folks was significant, any speed test across the contry on bell I get almost my max speeds or better. login-to-view

[stephenw10]

Nice, I can only dream of a connection like that.

Yeah, if you have a PPPoE connection that fast you're unlikely to fill it with anything FreeBSD based while that single thread restriction exists. You would need something very fast for single thread use. Up to 1Gbps is possible on reasonable hardware though.

Steve

[techanalyst]

@stephenw10 said in How to get Bell Fibe in Quebec/Ontario (Internet and IPTV) working with pfSense:

Yeah, if you have a PPPoE connection that fast you're unlikely to fill it with anything FreeBSD based while that single thread restriction exists. You would need something very fast for single thread use. Up to 1Gbps is possible on reasonable hardware though.

Don't get me wrong, I love pfsense, theres so much it does out of box that just works and just works well. FreeBSD is a minor limiter here, nothing to do with pfsense. The machine I'm running this on is a dual 2699v4 :) I was able to tweak to ALMOST match, download actually wasn't any different, upload is where I saw my speed limits but were talking getting 890 vs 1000, keeping in mind sure I have a good upload but we don't see the drops on the ONT.

Linux just seems more flexible, BBR alone made all the world a difference for me and hosting my plex server, people can now stream 140Mbit remux blurays off me, with reno/cubic they can't.

[stephenw10]

Interesting. BBR should make no significant difference on a firewall/router as the TCP connections are not terminated there. It would all be on the forwarded hosts.

[techanalyst]

@stephenw10 Thats what you'd think and thats the theory, but I retested new reno/cubic on linux, folks max out at 35-55mbit from me out of the local province, within province they're fine, outside, the most stable bitrate is 20mbit, with BBR on the firewall, the absolutely only change I've made, they can pull original at 140 no issue, also did FTP/file share tests, same speed limit. So I agree with you, but the results tell us something different. We can saturate my pipe with BBR, anything else we can't and the only change is BBR, nothing on clients etc (cant change the dummy clients).

[techanalyst]

@stephenw10 Here's an example, this guy is located in Florida, he was never able to do more than a 10Mbit stream on me (Pfsense, reno or cubic, linux anything reno or cubic), with BBR, he's been doing these streams, starts as he says "instantly", never buffers and never pauses, totally not doable otherwise. He's also doing it over a wireless ISP and his apple tv is wireless....so its impressive.

login-to-view

[Patian]

@techanalyst
I followed the work around method as described under "PPPoE with Multi-Queue NICs" and added the 3 parameters into the Advance>System Tunable.

net.isr.dispatch=deferred
net.isr.maxthreads=4 (tried for 2 and 1, not much difference)
net.isr.numthreads=4 (tried for 2 and 1, not much difference)

I saw a slight improvement on the speed. 767down, 607up.
The speed test was done on using speedtest website on Firefox.

But if I use the speedtest for MacOS apps, the result was close to the ISP said value, 891down, 756up.

Is the apps more accurate then using the website speedtest?

Do I need to try on other parameters as described on the page?
ie, manipulate the /boot/loader.conf.local file?

The SG-5100 from netgate store should have been optimized to make its compatible with pfsense with minimize tuning needed.

[techanalyst]

@patian install and test with the cli version or the app, browsers can introduce some pool

[techanalyst]

@techanalyst https://www.speedtest.net/apps/cli

Test from clients, also if you're running Win 10 change the congestion control provider to CUBIC. I was able to test 10Gbit (actually was getting about 9.1-9.3Gbps), so pfsense has the ability to forward as crazy awesome rates. Ill be back once they go with FreeBSD 13

[Patian]

@techanalyst
I followed the link and tried to install the freeBSD version of speedtest on the box, using command prompt, but failed. I typed the same command as on the website for freeBSD version.
Anyway, on the netgate forum, I read a post on this and tried

pkg update ; pkg install -y py37-speedtest-cli

Installation successful, Then

speedtest

Failed with error:
Retrieving speedtest.net configuration...
Traceback (most recent call last):
File "/usr/local/bin/speedtest", line 11, in <module>
load_entry_point('speedtest-cli==2.1.2', 'console_scripts', 'speedtest')()
File "/usr/local/lib/python3.7/site-packages/speedtest.py", line 1986, in main
shell()
File "/usr/local/lib/python3.7/site-packages/speedtest.py", line 1875, in shell
secure=args.secure
File "/usr/local/lib/python3.7/site-packages/speedtest.py", line 1091, in init
self.get_config()
File "/usr/local/lib/python3.7/site-packages/speedtest.py", line 1174, in get_config
map(int, server_config['ignoreids'].split(','))
ValueError: invalid literal for int() with base 10: ''