pfSense router for fiber 10GB instead of the one provided by my ISP?
-
@keyser Thanks, I think I will go with the 12600h version of the MS-01. It was about the same price as the SG-6100 and I would like to in theory pass along the whole 10GB bandwidth to my server (even though I know that most of the time it will not be used).
/Erik
-
@eribob said in pfSense router for fiber 10GB instead of the one provided by my ISP?:
@keyser Big thank you for the quick reply!
I will call my ISP tomorrow and ask them.I have Bahnhof here in Sweden if by chance anyone at the forms is familiar with them.
I will keep you updated on the progress :)
Hej eribob!
I'm also on Bahnhof, and have 10G running on pfsense and it works like a charm. The only thing you need is to clone the MAC from the Zyxel device they sent you and then you are all set.
I have a 1U server that I built and run pfsense virtualized on Proxmox but have dedicated the NICs (IOMMU) to make sure I don't limit performance in any way. I found some used Fujitsu X520-DA2 which I bought from this site:
https://www.ggsp.se/10gbe/fujitsu-intel-x520-da2-10gbe-dualport-pcie-lp-d2755-a11What does the rest of your network look like, in terms of switches and cabling? It can be quite costly to upgrade...
-
BTW, I'm using an i5-11400 and top out at around 8+ Gbit with Suricata running in legacy mode... Doesn't really improve if I turn off Suricata completely but it drops down to around 5 Gbit if I change to inline mode...
With the 12600h you have so many cores and power that I'd look at running Proxmox to be able to host other stuff on that machine...
-
@Gblenn Awsome! Big thanks for this update, it really helps! So if I get you right, I will do the following:
- Ask for a public IP at Bahnhof (https://bahnhof.se/privat/kundservice/bestall-publik-ip/) and type the MAC address of my Zyxel router
- Install proxmox on the MS-01
- Create a pfSense VM and pass through the SFP+ NIC that is built in to the MS-01.
- Clone the MAC address from the Zyxel to the WAN port on the SFP+ NIC (I did not know that you could do this, is it easy in pfSense?).
I hope that the 12600h will be powerful enough to run the pfSense with 10GBit. I was hoping to move some other services to it as well, such as my reverse proxy, authentication (Authelia docker), LLDAP. At the moment I am using pfBlockerNg on pfSense and the built in DNS resolver, but maybe I should install pihole for DNS and DNS blocking? Dont know which would be best...
I did not know about Suricata at all, sorry for beeing a networking beginner hehe. It is some kind of security tool? Is it on by default in pfSense? Would you highly recommend using it?
My network is composed of CAT6 RJ45 cables that are wired in my house. I have 2, 1GBit switches (one on each floor) and an older ASUS wireless router that I use as wifi hotspot.
Connected devices except for my server are an apple TV and an Nvidia Shield. I use an apple laptop as my workstation and a Fedora VM running on the server with GPU passthrough. I have considered getting a 10GBit thunderbird adapter for the macbook but they are quite expensive so I have been hesitant.
My priority is to get 10GBit to my server for potentially fast downloads to it. 10GBit switches are expensive as you say. Perhaps I should go for 2.5GBit for the rest of the network and just try to connect the server to the router with 10GBit. In that case I could look for a switch with only 2, 10GBit ports.
An alternative would be to get a PCI-e card with 2 10GBit RJ45 ports and install it in the PCI-e x8 slot on the MS-01 and pass it through to the pfSense VM in addition to the 2 SFP+ ports. That way I would have 3 10GBit ports that could be LAN ports on the router, do you think that would work?
Again, big thanks!
-
@eribob said in pfSense router for fiber 10GB instead of the one provided by my ISP?:
@Gblenn Awsome! Big thanks for this update, it really helps! So if I get you right, I will do the following:
- Ask for a public IP at Bahnhof (https://bahnhof.se/privat/kundservice/bestall-publik-ip/) and type the MAC address of my Zyxel router
I'm guessing you already have a public IP from Bahnhof actually? So when they send out a device to a customer, they have registered the MAC (in your case the Zyxel they sent you) so when you plug that it, it will automagically get an IP and you are all set. IF you want, you can of course call them and provide them with the MAC of the unit you want to use, but why bother...
Cloning is super easy, you just enter it into the Interfaces / WAN page in pfsense (where it sais MAC Address) and that's it. Now their router will see the registered MAC that they think is the Zyxel. So any time you change or want to test some other firewall or build, just make sure to use that MAC and it will work with Bahnhof.- Install proxmox on the MS-01
Yup, make sure to activate all the right things in BIOS, to support virtualization and IOMMU. There are lot's of great guides on the internet/youtube.
- Create a pfSense VM and pass through the SFP+ NIC that is built in to the MS-01.
When you create the VM, just create it without an interface and add it afterwards when it has finished creating it, but before you start it up. Proxmox will read the NICs on that card starting with the upper one first. So the 0000:01:00.0 is first and then 0000:01:00.1
And pfsense will see them in the order that you add them from Proxmox so if you add them in that order, the top one will be WAN. Of course that's super easy to change later but good to know. I noticed some other Firewalls (Sophos) read things in a different order...- Clone the MAC address from the Zyxel to the WAN port on the SFP+ NIC (I did not know that you could do this, is it easy in pfSense?).
Like I said, it's super simple...
I hope that the 12600h will be powerful enough to run the pfSense with 10GBit. I was hoping to move some other services to it as well, such as my reverse proxy, authentication (Authelia docker), LLDAP. At the moment I am using pfBlockerNg on pfSense and the built in DNS resolver, but maybe I should install pihole for DNS and DNS blocking? Dont know which would be best...
It's powerful enought, for sure, also to run other stuff. I run pfBlocker_NG in pfsense and I run Ntop-NG as a separate VM on the same Proxmox machine.
I did not know about Suricata at all, sorry for beeing a networking beginner hehe. It is some kind of security tool? Is it on by default in pfSense? Would you highly recommend using it?
Yes it's an Intrusion Detection and Intrusion Prevention System, but don't bother just yet. The Firewall will block anything from the outside anyway. You can start playing with that when you are getting up to speed. It's just that it will load the CPU's so that's why I mentioned it, as it can limit throughput...
My network is composed of CAT6 RJ45 cables that are wired in my house. I have 2, 1GBit switches (one on each floor) and an older ASUS wireless router that I use as wifi hotspot.
That should work, although SFP+ RJ45 modules have limited range (max 30 meters) unless you buy the more expensive versions. I am using TP-Link Switches (Omada) and have one SFP+ only 8p switch which I use only for connecting inside the cabinet. And then a 6 Port (4 RJ45, 2 SFP+) that I use to connect to rooms in the house, like my PC on a different floor.
Connected devices except for my server are an apple TV and an Nvidia Shield. I use an apple laptop as my workstation and a Fedora VM running on the server with GPU passthrough. I have considered getting a 10GBit thunderbird adapter for the macbook but they are quite expensive so I have been hesitant.
My priority is to get 10GBit to my server for potentially fast downloads to it. 10GBit switches are expensive as you say. Perhaps I should go for 2.5GBit for the rest of the network and just try to connect the server to the router with 10GBit. In that case I could look for a switch with only 2, 10GBit ports.
I have put the same type of card that I linked to, also on my main server running e.g. TrueNAS, NextCloud, Plex etc. (again Proxmox). And with enough RAM your uploads of large files to TrueNAS from your PC are super fast. Then of course with HDD's you will max out on them in the end...
An alternative would be to get a PCI-e card with 2 10GBit RJ45 ports and install it in the PCI-e x8 slot on the MS-01 and pass it through to the pfSense VM in addition to the 2 SFP+ ports. That way I would have 3 10GBit ports that could be LAN ports on the router, do you think that would work?
Again, big thanks!
-
@Gblenn wow that was really awsome! Now I definitely feel like this will go quite smoothly. A big thank you! The ms-01 will arrive in a couple of days and I will start setting things up. Might post here again if I hit a roadblock :) will also update with the results for sure.
This will be my first experience with proxmox actually, I am using unraid on my main server. They made licenses much more expensive recently though, so I want to move to proxmox.
-
@Gblenn said in pfSense router for fiber 10GB instead of the one provided by my ISP?:
top out at around 8+ Gbit with Suricata running in legacy mode
Btw does this mean that you are able to reach speeds close to 10GBit when downloading from the internet?
-
@eribob said in pfSense router for fiber 10GB instead of the one provided by my ISP?:
@Gblenn said in pfSense router for fiber 10GB instead of the one provided by my ISP?:
top out at around 8+ Gbit with Suricata running in legacy mode
Btw does this mean that you are able to reach speeds close to 10GBit when downloading from the internet?
Not really, the best I have seen is around 3.5-4 Gbit from e.g. Battle.net or Steam. But most often they max out below 3...
Of course that's not bad at all, and it means I typically finish downloading an upgrade in a few minutes rather than an hour which could be the case on a 100 Mbit line...
In reality there is very little "out there" that is able to push that much, and I think ISP's would be better off selling 2,5 to consumers... which then would be possible to make use of. And the cost of upgrading your home wouldn't be so steep either.
-
Hello again,
I happen to have, temporarily, two public IPs and I have another firewall on that additional IP for testing. So I decided to make a test through pfsense using iPerf instead of Speedtest. To see if I can reach even better numbers than I have seen so far...
I set up an iperf3 server on a Linux VM running under Proxmox on the pfsense LAN side, and created a NAT rule to forward the default iperf port, 5201, to that server.
I then changed my PC over to the LAN side of the second WAN IP's firewall (Sophos) and ran "iperf3 -c pfsenseWAN-IP -P 10" and this is the result... with several of the individual "runs" reaching 9.3 G and more...
I don't think one can expect any better result than this actually. Especially considering pfsense is running Suricata in legacy mode and reports CPU going up to and over 90% percent during the testing. (4 cores on an i5-1140).
-
@Gblenn Wow, that is impressive! So that means your traffic is going through bahnhof from one public IP to the other?
I am still waiting for my minisforum, and today I ordered a 10GBe card for my server. Within 1-2 weeks I hope to be able to test the full speed. For now I use the included zyxel router but are limited to 1Gbit due to no 10GB card in the server. I am able to saturate that using speedtests at least =)
-
@eribob Yes it's quite some performance from a virtualized machine...
However, the way I have it set up is that I have the fiber going directly into a Mikrotik 10G switch (CRS310-1G-5S-4S+). Each firewall is then connected with a DAC cable from the switch to their respective WAN ports. And Bahnhof hands out the two IP's based on the different MAC's that I provided them with, exactly as we discussed earlier.
But since there is a switch there, that also means that the traffic only goes out one WAN and then back in to the other WAN via the switch, without involving Bahnhof equipment at all. I could actually pull the fiber, if I had set static IP's, and it would still work...
-
@Gblenn Quick update! Got it working now. Installed proxmox (my first time using it) and set it up with some online guides. Created a router VM and passed the x710 NIC in the ms-01. Also bought a x540 PCI-e card with 2 RJ45 10GB ports and passed that through as well. I have created a LAN bridge now. I hear that that is not optimal for performance, and will probably buy a 10G switch at some point.
Anyway, getting good speeds (ookla speedtest):
Overall happy and will continue to tinker =)
-
@eribob Wow, really good numbers, congratulations!