Hardware recommendations for 40gb internet, 100gb lan

mercer2

it is kind of crazy, our house is 15k sqft, with 30 tons of ac. our electrical service panel is 1200amps and this 3 racks have 7 x 30 amps dedicated circuits plus a dedicated 2 ton ducted ac.

any recommendations on hardware components for a pfsense capable that also could be migrated to tnsr and snort?

thanks

flat4

@mercer2
Sir what do you do for a living?
15k House, 30 tons of ac,
going to assume that's in a basement?

luckman212

@mercer2 said in Hardware recommendations for 40gb internet, 100gb lan:

any recommendations on hardware components

uh yeah this should fit nicely in your 15k sqft home: PA-7080 (and will only set you back $170K)

mercer2

I wish!!!

trying to be cost effective and wife not mad about it.

luckman212

@mercer2 Ehh.. if she's not mad about 3 full racks, the 600lb workgroup printer or the electric bill from those 2100 Amps...for your homelab, I can't see her complaining about anything.

My wife got mad at the little LED on our Apple TV, I had to cover it with electrical tape

stephenw10

In all seriousness firewall/routing at 40Gbps is beyond pfSense scale. TNSR can do it though and is very cost effective at those bandwidths compared with other enterprise routers, like that PA.

Steve

Derelict

@derelict said in Hardware recommendations for 40gb internet, 100gb lan:

@mercer2 What does the ISP providing the 40Gb service recommend?

Still no answer to this. Genuinely curious.

Patch

@mercer2 said in Hardware recommendations for 40gb internet, 100gb lan:

any recommendations on hardware components for a pfsense capable that also could be migrated to tnsr and snort

As reference point I would start with Netagate characterised hardware
https://www.netgate.com/tnsr-software/how-to-buy#appliances
https://docs.netgate.com/tnsr/en/latest/platforms/ and
https://www.netgate.com/tnsr-software/performance#get-to-know

For software in a home lab; Netgate have a free option which would be a reasonable place to start if you are just trying it for fun https://www.netgate.com/pricing-tnsr-software

Similarly pfsense is available in a free community edition https://www.pfsense.org/download/ or a free home lab versions https://www.netgate.com/pricing-pfsense-plus

If you want to start with more capacity you will need to scale it up from there, which probably involves adding a 40Gb network interface and more Xeon processes.

Personally I would not push the hardware specs till I had experimented with pfsense & tnsr to see if I was comfortable working with them. Both pfsense and tnsr run on most recent 64-bit x86 processor with SSE4.2 extensions. You may already have under used hardware meeting the minimum requirements.

Sergei_Shablovsky

@mercer2 said in Hardware recommendations for 40gb internet, 100gb lan:

I’m extremely lucky to have a 40gb internet fiber installed in my house

I know this is way over the top, and greatly underused. I’m a typical user, but my hobby is home automation and homelab

I would like to hit a Speedtest alternative service and get the full 40gb, with a chance to maybe upgrade to 100gb internet in the future.

According equipments in a racks more looks like this is a online + terrestrial radio station site:
0. EXTRA-HUGE 40Gb / 100Gb UPLINK (extremely too much for ANY work, even online trading need only 200Mb but with small jitter/delay, only ISP need speeds like that...)

A lot of UPS (ok, not Symmetra or 50kva GE, but anyway this is a Houston, not a Siberia or India with a “electricity cut-offs”...);
Rackmount PROFESSIONAL sound processors (I recognize 2 big (and costly!!!!) (Only this two may cost much more than all this setup, even full of Dell Servers...)
Not so much powerful Servers and Svitches in a racks, no fiber connectors.. (Why You need even 40G if a You have no fiber-connected rack equipment?)
Audio streamer amplifiers (or may be CD changers, but no one wiring even middle-class amplifier from underground to living room 2 floor ahead, because even You using $100/ft cable to Dali's sound speaker, as a result You receive a CRAPPY sound). I could recognize some modern DENONs and SONOS..
Rackmount storage (I not see the lights on HDD caddies, may be not filled or switched on...)
Some sort of car shortwave radio stations (or may be receiver/processors for wireless studio mic)
A huge copy/print station on a background
A lot of LCD remote for climate control systems on a wall

But also I see

unlabeled patches at the top of each rack;
no any fire-stop system (strongly needed if You have such amount of UPS in same room) near racks;
NO PIPES FOR COOLING SYSTEM for this amount of equipment (especially for a bunch of 10/14k speed 2,5' HDD that You have);

And You wrote that are “32 audio zones” in a building....

At the same time even You have so much space and zones in a living house, all of Yours imagination able to be realized by AppleTV / SONOS / RUSOUND / DENON media players/streamers + MESH WiFi (from any good brand, if You have no ability to drilling walls and cabling all inside house).

Are You sure this is room in Your house? Who are You, mr. President? :)

Sergei_Shablovsky

@flat4 said in Hardware recommendations for 40gb internet, 100gb lan:

@mercer2
Sir what do you do for a living?
15k House, 30 tons of ac,

Just as a joke: a lot of Russian oligarchs now escaping with A HUGE MONEY from Putin's dictatorship... I read from news half of a year ago one FSB-related person was arrested and in his 4-room flat in center if Moscow city 10 TON OF GOLDS was arrested! Another one time: 10 TON OF GOLD !!!! In a 4-room flat!!!!
And need to note this person was ordinary authority, even not top-level. Imagine how much top-level persons have?

Sergei_Shablovsky

@patch said in Hardware recommendations for 40gb internet, 100gb lan:

@mercer2 said in Hardware recommendations for 40gb internet, 100gb lan:

so would like something easy to setup and monitor.

A Lan with 40-100Gb capacity sounds generous but possibly useful.

To size your Wan, do not use what your ISP can provide and what Speedtest can measure (although at that speed you would probably have to run multiple concurrently).

Instead look at how many concurrent video channels you will want to access remotely. Or what is the actual high bandwidth concurrent tasks you will actually do.

The result of this calculation is likely to be within high end pfsense hardware capacity. If not someone is going to have to manage an enterprise system.

If you want it just for fun, then try playing with TNSR

ABSOLUTELY AGREE with Your opinion.

More than this, from our experience, the setup like this home (You may see building plan on a wall opposite to racks), need not more 5Gb for ALL NEEDS, even
SIMULTANEOUSLY each person (7 persons at all) in each separate room

see 4K stream from Netflix/Amazon Prime/Apple/YouTube
listen 32-bit ALAC/FLAC sound on external media player
playing a network Game in 4k
downloading 200-300 torrents
uploading streams from Apple iPhone/iPad to FaceBook, TikTok, YouTube at one time
making a group conversation on a stand-alone IP-telephone
surfing a 50 web sites with a huge graphics and video content
making “heavy” banking (online trading means)
have a separate personal web server (shopping site, blog, etc...) with 300 users online

Imagine? 5Gb for all of that mess!

Of course, in real life only 1-2Gb would be used.

Patch

@sergei_shablovsky
Not sure why you are trying to define adequate requirements. @mercer2 has already said he does not need the capacity but is doing it because he likes playing with technology and his ISP is offering it.

Sergei_Shablovsky

@patch said in Hardware recommendations for 40gb internet, 100gb lan:

@sergei_shablovsky
Not sure why you are trying to define adequate requirements. @mercer2 has already said he does not need the capacity but is doing it because he likes playing with technology and his ISP is offering it.

Just try to be reasonable.

Anyway, even this is like a toy, this is dangerous for himself and family members and neighbors, if they have no using fire-stop system and properly cooling for such amount UPS and equipment. Agree?

TSNR on big x86_64 hardware might be that you should have a look on!

SuperServer SYS-110D-20C-FRDN8TP offers 25 GBs Ports
and perhaps there will be also some well supported 40 GBs
ports. All in all I would directly ask at Netgate support to be sure to get the hands on the right hardware.

jamesdwi

As others have said Enterprise grade gear would work for this setup the problem is, Yes its a huge pipe to the internet, but you really don't have any other needs for the firewall. Enterprise gear is made for 100's of 1000's of active sessions. thousands of firewall rules. Your setup just doesn't need that level of hardware and complexity. Your setup may have 25 rules in your firewall. Only a torrent box will have more than 100 active session. A user would be burned out just trying to keep a 100 active sessions on a PC. The firewall would Block everything by default, allow into a couple servers and data for active sessions. , if you want to host a website or two.

Your house while impressive, it is going to have a hard time utilizing that pipe, 30 or even 60 netflix streams isn't going to fill 40gbit, 4k netflix stream is about 25 megabits, 60 of them come to about 1.5 gigabit/second. Down loading from even large internet sites is probably going to limit you to 1gbit/s per transfer, unless its from your own cloud machines.

I would get a AMD EPYC 7313 Processor 16-core 3.00GHz 128MB Cache 32 threads, 2x 32 GB of the fastest DDR4 dimms you can find, yes its over kill, you need dimms in each memory channel so you can get maximum number of memory channels and bandwidth. PCIe version 4, NVME drives, fastest... 1TB each, in case you get crazy or accidently enable logging of all traffic. 40 Gbit nic in and 100gbit nic to the lan. Make sure you block all internal traffic from hitting your pfsense box. While you are shopping for this server probably want to pick up a couple more EPYC servers, if you want to to host and download torrents. I expect that either you can't generate or receive enough traffic from the internet to fill your pipe, pfsense firewall with this hardware may be up to the task. Or it will do just fine with the network load if all the pfsense fits in the cpu cache. 128MB of L3 cache.

You may want to check out Linux Tech Tips video "A $15,000 Network Switch?? - HOLY $H!T - 100GbE Networking" It shows what it takes to max out 25 and 100 gigabit nics using multiple 12k and 8k raw video from a machine with 2x EPYC 64 core cpus, and 40 NVME drives. It can serve the data. But how to get steaming content that needs that amount of bandwidth. For businesses even 10 gigabit internet connections aren't cheap. So you will probably need to pull traffic from 80+ internet sites to even slightly load down your internet connection..

mercer2

@jamesdwi
Hi James

I found this 100g router, which I received last week

https://youtu.be/7_uLxZYYEpQ

will test to see how it goes.

in regards to saturating a 100g local link and a 40g internet link.

that’s the journey i will go trough, and will enjoy it, since is a hobby for me.

Sergei_Shablovsky

@jamesdwi said in Hardware recommendations for 40gb internet, 100gb lan:

As others have said Enterprise grade gear would work for this setup the problem is, Yes its a huge pipe to the internet, but you really don't have any other needs for the firewall. Enterprise gear is made for 100's of 1000's of active sessions. thousands of firewall rules. Your setup just doesn't need that level of hardware and complexity. Your setup may have 25 rules in your firewall. Only a torrent box will have more than 100 active session. A user would be burned out just trying to keep a 100 active sessions on a PC. The firewall would Block everything by default, allow into a couple servers and data for active sessions. , if you want to host a website or two.

Your house while impressive, it is going to have a hard time utilizing that pipe, 30 or even 60 netflix streams isn't going to fill 40gbit, 4k netflix stream is about 25 megabits, 60 of them come to about 1.5 gigabit/second. Down loading from even large internet sites is probably going to limit you to 1gbit/s per transfer, unless its from your own cloud machines.

I would get a AMD EPYC 7313 Processor 16-core 3.00GHz 128MB Cache 32 threads, 2x 32 GB of the fastest DDR4 dimms you can find, yes its over kill, you need dimms in each memory channel so you can get maximum number of memory channels and bandwidth. PCIe version 4, NVME drives, fastest... 1TB each, in case you get crazy or accidently enable logging of all traffic. 40 Gbit nic in and 100gbit nic to the lan. Make sure you block all internal traffic from hitting your pfsense box. While you are shopping for this server probably want to pick up a couple more EPYC servers, if you want to to host and download torrents. I expect that either you can't generate or receive enough traffic from the internet to fill your pipe, pfsense firewall with this hardware may be up to the task. Or it will do just fine with the network load if all the pfsense fits in the cpu cache. 128MB of L3 cache.

You may want to check out Linux Tech Tips video "A $15,000 Network Switch?? - HOLY $H!T - 100GbE Networking" It shows what it takes to max out 25 and 100 gigabit nics using multiple 12k and 8k raw video from a machine with 2x EPYC 64 core cpus, and 40 NVME drives. It can serve the data. But how to get steaming content that needs that amount of bandwidth. For businesses even 10 gigabit internet connections aren't cheap. So you will probably need to pull traffic from 80+ internet sites to even slightly load down your internet connection..

Very argumentative reply. Thank You!
(And a little bit stat for me ;)

But anyway as topicstarter wrote - this is just toy, not about TCO, business, or something serious. Just a toy like GameBoy or Tetris.

Sergei_Shablovsky

@mercer2 said in Hardware recommendations for 40gb internet, 100gb lan:

@jamesdwi
Hi James

I found this 100g router, which I received last week

https://youtu.be/7_uLxZYYEpQ

will test to see how it goes.

in regards to saturating a 100g local link and a 40g internet link.

For this kind of speeds better to use Emerson, Nokia, Huawei routers from ISP-grade lines, or a little bit cheaper F5, Extreme, Juniper.
But not SOHO like Ubiquity, Microtik, D-Link, etc...

As a You demonstrate, money, electricity uplinks and rack space - not a big problem for You. Because this equipment are ISP-grade, anyone local distributor would be happy ship, install and give You 1-2 weeks for test, for free.

devin

@mercer2 Super cool router that you bought. That is absolutely my dream router. Hows it been treating you? Did you get the speeds you were expecting?