Topology suggestions for a home network please

eiger3970 0

Hello,
I'm looking to build a home network, that optimises hardware required.

Currently run 1 desktop machine with 1 LAN and 1 WAN NIC, with a hypervisor running a 24 hour vm router pfSense, home assistant and security camera operations and data management.

However the network works well for months and then a mystery causes bugs which is difficult to solve.

If a separate conventional hardware router (25 Watts) with pfSense was integrated into the home network topology, this would be 24 hours on and the desktop (700 Watts) could be shutdown.
However, I'm unclear how the other 24 hour systems would then run? Buy more hardware or install onto the new hardware running pfSense?
Seems like duplicating what I have already?

johnpoz

@eiger3970-0 your desktop is drawing 700W? Or that is the max it can draw.. That seems like a really heavy draw..

Do you have like a kill-a-watt meter showing that type of usage? Doesn't really matter in the big picture if it is drawing that or 1/8 of it, etc. A hardware router is going to draw way way less.. My getting a bit old in the tooth netgate sg4860 draws like 7w most of the time. Been a while since I put my kill-a-watt meter on it, etc.

But my nas which is running a VM 24/7/365 and others on and off and multiple containers and serves up plex 24/7 draws like 55W on average.

I am a fan of lowering electric consumption so I am with you going that route.. But that 700W seems a bit of an over estimate ;)

What exactly happens when your network goes into issues mode?

I use to run pfsense for years as a VM on multiple platforms, latest was esxi for many years until I finally switched to hardware.. I don't think I would go back, I can still run pfsense VMs for testing stuff, etc. But my network doesn't depend on my VM host being up.. And since the network equip is low power, modem, router, switch a few AP, etc. I can run for quite some time on UPS during power outages. My PC and NAS on different UPS, etc. I can go good 20-30 minutes without even noticing from PC and network that power is out.. So yeah some advantages for sure for low power use, let alone the reduction in electric bill, etc.

You mention Cameras, I have a few of those.. That network is isolated behind the NVR, and the cameras are poe.. So even on a power outage my cameras function.. I haven't timed how long that is - tmrw have to go take a look at what they are drawing on the UPS and do a bit of math ;)

eiger3970 0

@johnpoz The vm router network issues are these.

I'm still unclear if I move the vm router to a separate hardware device, what happens to my other 24 hour vm systems like camera operations and data storage and more importantly Home Automation controls and data storage. Seems like more hardware devices to be set up for these, rather than the current set up with 1 machine for all.

johnpoz

@eiger3970-0 leave those on the VM host.. but if the vm host goes down or has issues you don't loose internet.

eiger3970 0

@johnpoz yes, however if I'm using the host which doubles as a desktop for <24 hours, it may be less electricity and hardware wear and tear wasteful.

So, if I'm going to have some systems needing 24 hour On state, the dilemma is why move the vm router to its own hardware...why not try and configure the network to work with less hardware parts...less problems?

Possible digression, however this is the security measurability...store a secret private key on 1 item is less risk than on 3 items.

johnpoz

@eiger3970-0 not sure what circle your going in.. But a main reason to move your router/firewall outside your VM host, at least for me is that when I need to take down my VM host I still have internet.

If something goes wrong on my VM host or VMs again still have internet. Still can route between my networks, etc.

Sure there are some advantages to running it all VM, like I said I did that for many years.. But also as mentioned anything to do with the VM host and you have no internet.. Which is a big disadvantage, if you ask me.

eiger3970 0

@johnpoz yes, I appreciate having the router on 24 hours and the host/desktop off when not needed.

However, I'm trying to plan how to run the other 24 hour systems like Home Assistant and Security Camera stuff. Could the hardware router run the other 24 hour systems too? Seems like I have that build already, but need a separate desktop machine for on/off operations.
If I understand this architecture correctly, this means the 24 hour hardware item will be some type of hypervisor type 1 to run the router and other 24 hour systems.
Therefore, the network issues I have now with the vm router running seems similar to running a router hardware (with other 24 hour systems somehow running on the router hardware?).
Yes, I'm a bit confused and probably running around in circles...sorry.

johnpoz

@eiger3970-0 your going in circles.. Sounds like you just want a VM host that doesn't suck as much power as you think the box your currently running them on.

I again doubt your current PC/VM host is sucking 700W to be honest just sitting there doing its thing..

Sure you can move your setup to a lower power VM host.. Like I said I run some VM and Containers on my NAS and it only draws about 55W..

But my router is hardware, my cameras are hardware (NVR).. Sure my unifi controller is on my NAS via a VM.. But the unifi controller being down doesn't stop wifi from working. Just not getting any stats and can't make any changes while the controller is down.

If your current VM host is spinning your electric meter, then get a different box to use as your host that doesn't spin the dial so fast ;) Be it you run your router on that is up to you.

stephenw10

Mmm, I'd be amazed if that's actually 700W.

johnpoz

@stephenw10 I would be shocked.. So I don't have a meter on just my pc.. But the ups its plugged into with monitors, maybe my switch or few other things.. Running around 200W total off the ups..

If I max out all 8 cores with cpustres.. I go to a bit over 300W total on that ups.. It would be insane to be sucking 700W on some PC..

My whole rack of gear DAS and NAS and pi and 2 AP and modem and router and 28 port switch, a little hub for some lutron switches my laptop is plugged in there, my I9 PC, 2 32" monitors etc.. If you add up both UPS load its like 300W (95W on one, 195W on the other) sitting here while I type this out.. Even if the thing was at 100% cpu the whole time I just can not fathom it drawing 700W..

My guess would be that is the rating on the power supply in the PC and not what it uses just doing its thing.

eiger3970 0

@johnpoz Fwiw, I just ran these random commands to find the wattage used on the 700W PSU

linuxmint@linuxmint:~$ sudo cat /sys/class/powercap/*/energy_uj
[sudo] password for linuxmint:            
65772978044
208569600642
201593212191
linuxmint@linuxmint:~$ time=1
linuxmint@linuxmint:~$ declare T0=($(sudo cat /sys/class/powercap/*/energy_uj)); sleep $time; declare T1=($(sudo cat /sys/class/powercap/*/energy_uj))
linuxmint@linuxmint:~$ for i in "${!T0[@]}"; do echo - | awk "{printf \"%.1f W\", $((${T1[i]}-${T0[i]})) / $time / 1e6 }" ; done
2.5 W0.6 W4.5 W

The main concern is the VM router works but then has problems many people struggle to fix.

The 2nd concern is the Desktop tower has fans which are not silent.
ARM and RISC-V SBC Pico form factors are silent and have the power for desktops these days...so thinking how to update my setup.

Imho, less hardware = less problems.
However I need 24 hour running hardware for a router, home automation and security cam operations and data.
If I can't run it all on this machine due to network bugs, I'm unclear how to rebuild at this point in time.

The current network is very confusing.
Desktop 192.168.1.120 pinging 192.168.1.110 Destination Host Unreachable
Desktop 192.168.1.120 pinging 192.168.1.160 Destination Host Unreachable
Desktop 192.168.1.120 pinging 1.1.1.1 0% packet loss.

VM router 192.168.1.170 pinging 192.168.1.110 Host is down
VM router 192.168.1.170 pinging 192.168.1.160 Host is down
VM router 192.168.1.170 pinging 192.168.1.120 0% packet loss
VM router 192.168.1.170 pinging 1.1.1.1 0% packet loss

192.168.1.110 pinging 192.168.1.120 Host is down
192.168.1.110 pinging 192.168.1.160 0% packet loss
192.168.1.110 pinging 192.168.1.170 Host is down
192.168.1.110 pinging 1.1.1.1 No route to host
Ethernet connection is Connected to IP 192.168.1.110/24 Router 192.168.1.170 DNS Server 192.168.1.170

192.168.1.120 ip neigh

linuxmint@linuxmint:~$ ip neighbo
192.168.1.254 dev br0 lladdr 4c:fc:aa:91:db:ed STALE
192.168.1.147 dev br0 lladdr 52:54:00:a1:d1:e2 STALE
192.168.1.110 dev br0  FAILED
192.168.1.170 dev br0 lladdr 52:54:00:09:d3:92 REACHABLE
192.168.1.146 dev br0 lladdr 52:54:00:ac:44:b5 STALE
192.168.1.230 dev br0  FAILED
192.168.1.200 dev br0 lladdr cc:f9:57:2c:80:25 STALE
192.168.1.160 dev br0  FAILED

stephenw10

That's all on the same subnet so either those hosts are actually down or there some layer 2 disconnection.

eiger3970 0

@stephenw10
.110 is on as I manually turned the machine on to run the tests to prove what it could and couldn't ping.
.160 is the printer and it does deep sleep, however the display has been lit up with Pause for maybe 24 hours.
.170 is the vm router pfSense (where I suspect the bugs)
.230 is the Wi-Fi AP by UniFi which is currently in adopt mode with a solid white light, connected by Ethernet to the 24 port Switch.
The Netgear switch ports show an orange light, indicating connection.
I have physically tested with a VDV Multimedia cable tester and all cables and wall plates pass with 8/8 wires.

johnpoz

@eiger3970-0 said in Topology suggestions for a home network please:

.170 is the vm router pfSense (where I suspect the bugs)

again devices on the same network talking to each other have nothing to do with the router - nothing!!! Device A (you) want to talk to B (printer). A says hey I want to talk to 192.168.1.160, that is on my network since it falls under the network mask of my IP /24 so I will send an ARP... Oh hey .160 answered his mac is xyz.. Device A then sends its traffic to mac xyz..

The router has nothing to do with this..

eiger3970 0

@johnpoz thanks for the reminder. I guess I'll re test the LAN physical layer cables, wall plates and 24 port switch lights.

keyser

@eiger3970-0 I had a similar host/vm setup like you, but grew tired of paying the electricity bill + loosing internet every time played around/changed some things.

So I got myself a little pfSense box - for just pfsense - and installed my homeautomation/fileserver/monitoring software/camera NVR on a Raspberry Pi 5 with a NVMe hat holding a 2Tb NVMe SSD.

It pulls between 3 and 6 watts at the plug 24/7 and performs very very well.
My pfsense box is a SG-2100 that uses around 4w, so all in all - around 10w round the clock and has both duty separation plus excellent performance.

I often wonder why it took me soo long to separate these duties and get some really power efficient hardware.

stephenw10

Could be the switch blocking things if there's a loop somewhere.

Could be a wifi issue if any of those devices are wireless.

eiger3970 0

@keyser Yes, I originally had a mini-box APU however it broke after some time so I moved to less hardware devices.

The Pi5 with NVMe is awesome, so fast.
I need a desktop though, so I'm yet to test the Pi5 desktop running hypervisor type 1s with VMs for home automation and security cam management.

The Pi5 does heat up to 55C without an external fan (internal fan running), which keeps it around room temperature.

Your suggestion would mean 2 extra hardware devices as the 3rd would be a desktop.
I might try a small router again, but for now I'll check the LAN network.

The issue seems to be connecting the Wi-Fi AP Unifi UAP-HD-AC.

.230 is the Wi-Fi AP, so unsure how to stop it messing up the switch?

keyser

@eiger3970-0 My Pi 5 is passively cooled with a fairly intelligently designed case that has a thick aluminium upper part that serves as a radiator as well as a case.
Works beautifully - and ensures no moving parts/noise/additional power usage.

I opted against using VMs on Pi for the different services. the performance hit is too big. Personally I think containers are to "complicated" in design/setup, so I just run the OpenHAB, NFS server, Zabbix Server and NVR software native on the raspbian OS install. Has both better performance and much lower power usage (less CPU heating too).

Running a full working desktop on pi 5 is not recommended. Use your old PC as the desktop and turn it on/off when needed. Run the Pi with Raspbian PI OS lite (No GUI) for both performance and stability. That is by far the best solution and makes the Pi a "Server" only.

eiger3970 0

@keyser
I have a few Pi5's with the standard red/white case and internal fan.
Happily run a full desktop Raspberry Pi OS desktop (an external fan is needed in summer to bring the 55C CPU down to room temperature around 23C).
1 Pi5 runs hypervisor type1 and VMs running on it like this machine are very optimised running on the kernel with KVM (CLI) and Qemu (GUI).
Yes, some other hypervisors type 2, some type 1's and containers do chew up resources.

I'm running out of old desktops, so I'm moving to SBCs with Pico form factors, the RISC-V SGxxx comes out next year. SBCs are now powerful enough for desktops if you don't do gaming.

Once I sort out this LAN network issue, I'm building another Pi5 with hypervisors and VMs...no routers needed, just hotspot for Desktop Internet stuff and LAN control only (definitely no 3rd party cloud control complications/nonsense) via Ethernet and Wi-Fi APs (maybe some Zigbee too).
The tricky bit is creating and configuring some DHCP server or something like that.