Great to hear.

You could try logging in (root/<your admin password>) or you could unplug/plug the device power. Usually that kind of error only happens if something severe went wrong, like an error early in the boot process, a filesystem error, etc. If you open a support case at https://go.netgate.com the crew there can get you a link to the installer image to reinstall if you need to.

@tawsenior Nevermind, I went into the Package Manager and found it. Thanks.

Glad to read that Netgate as FW manufacturer keep attention on this. :)

The OpenVPN option text should probably be renamed. The engine command in OpenVPN isn't required. When it's unset then it automatically selects a device which supports accelerating whatever cipher it's trying to use. When it's set to a specific engine, it's supposed to prefer that engine but I don't believe it's restricted to only using that engine. Since most things only have 0-1 available usable engine types, that's not so easy to test. So really the No Hardware Crypto Acceleration line should be Use any available cryptographic hardware device or something along those lines.

That's a known issue and is harmless. But thanks for reporting it anyway. https://redmine.pfsense.org/issues/9975 Steve

@jimp That would be my next step. I just figured it would be good information for the community to know about how the support works. I often feel there is a disconnect between the community and Netgate and so I thought it would be nice if that information was out there in case someone else ran into this situation.

You cannot add the SFP ports to the existing lagg and you need the lagg to the switch anyway. So, yes, I would create a new lagg (lagg1) with ix0 and ix1 in it and connect that to the Ubiquity Switch. You can then use LACP on that lagg which would be a far better choice there than loadbalance. Steve

There's little point using a mesh type setup when all the traffic is from the remote sites to the central site in my opinion. You might be able route traffic via another site if a link goes down providing some redundancy. It would be a far more complex setup though. You might check the TINC package for a mesh setup. Steve

@stephenw10 That's what I did, I just didn't want to specify how I've done the private subnetting. But that's a good point for anyone who stumbles on this looking for a leg up.

I wouldn't necessarily recommend doing this, but you could add (or create) an entry in /boot/loader.conf.local which sets pfsense.fsck.force=5 or so. Then on every boot it would perform that many iterations of fsck to check/repair potential problems, even when the filesystem is marked clean. It would drastically slow down the boot process and is typically unnecessary, but it might at least help with some of these situations. It's definitely not something we'd ever ship with set by default.

@stephenw10 Ok I think I may have figured it out. I had two switches Daisy chained and when I disconnected the second switch everything worked fine. I tried reconnecting and then the switches lost connection again. I tried several different cables with the same results. I ended up configuring the opt1 port and connected the second switch to that. I am not sure what made the switches stop working while Daisy chained but they showed they were still communicating. I am going to have to dig into this more and find out what caused the issue. Thank you for the help @stephenw10 .

Have you tried from multiple browsers, multiple remote clients?

Turns out, the SFP port needs to have speed set explicitly. Steve mentioned this earlier, but I ignored the advice. When I explicitly set the speed to 10G, Comcast said they were getting ARPs, so they disabled auto-negotiation on their side. When they did, the link came up. So, in the end, an easy fix. But Comcast had advised me to set the link at auto select. Learn from my mistake.

Not with LACP. As far as I know the switches need to be 'stacked' in order to have lagg links split across different physical units. You could connect everything to everything else and rely in spanning tree to prevent a loop. But.... That also means bridging the ix ports in pfSense. I would consider an LACP lagg from the 7100 to the first switch and a second lagg from that switch to the other switch. It won't help if the first switch fails entirely but you would have port/cable redundancy between everything. Steve

Yes the /32 mask was the problem... changed to /24, enabled DHCP, and good. Thank you all!

is there a fiber option? i could get a 1m premade if that would work

Just for reference I was able to see >300Mbps though a 3100 using iperf3 in local testing, so very low latency. That was using AES-CBC 128 and SHA1.