@SteveITS I have had ZFS since it was available for that reason, and I always reformat the SSD so pfsense install does ZFS from scratch. Yes, I have a very large UPS for many years, small car battery size. The problem is it lasts for a couple of hours since it handles the modem, router, HP 24 port switch, Mac Mini phone system, etc., whereas our power failures average 3 to 8 hours. Sometimes multiple days, one time almost a week! 2.6 always recovered until the modem change, strange but true. Trying to login to pfsense 2.6 only returned the dreaded "502 Bad Gateway Nginx error". Had to power off/on. Waiting to see what happens to 2.7.2. I'm thinking possible ethernet driver issue with a different chip in the S34 than the S33, which may be fixed in 2.7.2. The next power failure will be the test. My plan is after Pfsense 2.8 is released I will buy a new box for it with 2.5G ethernet to the modem which hopefully will be fine. Thanks for the comments.

@SteveITS 60% of the time it works every time.

If you're policy routing traffic via the VPN then traffic meant for other local subnets would be forced that way unless you have bypass rules to allow it to be locally routed. But that doesn't apply to traffic in the same subnet, that doesn't go through pfSense at all. So I would confirm that they really are in the same subnet. Make sure the mask is set correctly on all devices.

Hanlon's Razor applies here. It was probably just a mistake somewhere. Or perhaps some client thought they could just add more IPs to use and it wouldn't matter. If they didn't use them all the time that might explain it. Anyway let us know if you still see any issues now that can't happen.

Wireguard produces almost no logs which makes troubleshooting....interesting! So there are no WG specific logs. You can only see the interfaces connection in the system logs or check the states for passing traffic etc.

Then open a TAC ticket: https://www.netgate.com/tac-support-request It sounds like that unit has a faulty LED or controller. Though, as I say, it's very unlikely it's anything other than cosmetic.

It could be Kea via some affected process but not directly. If dhclient shows failing to pull a new lease at release time then that's certainly a problem.

For reference: https://redmine.pfsense.org/issues/16103

@viragomann that was exactly what was needed, thank you.

@greedj Thank You! Primary I mean running pfSense only on bare metal servers w/ 2 CPUs. No any reason to run virtualization because of highloading, even more: better to make HA cluster of pfSense (with two(2) independent online-interactive UPS - each to one of server’s power supply, and more than 2 uplinks to power provider).

@stephenw10 Stephen... thanks for jumping in.. removed and now all good.. internet available. thanks for everyones help

@stephenw10 Then I'm just going to stick with my current setup and see if there is anything on the console the next time this happens, if happens. Thank you for your help, much appreciated!

@Bryan81 Especially if using pfBlocker set that to something like 2 million and adjust upward if necessary.

@Bryan81 https://forum.netgate.com/user/bryan81/settings has a Notification section to disable notifications, if that's what you're looking for. There is a Mark All Read button if you click the bell.

Extremely unlikely! We don't even have 802.11ac in FreeBSD yet.

@stephenw10 said in Slow Iperf3 Results: Could have been some sort of loop then. Or maybe some asymmetry. If it was a loop/flood you'd see it in the traffic graphs from the time. If it was going through pfSense at least. Must have been a loop, just flooded the 1G connection and monitored on the switch and it didn't once loose connection and had to reconnect. Very strange.

If you're authenticating against Freeradius the users only need to exist there. If you have 100s of users though I'd consider using an external radius server. The Freeradius package in pfSense is not really optimised for large numbers like that.

Doing so removes all filtering. You can have filtering as long as you have the rules to pass traffic you need.

@Bob-Dig Thanks for the reply. Subsequent error messages appear to show the SMS is being blocked as spam. (AUP = Acceptable Use Policy, CNCT = Concurrent Connections, MXRT = Max Rate) Final-recipient: rfc822; XXXXXXXXX@mms.att.net Diagnostic-Code: smtp; 421 att-e2xms-ibgw-6001a.ext.cloudfilter.net cmsmtp 96.102.19.37 blocked AUP#CNCT Final-recipient: rfc822; XXXXXXXXXX@txt.att.net Diagnostic-Code: smtp; 451 4.2.0 <XXXXXXX@comcast.net> server temporarily unavailable AUP#MXRT I likely have a ton of messages in queue and will wait for them to fail out and before testing again. Just leaving this for anyone who is having similar issues. Searches found many instances of this problem with other providers. For example.: Anyway, thanks again for the comments.

Sounds good and thanks again for helping out!