Many thanks for the detailed options and pointing me in the right direction bmeeks! You have definitely given me some good areas to research further.

@jimp
Yes, I have some of the recommended patches applied, including the one you mentioned, so a version of 22.9 is expected.
I see that https://docs.netgate.com/pfsense/en/latest/releases/versions.html now documents 23.05 as config version 22.9.
Thank you for your clarifications to resolve my concern.
Regards.

@stephenw10 thank you kindly sir I was downvoted on the reddit for this same question I appreciate your time.

Typical filename looks like this: config-pfsense.mydomain.biz-20220222183909.xml, where pfsense.mydomain.biz is to replace with the pfSense name followed by FQDN.

Regards

@stephenw10 Got it. Very much appreciate the info! Thank you!

@bmeeks said in Weird issue with SMTP:

https://redmine.pfsense.org/issues/14031#note-15.

Aha !
So, a non root instance couldn't overwrite with an empty queue file so it kept on sending the same queue content.
Nice catch !

Thanks for the patch code 👍

What gateway are you being passed by the ISP? No route error like that implies it might be outside the WAN subnet. There's a advanced gateway setting for allowing a gateway like that: 'use non-local gateway'. If that's what is happening.

Yeah if you don't disable the blacklist you will only get one queue for Rx and Tx on a vmx NIC. As you say for a PPP connection that won't make any difference but for anything else it will.

That is fixed in 23.05: https://redmine.pfsense.org/issues/14117

The patch for it is also included in the recommend patches list in the System Patches package.

Steve

Me too! 🤞

@stephenw10
I just started to remove all additional scripts running from /usr/local/etc/rc.d
and found that removing dyndns.sh does help.
I can now reboot the system without problem.

#!/usr/local/bin/bash while true; do IP_ADDRESS=$(ifconfig pppoe0 | grep "inet " | awk '{print $2}') if [ -z "$IP_ADDRESS" ]; then # PPPoE connection does not have a valid IP address logger "PPPoE connection does not have a valid IP address" else # PPPoE connection has a valid IP address /etc/rc.dyndns.update logger "PPPoE connection has a valid IP address, force DYNDNS" fi sleep 3600 done

I don't really remember if that some manual script I have been some years ago installed, or it is part of pfSense+ but it is the same on the secondary firewall and just works… Can not explain what exactly triggering this issue with reboot. My clean VM just do not have any scripts… but… it's not PPPoE and there are no hosts configured…
Ok. Changed this to:

#!/usr/local/bin/bash case "$1" in start) while true; do IP_ADDRESS=$(ifconfig pppoe0 | grep "inet " | awk '{print $2}') if [ -z "$IP_ADDRESS" ]; then # PPPoE connection does not have a valid IP address logger "PPPoE connection does not have a valid IP address" else # PPPoE connection has a valid IP address /etc/rc.dyndns.update logger "PPPoE connection has a valid IP address, force DYNDNS" fi sleep 3600 done ;; stop) exit 0 ;; esac exit 0

And reboot works just fine... so it possible that sometime ago I have just generated this problem that was so hard to debug. Anyway, thank you for trying to help me!
Edited:
Yes, definitely it was manual script added, just because for some reason dynDNS was not updated.

Backtrace:

db:1:pfs> bt Tracing pid 79686 tid 100334 td 0xfffffe010ce053a0 kdb_enter() at kdb_enter+0x32/frame 0xfffffe010bfa8900 vpanic() at vpanic+0x182/frame 0xfffffe010bfa8950 panic() at panic+0x43/frame 0xfffffe010bfa89b0 trap_fatal() at trap_fatal+0x409/frame 0xfffffe010bfa8a10 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe010bfa8a70 calltrap() at calltrap+0x8/frame 0xfffffe010bfa8a70 --- trap 0xc, rip = 0xffffffff80f9352c, rsp = 0xfffffe010bfa8b40, rbp = 0xfffffe010bfa8b70 --- X_ip_mrouter_done() at X_ip_mrouter_done+0x31c/frame 0xfffffe010bfa8b70 rip_detach() at rip_detach+0x3f/frame 0xfffffe010bfa8ba0 sorele_locked() at sorele_locked+0x89/frame 0xfffffe010bfa8bc0 soclose() at soclose+0xeb/frame 0xfffffe010bfa8c20 _fdrop() at _fdrop+0x11/frame 0xfffffe010bfa8c40 closef() at closef+0x24b/frame 0xfffffe010bfa8cd0 fdescfree() at fdescfree+0x4b3/frame 0xfffffe010bfa8d90 exit1() at exit1+0x4c7/frame 0xfffffe010bfa8df0 sys_exit() at sys_exit+0xd/frame 0xfffffe010bfa8e00 amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe010bfa8f30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe010bfa8f30 --- syscall (1, FreeBSD ELF64, sys_exit), rip = 0x822b5786a, rsp = 0x820a03288, rbp = 0x820a032a0 ---

Panic:

Fatal trap 12: page fault while in kernel mode cpuid = 6; apic id = 06 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff80f9352c stack pointer = 0x28:0xfffffe010bfa8b40 frame pointer = 0x28:0xfffffe010bfa8b70 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 79686 (pimd) rdi: fffffe00e204ad18 rsi: 4 rdx: 1 rcx: 0 r8: 0 r9: fffff80010067000 rax: 100 rbx: fffffe010ce053a0 rbp: fffffe010bfa8b70 r10: 0 r11: 800000044d83ed99 r12: fffffe010ce053a0 r13: 0 r14: fffff805734b4700 r15: 0 trap number = 12 panic: page fault cpuid = 6 time = 1680781157 KDB: enter: panic

Console also shows:

config_aqm Unable to configure flowset, flowset busy! config_aqm Unable to configure flowset, flowset busy!

It's probably this or related to it: https://redmine.pfsense.org/issues/12079
Except it's in pimd rather than igmpproxy hence the differences.

Steve

If you're able to replicate it then a bug would be helpful. We would need to know what the config was in 22.01 in order to prevent it failing it upgrade.

Steve

@nocling I can't thank you enough. This worked! I've read so much documentation, posted in numerous forums, etc. No one brought up the switch aspect. Thanks!!!

@soupdiver said in Unable to Register pfSense Plus:

I guess some kind of user error would be nice here

Hmm, I agree. Let me see what we can do there.

@riggi Yes, the first boot after you restore the config to the bare metal device, pfSense will prompt you to correct/assign interfaces.

You can also edit the config.xml file to change the interface names before restoring with a tool like NotePad++. Being careful to replace the names individually and not just do a lazy-man 'replace all'. Simple and effective.

@stephenw10
Well I have not tested on 23.01 but I used to get similar issues for many of my installations with 2.6.

Yes, Ofcourse, Rebooting firewall or restarting service makes the gateway come online.

Recently I found a work around, if it gives some kind of pointer. I have set the WAN as static IP instead of dhcp. This solves the pending issue. I guess it is more of an issue in uplink modem, unable to assign a dhcp address to WAN port of firewall.

So i guess there is no issue with pfsense.

@jknott The earliest computer I had was an At&t PC6300 it had a DB-9 for the keyboard, monochrome guy. I also remember my Tandy 102 my uncle got us one christmas had a DB25 on the back. My Dad had a Commodore 64 I never got to play with it. The thing was disconnected by the time I was able to. Again, the monitor was dead that went with it and it was outdated at that point but that guy had some connections on the back also. Today I have the C64 mini so I got to play with it in the end, Thank you Santa!!!

You could do this using an alias with all the client subnets in it and then use that as the source in the firewall rule at site A on the IPSec tab.
That wouldn't filter clients that are at site A that don't use tunnel so you'd still need a rule on the client VLAN there directly.
Or as you say you could put that rule as floating outbound on the resources VLAN at site A.

There are some ways to realize it;

Each LAN Port gets an own subnet like
192.168.1.0/24 and on the next one 192.168.2.0/24 You can also add a switch to each LAN port and enrich
that scenario for more users or devices. You may be able to work with VLANs for privat and home
VLAN10 = Home - 192.168.1.0/24
VLAN20 = Work - 192.168.2.0/24
VLAN30 = WiFi - 172.xxx

You may be able to set up behind the pfSense also a small
MikroTik router for each network if you want.

There are many ways you may be able to walk on.