non-ENA0 as WAN
-
This is in AWS though correct?
Generally all interfaces are DHCP in AWS so the addressing is all defined by the AWS dhcp servers. If you try to use static addressing it often doesn't work if AWS doesn't see your lease.
How does it appear before and after reboot?
I'm not aware of any issue with NAT and IPSec that would cause a crash. Do you have any details from that?
-
About AWS and DHCP, not quite right - you can assign static IPs that were given to you by AWS via DHCP in the first instance. AWS allocates addresses per interface, and you can also add a secondary address. If you create an interface, you can directly see the interface via the EC2 CLI and use this address on the firewall; this will be fine. You can go as far as trying to allocate a specific address within the subnet (if it is available), while creating an interface. That's if you want to have a particular single address. However, if you're going to have an additional address, then you can also allocate it as a secondary and attach on the interface parallel to the primary. Then it should be usable on the firewall. This behavior is dictated by the AWS fabric - it doesn't support L2 protocols, and your ARP doesn't work.
About NAT, IPSec, and crash: The firewall reboots once it gets the route via BGP in IPSec on an attempt to push the traffic down the tunnel - a single packet, and it's gone. I don't have what precisely causes it, but my guess would be NAT - it is the only differentiator between the traffic pattern, because BGP P2P over the tunnel functions just fine. I have one of the units still unupgraded and available for testing, if required. Please let me know what to run. It was fixed moving to the 25.03, however, I have NAT issues now (https://forum.netgate.com/topic/197223/packets-are-not-nat-ted-and-encrypted-when-sent-over-ipsec2-interface). The crash is as follows:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x99
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80ffd293
stack pointer = 0x28:0xfffffe004a6180f0
frame pointer = 0x28:0xfffffe004a6181e0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 54023 (ping)
rdi: fffffe004a6180f4 rsi: 0000000000000000 rdx: fffff800019dfe78
rcx: fffff800019dfe70 r8: fffffe004a6180f8 r9: 0000000000000010
rax: 0000000000000000 rbx: 0000000000000002 rbp: fffffe004a6181e0
r10: 0000001000000000 r11: 0000000000000000 r12: 0000000000000000
r13: 0000000000000002 r14: fffff800093fc1e0 r15: 0000000000000001
trap number = 12
panic: page fault
cpuid = 0
time = 1744365701
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe004a617de0
vpanic() at vpanic+0x13f/frame 0xfffffe004a617f10
panic() at panic+0x43/frame 0xfffffe004a617f70
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe004a617fd0
trap_pfault() at trap_pfault+0x46/frame 0xfffffe004a618020
calltrap() at calltrap+0x8/frame 0xfffffe004a618020
--- trap 0xc, rip = 0xffffffff80ffd293, rsp = 0xfffffe004a6180f0, rbp = 0xfffffe004a6181e0 ---
pfr_pool_get() at pfr_pool_get+0x303/frame 0xfffffe004a6181e0
pf_map_addr() at pf_map_addr+0x7a3/frame 0xfffffe004a618270
pf_get_sport() at pf_get_sport+0x5d/frame 0xfffffe004a618320
pf_get_translation() at pf_get_translation+0x3b3/frame 0xfffffe004a6183a0
pf_test_rule() at pf_test_rule+0x301/frame 0xfffffe004a618830
pf_test() at pf_test+0x12c8/frame 0xfffffe004a618a00
pf_check_out() at pf_check_out+0x22/frame 0xfffffe004a618a20
pfil_mbuf_out() at pfil_mbuf_out+0x38/frame 0xfffffe004a618a50
ip_output() at ip_output+0xbf5/frame 0xfffffe004a618b50
rip_send() at rip_send+0x400/frame 0xfffffe004a618bc0
sosend_generic() at sosend_generic+0x643/frame 0xfffffe004a618c80
sousrsend() at sousrsend+0x5f/frame 0xfffffe004a618ce0
kern_sendit() at kern_sendit+0x144/frame 0xfffffe004a618d60
sendit() at sendit+0x1a8/frame 0xfffffe004a618db0
sys_sendto() at sys_sendto+0x4d/frame 0xfffffe004a618e00
amd64_syscall() at amd64_syscall+0x115/frame 0xfffffe004a618f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe004a618f30
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x15a2744d787a, rsp = 0x15a26f3738b8, rbp = 0x15a26f373920 ---
Uptime: 2m52s
Automatic reboot in 15 seconds - press a key on the console to abort -
I cannot really re-add them - hypervisor is AWS and it has its limitations. I have only two "physical" ena interfaces and one additional logical interface (ipsec2). That's the whole point - changing doesn't survive the reboot.
Regards
-
Huh, interesting. But 25.03 beta doesn't crash in the same situation?
-
Nope, but it has another NAT issue, as per here (https://forum.netgate.com/topic/197223/packets-are-not-nat-ted-and-encrypted-when-sent-over-ipsec2-interface). I'm stuck with the last one - it needs to be moved.
Regards
-
But in both cases you're seeing the WAN assignment change to ena0 across a reboot?
-
Yes
-
Is that shown in the config history? Is there a reason shown?
What exactly are you seeing change? I'm having a hard time imagining what could be happening here!
-
Please see below - it loads like this. No exceptions - just brings a single interface up and then puts all IP's as aliases on it (you see LAN and WAN together on one ena0). Then you can see me logging in as admin and re-assigning the interfaces. Right after, you see IPSec coming up as an indicator that WAN interface is in the right place.
P.S.: I would really appreciate some help with another issue I mentioned before - I am in a jam there. Cheers mate!
-
Hmm, that's the output in the main system log?
It's odd that it ena1 going up only after you connect.
But also back to my earlier question; what does the config history show when you initially log in?
I would also expect to see a whole bunch of other scripts triggered if you re-assign an interface like that.
Is LAN unassigned at that point? Assigned as ena0 also?
-
LAN and WAN are collapsed into one interface. I have to manually login and re-assign WAN to be on ena1. About scripts - I don't see anything
special there.Let me know where I can send you the debug information - I don't want to post it publicly.
-
Sure, please upload anything you can here: https://nc.netgate.com/nextcloud/s/eF4YsKErrP97X6A
-
Added logs, config and network environment files into the drop point. let me know if you need anything else.
Regards
-
Ok, so when were those files taken?
The system log looks like you restored a config file after the initial boot?
I assume that was not the config file that you uploaded?
Those also look like 24.11 logs only. I assume 25.03 shows similarly?
-
After encountering the VPN issue, I attempted to use the paid PFSense and was indeed restoring the configuration from the community edition. Files were taken this morning, while I was restoring them about 24 hours before. I will be restoring the 25.03 version because of the bugs, and can send these as well - they should be looking similar.
-
Yes, I'd like to see that. I'd also particularly like to see the configuration history after it fails. The only thing I can imagine so far is that is fails to detect ena1 and somehow assigns all interfaces to the only remaining NIC.
But if that did happen it should drop to the interfaces assign prompt because the config contains interfaces no longer in the system.
In the config you uploaded though both WAN and LAN are assigned as ena0 and that shouldn't be possible.
