non-ENA0 as WAN
-
Hi Guys,
We are running PFSense as a virtualized firewall, and I have noticed a very peculiar behavior with the interface assignations—the firewall notoriously assigns the WAN interface upon the reboot to be ENA0, or the first interface in the system, while we have it as a second. The LAN interface is also being assigned as a Is this something that is expected, or is it a bug?
Regards
-
@Blade1024 said in non-ENA0 as WAN:
ENA0
What happens if you log in to the shell and change the assignment there (menu item #1). Doesn't that survive a reboot?
An alternative, depending on your Hypervisor is to remove the interfaces from the VM and assign them again in the order you want, and then reboot. First in the list will be WAN and the second is LAN... -
I assume this is in AWS? Which pfSense version?
It shouldn't do that. The interface assignment should not change across a reboot. Only if the config is defaulted should anything change.
-
Hi Steph,
It simply adds WAN addressing to the ena0 interface. I have tried in two versions - 24.11 and 25.03. I had to upgrade from 24.11 because it was crashing when attempting to NAT the traffic that should be sent into the IPSec tunnel, but both have the same behaviour. I tried to lock this down based on the MAC addresses of the unit, but then it simply doesn't want to start.
For changing them around, yes, I thought of that. From my point of view, ENA0 should not be a WAN. You start by configuring the MGMT or, if the box doesn't support it, the LAN interface, and only when you are sure, you configure the WAN. I think it would be fair to address this issue or, at the very least, define ENA0 as an explicit assignment that cannot be changed.Cheers
-
Hi,
I would have to try that explicitly, but from what I remember when I started it up, the behavior was the same.
Regards
-
@Blade1024 said in non-ENA0 as WAN:
Hi,
I would have to try that explicitly, but from what I remember when I started it up, the behavior was the same.
Regards
Whatever setting you define, should survive a reboot, once you have gone through the installation process, or change at some later point.
During install the order that pfsense assigns them is NIC0=WAN, NIC1=LAN, NIC2,3=OPT1, 2 etc. Not sure how things are ordered if you have a mix of NIC's, for example ix0, igb0 and a virtualized NIC like vtnet0 or ENA0 in your case.
@Blade1024 said in non-ENA0 as WAN:
You start by configuring the MGMT or, if the box doesn't support it, the LAN interface, and only when you are sure, you configure the WAN. I think it would be fair to address this issue or, at the very least, define ENA0 as an explicit assignment that cannot be changed.
Views may differ and pfsense has one way of doing it, and other FW's may do it differently. But, if you are not happy with the order, you simply change it. And you can either change from within pfsense if you SSH into it. Or you can remove and re-add the interfaces in a different order from the Hypervisor UI.
-
This is in AWS though correct?
Generally all interfaces are DHCP in AWS so the addressing is all defined by the AWS dhcp servers. If you try to use static addressing it often doesn't work if AWS doesn't see your lease.
How does it appear before and after reboot?
I'm not aware of any issue with NAT and IPSec that would cause a crash. Do you have any details from that?
-
About AWS and DHCP, not quite right - you can assign static IPs that were given to you by AWS via DHCP in the first instance. AWS allocates addresses per interface, and you can also add a secondary address. If you create an interface, you can directly see the interface via the EC2 CLI and use this address on the firewall; this will be fine. You can go as far as trying to allocate a specific address within the subnet (if it is available), while creating an interface. That's if you want to have a particular single address. However, if you're going to have an additional address, then you can also allocate it as a secondary and attach on the interface parallel to the primary. Then it should be usable on the firewall. This behavior is dictated by the AWS fabric - it doesn't support L2 protocols, and your ARP doesn't work.
About NAT, IPSec, and crash: The firewall reboots once it gets the route via BGP in IPSec on an attempt to push the traffic down the tunnel - a single packet, and it's gone. I don't have what precisely causes it, but my guess would be NAT - it is the only differentiator between the traffic pattern, because BGP P2P over the tunnel functions just fine. I have one of the units still unupgraded and available for testing, if required. Please let me know what to run. It was fixed moving to the 25.03, however, I have NAT issues now (https://forum.netgate.com/topic/197223/packets-are-not-nat-ted-and-encrypted-when-sent-over-ipsec2-interface). The crash is as follows:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0x99
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80ffd293
stack pointer = 0x28:0xfffffe004a6180f0
frame pointer = 0x28:0xfffffe004a6181e0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 54023 (ping)
rdi: fffffe004a6180f4 rsi: 0000000000000000 rdx: fffff800019dfe78
rcx: fffff800019dfe70 r8: fffffe004a6180f8 r9: 0000000000000010
rax: 0000000000000000 rbx: 0000000000000002 rbp: fffffe004a6181e0
r10: 0000001000000000 r11: 0000000000000000 r12: 0000000000000000
r13: 0000000000000002 r14: fffff800093fc1e0 r15: 0000000000000001
trap number = 12
panic: page fault
cpuid = 0
time = 1744365701
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe004a617de0
vpanic() at vpanic+0x13f/frame 0xfffffe004a617f10
panic() at panic+0x43/frame 0xfffffe004a617f70
trap_fatal() at trap_fatal+0x40b/frame 0xfffffe004a617fd0
trap_pfault() at trap_pfault+0x46/frame 0xfffffe004a618020
calltrap() at calltrap+0x8/frame 0xfffffe004a618020
--- trap 0xc, rip = 0xffffffff80ffd293, rsp = 0xfffffe004a6180f0, rbp = 0xfffffe004a6181e0 ---
pfr_pool_get() at pfr_pool_get+0x303/frame 0xfffffe004a6181e0
pf_map_addr() at pf_map_addr+0x7a3/frame 0xfffffe004a618270
pf_get_sport() at pf_get_sport+0x5d/frame 0xfffffe004a618320
pf_get_translation() at pf_get_translation+0x3b3/frame 0xfffffe004a6183a0
pf_test_rule() at pf_test_rule+0x301/frame 0xfffffe004a618830
pf_test() at pf_test+0x12c8/frame 0xfffffe004a618a00
pf_check_out() at pf_check_out+0x22/frame 0xfffffe004a618a20
pfil_mbuf_out() at pfil_mbuf_out+0x38/frame 0xfffffe004a618a50
ip_output() at ip_output+0xbf5/frame 0xfffffe004a618b50
rip_send() at rip_send+0x400/frame 0xfffffe004a618bc0
sosend_generic() at sosend_generic+0x643/frame 0xfffffe004a618c80
sousrsend() at sousrsend+0x5f/frame 0xfffffe004a618ce0
kern_sendit() at kern_sendit+0x144/frame 0xfffffe004a618d60
sendit() at sendit+0x1a8/frame 0xfffffe004a618db0
sys_sendto() at sys_sendto+0x4d/frame 0xfffffe004a618e00
amd64_syscall() at amd64_syscall+0x115/frame 0xfffffe004a618f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe004a618f30
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x15a2744d787a, rsp = 0x15a26f3738b8, rbp = 0x15a26f373920 ---
Uptime: 2m52s
Automatic reboot in 15 seconds - press a key on the console to abort -
I cannot really re-add them - hypervisor is AWS and it has its limitations. I have only two "physical" ena interfaces and one additional logical interface (ipsec2). That's the whole point - changing doesn't survive the reboot.
Regards
-
Huh, interesting. But 25.03 beta doesn't crash in the same situation?
-
Nope, but it has another NAT issue, as per here (https://forum.netgate.com/topic/197223/packets-are-not-nat-ted-and-encrypted-when-sent-over-ipsec2-interface). I'm stuck with the last one - it needs to be moved.
Regards
-
But in both cases you're seeing the WAN assignment change to ena0 across a reboot?
-
Yes
-
Is that shown in the config history? Is there a reason shown?
What exactly are you seeing change? I'm having a hard time imagining what could be happening here!
-
Please see below - it loads like this. No exceptions - just brings a single interface up and then puts all IP's as aliases on it (you see LAN and WAN together on one ena0). Then you can see me logging in as admin and re-assigning the interfaces. Right after, you see IPSec coming up as an indicator that WAN interface is in the right place.
P.S.: I would really appreciate some help with another issue I mentioned before - I am in a jam there. Cheers mate!
-
Hmm, that's the output in the main system log?
It's odd that it ena1 going up only after you connect.
But also back to my earlier question; what does the config history show when you initially log in?
I would also expect to see a whole bunch of other scripts triggered if you re-assign an interface like that.
Is LAN unassigned at that point? Assigned as ena0 also?
-
LAN and WAN are collapsed into one interface. I have to manually login and re-assign WAN to be on ena1. About scripts - I don't see anything
special there.Let me know where I can send you the debug information - I don't want to post it publicly.
-
Sure, please upload anything you can here: https://nc.netgate.com/nextcloud/s/eF4YsKErrP97X6A
-
Added logs, config and network environment files into the drop point. let me know if you need anything else.
Regards
-
Ok, so when were those files taken?
The system log looks like you restored a config file after the initial boot?
I assume that was not the config file that you uploaded?
Those also look like 24.11 logs only. I assume 25.03 shows similarly?