2.5 RC Testing

tomahhunt · Feb 10, 2021, 8:12 AM

I thought I would give the 2.5 RC a try.
Upgraded last night which was flawless.
All working fine.
Went to bed.
When i woke up I had enough functioning internet to vpn to work.
But web browsing was broken.
Restarted pfsense and it's working again.

This is a pfsense running in a proxmox VM.

Any useful logs and or debug I can do IF/WHEN it happens again?

jimp · Feb 10, 2021, 1:48 PM

Without more information it's impossible to say. Did you not keep any logs?

A few things to check if it happens again:

Check every log tab for errors, starting with the main System tab
Are all the expected services up and running under Status > Services?
What is the gateway status?
Can you ping out by IP address?
Does DNS work?

That kind of basic stuff.

tomahhunt · Feb 10, 2021, 1:52 PM

Thanks. Yes I had to restart quickly because I was in trouble with the other half for breaking the internet :)

In the minutes I had:
I checked all services were running which they were and I specifically restarted DNS as I figured that might be the issue.
Gateway status was good.
My vpn was connected to my work via openvpn fine and I could access my work machine.
i didn't try direc try direct pin.

It does feel like a DNS problem from the above symptoms.
Luckily no issues today and I will try and grab logs if it happens again.

Cheers.

zleppy · Feb 11, 2021, 2:51 PM

@tomahhunt I've seen something similar happening. But mostly after a restart. Try restarting the "dns resolver" service next time under the service tab.

The service icon will be showing green but it wont resolve anything. Couldn't find anything specific in the logs either.

JD 0 · Feb 11, 2021, 4:39 PM

I did an upgrade-in-place from 2.4.5 on wed using the 2nd RC spin. So far it's been stable. The upgrade process itself was a total snore (that's a compliment) -- can't recall the last time an upgrade "just worked" without issues and I wasn't upgrading a mobile device.

I see a new spin dropped this morning, I'll roll that one when I can find a window in the workday while simultaneously distracting my family away from the network. :-)

ESX VM, Dual WAN (1 Gb Pri, 50 Mb failover), 5 subnets, OpenVPN

JD 0 · Feb 11, 2021, 5:03 PM

@jd-0 And no sooner than I wrote the previous, our Gb link started taking some errors (work going on in the neighborhood). As expected, pfsense smoothly dropped that gateway and favored traffic over to the backup, then switched back when the packet errors cleared.

JD 0 · Feb 11, 2021, 5:15 PM

@jd-0 One small glitch -- my running 2.5.0 build didn't swap dynamic dns (easyDNS) for VPN back from the failover event after the primary was restored. Going into the Dynamic DNS client page and forcing save/update swapped it back. Will update if this recurs on future spins. Looks like it simply didn't trigger at all as I have email notifications set up and I normally receive notification of the DNS switch when the the links failover/giveback, but nothing this time after it switched back beyond the normal notification that the gateway had been re-added to the group.

JD 0 · Feb 11, 2021, 5:26 PM

@jd-0 Here are the event notification in order for reference:

Notifications in this message: 1

Notifications in this message: 1

10:44:54 DynDNS updated IP Address on 02_ATT_WAN_50MB (vmx1) to 108.195.xx.xx

Notifications in this message: 1

** Manual update of dynamic DNS **

Notifications in this message: 1

11:07:59 DynDNS updated IP Address on 01_COMCAST_WAN_1GB (vmx3) to 68.47.xx.xx

bimmerdriver · Feb 14, 2021, 6:00 PM

I upgraded my test system to 2.5.0 RC yesterday. Today when I returned to take a look, the pfsense webgui was dead. The console was still operational. I rebooted and it came back. There was a new release, so I upgraded to it.

tomahhunt · Feb 14, 2021, 6:05 PM

To follow up on my original post.
It's not happened again. Solid ever since the one incident.
I've also kept up to date with the dailys which means I've restarted often. So we shall see as the releases settle.

tomahhunt · Feb 19, 2021, 8:13 AM

After upgrading to the final 2.5.0, all was well, but then I had a crash overnight.
Crashlogs attached.

info.0
textdump.tar.0

jimp · Feb 19, 2021, 4:07 PM

That one doesn't look familiar to me...

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address	= 0x0
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80e34921
stack pointer	        = 0x28:0xfffffe00401f67e0
frame pointer	        = 0x28:0xfffffe00401f6820
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (swi1: netisr 0)
trap number		= 12
panic: page fault
cpuid = 2
time = 1613691991
KDB: enter: panic

db:0:kdb.enter.default>  show pcpu
cpuid        = 2
dynamic pcpu = 0xfffffe007f159380
curthread    = 0xfffff800053c1000: pid 12 tid 100039 "swi1: netisr 0"
curpcb       = 0xfffff800053c15a0
fpcurthread  = none
idlethread   = 0xfffff80005341000: tid 100005 "idle: cpu2"
curpmap      = 0xffffffff8368d5a8
tssp         = 0xffffffff837176f0
commontssp   = 0xffffffff837176f0
rsp0         = 0xfffffe00401f6cc0
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff8371df08
ldt          = 0xffffffff8371df48
tss          = 0xffffffff8371df38
tlb gen      = 372632
curvnet      = 0xfffff8000508fc80
db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100039 td 0xfffff800053c1000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00401f64a0
vpanic() at vpanic+0x197/frame 0xfffffe00401f64f0
panic() at panic+0x43/frame 0xfffffe00401f6550
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00401f65b0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00401f6600
trap() at trap+0x286/frame 0xfffffe00401f6710
calltrap() at calltrap+0x8/frame 0xfffffe00401f6710
--- trap 0xc, rip = 0xffffffff80e34921, rsp = 0xfffffe00401f67e0, rbp = 0xfffffe00401f6820 ---
sbappendaddr_locked_internal() at sbappendaddr_locked_internal+0x81/frame 0xfffffe00401f6820
sbappendaddr_locked() at sbappendaddr_locked+0x93/frame 0xfffffe00401f6860
rip_append() at rip_append+0xd2/frame 0xfffffe00401f68a0
rip_input() at rip_input+0x27a/frame 0xfffffe00401f6940
icmp_input() at icmp_input+0x1a1/frame 0xfffffe00401f6a30
ip_input() at ip_input+0x168/frame 0xfffffe00401f6ae0
swi_net() at swi_net+0x12b/frame 0xfffffe00401f6b50
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00401f6bb0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00401f6bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00401f6bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

What kind of features do you have enabled on there?

I don't see any exact matches on the code path there but it appears to be something working with raw IP sockets. Do you maybe have NAT+Proxy reflection enabled?

tomahhunt · Feb 19, 2021, 5:48 PM

@jimp I would say my setup is pretty vanilla.
Extra pakages installed are:

acme
bandiwdthd
iperf
openvpn-client-export
pfBlockerNG-devel

But pfblocker/iperf is disabled and acme not in use.

No NAT rules in use.

The only oddity is that it runs inside proxmox.