Some hosts aren't connecting to the internet but others are

[nimrod]

@mechtheist

This looks like a hardware issue to me. Check the storage device on the system that hosts your VM. Check the system RAM as well. The fact that you reinstalled pfSense, and you had those issues where CPU load status indicator is not loading, is leading me to belieive there might be something wrong with your hardware. We already know that NIC is fine, thats one thing ruled out, check everything else. Verify your VM settings as well. There could be an update that reverted your CPU/RAM/Storage settings for your VM. Its pointless looking for an issue in pfSense until the host system is not checked.

[stephenw10]

Hmm, this seems more like VBox issue to me.

Can you try running in Hyper-V instead?

Or, better, running bare metal as a test?

Steve

[mechtheist]

@nimrod said in Some hosts aren't connecting to the internet but others are:

@mechtheist

This looks like a hardware issue to me. Check the storage device on the system that hosts your VM. Check the system RAM as well. The fact that you reinstalled pfSense, and you had those issues where CPU load status indicator is not loading, is leading me to belieive there might be something wrong with your hardware. We already know that NIC is fine, thats one thing ruled out, check everything else. Verify your VM settings as well. There could be an update that reverted your CPU/RAM/Storage settings for your VM. Its pointless looking for an issue in pfSense until the host system is not checked.

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

Hmm, this seems more like VBox issue to me.

Can you try running in Hyper-V instead?

Or, better, running bare metal as a test?

Steve

I've had this problem for many months and looked but have never found anything that I could determine to be a possible cause in pfsense or the PC. It's weird that doing a fresh install and little configuration dropped the already low bandwidth by a factor of almost 10, from about 17Mbps to about 1.7 Mbps. I have v v little experience with Hyper-V and none with any other VM software but I was starting to think about trying Hyper-V . For about 6 years at least I've done this with VirtualBox and had no real problems, pfsense just worked, it's always kinda amazed and really impressed me.

That said, a couple of hours ago it crashed, below is the report copied off the web page, I don't even remember getting one of these before and definitely don't have the knowledge to understand what it's telling me, anything stick out to you guys? I don't know how to fix the nonprinting characters, sorry.
Had to cut a lot out, too long, so attached file. Thanks for all the help!

Crash report begins.  Anonymous machine information:

amd64
12.2-STABLE
FreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense

Crash report details:

No PHP errors found.

Filename: /var/crash/info.0
Dump header from device: /dev/ada0p2
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 69632
  Blocksize: 512
  Compression: none
  Dumptime: Sun Nov 14 14:03:31 2021
  Hostname: pfSense.domain.org
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense
  Panic String: page fault
  Dump Parity: 3658854709
  Bounds: 0
  Dump Status: good

Filename: /var/crash/textdump.tar.0
ddb.txt--run of nonprointing characters--db:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo>  show alllocks
No such command; use "help" to list available commands
db:1:lockinfo>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 6
dynamic pcpu = 0xfffffe007f83f380
curthread    = 0xfffff800054ad000: pid 12 tid 100055 "swi4: clock (0)"
curpcb       = 0xfffff800054ad5a0
fpcurthread  = none
idlethread   = 0xfffff8000540e000: tid 100009 "idle: cpu6"
curpmap      = 0xffffffff8368d5a8
tssp         = 0xffffffff83717890
commontssp   = 0xffffffff83717890
rsp0         = 0xfffffe00005a7cc0
kcr3         = 0x80000000040d4003
ucr3         = 0xffffffffffffffff
scr3         = 0x13d79ee25
gs32p        = 0xffffffff8371e0a8
ldt          = 0xffffffff8371e0e8
tss          = 0xffffffff8371e0d8
tlb gen      = 15968
curvnet      = 0xfffff800050bab80
db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100055 td 0xfffff800054ad000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00005a7520
vpanic() at vpanic+0x197/frame 0xfffffe00005a7570
panic() at panic+0x43/frame 0xfffffe00005a75d0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00005a7630
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00005a7680
trap() at trap+0x286/frame 0xfffffe00005a7790
calltrap() at calltrap+0x8/frame 0xfffffe00005a7790
--- trap 0xc, rip = 0xffffffff810343b3, rsp = 0xfffffe00005a7860, rbp = 0xfffffe00005a78b0 ---
in6_unlink_ifa() at in6_unlink_ifa+0x63/frame 0xfffffe00005a78b0
in6_purgeaddr() at in6_purgeaddr+0x309/frame 0xfffffe00005a7a30
nd6_timer() at nd6_timer+0xc2/frame 0xfffffe00005a7a80
softclock_call_cc() at softclock_call_cc+0x141/frame 0xfffffe00005a7b30
softclock() at softclock+0x79/frame 0xfffffe00005a7b50
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00005a7bb0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00005a7bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005a7bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db:0:kdb.enter.default>  ps

...
long list of devices and options cut out because it was too long
...

--run of nonprointing characters--
---<<BOOT>>---
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense amd64
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
VT(vga): text 80x25
CPU: Intel(R) Core(TM) i7-5960X CPU @ 3.00GHz (3000.19-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x5eda2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x2421<FSGSBASE,AVX2,INVPCID,NFPUSG>
  Structured Extended Features3=0x10000400<MD_CLEAR,L1DFL>
  TSC: P-state invariant
real memory  = 11149508608 (10633 MB)
avail memory = 10224226304 (9750 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <VBOX   VBOXAPIC>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 cache groups x 1 core(s)
random: unblocking device.
ioapic0: MADT APIC ID 8 != hw id 0
ioapic0 <Version 2.0> irqs 0-23 on motherboard
Launching APs: 6 3 7 4 5 1 2
wlan: mac acl policy registered
random: entropy device external interface
module_register_init: MOD_LOAD (vesa, 0xffffffff8140c3e0, 0) error 19
kbd1 at kbdmux0
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
[ath_hal] loaded
000.000060 [4336] netmap_init               netmap: loaded module
mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
padlock0: No ACE support.
acpi0: <VBOX VBOXXSDT> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
vgapci0: <VGA-compatible display> port 0xd000-0xd00f mem 0xe0000000-0xe7ffffff,0xf0000000-0xf01fffff irq 18 at device 2.0 on pci0
vgapci0: Boot video device
virtio_pci0: <VirtIO PCI Network adapter> port 0xd020-0xd03f irq 19 at device 3.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
<6>vtnet0: Ethernet address: 08:00:27:b1:44:9a
<6>vtnet0: netmap queues/slots: TX 1/128, RX 1/128
000.000243 [ 445] vtnet_netmap_attach       vtnet attached txq=1, txd=128 rxq=1, rxd=128
pcm0: <Intel ICH (82801AA)> port 0xd100-0xd1ff,0xd200-0xd23f irq 21 at device 5.0 on pci0
pcm0: <SigmaTel STAC9700/83/84 AC97 Codec>
pci0: <bridge> at device 7.0 (no driver attached)
virtio_pci1: <VirtIO PCI Network adapter> port 0xd240-0xd25f irq 16 at device 8.0 on pci0
vtnet1: <VirtIO Networking Adapter> on virtio_pci1
<6>vtnet1: Ethernet address: 08:00:27:19:a7:6d
<6>vtnet1: netmap queues/slots: TX 1/128, RX 1/128
000.000245 [ 445] vtnet_netmap_attach       vtnet attached txq=1, txd=128 rxq=1, rxd=128
virtio_pci2: <VirtIO PCI Network adapter> port 0xd260-0xd27f irq 17 at device 9.0 on pci0
vtnet2: <VirtIO Networking Adapter> on virtio_pci2
<6>vtnet2: Ethernet address: 08:00:27:99:d8:40
<6>vtnet2: netmap queues/slots: TX 1/128, RX 1/128
000.000246 [ 445] vtnet_netmap_attach       vtnet attached txq=1, txd=128 rxq=1, rxd=128
virtio_pci3: <VirtIO PCI Network adapter> port 0xd280-0xd29f irq 18 at device 10.0 on pci0
vtnet3: <VirtIO Networking Adapter> on virtio_pci3
<6>vtnet3: Ethernet address: 08:00:27:70:a7:e2
<6>vtnet3: netmap queues/slots: TX 1/128, RX 1/128
000.000247 [ 445] vtnet_netmap_attach       vtnet attached txq=1, txd=128 rxq=1, rxd=128
ahci0: <Intel ICH8M AHCI SATA controller> port 0xd2a0-0xd2a7,0xd2a8-0xd2ab,0xd2b0-0xd2b7,0xd2b8-0xd2bb,0xd2c0-0xd2cf mem 0xf0804000-0xf0805fff irq 21 at device 13.0 on pci0
ahci0: AHCI v1.10 with 2 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
acpi_acad0: <AC Adapter> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xe2000-0xe2fff pnpid ORM0000 on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 10.000 msec
Trying to mount root from zfs:pfSense/ROOT/default []...
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
ada0 at ahcich1 bus 0 scbus1 target 0 lun 0
ada0: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada0: Serial Number VBa60260db-6c82640f
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 76341MB (156346752 512 byte sectors)
cd0 at ahcich0 bus 0 scbus0 target 0 lun 0
cd0: <VBOX CD-ROM 1.0> Removable CD-ROM SPC-3 SCSI device
cd0: Serial Number VB0-1a2b3c4d
cd0: 300.000MB/s transfers (SATA 2.x, UDMA6, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
<118>Configuring crash dumps...
<118>Using /dev/ada0p2 for dump device.
<118>Filesystems are clean, continuing...
<118>Mounting filesystems...
<118>
<118>        __
<118> _ __  / _|___  ___ _ __  ___  ___
<118>| '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \
<118>| |_) |  _\__ \  __/ | | \__ \  __/
<118>| .__/|_| |___/\___|_| |_|___/\___|
<118>|_|
<118>
<118>
<118>Welcome to pfSense 2.5.2-RELEASE...
<118>
<118>No core dumps found.
<118>...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.32/mach/CORE
<118>32-bit compatibility ldconfig path:
<118>done.
<118>>>> Removing vital flag from php74... done.
<118>External config loader 1.0 is now starting... ada0p1 ada0p3
<118>Launching the init system...Updating CPU Microcode...
CPU: Intel(R) Core(TM) i7-5960X CPU @ 3.00GHz (3000.19-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x5eda2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x2421<FSGSBASE,AVX2,INVPCID,NFPUSG>
  Structured Extended Features3=0x10000400<MD_CLEAR,L1DFL>
  TSC: P-state invariant
<118>Done.
<118>.... done.
<118>Initializing.................. done.
<118>Starting device manager (devd)...
intsmb0: <Intel PIIX4 SMBUS Interface> irq 23 at device 7.0 on pci0
intsmb0: intr IRQ 9 enabled revision 0
smbus0: <System Management Bus> on intsmb0
<118>done.
<118>Loading configuration......done.
<118>Updating configuration...done.
<118>Checking config backups consistency.................................done.
<118>Setting up extended sysctls...done.
<118>Setting timezone...done.
<118>Configuring loopback interface...
<6>lo0: link state changed to UP
<118>done.
<118>Starting syslog...done.
<118>Starting Secure Shell Services...done.
<118>Setting up interfaces microcode...done.
<118>Starting PC/SC Smart Card Services...done.
<118>Configuring loopback interface...done.
<118>Creating wireless clone interfaces...done.
<118>Configuring LAGG interfaces...done.
<118>Configuring VLAN interfaces...done.
<118>Configuring QinQ interfaces...done.
<118>Configuring WAN interface...
<6>vtnet0: link state changed to UP
<118>done.
<118>Configuring LAN interface...
<6>vtnet1: link state changed to UP
<118>done.
<118>Configuring SEC interface...
<6>vtnet2: link state changed to UP
<118>done.
<118>Configuring MINTSERVER interface...
<6>vtnet3: link state changed to UP
<118>done.
<118>Configuring IPsec VTI interfaces...done.
<118>Configuring CARP settings...done.
<118>Syncing OpenVPN settings...done.
<118>Configuring firewall.
<6>pflog0: promiscuous mode enabled
<118>.....done.
<118>Starting PFLOG...done.
<118>Setting up gateway monitors...done.
<118>Setting up static routes...done.
<118>Setting up DNSs...
<118>Starting DNS Resolver...done.
<118>Synchronizing user settings...done.
<118>Starting webConfigurator...done.
<118>Configuring CRON...done.
<118>Starting NTP Server...done.
<118>Starting DHCP service...done.
<118>Configuring firewall......done.
<118>Generating RRD graphs...done.
<118>Starting syslog...done.
<118>Starting CRON... done.
<118>pfSense 2.5.2-RELEASE amd64 Fri Jul 02 15:33:00 EDT 2021
<118>Bootup complete


Fatal trap 12: page fault while in kernel mode
cpuid = 6; apic id = 06
fault virtual address	= 0x28
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff810343b3
stack pointer	        = 0x28:0xfffffe00005a7860
frame pointer	        = 0x28:0xfffffe00005a78b0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (swi4: clock (0))
trap number		= 12
panic: page fault
cpuid = 6
time = 1636920211
KDB: enter: panic
more nonprinting ch.[textdump.tar-11-14-2021.0](/assets/uploads/files/1636926910680-textdump.tar-11-14-2021.0) 	

[nimrod]

@mechtheist

At this point, the fact that some devices are connecting and some dont, is completely irrelevant. You did a fresh install of pfSense, and your traffic speed was cut anyway. Thats not how clean install of pfSense performs under normal conditions.

Lets try again.

1. Did you check VirtualBox VM settings ?
2. Did you test system RAM on the machine that hosts your VM?
3. Did you check the storage on the machine that hosts your VM/VM Images and OS ?

[stephenw10]

@mechtheist said in Some hosts aren't connecting to the internet but others are:

db:0:kdb.enter.default> bt
Tracing pid 12 tid 100055 td 0xfffff800054ad000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00005a7520
vpanic() at vpanic+0x197/frame 0xfffffe00005a7570
panic() at panic+0x43/frame 0xfffffe00005a75d0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00005a7630
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00005a7680
trap() at trap+0x286/frame 0xfffffe00005a7790
calltrap() at calltrap+0x8/frame 0xfffffe00005a7790
--- trap 0xc, rip = 0xffffffff810343b3, rsp = 0xfffffe00005a7860, rbp = 0xfffffe00005a78b0 ---
in6_unlink_ifa() at in6_unlink_ifa+0x63/frame 0xfffffe00005a78b0
in6_purgeaddr() at in6_purgeaddr+0x309/frame 0xfffffe00005a7a30
nd6_timer() at nd6_timer+0xc2/frame 0xfffffe00005a7a80
softclock_call_cc() at softclock_call_cc+0x141/frame 0xfffffe00005a7b30
softclock() at softclock+0x79/frame 0xfffffe00005a7b50
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00005a7bb0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00005a7bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005a7bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

That's the important part there, the backtrace, and it's inconclusive.
But it is showing an issue in IPv6. Maybe your old config didn't have IPv6 enabled at all?
Maybe your ISP jusy turned on IPv6 (or some port of it) and it's broken? Or your config is somehow not compatible with it?

I would be testing pfSense bare metal if you can't try hyper-v.

Steve

[mechtheist]

@nimrod said in Some hosts aren't connecting to the internet but others are:

@mechtheist

At this point, the fact that some devices are connecting and some dont, is completely irrelevant. You did a fresh install of pfSense, and your traffic speed was cut anyway. Thats not how clean install of pfSense performs under normal conditions.

Lets try again.

1. Did you check VirtualBox VM settings ?
2. Did you test system RAM on the machine that hosts your VM?
3. Did you check the storage on the machine that hosts your VM/VM Images and OS ?

1>I've been checking the VM settings again and again for months, I can't find anything that looks anomalous to my fairly ignorant eyes. I have been doing this kind of setup for years and never had this issue until recently. Do you know if something changed in pfsense basic setup in the past year that would adversely affect compatibility with basic, typical VM settings?
2> Do you mean a full blown MEMTest kind of thing, running it for many hours kind of thing? No, that I have not done. I haven't had any issues that look like memory problems, it's not unusual to use a good percentage of 64GB. The problem has persisted for a long time with numerous substantial changes in the VM, the VM is loaded and run in a LOT of different orders wrt the other programs running on the PC, which is used for a LOT of different applications. I don't see how memory problems would be at all likely considering these facts, but if you think it's still worth doing, I will try to do but doing one of those is a hassle.

3>I'm not sure what 'check storage' implies, I just did a windows check drive and it found no errors. It's a newish Samsung Evo NVMe. This problem has persisted through multiple virtual disks created for the pfsense VM, new ones with the old ones left alone. I've never had any kind of indication that there was any kind of disk problem for this drive. I've probably gone through at least 6 completely new VMs, new virtual disks, etc. Considering this scenario, if there were disks problems, it sure seems like I would have had some indications, it doesn't seem like I would have very similar problems across multiple virtual disks which would be very different locations on the disk especially with it being an NVMe. Also, this is a network bandwidth issue, not sure how a disk problem would degrade throughput on the network.

I appreciate your efforts, thank you. I will have another post shortly that has some more info.

[mechtheist]

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

@mechtheist said in Some hosts aren't connecting to the internet but others are:

db:0:kdb.enter.default> bt
Tracing pid 12 tid 100055 td 0xfffff800054ad000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00005a7520
vpanic() at vpanic+0x197/frame 0xfffffe00005a7570
panic() at panic+0x43/frame 0xfffffe00005a75d0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00005a7630
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00005a7680
trap() at trap+0x286/frame 0xfffffe00005a7790
calltrap() at calltrap+0x8/frame 0xfffffe00005a7790
--- trap 0xc, rip = 0xffffffff810343b3, rsp = 0xfffffe00005a7860, rbp = 0xfffffe00005a78b0 ---
in6_unlink_ifa() at in6_unlink_ifa+0x63/frame 0xfffffe00005a78b0
in6_purgeaddr() at in6_purgeaddr+0x309/frame 0xfffffe00005a7a30
nd6_timer() at nd6_timer+0xc2/frame 0xfffffe00005a7a80
softclock_call_cc() at softclock_call_cc+0x141/frame 0xfffffe00005a7b30
softclock() at softclock+0x79/frame 0xfffffe00005a7b50
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00005a7bb0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00005a7bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005a7bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

That's the important part there, the backtrace, and it's inconclusive.
But it is showing an issue in IPv6. Maybe your old config didn't have IPv6 enabled at all?
Maybe your ISP jusy turned on IPv6 (or some port of it) and it's broken? Or your config is somehow not compatible with it?

I would be testing pfSense bare metal if you can't try hyper-v.

Steve

I always have pfsense block all ip6 traffic, i have enough problems [ignorance and stupidity] with understanding what's going on with ip4. There's always ip6 crap going on though and I'm woefully ignorant about what it means or actually even is, though most of it that I can figure out is linklocal kind of stuff. That said, on the terminal screen for pfsense, it started listing an ip6 entry for the WAN along with the other interfaces, but I think that was when I had the WAN cable disconnected and was trying to do other things.

Is Hyper-V considered bare metal? I've used VirtualBox for a long time and I'm fairly confortable with it and it has always worked well and without issues, I hate the idea of going with something different and facing the learning curve.

[mechtheist]

As I was dicking around with it, the pfsensee VM was getting really sluggish, Running top from the terminal/console, there was a lot of heavy cpu usage that wasn't normal, there was not anything like heavy traffic for it to contend with, but I was getting a lot of different applications showing very high cpu usage, a lot of it was php-fpm, also unbound, bzip, sh, rate, even ps. Often numbers in the 60s and 70s, often higher, sometimes over 100% and a couple of times I'd swear it said something like 1000%, but I'm not sure if I was seeing that right. I don't really have the time to be fooling around with this so I'm back on a router for now and I'm not sure what to do or if I want to spend much more time with it. This is the kind of times that defenestration starts to look good.

[nimrod]

@mechtheist

OK. So VM settings are ok. You dont have to run full blown mem test. That will take days to complete with 64gb of RAM. Just run quick test from pure DOS mode which will test CPU/RAM memory controller. As for storage, windows check only verifies integrity of files and file system, and in rare cases, it will run surface test if windows itself suspects that there are bad sectors on the drive. Anyway, thats not enough. Since you have a nice Samsung nVME, just download and install Samsung Magician SSD tool from Samsung website. That tool will read S.M.A.R.T parameters and perform quick test to see if your drive controller and drive itself are OK.

In your last post, you mentioned high CPU usage and sluggishness. None of such things can be caused by default pfSense installation. It all indicates issue with either hardware or host operating system. Try removing Windows network loopback driver and reinstalling it again.

When we say "bare metal" we refer to pure hardware. No virtualization. This is to rule out any potential missconfiguration or issues with hardware or software. Hyper-V is just virtualization software that Microsoft built into the Windows 10. Its similar to VirtualBox. You can try it, but as @stephenw10 already said, its better to try on bare metal.

[stephenw10]

@mechtheist said in Some hosts aren't connecting to the internet but others are:

it started listing an ip6 entry for the WAN along with the other interfaces

A real IPv6 address? That didn't exist before?

That's the sort of change that could be introduced by your ISP and if not setup right (or completely disabled) can make the experience at a client seem very bad. Though it usually presents as a delay before anything starts loading rather than slow throughput.

Steve

[mechtheist]

@nimrod said in Some hosts aren't connecting to the internet but others are:

@mechtheist

OK. So VM settings are ok. You dont have to run full blown mem test. That will take days to complete with 64gb of RAM. Just run quick test from pure DOS mode which will test CPU/RAM memory controller. As for storage, windows check only verifies integrity of files and file system, and in rare cases, it will run surface test if windows itself suspects that there are bad sectors on the drive. Anyway, thats not enough. Since you have a nice Samsung nVME, just download and install Samsung Magician SSD tool from Samsung website. That tool will read S.M.A.R.T parameters and perform quick test to see if your drive controller and drive itself are OK.

In your last post, you mentioned high CPU usage and sluggishness. None of such things can be caused by default pfSense installation. It all indicates issue with either hardware or host operating system. Try removing Windows network loopback driver and reinstalling it again.

When we say "bare metal" we refer to pure hardware. No virtualization. This is to rule out any potential missconfiguration or issues with hardware or software. Hyper-V is just virtualization software that Microsoft built into the Windows 10. Its similar to VirtualBox. You can try it, but as @stephenw10 already said, its better to try on bare metal.

Days for a memtest, that's depressing because my other main PC IS having a subtle thing going on that is likely a memory issue and I've been putting off testing due to how much time it took, it's also 64GB, and I thought maybe it was 8-12 hrs, it's been a really long time since I've done one of those. I'll try a DOS mode test.

I used to have Magician installed but removed it, forgot why, I tried to reinstall it with latest version I just downloaded and got this:

Not helpful, I can't even figure out who's objecting, Windows doesn't mention it in its virus security settings nor does Malwarebytes, but I put in an exception in both of those and still getting it, it must be part of the install program itself [probably likely as it has that 'Setup' bit showing] and not sure what to do about that, maybe there's a log entry, will have to deal with later. On a positive note, I have CrystalDiskInfo that I run from startup and it isn't reporting any errors, so shouldn't be any SMART issues.

I wasn't sure about bare metal, I think I've read about some of the higher end VM tech is much better at bringing your OSes closer to 'bare metal' so I thought maybe that's what you meant. I have like 4 or 5 older PCs I might drag back out and try. There is an issue I will mention in next post in answer to stephenw10.

Thanks again for your efforts!

[mechtheist]

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

@mechtheist said in Some hosts aren't connecting to the internet but others are:

it started listing an ip6 entry for the WAN along with the other interfaces

A real IPv6 address? That didn't exist before?

That's the sort of change that could be introduced by your ISP and if not setup right (or completely disabled) can make the experience at a client seem very bad. Though it usually presents as a delay before anything starts loading rather than slow throughput.

Steve

Not sure what 'real' ip6 implies, I should have done a screen shot. It was there only when I was first dicking with the new pfsense install and didn't have the WAN connected, so it can't be an ISP thing.

I had an idea that made me want to facepalm, maybe it's the NIC after all, I'm using a Intel PRO/1000 PT Quad Port Low Profile Server Adapter, it uses a 82571EB Gigabit controller, I searched the forum for any info and found one of your posts from just a couple of months ago here:

PCIe 1 vs 2

Some virtualisation support in the 82580.

But 82571 is an em(4) NIC but 82580 is an igb(4) muti-queue NIC. If you have multiple CPU cores that is going to load them far more efficiently.

There is non-low profile version of this NIC that only supports PCIe1 but the LowProfile version is supposed to have the ability to work with PCIe2 through some kind of addon chip or emulation or ??? Could that be the problem? With the PC connected to the LAN through the quad NIC it can do 100Mbps just fine. That's from the internet with Ookla speedtest. I ran a local test and got this:

and that's while using RDP to run the test so it had that traffic to compete with. The FreeBSD manual page for the em driver doesn't actually list the lowprofile version, it has a LOT of the similar cards though. Mine is the 'PT' variety, not sure what the difference is with the PT, MT, PF, MF etc versions. Does any of that mean anything to you and is it a possible reason for what I'm seeing, could the low profile kludge be the culprit? I appreciate all the help, thanx.

[stephenw10]

@mechtheist said in Some hosts aren't connecting to the internet but others are:

Not sure what 'real' ip6 implies

I mean a globally routable public IP as opposed to a link-local IP based in the MAC address which every interface has.

An 82571 NIC will easily pass 1Gbps with any vaguely recent processor. Unless it's something ultra low power maybe, but any system designed for that wouldn't be using that NIC.
I wouldn't expect any issue with that.

Steve

[mechtheist]

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

@mechtheist said in Some hosts aren't connecting to the internet but others are:

Not sure what 'real' ip6 implies

I mean a globally routable public IP as opposed to a link-local IP based in the MAC address which every interface has.

An 82571 NIC will easily pass 1Gbps with any vaguely recent processor. Unless it's something ultra low power maybe, but any system designed for that wouldn't be using that NIC.
I wouldn't expect any issue with that.

Steve

OK, I switched the LAN and WAN to the mobo's onboard NICs and no change, then did a new install of pfsense and no change, if anything it was even slower.

I used to drool at the thought of getting a 1Mbps internet connection, a 'T1' I think I always heard them called, but that was a long time ago. So it's not the NIC, here the quad NIC isn't connected in any way. I was also immediately after the new install getting anomalous cpu usage again:

That's what it looks like, the right two are from earlier examples, the left one is from this new install, I saw a number of 3-400% figures, just didn't catch with the screenshot.

The ip6 address popped up again and I see now what happened, pfsense seems to have a lot more ip6 stuff going on and set up as default, there's at least 2 or 3 settings I had to switch off, kept getting 'you can't do that because X is turned on' kinda thing, I really don't recall having to do any of that before.

One strangeness I noted was the syslog was full of this:

Nov 16 14:29:34 	nginx 		2021/11/16 14:29:34 [error] 33672#100684: *11813 open() "/usr/local/www/cgi-bin/luci/;stok=df714b094f1a294eb62bd685e32d868d/admin/status" failed (2: No such file or directory), client: 10.0.0.40, server: , request: "POST /cgi-bin/luci/;stok=df714b094f1a294eb62bd685e32d868d/admin/status?form=wan_speed HTTP/2.0", host: "10.0.0.1", referrer: "https://10.0.0.1/webpages/index.html?t=0093dbde"


Nov 16 14:29:38 	nginx 		2021/11/16 14:29:38 [error] 33899#100585: *13843 open() "/usr/local/www/cgi-bin/luci/;stok=df714b094f1a294eb62bd685e32d868d/admin/status" failed (2: No such file or directory), client: 10.0.0.40, server: , request: "POST /cgi-bin/luci/;stok=df714b094f1a294eb62bd685e32d868d/admin/status?form=wan_speed HTTP/2.0", host: "10.0.0.1", referrer: "https://10.0.0.1/webpages/index.html?t=0093dbde"

There were 4 of the first line followed by 496 of the second one. 10.0.0.40 is my main PC that I'm on now, not the PC with the VMs. So the whole log file was just this crap which means absolutely nothing to me.

I realized trying it bare-metal meant to run pfsense on the same PC I have it virtualized on, it makes no sense to use another PC, I've been thinking of doing that to 'solve' this problem but that wouldn't help figure out what's going on with the VMs. I could just try running it live, that shouldn't be too intrusive, installing it would make me really nervous I'd break something, I'm not that good with boot loaders and the like and zfs is utterly opaque to me.

[stephenw10]

Hmm that's a lot of awking! I'd try running ps -auxwwd to see what it calling that command.

Those errors are something on 10.0.0.40 trying to open a page that does not exist on pfSense.
And the page it's looking for, /cgi-bin/luci/, is from OpenWRT so I suspect you have ha another router at 10.0.0.1 running that at some time.

A restriction to ~1Mbps almost has to be some link issue or something in the VM setup.

Steve

[mechtheist]

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

Hmm that's a lot of awking! I'd try running ps -auxwwd to see what it calling that command.

Those errors are something on 10.0.0.40 trying to open a page that does not exist on pfSense.
And the page it's looking for, /cgi-bin/luci/, is from OpenWRT so I suspect you have ha another router at 10.0.0.1 running that at some time.

A restriction to ~1Mbps almost has to be some link issue or something in the VM setup.

Steve

I appreciate your patience and help. You're probably right about the clogging of the log, I think must be because when I switched from router to pfsense I just took the WAN cable from the router and plugged it in the NIC for pfsense and then unplugged the router from the LAN, so my PC was probably still trying to doing something with the router. I thought the same thing about the awk cpu usage. The one with awk at 99% also has 'unslogd' right under it at 11% so I was thinking it had something to do with diddling with the log file archiving. I couldn't find anything on unslogd but it seems a plausible guess at what unlogging might do;)

You've been a great help and I'm hoping you can give me a little more, what would I look for if it's a link issue? Are we talking packet capture? Are there any kind of errors you think I should look for? The VM settings thing is really frustrating and why I keep repeating about doing this for a long time, as far as I know, I'm setting up the VMs the same way I've always done except switching to the vertio for the networks, and I did that when I was already having the problem hoping it might improve the performance and it did by some, maybe 5-10% at best. I ordered one of the newer NICs with the 82580 controllers. It's not the NIC causing the problem but maybe something will popup by switching to the newer card.

In case anyone can make sense of it, this is the setup info on virtualbox:

C:\Users\rob>vboxmanage showvminfo "pfsense_octo"
Name:                        pfsense_octo
Groups:                      /
Guest OS:                    FreeBSD (64-bit)
UUID:                        e73eb791-860c-40f9-94a5-b02b4073ec44
Config file:                 H:\VirtualBoxVMs\pfsense_octo\pfsense_octo.vbox
Snapshot folder:             H:\VirtualBoxVMs\pfsense_octo\Snapshots
Log folder:                  H:\VirtualBoxVMs\pfsense_octo\Logs
Hardware UUID:               e73eb791-860c-40f9-94a5-b02b4073ec44
Memory size:                 10121MB
Page Fusion:                 disabled
VRAM size:                   126MB
CPU exec cap:                100%
HPET:                        disabled
CPUProfile:                  host
Chipset:                     piix3
Firmware:                    BIOS
Number of CPUs:              8
PAE:                         enabled
Long Mode:                   enabled
Triple Fault Reset:          disabled
APIC:                        enabled
X2APIC:                      disabled
Nested VT-x/AMD-V:           disabled
CPUID Portability Level:     0
CPUID overrides:             None
Boot menu mode:              message and menu
Boot Device 1:               DVD
Boot Device 2:               HardDisk
Boot Device 3:               Not Assigned
Boot Device 4:               Not Assigned
ACPI:                        enabled
IOAPIC:                      enabled
BIOS APIC mode:              APIC
Time offset:                 0ms
RTC:                         local time
Hardware Virtualization:     enabled
Nested Paging:               enabled
Large Pages:                 enabled
VT-x VPID:                   enabled
VT-x Unrestricted Exec.:     enabled
Paravirt. Provider:          Default
Effective Paravirt. Prov.:   None
State:                       powered off (since 2021-11-16T16:34:52.162000000)
Graphics Controller:         VMSVGA
Monitor count:               1
3D Acceleration:             disabled
2D Video Acceleration:       disabled
Teleporter Enabled:          disabled
Teleporter Port:             0
Teleporter Address:
Teleporter Password:
Tracing Enabled:             disabled
Allow Tracing to Access VM:  disabled
Tracing Configuration:
Autostart Enabled:           disabled
Autostart Delay:             0
Default Frontend:
VM process priority:         default
Storage Controller Name (0):            AHCI
Storage Controller Type (0):            IntelAhci
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0):  30
Storage Controller Port Count (0):      2
Storage Controller Bootable (0):        on
AHCI (0, 0): Empty
AHCI (1, 0): H:\VirtualBoxVMs\pfsense_octo\pfsense-2nd-replacement-11-11-2021--1333.vhd (UUID: a60260db-d839-44c6-8434-345e0f64826c)
NIC 1:                       MAC: 080027B1449A, Attachment: Bridged Interface 'Intel(R) I211 Gigabit Network Connection', Cable connected: on, Trace: off (file: none), Type: virtio, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none
NIC 2:                       MAC: 08002719A76D, Attachment: Bridged Interface 'Intel(R) Ethernet Connection (2) I218-V', Cable connected: on, Trace: off (file: none), Type: virtio, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none
NIC 3:                       disabled
NIC 4:                       disabled
NIC 5:                       disabled
NIC 6:                       disabled
NIC 7:                       disabled
NIC 8:                       disabled
Pointing Device:             USB Tablet
Keyboard Device:             PS/2 Keyboard
UART 1:                      disabled
UART 2:                      disabled
UART 3:                      disabled
UART 4:                      disabled
LPT 1:                       disabled
LPT 2:                       disabled
Audio:                       enabled (Driver: DSOUND, Controller: AC97, Codec: STAC9700)
Audio playback:              enabled
Audio capture:               disabled
Clipboard Mode:              disabled
Drag and drop Mode:          disabled
VRDE:                        enabled (Address 0.0.0.0, Ports 33891, MultiConn: on, ReuseSingleConn: off, Authentication type: null)
Video redirection:           disabled
VRDE property               : TCP/Ports  = "33891"
VRDE property               : TCP/Address = <not set>
VRDE property               : VideoChannel/Enabled = <not set>
VRDE property               : VideoChannel/Quality = <not set>
VRDE property               : VideoChannel/DownscaleProtection = <not set>
VRDE property               : Client/DisableDisplay = <not set>
VRDE property               : Client/DisableInput = <not set>
VRDE property               : Client/DisableAudio = <not set>
VRDE property               : Client/DisableUSB = <not set>
VRDE property               : Client/DisableClipboard = <not set>
VRDE property               : Client/DisableUpstreamAudio = <not set>
VRDE property               : Client/DisableRDPDR = <not set>
VRDE property               : H3DRedirect/Enabled = <not set>
VRDE property               : Security/Method = <not set>
VRDE property               : Security/ServerCertificate = <not set>
VRDE property               : Security/ServerPrivateKey = <not set>
VRDE property               : Security/CACertificate = <not set>
VRDE property               : Audio/RateCorrectionMode = <not set>
VRDE property               : Audio/LogPath = <not set>
OHCI USB:                    disabled
EHCI USB:                    disabled
xHCI USB:                    enabled

USB Device Filters:

<none>

Bandwidth groups:  <none>

Shared folders:

Name: 'shared', Host path: 'H:\VirtualBoxVMs\pfsense_octo\shared' (machine mapping), writable, auto-mount, mount-point: '/root/octo'

Capturing:                   active
Capture audio:               not active
Capture screens:             0
Capture file:                H:\VirtualBoxVMs\pfsense_octo\pfsense_octo.webm
Capture dimensions:          1024x768
Capture rate:                512kbps
Capture FPS:                 25kbps
Capture options:             vc_enabled=true,ac_enabled=false,ac_profile=med

Guest:

Configured memory balloon size: 0MB

[stephenw10]

I'm not really very familiar with VBox but nothing there jumps out. I do run it here occasionally but in Linux. What is the host processor? Nothing would really make that much difference but that would be a difference.
You should be able to run it in hyper-v on the same host relatively easily. It you can't run it bare metal that's the next thing I would try.
https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-hyper-v.html

Steve

[mechtheist]

@stephenw10
I was actually thinking of using the linux VM I use for a web server to run a pfsense VM on, there's probably more than enough cpu power to work adequately. The PC is win10 on a i7-5960X w/ 64GB RAM with an NVIDIA 2080 gpu hopefully that will be enough so it's not too painful to work with.

I wrote that yesterday, still working on getting a VM in a VM going. I'm definitely going to try HyperV, one reason for the hesitation is you have to turn all of it off in Windows or it interferes with VirtualBox. The last time I used it I think was back in win8 when Windows basically came with a VM XP. The new card should get here Monday.

[stephenw10]

Hmm, VM in a VM is double the potential issues IMO.

If there's any way you can test it bare metal however temporarily I would do that.

At least moving to Hyper-V changes the potential issues so if you still see it the issue is probably on the host or external.

Steve

[mechtheist]

@stephenw10 Yeah, I'm going to do a live instance and see how that goes. Thanks for helping out and being patient.