Some hosts aren't connecting to the internet but others are

mechtheist

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

Are you running any dynamic packages list Snort, Suricata or pfBlocker?

Do you see any blocked traffic in the firewall logs when this happens?

What's the actual error shown if you try to ping out from one of these hosts when they fail?

Steve

Hi, thanks for the reply. I have pfBlocker but there are no blocked entries for anything to these IPs. I usually get a host unreachable if I remember correctly. Just before I started to reply to this, I had another glitch, nothing was talking to pfsense even though it said all the interfaces were up. I could talk to the PC running the VM, but even that PC could not talk to pfsense. I rebooted pfsense and it came back up but now, wifi clients on two different routers are not getting assigned IPs but a wired new connection did..

It's just getting weirder. One of the wifi connections has a static IP set up in pfsense and the DHCP status shows it's online:

10.0.0.66 	c8:58:c0:12:1d:b7 	sagerII_wifi6 	sagerII_wifi6 	sagerII_wifi6 	n/a 	n/a 	online 	static

but it isn't getting ti's IP from the dhcp server even though it's set up to, it's actually getting set to 169.254.217.222, one of those auto-config IPs. There are no log entries related to the MACs except an old one where it shows it connected.

So I'm even more confused now. I have never seen it do anything like this with DHCP acting weird, but I have seen the problem where pfsense says its interfaces are all up and not been able to talk to it even though the rest of the LAN was fine including the PC with pfsense running in a VM. I see nothing in the system logs that would hint at a problem, but again, I'm not all that knowledgeable. I tried searching the system log for 'dhcp' and nothing relevant came up. Any ideas?

stephenw10

The 'online' status there indicates it is present in the ARP table. Check that in Diag > ARP.

Behaviour like this can often be due to a rogue dhcp server somewhere on your network. Some clients start pulling leases fro there instead and end up with no valid route.

However that doesn't explain clients that end up with an APIPA address.

I would be running packet captures at this point to see where the traffic is failing. You should be able to see the dhcp requests and responses on the interface in pfSense.

Steve

mechtheist

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

The 'online' status there indicates it is present in the ARP table. Check that in Diag > ARP.

Behaviour like this can often be due to a rogue dhcp server somewhere on your network. Some clients start pulling leases fro there instead and end up with no valid route.

However that doesn't explain clients that end up with an APIPA address.

I would be running packet captures at this point to see where the traffic is failing. You should be able to see the dhcp requests and responses on the interface in pfSense.

Steve

It's there. Does that mean you'll see 'online' if it has a entry in the table regardless of whether it's actually online? I will try doing some packet captures and see what I can find. I tired before and don't think there were any errant rogue dhcp replies but I could have missed it. Thank you for the help.

stephenw10

The ARP table updates pretty fast but things can show as on-line there after they have shutdown and before the ARP entry expires. If it shows off-line there it really is off-line or at least that pfSense has not seen any traffic from it in the last 15mins.
You should see the valid dhcp queries and replies also. So if clients are failing to pull an IP and those queries do not appear in a pcap on LAN then something else is filtering them. Or the clients are not sending them (to the pfSense LAN).

Steve

mechtheist

@stephenw10 OK, I hate it when these things happen, this new problem turns out to be ridiculous--I was focusing on the original issue so I thought this new thing that just started had to be related and I didn't even notice the router ethernet cable had come loose, I'm blaming the cats! and the fact that the RJ45 connector's locking tab is broken off, I'm going to correct that. I apologize for the diversion, the original problem remains, though at the moment it isn't happening, and I hate it when THAT happens. When it was, I kept getting tons of these in the resolver log:

Aug  3 16:44:27 pfSense unbound[85165]: [85165:4] notice: sendmsg failed: Invalid argument
Aug  3 16:44:27 pfSense unbound[85165]: [85165:4] notice: remote address is 10.0.0.209 port 11080

10.0.0.209 is the IP for my laptop wireless. At that time, there were 3 IPs getting these errors, over 600 total in the whole log. Also, got this in the dhcp log:

Aug  3 16:41:37 pfSense dhcpd[98304]: icmp_echorequest 10.0.0.209: Invalid argument
Aug  3 16:41:38 pfSense dhcpd[98304]: DHCPOFFER on 10.0.0.209 to c8:58:c0:12:1d:b7 (DESKTOP-LK72M2K) via vtnet1
Aug  3 16:41:38 pfSense dhcpd[98304]: DHCPREQUEST for 10.0.0.209 (10.0.0.1) from c8:58:c0:12:1d:b7 (DESKTOP-LK72M2K) via vtnet1
Aug  3 16:41:38 pfSense dhcpd[98304]: DHCPACK on 10.0.0.209 to c8:58:c0:12:1d:b7 (DESKTOP-LK72M2K) via vtnet1

c8:58:c0:12:1d:b7 (DESKTOP-LK72M2K) --this is the laptops wireless MAC, it was the only mention in the dhcp log. It was getting an IP, it just couldn't talk to the internet.

I also can't get any better than about 17Mbps through pfsense, it's been like that for some while. My connection is 100Mbps and I can get that through the cable modem. The PC I'm running virtualbox is about 6 years old but it's an Intel Core i7-5960X 8-core beast for its time. Previously had a similar setup, with the same Intel quad-NIC on a really old, like 2007 or so, Intel Core i5-2400 4 Core Processor, with pfsense on one of two VMs and it had no problem keeping up with 100Mbps. I have no idea if these problems might be related. Any idea what could be limiting the bandwidth on the newer faster machine?

stephenw10

A bandwidth throttle that bad is usually something at layer 1, like a bad cable or a mismatched speed/duplex. Check Status > Interfaces for errors/collisions.

You should be able to pass 100Mbps with like 1% CPU use on that beast!

Steve

mechtheist

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

A bandwidth throttle that bad is usually something at layer 1, like a bad cable or a mismatched speed/duplex. Check Status > Interfaces for errors/collisions.

You should be able to pass 100Mbps with like 1% CPU use on that beast!

Steve

I checked everything I knew to check and don't see problems, there are no indications of errors on the interface status page. Unfortunately, pfsense has gotten completely screwed up. The web configurator started not completely loading, the version and cpu usage on the dashboard never finished loading and the right of 3 columns just wasn't there. After that went on for a few hours, not the web page isn't coming up at all. If I ssh in, everything looks OK, but if I try to see the log, it never finished loading, it gets to about 6 hours ago and stops, but after a while, the connection dies. I can see the log on the terminal in the VM terminal for pfsense and it says the client timed out. For some time, the system has just seemed slow, more than just the bandwidth, the web pages started getting really slow to appear. Just now, when I was trying to get see the system log on SSH, it was getting sent ridiculously slowly, like 20 or so lines would show, then a pause and some more, and it continued like that until it just gave up. If I run top, the machine is almost always showing 98+ and usually 99+% idle.

I restored to about 2 weeks ago and all this new crap went away. Don't have any idea what might have happened, the only change I think could have screwed things up that bad was I unchecked the 3 disable hardware checksum, TCP segmentation , and large receive offloads in advanced setup, but it ran OK for some time after doing that. And, I'm still stuck at 18-19Mbps.

I'm just more confused. All of this BS and nothing fixed except stuff that wasn't broken when I started.

Well, this is where my reserve will-to-live starts to deplete to scary levels. I hadn't checked the cable from the cable modem to the NIC on the PC running pfsense VM so I unplugged it from the WAN and plugged it into the LAN NIC and it worked, got 105Mbps, so I switched the cables back and now the LAN isn't working. Aaaarrrggghhhh. It's like it isn't there on the network. A network scanner doesn't see it, but it sees the PC running the VM, but that PC isn't seeing it. The pfsense console says it's up, ifconfig says it's up and active and OK, the switch it's connected to says it's up and has the right speed, but I can't get to the web pages, can't SSH. I've rebooted pfsense 3 times now and still nothing. I'm at a loss, can only think of a hardware issue in the Intel quad NIC at his point.

stephenw10

Hmm, this starts to sound like something in the VM setup. Like maybe both NICs are connected together somehow or swapped. I would not expect to get any connection if you move the link between the WAN and LAN. Unless the LAN side clients are just connecting directly to whatever is upstream of pfSense, bypassing it completely it that situation.
I would try connecting out from pfSense itself at the console.
Try connecting only the WAN cable and make sure it can still connect out and that LAN clients cannot connect at all.

Steve

mechtheist

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

Hmm, this starts to sound like something in the VM setup. Like maybe both NICs are connected together somehow or swapped. I would not expect to get any connection if you move the link between the WAN and LAN. Unless the LAN side clients are just connecting directly to whatever is upstream of pfSense, bypassing it completely it that situation.
I would try connecting out from pfSense itself at the console.
Try connecting only the WAN cable and make sure it can still connect out and that LAN clients cannot connect at all.

Steve

That was just testing the cable, that it wasn't causing the throttled bandwidth somehow, so it was like you said, I just unplugged LAN to switch cable, took cable from WAN to cable modem and connected it to the LAN port, which was now connected directly to the cable modem and could act like a normal connection, the pfsense VM would be out of the picture. Then did a quick ookla speedtest and it went to over 100Mbps so I just reconnected everything to what it had been before.

That's when pfsense and the rest of the network stopped communicating or whatever it was doing. I can see the PC running the VM, it's on 10.0.0.88. I can run NIRSoft Tools Wireless Network Watcher, they have a lot of great tools for windows, and it sees 10.0.0.88 but not the pfsense VM which is on 10.0.0.1, which is supposed to be bridged to the LAN, I should see both IPs in the scanner. So, I can't SSH or use the web page because the rest of the LAN can't see 10.0.0.1. BUT, at the console, ifconfig says everything is up and running just fine. I did nothing to pfsense, I only switched the cables and switched them back. I rebooted pfsense a few times, rebooted the PC running the VM, and no change.

I'm not savvy enough to do much troubleshooting with only the console, I wouldn't even know where to begin when it's saying the interface is up and running but everything else isn't seeing that at all. So I got disgusted and just threw one of my routers in and said screw it for now, I had a lot of things to do and needed my network to be functioning. I hate it when I'm not connected through pfsense. it makes me feel naked, vulnerable, and blind.
I really appreciate your trying to help. I'm thinking maybe it's a hardware issue but I will need to dick with it some more, just haven't had the time.

stephenw10

Yeah, you should see it responding to arp requests if it's actually conected. Assuming that's what that tool does.

At the console menu choose option 8 to drop to the command prompt.
Then try to ping something local. For example:

ping -c 3 10.0.0.88

If it's connected correctly, and the host machine responds to ping, that should work.

Now try to ping some other things. Try to ping an external IP.

Check the routing. Run:

netstat -rn

Make sure it has a valid default route.

Steve

mechtheist

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

Yeah, you should see it responding to arp requests if it's actually conected. Assuming that's what that tool does.

At the console menu choose option 8 to drop to the command prompt.
Then try to ping something local. For example:
ping -c 3 10.0.0.88
If it's connected correctly, and the host machine responds to ping, that should work.

Now try to ping some other things. Try to ping an external IP.

Check the routing. Run:
netstat -rn
Make sure it has a valid default route.

Steve

Thanks, I'll give that a try when I next stick it back together again. I could always do a tcpdump and see what's coming across the interface.

mechtheist

@stephenw10 It's a little late, I've been distracted. I decided to just do a fresh install, hoping maybe some setting was off. After installing and almost no configuring at all and only WAN and LAN, I fired it up and it made a huge difference, instead of getting only about 17Mbps on a 100Mbps bandwidth internet connection, I was now getting a whopping 1.6 to 1.8 Mbps! I've checked as well as I know how to find some mismatch in the settings and can find nothing. I got disgusted and swapped my router back in. Using the NIC directly on the PC that was running pfsense with pfsense not running, I can get the full bandwidth, so it's not some problem with the NIC. If you or anyone else has any ideas I can swap pfsense back in easily enough to try them out but otherwise, I'm going to put it aside for now and when I have more time I can try it again, maybe start a new thread if the problem persists. Thanks for you help, it's been appreciated.

Giovanni 0

I had this problem for about 6 months,
I had this problem with cell phones, TVs, routers and some virtual machines
Every day I got this problem and the only temporary solution I had found was to clear the arp table, so 99% of things worked, until they were turned off
Earlier this month I turned it off, installed another firewall and left my network like this:
pfSense on the 192.168.5.1 network
another firewall on the network
192.168.4.1

And so I was changing device by device from one network to another, until I moved my loadbalance and everything stopped, I moved it back to the old network and it started working again

I recommend that you do this manual method, if it is interference, it is the best way to find it

mechtheist

@giovanni-0
Thanks for the reply. My network really isn't that complex, I've never set up load balancing. In this latest attempt, there was little else going on with anything other than pfsense and the PC its VM was installed on that I ran the tests with. It was a fresh install with almost no config changes and it got only 1.8% of the actual bandwidth available. As I mentioned in OP, I've been doing this for years with much less capable PCs and never had problems. It makes me thing there's some box checked off somewhere, or not checked off and I'm just too stupid to see it.

Thanks again.

nimrod

@mechtheist

This looks like a hardware issue to me. Check the storage device on the system that hosts your VM. Check the system RAM as well. The fact that you reinstalled pfSense, and you had those issues where CPU load status indicator is not loading, is leading me to belieive there might be something wrong with your hardware. We already know that NIC is fine, thats one thing ruled out, check everything else. Verify your VM settings as well. There could be an update that reverted your CPU/RAM/Storage settings for your VM. Its pointless looking for an issue in pfSense until the host system is not checked.

stephenw10

Hmm, this seems more like VBox issue to me.

Can you try running in Hyper-V instead?

Or, better, running bare metal as a test?

Steve

mechtheist

@nimrod said in Some hosts aren't connecting to the internet but others are:

@mechtheist

This looks like a hardware issue to me. Check the storage device on the system that hosts your VM. Check the system RAM as well. The fact that you reinstalled pfSense, and you had those issues where CPU load status indicator is not loading, is leading me to belieive there might be something wrong with your hardware. We already know that NIC is fine, thats one thing ruled out, check everything else. Verify your VM settings as well. There could be an update that reverted your CPU/RAM/Storage settings for your VM. Its pointless looking for an issue in pfSense until the host system is not checked.

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

Hmm, this seems more like VBox issue to me.

Can you try running in Hyper-V instead?

Or, better, running bare metal as a test?

Steve

I've had this problem for many months and looked but have never found anything that I could determine to be a possible cause in pfsense or the PC. It's weird that doing a fresh install and little configuration dropped the already low bandwidth by a factor of almost 10, from about 17Mbps to about 1.7 Mbps. I have v v little experience with Hyper-V and none with any other VM software but I was starting to think about trying Hyper-V . For about 6 years at least I've done this with VirtualBox and had no real problems, pfsense just worked, it's always kinda amazed and really impressed me.

That said, a couple of hours ago it crashed, below is the report copied off the web page, I don't even remember getting one of these before and definitely don't have the knowledge to understand what it's telling me, anything stick out to you guys? I don't know how to fix the nonprinting characters, sorry.
Had to cut a lot out, too long, so attached file. Thanks for all the help!

Crash report begins.  Anonymous machine information:

amd64
12.2-STABLE
FreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense

Crash report details:

No PHP errors found.

Filename: /var/crash/info.0
Dump header from device: /dev/ada0p2
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 69632
  Blocksize: 512
  Compression: none
  Dumptime: Sun Nov 14 14:03:31 2021
  Hostname: pfSense.domain.org
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense
  Panic String: page fault
  Dump Parity: 3658854709
  Bounds: 0
  Dump Status: good

Filename: /var/crash/textdump.tar.0
ddb.txt--run of nonprointing characters--db:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo>  show alllocks
No such command; use "help" to list available commands
db:1:lockinfo>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 6
dynamic pcpu = 0xfffffe007f83f380
curthread    = 0xfffff800054ad000: pid 12 tid 100055 "swi4: clock (0)"
curpcb       = 0xfffff800054ad5a0
fpcurthread  = none
idlethread   = 0xfffff8000540e000: tid 100009 "idle: cpu6"
curpmap      = 0xffffffff8368d5a8
tssp         = 0xffffffff83717890
commontssp   = 0xffffffff83717890
rsp0         = 0xfffffe00005a7cc0
kcr3         = 0x80000000040d4003
ucr3         = 0xffffffffffffffff
scr3         = 0x13d79ee25
gs32p        = 0xffffffff8371e0a8
ldt          = 0xffffffff8371e0e8
tss          = 0xffffffff8371e0d8
tlb gen      = 15968
curvnet      = 0xfffff800050bab80
db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100055 td 0xfffff800054ad000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00005a7520
vpanic() at vpanic+0x197/frame 0xfffffe00005a7570
panic() at panic+0x43/frame 0xfffffe00005a75d0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00005a7630
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00005a7680
trap() at trap+0x286/frame 0xfffffe00005a7790
calltrap() at calltrap+0x8/frame 0xfffffe00005a7790
--- trap 0xc, rip = 0xffffffff810343b3, rsp = 0xfffffe00005a7860, rbp = 0xfffffe00005a78b0 ---
in6_unlink_ifa() at in6_unlink_ifa+0x63/frame 0xfffffe00005a78b0
in6_purgeaddr() at in6_purgeaddr+0x309/frame 0xfffffe00005a7a30
nd6_timer() at nd6_timer+0xc2/frame 0xfffffe00005a7a80
softclock_call_cc() at softclock_call_cc+0x141/frame 0xfffffe00005a7b30
softclock() at softclock+0x79/frame 0xfffffe00005a7b50
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00005a7bb0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00005a7bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005a7bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db:0:kdb.enter.default>  ps

...
long list of devices and options cut out because it was too long
...

--run of nonprointing characters--
---<<BOOT>>---
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense amd64
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
VT(vga): text 80x25
CPU: Intel(R) Core(TM) i7-5960X CPU @ 3.00GHz (3000.19-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x5eda2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x2421<FSGSBASE,AVX2,INVPCID,NFPUSG>
  Structured Extended Features3=0x10000400<MD_CLEAR,L1DFL>
  TSC: P-state invariant
real memory  = 11149508608 (10633 MB)
avail memory = 10224226304 (9750 MB)
Event timer "LAPIC" quality 100
ACPI APIC Table: <VBOX   VBOXAPIC>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 cache groups x 1 core(s)
random: unblocking device.
ioapic0: MADT APIC ID 8 != hw id 0
ioapic0 <Version 2.0> irqs 0-23 on motherboard
Launching APs: 6 3 7 4 5 1 2
wlan: mac acl policy registered
random: entropy device external interface
module_register_init: MOD_LOAD (vesa, 0xffffffff8140c3e0, 0) error 19
kbd1 at kbdmux0
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
[ath_hal] loaded
000.000060 [4336] netmap_init               netmap: loaded module
mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
padlock0: No ACE support.
acpi0: <VBOX VBOXXSDT> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
cpu0: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
vgapci0: <VGA-compatible display> port 0xd000-0xd00f mem 0xe0000000-0xe7ffffff,0xf0000000-0xf01fffff irq 18 at device 2.0 on pci0
vgapci0: Boot video device
virtio_pci0: <VirtIO PCI Network adapter> port 0xd020-0xd03f irq 19 at device 3.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
<6>vtnet0: Ethernet address: 08:00:27:b1:44:9a
<6>vtnet0: netmap queues/slots: TX 1/128, RX 1/128
000.000243 [ 445] vtnet_netmap_attach       vtnet attached txq=1, txd=128 rxq=1, rxd=128
pcm0: <Intel ICH (82801AA)> port 0xd100-0xd1ff,0xd200-0xd23f irq 21 at device 5.0 on pci0
pcm0: <SigmaTel STAC9700/83/84 AC97 Codec>
pci0: <bridge> at device 7.0 (no driver attached)
virtio_pci1: <VirtIO PCI Network adapter> port 0xd240-0xd25f irq 16 at device 8.0 on pci0
vtnet1: <VirtIO Networking Adapter> on virtio_pci1
<6>vtnet1: Ethernet address: 08:00:27:19:a7:6d
<6>vtnet1: netmap queues/slots: TX 1/128, RX 1/128
000.000245 [ 445] vtnet_netmap_attach       vtnet attached txq=1, txd=128 rxq=1, rxd=128
virtio_pci2: <VirtIO PCI Network adapter> port 0xd260-0xd27f irq 17 at device 9.0 on pci0
vtnet2: <VirtIO Networking Adapter> on virtio_pci2
<6>vtnet2: Ethernet address: 08:00:27:99:d8:40
<6>vtnet2: netmap queues/slots: TX 1/128, RX 1/128
000.000246 [ 445] vtnet_netmap_attach       vtnet attached txq=1, txd=128 rxq=1, rxd=128
virtio_pci3: <VirtIO PCI Network adapter> port 0xd280-0xd29f irq 18 at device 10.0 on pci0
vtnet3: <VirtIO Networking Adapter> on virtio_pci3
<6>vtnet3: Ethernet address: 08:00:27:70:a7:e2
<6>vtnet3: netmap queues/slots: TX 1/128, RX 1/128
000.000247 [ 445] vtnet_netmap_attach       vtnet attached txq=1, txd=128 rxq=1, rxd=128
ahci0: <Intel ICH8M AHCI SATA controller> port 0xd2a0-0xd2a7,0xd2a8-0xd2ab,0xd2b0-0xd2b7,0xd2b8-0xd2bb,0xd2c0-0xd2cf mem 0xf0804000-0xf0805fff irq 21 at device 13.0 on pci0
ahci0: AHCI v1.10 with 2 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
acpi_acad0: <AC Adapter> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xe2000-0xe2fff pnpid ORM0000 on isa0
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff pnpid PNP0900 on isa0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 10.000 msec
Trying to mount root from zfs:pfSense/ROOT/default []...
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
ada0 at ahcich1 bus 0 scbus1 target 0 lun 0
ada0: <VBOX HARDDISK 1.0> ATA-6 SATA 2.x device
ada0: Serial Number VBa60260db-6c82640f
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 76341MB (156346752 512 byte sectors)
cd0 at ahcich0 bus 0 scbus0 target 0 lun 0
cd0: <VBOX CD-ROM 1.0> Removable CD-ROM SPC-3 SCSI device
cd0: Serial Number VB0-1a2b3c4d
cd0: 300.000MB/s transfers (SATA 2.x, UDMA6, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
<118>Configuring crash dumps...
<118>Using /dev/ada0p2 for dump device.
<118>Filesystems are clean, continuing...
<118>Mounting filesystems...
<118>
<118>        __
<118> _ __  / _|___  ___ _ __  ___  ___
<118>| '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \
<118>| |_) |  _\__ \  __/ | | \__ \  __/
<118>| .__/|_| |___/\___|_| |_|___/\___|
<118>|_|
<118>
<118>
<118>Welcome to pfSense 2.5.2-RELEASE...
<118>
<118>No core dumps found.
<118>...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/ipsec /usr/local/lib/perl5/5.32/mach/CORE
<118>32-bit compatibility ldconfig path:
<118>done.
<118>>>> Removing vital flag from php74... done.
<118>External config loader 1.0 is now starting... ada0p1 ada0p3
<118>Launching the init system...Updating CPU Microcode...
CPU: Intel(R) Core(TM) i7-5960X CPU @ 3.00GHz (3000.19-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x5eda2203<SSE3,PCLMULQDQ,SSSE3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x2421<FSGSBASE,AVX2,INVPCID,NFPUSG>
  Structured Extended Features3=0x10000400<MD_CLEAR,L1DFL>
  TSC: P-state invariant
<118>Done.
<118>.... done.
<118>Initializing.................. done.
<118>Starting device manager (devd)...
intsmb0: <Intel PIIX4 SMBUS Interface> irq 23 at device 7.0 on pci0
intsmb0: intr IRQ 9 enabled revision 0
smbus0: <System Management Bus> on intsmb0
<118>done.
<118>Loading configuration......done.
<118>Updating configuration...done.
<118>Checking config backups consistency.................................done.
<118>Setting up extended sysctls...done.
<118>Setting timezone...done.
<118>Configuring loopback interface...
<6>lo0: link state changed to UP
<118>done.
<118>Starting syslog...done.
<118>Starting Secure Shell Services...done.
<118>Setting up interfaces microcode...done.
<118>Starting PC/SC Smart Card Services...done.
<118>Configuring loopback interface...done.
<118>Creating wireless clone interfaces...done.
<118>Configuring LAGG interfaces...done.
<118>Configuring VLAN interfaces...done.
<118>Configuring QinQ interfaces...done.
<118>Configuring WAN interface...
<6>vtnet0: link state changed to UP
<118>done.
<118>Configuring LAN interface...
<6>vtnet1: link state changed to UP
<118>done.
<118>Configuring SEC interface...
<6>vtnet2: link state changed to UP
<118>done.
<118>Configuring MINTSERVER interface...
<6>vtnet3: link state changed to UP
<118>done.
<118>Configuring IPsec VTI interfaces...done.
<118>Configuring CARP settings...done.
<118>Syncing OpenVPN settings...done.
<118>Configuring firewall.
<6>pflog0: promiscuous mode enabled
<118>.....done.
<118>Starting PFLOG...done.
<118>Setting up gateway monitors...done.
<118>Setting up static routes...done.
<118>Setting up DNSs...
<118>Starting DNS Resolver...done.
<118>Synchronizing user settings...done.
<118>Starting webConfigurator...done.
<118>Configuring CRON...done.
<118>Starting NTP Server...done.
<118>Starting DHCP service...done.
<118>Configuring firewall......done.
<118>Generating RRD graphs...done.
<118>Starting syslog...done.
<118>Starting CRON... done.
<118>pfSense 2.5.2-RELEASE amd64 Fri Jul 02 15:33:00 EDT 2021
<118>Bootup complete


Fatal trap 12: page fault while in kernel mode
cpuid = 6; apic id = 06
fault virtual address	= 0x28
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff810343b3
stack pointer	        = 0x28:0xfffffe00005a7860
frame pointer	        = 0x28:0xfffffe00005a78b0
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (swi4: clock (0))
trap number		= 12
panic: page fault
cpuid = 6
time = 1636920211
KDB: enter: panic
more nonprinting ch.[textdump.tar-11-14-2021.0](/assets/uploads/files/1636926910680-textdump.tar-11-14-2021.0)

nimrod

@mechtheist

At this point, the fact that some devices are connecting and some dont, is completely irrelevant. You did a fresh install of pfSense, and your traffic speed was cut anyway. Thats not how clean install of pfSense performs under normal conditions.

Lets try again.

Did you check VirtualBox VM settings ?
Did you test system RAM on the machine that hosts your VM?
Did you check the storage on the machine that hosts your VM/VM Images and OS ?

stephenw10

@mechtheist said in Some hosts aren't connecting to the internet but others are:

db:0:kdb.enter.default> bt
Tracing pid 12 tid 100055 td 0xfffff800054ad000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00005a7520
vpanic() at vpanic+0x197/frame 0xfffffe00005a7570
panic() at panic+0x43/frame 0xfffffe00005a75d0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00005a7630
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00005a7680
trap() at trap+0x286/frame 0xfffffe00005a7790
calltrap() at calltrap+0x8/frame 0xfffffe00005a7790
--- trap 0xc, rip = 0xffffffff810343b3, rsp = 0xfffffe00005a7860, rbp = 0xfffffe00005a78b0 ---
in6_unlink_ifa() at in6_unlink_ifa+0x63/frame 0xfffffe00005a78b0
in6_purgeaddr() at in6_purgeaddr+0x309/frame 0xfffffe00005a7a30
nd6_timer() at nd6_timer+0xc2/frame 0xfffffe00005a7a80
softclock_call_cc() at softclock_call_cc+0x141/frame 0xfffffe00005a7b30
softclock() at softclock+0x79/frame 0xfffffe00005a7b50
ithread_loop() at ithread_loop+0x23c/frame 0xfffffe00005a7bb0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00005a7bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00005a7bf0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

That's the important part there, the backtrace, and it's inconclusive.
But it is showing an issue in IPv6. Maybe your old config didn't have IPv6 enabled at all?
Maybe your ISP jusy turned on IPv6 (or some port of it) and it's broken? Or your config is somehow not compatible with it?

I would be testing pfSense bare metal if you can't try hyper-v.

Steve

mechtheist

@nimrod said in Some hosts aren't connecting to the internet but others are:

@mechtheist

At this point, the fact that some devices are connecting and some dont, is completely irrelevant. You did a fresh install of pfSense, and your traffic speed was cut anyway. Thats not how clean install of pfSense performs under normal conditions.

Lets try again.

Did you check VirtualBox VM settings ?

Did you test system RAM on the machine that hosts your VM?

Did you check the storage on the machine that hosts your VM/VM Images and OS ?

1>I've been checking the VM settings again and again for months, I can't find anything that looks anomalous to my fairly ignorant eyes. I have been doing this kind of setup for years and never had this issue until recently. Do you know if something changed in pfsense basic setup in the past year that would adversely affect compatibility with basic, typical VM settings?
2> Do you mean a full blown MEMTest kind of thing, running it for many hours kind of thing? No, that I have not done. I haven't had any issues that look like memory problems, it's not unusual to use a good percentage of 64GB. The problem has persisted for a long time with numerous substantial changes in the VM, the VM is loaded and run in a LOT of different orders wrt the other programs running on the PC, which is used for a LOT of different applications. I don't see how memory problems would be at all likely considering these facts, but if you think it's still worth doing, I will try to do but doing one of those is a hassle.

3>I'm not sure what 'check storage' implies, I just did a windows check drive and it found no errors. It's a newish Samsung Evo NVMe. This problem has persisted through multiple virtual disks created for the pfsense VM, new ones with the old ones left alone. I've never had any kind of indication that there was any kind of disk problem for this drive. I've probably gone through at least 6 completely new VMs, new virtual disks, etc. Considering this scenario, if there were disks problems, it sure seems like I would have had some indications, it doesn't seem like I would have very similar problems across multiple virtual disks which would be very different locations on the disk especially with it being an NVMe. Also, this is a network bandwidth issue, not sure how a disk problem would degrade throughput on the network.

I appreciate your efforts, thank you. I will have another post shortly that has some more info.