Some hosts aren't connecting to the internet but others are

[mechtheist]

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

@mechtheist said in Some hosts aren't connecting to the internet but others are:

Not sure what 'real' ip6 implies

I mean a globally routable public IP as opposed to a link-local IP based in the MAC address which every interface has.

An 82571 NIC will easily pass 1Gbps with any vaguely recent processor. Unless it's something ultra low power maybe, but any system designed for that wouldn't be using that NIC.
I wouldn't expect any issue with that.

Steve

OK, I switched the LAN and WAN to the mobo's onboard NICs and no change, then did a new install of pfsense and no change, if anything it was even slower.

I used to drool at the thought of getting a 1Mbps internet connection, a 'T1' I think I always heard them called, but that was a long time ago. So it's not the NIC, here the quad NIC isn't connected in any way. I was also immediately after the new install getting anomalous cpu usage again:

That's what it looks like, the right two are from earlier examples, the left one is from this new install, I saw a number of 3-400% figures, just didn't catch with the screenshot.

The ip6 address popped up again and I see now what happened, pfsense seems to have a lot more ip6 stuff going on and set up as default, there's at least 2 or 3 settings I had to switch off, kept getting 'you can't do that because X is turned on' kinda thing, I really don't recall having to do any of that before.

One strangeness I noted was the syslog was full of this:

Nov 16 14:29:34 	nginx 		2021/11/16 14:29:34 [error] 33672#100684: *11813 open() "/usr/local/www/cgi-bin/luci/;stok=df714b094f1a294eb62bd685e32d868d/admin/status" failed (2: No such file or directory), client: 10.0.0.40, server: , request: "POST /cgi-bin/luci/;stok=df714b094f1a294eb62bd685e32d868d/admin/status?form=wan_speed HTTP/2.0", host: "10.0.0.1", referrer: "https://10.0.0.1/webpages/index.html?t=0093dbde"


Nov 16 14:29:38 	nginx 		2021/11/16 14:29:38 [error] 33899#100585: *13843 open() "/usr/local/www/cgi-bin/luci/;stok=df714b094f1a294eb62bd685e32d868d/admin/status" failed (2: No such file or directory), client: 10.0.0.40, server: , request: "POST /cgi-bin/luci/;stok=df714b094f1a294eb62bd685e32d868d/admin/status?form=wan_speed HTTP/2.0", host: "10.0.0.1", referrer: "https://10.0.0.1/webpages/index.html?t=0093dbde"

There were 4 of the first line followed by 496 of the second one. 10.0.0.40 is my main PC that I'm on now, not the PC with the VMs. So the whole log file was just this crap which means absolutely nothing to me.

I realized trying it bare-metal meant to run pfsense on the same PC I have it virtualized on, it makes no sense to use another PC, I've been thinking of doing that to 'solve' this problem but that wouldn't help figure out what's going on with the VMs. I could just try running it live, that shouldn't be too intrusive, installing it would make me really nervous I'd break something, I'm not that good with boot loaders and the like and zfs is utterly opaque to me.

[stephenw10]

Hmm that's a lot of awking! I'd try running ps -auxwwd to see what it calling that command.

Those errors are something on 10.0.0.40 trying to open a page that does not exist on pfSense.
And the page it's looking for, /cgi-bin/luci/, is from OpenWRT so I suspect you have ha another router at 10.0.0.1 running that at some time.

A restriction to ~1Mbps almost has to be some link issue or something in the VM setup.

Steve

[mechtheist]

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

Hmm that's a lot of awking! I'd try running ps -auxwwd to see what it calling that command.

Those errors are something on 10.0.0.40 trying to open a page that does not exist on pfSense.
And the page it's looking for, /cgi-bin/luci/, is from OpenWRT so I suspect you have ha another router at 10.0.0.1 running that at some time.

A restriction to ~1Mbps almost has to be some link issue or something in the VM setup.

Steve

I appreciate your patience and help. You're probably right about the clogging of the log, I think must be because when I switched from router to pfsense I just took the WAN cable from the router and plugged it in the NIC for pfsense and then unplugged the router from the LAN, so my PC was probably still trying to doing something with the router. I thought the same thing about the awk cpu usage. The one with awk at 99% also has 'unslogd' right under it at 11% so I was thinking it had something to do with diddling with the log file archiving. I couldn't find anything on unslogd but it seems a plausible guess at what unlogging might do;)

You've been a great help and I'm hoping you can give me a little more, what would I look for if it's a link issue? Are we talking packet capture? Are there any kind of errors you think I should look for? The VM settings thing is really frustrating and why I keep repeating about doing this for a long time, as far as I know, I'm setting up the VMs the same way I've always done except switching to the vertio for the networks, and I did that when I was already having the problem hoping it might improve the performance and it did by some, maybe 5-10% at best. I ordered one of the newer NICs with the 82580 controllers. It's not the NIC causing the problem but maybe something will popup by switching to the newer card.

In case anyone can make sense of it, this is the setup info on virtualbox:

C:\Users\rob>vboxmanage showvminfo "pfsense_octo"
Name:                        pfsense_octo
Groups:                      /
Guest OS:                    FreeBSD (64-bit)
UUID:                        e73eb791-860c-40f9-94a5-b02b4073ec44
Config file:                 H:\VirtualBoxVMs\pfsense_octo\pfsense_octo.vbox
Snapshot folder:             H:\VirtualBoxVMs\pfsense_octo\Snapshots
Log folder:                  H:\VirtualBoxVMs\pfsense_octo\Logs
Hardware UUID:               e73eb791-860c-40f9-94a5-b02b4073ec44
Memory size:                 10121MB
Page Fusion:                 disabled
VRAM size:                   126MB
CPU exec cap:                100%
HPET:                        disabled
CPUProfile:                  host
Chipset:                     piix3
Firmware:                    BIOS
Number of CPUs:              8
PAE:                         enabled
Long Mode:                   enabled
Triple Fault Reset:          disabled
APIC:                        enabled
X2APIC:                      disabled
Nested VT-x/AMD-V:           disabled
CPUID Portability Level:     0
CPUID overrides:             None
Boot menu mode:              message and menu
Boot Device 1:               DVD
Boot Device 2:               HardDisk
Boot Device 3:               Not Assigned
Boot Device 4:               Not Assigned
ACPI:                        enabled
IOAPIC:                      enabled
BIOS APIC mode:              APIC
Time offset:                 0ms
RTC:                         local time
Hardware Virtualization:     enabled
Nested Paging:               enabled
Large Pages:                 enabled
VT-x VPID:                   enabled
VT-x Unrestricted Exec.:     enabled
Paravirt. Provider:          Default
Effective Paravirt. Prov.:   None
State:                       powered off (since 2021-11-16T16:34:52.162000000)
Graphics Controller:         VMSVGA
Monitor count:               1
3D Acceleration:             disabled
2D Video Acceleration:       disabled
Teleporter Enabled:          disabled
Teleporter Port:             0
Teleporter Address:
Teleporter Password:
Tracing Enabled:             disabled
Allow Tracing to Access VM:  disabled
Tracing Configuration:
Autostart Enabled:           disabled
Autostart Delay:             0
Default Frontend:
VM process priority:         default
Storage Controller Name (0):            AHCI
Storage Controller Type (0):            IntelAhci
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0):  30
Storage Controller Port Count (0):      2
Storage Controller Bootable (0):        on
AHCI (0, 0): Empty
AHCI (1, 0): H:\VirtualBoxVMs\pfsense_octo\pfsense-2nd-replacement-11-11-2021--1333.vhd (UUID: a60260db-d839-44c6-8434-345e0f64826c)
NIC 1:                       MAC: 080027B1449A, Attachment: Bridged Interface 'Intel(R) I211 Gigabit Network Connection', Cable connected: on, Trace: off (file: none), Type: virtio, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none
NIC 2:                       MAC: 08002719A76D, Attachment: Bridged Interface 'Intel(R) Ethernet Connection (2) I218-V', Cable connected: on, Trace: off (file: none), Type: virtio, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: allow-all, Bandwidth group: none
NIC 3:                       disabled
NIC 4:                       disabled
NIC 5:                       disabled
NIC 6:                       disabled
NIC 7:                       disabled
NIC 8:                       disabled
Pointing Device:             USB Tablet
Keyboard Device:             PS/2 Keyboard
UART 1:                      disabled
UART 2:                      disabled
UART 3:                      disabled
UART 4:                      disabled
LPT 1:                       disabled
LPT 2:                       disabled
Audio:                       enabled (Driver: DSOUND, Controller: AC97, Codec: STAC9700)
Audio playback:              enabled
Audio capture:               disabled
Clipboard Mode:              disabled
Drag and drop Mode:          disabled
VRDE:                        enabled (Address 0.0.0.0, Ports 33891, MultiConn: on, ReuseSingleConn: off, Authentication type: null)
Video redirection:           disabled
VRDE property               : TCP/Ports  = "33891"
VRDE property               : TCP/Address = <not set>
VRDE property               : VideoChannel/Enabled = <not set>
VRDE property               : VideoChannel/Quality = <not set>
VRDE property               : VideoChannel/DownscaleProtection = <not set>
VRDE property               : Client/DisableDisplay = <not set>
VRDE property               : Client/DisableInput = <not set>
VRDE property               : Client/DisableAudio = <not set>
VRDE property               : Client/DisableUSB = <not set>
VRDE property               : Client/DisableClipboard = <not set>
VRDE property               : Client/DisableUpstreamAudio = <not set>
VRDE property               : Client/DisableRDPDR = <not set>
VRDE property               : H3DRedirect/Enabled = <not set>
VRDE property               : Security/Method = <not set>
VRDE property               : Security/ServerCertificate = <not set>
VRDE property               : Security/ServerPrivateKey = <not set>
VRDE property               : Security/CACertificate = <not set>
VRDE property               : Audio/RateCorrectionMode = <not set>
VRDE property               : Audio/LogPath = <not set>
OHCI USB:                    disabled
EHCI USB:                    disabled
xHCI USB:                    enabled

USB Device Filters:

<none>

Bandwidth groups:  <none>

Shared folders:

Name: 'shared', Host path: 'H:\VirtualBoxVMs\pfsense_octo\shared' (machine mapping), writable, auto-mount, mount-point: '/root/octo'

Capturing:                   active
Capture audio:               not active
Capture screens:             0
Capture file:                H:\VirtualBoxVMs\pfsense_octo\pfsense_octo.webm
Capture dimensions:          1024x768
Capture rate:                512kbps
Capture FPS:                 25kbps
Capture options:             vc_enabled=true,ac_enabled=false,ac_profile=med

Guest:

Configured memory balloon size: 0MB

[stephenw10]

I'm not really very familiar with VBox but nothing there jumps out. I do run it here occasionally but in Linux. What is the host processor? Nothing would really make that much difference but that would be a difference.
You should be able to run it in hyper-v on the same host relatively easily. It you can't run it bare metal that's the next thing I would try.
https://docs.netgate.com/pfsense/en/latest/recipes/virtualize-hyper-v.html

Steve

[mechtheist]

@stephenw10
I was actually thinking of using the linux VM I use for a web server to run a pfsense VM on, there's probably more than enough cpu power to work adequately. The PC is win10 on a i7-5960X w/ 64GB RAM with an NVIDIA 2080 gpu hopefully that will be enough so it's not too painful to work with.

I wrote that yesterday, still working on getting a VM in a VM going. I'm definitely going to try HyperV, one reason for the hesitation is you have to turn all of it off in Windows or it interferes with VirtualBox. The last time I used it I think was back in win8 when Windows basically came with a VM XP. The new card should get here Monday.

[stephenw10]

Hmm, VM in a VM is double the potential issues IMO.

If there's any way you can test it bare metal however temporarily I would do that.

At least moving to Hyper-V changes the potential issues so if you still see it the issue is probably on the host or external.

Steve

[mechtheist]

@stephenw10 Yeah, I'm going to do a live instance and see how that goes. Thanks for helping out and being patient.

[mechtheist]

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

Hmm, VM in a VM is double the potential issues IMO.

If there's any way you can test it bare metal however temporarily I would do that.

At least moving to Hyper-V changes the potential issues so if you still see it the issue is probably on the host or external.

Steve

The deed is done, not too many hiccups but one BIG one, pfsense doesn't do Live anymore, and it's been awhile, I had no idea. So I used a USB drive, worked flawlessly pretty much. Weirdly, I couldn't log on with either chrome or Firefox, chrome just took the ID and password and ignored them and presented a blank login page again and firefox simply wouldn't connect, it timed out every time even making sure I wasn't in https. speedtest-

I kinda forgot how zippy it feels when it's not virtualized. I was looking at the pfsense documentation and it discusses Type 1 and Type 2 virtualization engines, the Type 1 it calls 'bare metal'.

So it's sitting there all zippy and everything and I realized I had no idea what I should look for, I'm an idiot. The only thing I did was to look at ifconfig and it showed

em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=81009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER>
	ether a0:36:9f:29:18:28
	inet6 fe80::a236:9fff:fe29:1828%em1 prefixlen 64 scopeid 0x2
	inet6 ::a236:9fff:fe29:1828 prefixlen 64 autoconf
	inet6 ::cbda:70cf:b28a:8ad5 prefixlen 128
	inet 192.168.0.7 netmask 0xffffff00 broadcast 192.168.0.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

The one thing that stands out to me, if you look all the way up to my first post I mention how it was reporting a 10GbaseT connection in virtualized operation but bare-metal, it's showing the correct 1000baseT. I changed the properties of the NICs in windows to 1000 from 'auto' but that didn't change anything. Could this be the problem? I'd assume there's a setting in pfsense for this but it's never been an issue before and I didn't think of changing it there and I'm back on the router now.
So much fun.

[stephenw10]

@mechtheist said in Some hosts aren't connecting to the internet but others are:

I changed the properties of the NICs in windows to 1000 from 'auto'

You had set the NICs to 1G fixed? That could definitely cause a problem. When using Gigabit Ethernet you almost always should have the speed/duplex set to autoselect. The only time you would not is if the other end is also set fixed and that should only ever happen on a 100M link. Gigabit requires autoselect.
I suspect that is the problem here and the virtualisation is hiding it. VBox always present as a 10G NIC to pfSense so it can't see the real link speed.

Steve

[mechtheist]

@stephenw10 said in Some hosts aren't connecting to the internet but others are:

@mechtheist said in Some hosts aren't connecting to the internet but others are:

I changed the properties of the NICs in windows to 1000 from 'auto'

You had set the NICs to 1G fixed? That could definitely cause a problem. When using Gigabit Ethernet you almost always should have the speed/duplex set to autoselect. The only time you would not is if the other end is also set fixed and that should only ever happen on a 100M link. Gigabit requires autoselect.
I suspect that is the problem here and the virtualisation is hiding it. VBox always present as a 10G NIC to pfSense so it can't see the real link speed.

Steve

I only set it to 1G fixed recently to see if that made a difference and it didn't. I think the 10G thing must be a virtio thing, I never saw that until recently and I switched to using the virtio interfaces recently. If everyone is talking 1G fixed it's still an issue? That's the first time I ever remember changing that setting, I'll put them all back on auto. This is really frustrating, it's a x50 reduction in bandwidth FFS, it should be something glaringly obvious. Running top on that PC had the top running processes all mostly below .1% most of the time, not used to seeing numbers in top like .04%.

[mechtheist]

It's weird, I messed up the quote somehow and when i tried to edit it which only involved inserting a carriage return after the past line of the quote and my saved edit got flagged as spam!

[stephenw10]

Mmm, the forum hates too many edits! I made that edit.

Yeah, VBox with virtio always appears as 10G.

Yes, there would be no problem if both ends of both links are set to 1G fixed. However the issue is likely to be to a modem etc where it cannot be set fixed. You should really always leave it as autoselect for a Gig link.

Steve

[mechtheist]

@stephenw10 Well, I got Hyper-V working, kinda sorta, a few learning curve issues with the network settings using switches and how that affects which interface is doing what etc, and there were some weird glitches that I can't fathom but that's kinda SOP. But it looks good:

Thanks for the help, I'll probably never know what the hell went wrong with pfsense on VirtualBox but gotta move on.

[stephenw10]

Cool. Yeah looks like an issue in VBox then somehow.