PFSense on a DEC3840 (Netboard A20)

stephenw10

What's in device.hints? You can see it's loading that too.

Steve

bpl294

# $FreeBSD$
hint.fdc.0.at="isa"
hint.fdc.0.port="0x3F0"
hint.fdc.0.irq="6"
hint.fdc.0.drq="2"
hint.fd.0.at="fdc0"
hint.fd.0.drive="0"
hint.fd.1.at="fdc0"
hint.fd.1.drive="1"
hint.atkbdc.0.at="isa"
hint.atkbdc.0.port="0x060"
hint.atkbd.0.at="atkbdc"
hint.atkbd.0.irq="1"
hint.psm.0.at="atkbdc"
hint.psm.0.irq="12"
hint.sc.0.at="isa"
hint.sc.0.flags="0x100"
hint.uart.0.at="isa"
hint.uart.0.port="0x3F8"
hint.uart.0.flags="0x10"
hint.uart.0.irq="4"
hint.uart.1.at="isa"
hint.uart.1.port="0x2F8"
hint.uart.1.irq="3"
hint.ppc.0.at="isa"
hint.ppc.0.irq="7"
hint.atrtc.0.at="isa"
hint.atrtc.0.port="0x70"
hint.atrtc.0.irq="8"
hint.attimer.0.at="isa"
hint.attimer.0.port="0x40"
hint.attimer.0.irq="0"
hint.acpi_throttle.0.disabled="1"
hint.p4tcc.0.disabled="1"

stephenw10

Hmm, nothing unusual there either.....

bpl294

@stephenw10 do you have any other idea why the installer won't load? Is there any bios settings i should look for? Is there anyway to do a more verbose install?

stephenw10

You can interrupt the boot loader just before it gets to that point and force it to boot verbose at the prompt:

OK> boot -v

What NICs does that have?
If you install to the SSD in something else then move that across before the first boot it will come up normally if there is a profile for the NICs. So you would be able to hit the webgui even if the console doesn't work.

Steve

bpl294

@stephenw10 It has 4 Intel I210s and 2 AMD SFP+ ports. OPNSense looks like it load the ax drivers for them. Does PFSense support those? I can try disabling them in the bios for the PF install.

I installed pf on a donor machine and put the drive in the DEC3840, same issue. It hangs on loading the kernel. I tried disabling the SFP+ ports as well, no change in outcome.

bpl294

@stephenw10 Also, looks like there is another loader.conf in the /boot/defaults directory:

Not sure if it shows anything else there...

defaults_loader.txt

stephenw10

Yeah, if it didn't have drivers for the ax NICs it should still boot completely with igb0 as WAN and igb1 as LAN. Assuming you didn't configure any interfaces on the install box before moving it.

In that case this looks like not a console problem at all bit actually hanging booting the kernel for some reason. Might be time to check the FreeBSD forum for anything known for that CPU/platform. Or dig in the OPN code when they introduced it for any defaults they changed.

Steve

bpl294

@stephenw10 I contacted the Manufacturer of this appliance, they said the following:

"We don't support pfSense, in order to gain pfSense support you likely need to compile your own kernel at least for the 10gbps drivers to work, the serial console might have issues as well as the Epyc is only supported correctly in the current FreeBSD source if I'm correct.
You can always give it a try to downgrade to pfSense, but we can't really help you on that front.

If you're tech savvy enough, I don't mind sharing some of the upstream FreeBSD code references you will likely need in your kernel, just let me know in that case."

is any of that helpful in pointing me in the right direction?

stephenw10

Certainly support for the AMD SoC axgbe driver is not in FreeBSD 12 so that would require a back-port which is unlikely.
https://github.com/freebsd/freebsd-src/commit/7113afc84c0b68f1e531dbd6d57d024d868d11c0

Surprising it would require anything for serial console support but...
This maybe: https://bsdimp.blogspot.com/2018/07/how-to-get-memory-mapped-serial-console.html

If that was the case though I'd expect to see a loader value for it and the only one they have are the default values.

Can you see the console location in the OPN boot log?

Steve

bpl294

@stephenw10 the dmesg | grep tty didnt return anything in the serial console. In the /dev/ directory i do see quite a few "tty*" files. Only one was changed since i logged in, that was "ttyu0", see below...

Is this what you meant?

stephenw10

If it's this: https://reviews.freebsd.org/D16432
Then I expect uart0 to show as something other than that standard IO port in the boot log there.

Steve

bpl294

This post is deleted!

bpl294

@stephenw10 here is some of the code changes that he was referring to:

"To make sure the OS can find the serial port, you need to remove some pre production acpi hack, this https://github.com/freebsd/freebsd-src/commit/35af9331 should do the trick.

The 10gbps network card needs a driver, which AMD upstreams to https://github.com/freebsd/freebsd-src/tree/main/sys/dev/axgbe, we do have some additional bug fixes which will likely land later on (you can find them now in our repo, https://github.com/opnsense/src)"

can these changes be applied via a module or does it need to be applied in the kernel itself?

stephenw10

Ok, that is in 2.6 so the first thing to do there is just try a 2.6 snapshot:
https://github.com/pfsense/FreeBSD-src/commit/a7c68340584c942792188ad50593d4ef15cc8982#diff-96de3fc05e938f0fd1d95debb8e797e7c1da4645867d1722e01b1eff85e17186

Steve

bpl294

@stephenw10 the most recent snapshot of 2.6 stalls at the spot below:

stephenw10

So slightly further?

bpl294

@stephenw10 yes:

2.5.2

2.6

bpl294

@stephenw10 support from Decisio metioned the following as well:

"Try to add the following to loader.conf.local hw.uart.console="io:0x3f8,br:115200" "

Would this get added to the loader.conf.local after install on a donor system? or can it be done pre-install?

stephenw10

You could set that at the loader prompt before it boots.

OK> set hw.uart.console="io:0x3f8,br:115200"
OK> boot

You could add it to loader.conf.local on the other system before moving is across.

That was the only line we saw which looked like it could be doing anything so it's certainly worth trying. Those are the default com1 values though.
Having read through the commits it might be setting it back to the default values after the detection quirks that were added for earlier EPYC systems were interfering.

Steve