mpcie and NVME on 2100-MAX

JonathanLee

Hello everyone,

I’m reaching out today with a bit of a specialized challenge, hoping to tap into the collaborative, problem-solving spirit that made so many of us fall in love with pfSense in the first place.

I am currently running a Netgate 2100-MAX. As we know, this specific hardware doesn't natively come with an NVMe slot. However, using a bit of open-source style ingenuity, I successfully got a high-speed drive working perfectly using a custom mPCIE-to-NVMe hardware adapter. I rely heavily on this drive to handle my system swap space and aggressive Squid caching.

Unfortunately, I am completely locked out of updating to the latest versions of pfSense Plus. In recent software releases, the NVMe kernel drivers were stripped out of the software profile for this specific hardware branch, simply because the stock unit doesn't ship with one.

One of the greatest historical strengths of pfSense has always been its foundation in true open-source culture—the idea that our firewalls are highly customizable, adaptable, and meant to be pushed to their limits by the community. It’s always been about user freedom and empowering folks to build creative hardware solutions that fit their unique needs, rather than being confined to rigid manufacturer defaults.

With that original open-source philosophy in mind, has anyone found a clever workaround, loader trick, or a way to manually re-compile and inject the FreeBSD NVMe kernel module (nvme.ko) back into the newer pfSense Plus image for this hardware?

I would love to bring this system up to date without losing the storage architecture I worked so hard to build. Any ideas, guidance, or shared experiences from the community would be incredibly appreciated.

Thank you all for keeping the spirit of customization alive!

stephenw10

The problem is not that the nvme driver are not present. The issue is the the PCIe bus driver for the Marvell SoC broke upstream. It no longer attaches to the bus so no PCIe devices are seen.

In 26.07:

pcib0: <Marvell Armada 3700 PCIe Bus Controller> mem 0xd0070000-0xd008ffff irq 5 on simplebus0
pcib0: link never came up
pci0: <OFW PCI bus> on pcib0
pci0: failed to allocate bus number
device_attach: pci0 attach returned 6

So the fix here is the correct whatever is preventing that in that driver.

JonathanLee

@stephenw10 Thanks for the reply I hate to admit I was watching this all day for anything on the forum.

Just wanted to provide an update on this. Following up on the advice that this needs to be tracked and resolved by the upstream kernel team, I have officially submitted a formal problem report to the FreeBSD Bugzilla.

The upstream kernel bug report here:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=297536

The report targets the 16-CURRENT base system and isolates the regression to the pci_advk driver mechanics failing to map logical bus numbers on the Armada 3700 SoC under the new kernel constraints. Hopefully, an upstream developer can take a look at the allocation handler so the fix can eventually trickle down into future pfSense snapshots! Who knows I hope they do that way wifi can work again too.

JonathanLee

I was chasing a phantom driver issue when it was actually the underlying PCIe link layer dropping out during the pci0 Newbus attachment phase. Following your feedback, I went back to the FreeBSD Bugzilla report (Bug 297536) and updated it to clarify that the pci_advk.c driver is part of Netgate's out-of-tree vendor code for the Armada 3700 rather than upstream FreeBSD main. Because it is out-of-tree vendor code, I am closing that upstream ticket out to avoid wasting the FreeBSD team's time. The SYS_RES_BUS failure (error 6 / ENXIO) looks like a direct clash between legacy resource allocation methods in the vendor driver and the strict new Newbus tracking tracking rules introduced in recent FreeBSD 16-CURRENT snapshots.Since this lives in Netgate's private repository, is there a Redmine ticket tracking this PCIe bus regression for the 2100 development builds? I would love to follow along or help test any patch updates to advk_pcie_alloc_resource whenever the team gets a chance to align it with the updated kernel framework.

something like this??

struct resource *
advk_pcie_alloc_resource(device_t dev, device_t child, int type, int *rid,
    rman_res_t start, rman_res_t end, rman_res_t count, u_int flags)
{
    struct advk_pcie_softc *sc = device_get_softc(dev);

    /* Handle the modern FreeBSD 16-CURRENT sub-bus tracking */
    if (type == SYS_RES_BUS) {
        return (pci_domain_alloc_bus(sc->ap_domain, child, rid, 
            start, end, count, flags));
    }

    /* Fallback handler for memory and IRQ resources */
    return (bus_generic_alloc_resource(dev, child, type, rid, 
        start, end, count, flags));
}

JonathanLee

@stephenw10 Is there a way you can get some output for this command so the freebsd bugzilla can get some more info?

"Have you tried (at the FreeBSD loader prompt):

boot> set debug.rman_debug=1
boot> boot -v

and looking at the rman related output? For a time the
EDK2 draft's ACPI booting of RPI4B's was being rejected
and I figured out what part of the ACPI information was
being rejected starting from what rman reported about
its rejection. It helped to have a serial console I was
able to record and look at."

stephenw10

Ooo, that's fun. There's a lot of output but the relevant part appears to be:

pcib0: <Marvell Armada 3700 PCIe Bus Controller> mem 0xd0070000-0xd008ffff irq 5 on simplebus0
rman_reserve_resource: <I/O memory addresses> request: [0xd0070000, 0xd008ffff], length 0x20000, flags 0, device pcib0
rman_reserve_resource: trying 0xfff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xfff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0x3ffffff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0x53fffff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0x7ffffff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0x8238fff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0x7ffc5fff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0x7ffc6fff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xc7ffffff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xca77ffff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xcfffffff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd00105ff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd0010fff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd0011023 <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd001107f <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd00110a3 <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd0011fff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd00121ff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd0012fff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd00130ff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd00131ff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd00132ff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd00137ff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd00138ff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd0013fff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd001405f <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd0017fff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd00180ff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd00187ff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd00188ff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd002ffff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd0033fff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd003ffff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd0043fff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd0057fff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd005bfff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd005c00f <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd005dfff <0xd0070000,0x1ffff>
rman_reserve_resource: tried 0xd005ffff <0xd0070000,0x1ffff>
considering [0xd0060000, 0xd00d7fff]
truncated region: [0xd0070000, 0xd008ffff]; size 0x20000 (requested 0x20000)
candidate region: [0xd0070000, 0xd008ffff], size 0x20000
splitting region in three parts: [0xd0060000, 0xd006ffff]; [0xd0070000, 0xd008ffff]; [0xd0090000, 0xd00d7fff]
rman_reserve_resource: <Interrupts> request: [0x5, 0x5], length 0x1, flags 0, device pcib0
rman_reserve_resource: trying 0 <0x5,0>
rman_reserve_resource: tried 0 <0x5,0>
rman_reserve_resource: tried 0x1 <0x5,0>
rman_reserve_resource: tried 0x2 <0x5,0>
rman_reserve_resource: tried 0x3 <0x5,0>
rman_reserve_resource: tried 0x4 <0x5,0>
considering [0x5, 0x5]
truncated region: [0x5, 0x5]; size 0x1 (requested 0x1)
candidate region: [0x5, 0x5], size 0x1
candidate region is entire chunk
pcib0: link never came up
rman_manage_region: <(null)> request: start 0xe8000000, end 0xe8ffffff
rman_manage_region: <(null)> request: start 0xe9000000, end 0xe900ffff
rman_manage_region: <A3700 PCIe Interrupts> request: start 0, end 0xffffffffffffffff
pci0: <OFW PCI bus> on pcib0
pci0: failed to allocate bus number
device_attach: pci0 attach returned 6

JonathanLee

@stephenw10 Thanks added to the bugzilla

JonathanLee

@stephenw10 thanks again for the help

The upstream developers confirmed that the phrase "link never came up" does not exist anywhere in the standard FreeBSD source tree. Because this error originates from the out-of-tree pci_advk.c driver for the Marvell Armada 3700 SoC, the driver is failing its initial physical link-training / PHY sequence early in the boot cycle. As a result, the subsequent pci0: failed to allocate bus number (ENXIO) message is just a cascade failure because the child bus is never given a live lane to hook into. It looks like the driver's link-wait timeout or tracking logic is colliding with the newer resource guidelines in the updated core. Is there a Redmine issue tracking an update or a patch for pci_advk.c to fix the lane initialization on the 2100?

stephenw10

This is the public bug for it: https://redmine.pfsense.org/issues/14334

JonathanLee

@stephenw10

Thanks for the reply. I am a bit confused because this setup actually works perfectly for me on version 23.05.01. I am tracking this via the public Redmine tickets #15993 and #16080. Since those have been linked to the internal tracker ID #16655, could you please confirm if that internal ticket is still open? If the PCIe bus lane issue gets fixed, the upstream FreeBSD NVMe drivers should allow the drive to function normally again. Can I safely attempt the upgrade now, or will it still fail on the newer versions? It seems this was fixed in 23.05.01 but broken again in subsequent releases.

Thanks for the reply, again

https://redmine.pfsense.org/issues/15993

https://redmine.pfsense.org/issues/16080

"This issue has the internal tracker ID #16655."

stephenw10

Oh yes you're right that was where the driver was omitted and subsequently replaced for 23.05.

The internal ticket 16655 is for the ath driver panic which we no longer see because the PCIe bus itself no longer attaches.

I don't see an internal ticket for this specifically.

JonathanLee

@stephenw10 So in there a bus failed to attach ticket? Dang, I was hopeful that was fixed so I could move forward with the update

JonathanLee

@stephenw10 Just checking basically it's the same issue same errors as a result of the driver being omitted for the bus?

stephenw10

As far as I know the driver is present but ails to attach unlike when it was omitted in 23.01. Let me double check though....

stephenw10

Yes the files/drivers are still present in 26.07.

JonathanLee

@stephenw10 Thanks for checking, I was thinking if it works for the wifi card again I have then technically it should work again for the drive. Have a good day.

stephenw10

Yeah it definitely doesn't. The ath driver is still broken with a panic. I still have an ath card in a 2100 here and it doesn't hit it because it never sees the device on the bus.

JonathanLee

@stephenw10 I have a M2 SATA with a config on it for my wifi card still too. I enjoyed using it as a all in one system for a while.