D-Link DFE-580TX 4 port Server Adapter problem: only 2 of 4 ports

stephenw10

Here is another user struggling with the same problem. This thread (translated from Czech) has a possible workaround:
http://translate.google.co.uk/translate?hl=en&sl=cs&u=http://www.freebsd.cz/listserv/archive/users-l/2011q4/026764.html

Workaround is to modify the pci-pci driver to reset the noisa bit when it loads. However can that be loaded as a kernel module? Otherwise you would have to recompile the kernel. :-\

But as a hack that there is not such a problem FIXME:

Driver PCI-PCI bridge is sys / dev / pci / pci_pci.c

In it is a key feature pcib_attach_common ()

Well, you have to put it in a suitable place data sequence:

pci_write_config (dev, PCIR_BRIDGECTL_1,
sc-> bridgectl PCIB_BCR_ISA_ENABLE & ~, 2);
sc-> bridgectl pci_read_config = (dev, PCIR_BRIDGECTL_1, 2);

What is the "proper place" is concerned - there you will find the "case" escaped note
"Quirk handling" where he solves individual specifics chip
(Identified by devid).

Steve

jmserrano

@stephenw10:

Just spotted a pretty fundamental typo! :-[
[quote author=stephenw10 link=topic=54066.msg289161#msg289161 date=1348669111]
Could be no difference but it looks like you have missed a : in your pciconf test. It should be:
pciconf -rb pci0:0:2:0: 0x3e

The command should actually be:

pciconf -rb pci0:2:0:0: 0x3e

None of us spotted that 2 in the wrong place. See if you can read the correct bit with that before anything else.

Steve

Corrected the typo, this is the result

pciconf -rb pci0:2:0:0: 0x3e

returns 04

I can change (i suppose disable NOISA bit) with

pciconf -wb pci0:2:0:0: 0x3e 0

and

pciconf -rb pci0:2:0:0: 0x3e

now returns 00

pciconf works perfectly (in thread http://forums.freebsd.org/showthread.php?t=27611, there's an error in the address, it must be hexadecimal)

In ubuntu, after the hack, all ports are enabled (i need to reload sundance module).

Now, the question is:

can i do such opperation in pfsense (reload ste module)? or
can i do pciconf -wb … at boot time?

thanks in advance

stephenw10

As mentioned in the Czech freebsd thread it's not possible to unload/reload the ste driver because it's compiled into the kernel.
You would have to compile a new kernel without it. Probably not worth it!
It is possible to load some kernel modules at boot to override the in kernel version. However I've not seen the pci_pci driver as a module, I don't know if it's possible.

Steve

jmserrano

@stephenw10:

As mentioned in the Czech freebsd thread it's not possible to unload/reload the ste driver because it's compiled into the kernel.
You would have to compile a new kernel without it. Probably not worth it!
It is possible to load some kernel modules at boot to override the in kernel version. However I've not seen the pci_pci driver as a module, I don't know if it's possible.

Steve

Last chance.

Is there anyway to run the command "pciconf -wb …" at boot time.

Thanks again

stephenw10

Nope. You couldn't run that until after the bridge has been detected and then it's too late. You have a few options as I see it:
You could try to reload the ste(4) driver somehow though I don't know how.
You could try to patch the pci_pci driver and load it as a kernel module.
You could hack the BIOS to stop it setting the noisa bit. Not as difficult as you might think but risky.

Steve

wallabybob

Another option is to report the problem to Intel and ask for a BIOS update.

I suspect, not having written any kernel modules to behave this way, that it might be possible to write a kernel module to be loaded at boot time, run before the PCI device tree is walked and fixup the bridge register. Send me a message if you would like to discuss incentives that would persuade me to attempt it.

stephenw10

@wallabybob:

…discuss incentives that would persuade me to attempt it.

:D Ha.
I'd be interested to know how you might go about this.
From my own limited perspective it seems that since the pci_pci driver is not a standard kernel module it would require some fairly high level tinkering to make it into one. Can it even be done?
Perhaps it would be easier to include the workaround code in the ste(4) driver such that it ran before the driver attached?
Were you thinking of something completely different?

Steve

wallabybob

@stephenw10:

I'd be interested to know how you might go about this.

The kernel module declaration includes a numeric code giving the order in system startup in which the module should have its initialisation call (e.g. hard disk scan occurs after device driver initialisation and before root mount is attempted).

I was thinking of a small kernel module loaded by the boot loader because of a suitable entry in /boot/loader.conf. This module would be "initialised" before device drivers, would correct the bridge control register and then "exit". When device drivers run the bridge on the plugin card allows access to i/o registers on ALL the ste NICs.

This wouldn't require any changes to pfSense and would probably be fairly robust over normal FreeBSD version changes.

It would need a careful investigation to see if the kernel modules are sufficiently favourably segmented for there to be "room" to do this before PCI device drivers initialise.

stephenw10

Sounds promising.
Has nobody else done something like this before?
Have to get jmserrano working on those incentives…. ;)

Steve

stephenw10

Hmm, here's something that looks like it might work. It's a nasty workaround and you'd have to come up with the right way of running it at boot but….
Load the ste(4) driver as a kernel module at boot. This will override the in kernel driver with an identical driver.
To do this copy if_ste.ko into /boot/modules and add the line: if_ste_load='yes' to /boot/loader.conf.local
This allows you unload the driver using: kldunload if_ste
Issue the command to reset the NoISA bit on the bridge.
Reload the driver with: kldload if_ste
Punch the air or the wall depending on the result! ;)

Steve

jmserrano

@stephenw10:

Hmm, here's something that looks like it might work. It's a nasty workaround and you'd have to come up with the right way of running it at boot but….
Load the ste(4) driver as a kernel module at boot. This will override the in kernel driver with an identical driver.
To do this copy if_ste.ko into /boot/modules and add the line: if_ste_load='yes' to /boot/loader.conf.local
This allows you unload the driver using: kldunload if_ste
Issue the command to reset the NoISA bit on the bridge.
Reload the driver with: kldload if_ste
Punch the air or the wall depending on the result! ;)

Steve

Thanks Steve, you are great

It works now like a charm.

stephenw10

Nice. :)
When I tried it, with different NICs, I found that after reloading the driver all the interfaces were disabled in pfSense. I didn't experiment further but I imagine this would be resolved by reloading the interfaces or by reloading the driver during boot before the pfSense interfaces are loaded. Did you create a script to do this? Where/when did you run it?
It would be good to get all the details down for anyone else having similar trouble.

Steve

jmserrano

Hi
After a little standby, i can document the solution (excuse the little accuracy, i'm not a hardware expert).

This is the hardware configuration:

PFSense Version : 2.01 - i386
MotherBoard : Intel D2500HN
Multiport Ethernet PCI Board : D-Link DFE-580TX 4 port Server Adapter
Issue : PFSense can only detect 2 of the 4 ports.
Cause: I think it's a bogus BIOS setting incorrectly the noisa bit in the pci-to-pci bridge

Hack (thanks to Steve):
–---
1- Get if_ste kernel module for the correct version of the base freebsd (in this case 8.1) from here http://files.pfsense.org/jimp/ko-8.1/i386/if_ste.ko.
2- Copy it in /boot/modules
3- Add the line: if_ste_load='yes' to /boot/loader.conf.local.
This step failed, and the only way i have found to load the kernel module is to directly modify the file /boot/defaults/loader.conf, setting the line if_ste_load to "yes"
4- After reboot, the kernel module is loades correctly
5- Install package 'Shellcmd'
6- And configure the next 3 commands:

Command Type
–------- -----
kldunload if_ste earlyshellcmd
pciconf -wb pci0:2:0:0 0x3e 0 earlyshellcmd
kldload if_ste earlyshellcmd

From here, PFSense detects 4 of the 4 ports.

I must perform some test, but have all the ports assigned.

Again thanks to all

@stephenw10:

Nice. :)
When I tried it, with different NICs, I found that after reloading the driver all the interfaces were disabled in pfSense. I didn't experiment further but I imagine this would be resolved by reloading the interfaces or by reloading the driver during boot before the pfSense interfaces are loaded. Did you create a script to do this? Where/when did you run it?
It would be good to get all the details down for anyone else having similar trouble.

Steve

stephenw10

Excellent. :)
It's probably worth noting here for anyone replicating this that the pciconf command is system specific. Don't just copy it without reading through this whole thread.

Steve