D-Link DFE-580TX 4 port Server Adapter problem: only 2 of 4 ports

jmserrano

Your four port card has a PCI bridge (Intel 21152) between the four NICs and the CPU.

The BIOS is erroneously setting "ISA enable" in the bridge control register of the 21152.

On pfSense there is nothing you can do about the bridge control register value because the problem occurs during startup BEFORE you have any opportunity to correct the value from an application.

Are you running the latest BIOS (version 0154 released March 23, 2012)? The publicly available release notes give no indication the problem is fixed in that version but a search on the Intel web site for 21152 returned three hits including one giving information on the BIOS updates for the DN2800MT board but that page didn't have an obvious reference to the 21152.

The ISA enable bit in the Bridge Control register of the 21154 PCI Bridge is described as follows:

The 21154 supports ISA mode by providing an ISA enable bit in the bridge control register in
configuration space. ISA mode modifies the response of the 21154 inside the I/O address range in
order to support mapping of I/O space in the presence of an ISA bus in the system. This bit only
affects the response of the 21154 when the transaction falls inside the address range defined by the
I/O base and limit address registers, and only when this address also falls inside the first 64KB of I/
O space (address bits <31:16> are 0000h).
When the ISA enable bit is set, the 21154 does not forward downstream any I/O transactions
addressing the top 768 bytes of each aligned 1KB block. Only those transactions addressing the
bottom 256 bytes of an aligned 1KB block inside the base and limit I/O address range are
forwarded downstream. Transactions above the 64KB I/O address boundary are forwarded as
defined by the address range defined by the I/O base and limit registers.
Accordingly, if the ISA enable bit is set, the 21154 forwards upstream those I/O transactions
addressing the top 768 bytes of each aligned 1KB block within the first 64KB of I/O space. The
master enable bit in the command configuration register must also be set to enable upstream
forwarding. All other I/O transactions initiated on the secondary bus are forwarded upstream only
if they fall outside the I/O address range.
When the ISA enable bit is set, devices downstream of the 21154 can have I/O space mapped into
the first 256 bytes of each 1KB chunk below the 64KB boundary, or anywhere in I/O space above
the 64KB boundary.

(I quoted from the 21154 datasheet because in a short search I couldn't find a datasheet for the 21152 and the datasheet for the E7520 chipset included a very similar description, suggesting the ISA enable bit is a "standard" feature of PCI bridges.)

The BIOS sets ISA enable bit in the bridge upstream of the NICs and then assigns two of the NICs i/O addresses above the first 256 (0x100) bytes of a 1K chunk of I/O space (the chunk from 0x1000 to 0x13ff).

I'm running the lates bios (version 0072 released 08/08/2012), but there's no difference.

Is there anyway to change ISA+ bit on boot?

maybe with a custom kernel?

Thanks in advance

stephenw10

Check your bios. There isn't a specific guide that I could see but some Intel bioses have this:
@Intel:

ISA Enable Bit • Enabled/Disabled Some older expansion devices require this to be enabled.

Worth a try.

Steve

Edit: More detail than Intels glossary:
@http://www.techarp.com/showfreebog.aspx?lang=0&bogno=100:

ISA Enable Bit

Common Options : Enabled, Disabled

Quick Review

This BIOS feature allows you to determine if the system controller will perform ISA aliasing to prevent conflicts between ISA devices.

The default setting of Enabled forces the system controller to alias ISA addresses using address bits [15:10]. This restricts all 16-bit addressing devices to a maximum contiguous I/O space of 256 bytes.

When disabled, the system controller will not perform any ISA aliasing and all 16 address lines can be used for I/O address space decoding. This gives 16-bit addressing devices access to the full 64KB I/O space.

It is recommended that you disable ISA Enable Bit for optimal AGP (and PCI) performance. It will also prevent your AGP or PCI cards from conflicting with your ISA cards. Enable it only if you have ISA devices that are conflicting with each other.

jmserrano

@stephenw10:

Check your bios. There isn't a specific guide that I could see but some Intel bioses have this:
@Intel:

ISA Enable Bit • Enabled/Disabled Some older expansion devices require this to be enabled.

Worth a try.

Steve

Edit: More detail than Intels glossary:
@http://www.techarp.com/showfreebog.aspx?lang=0&bogno=100:

ISA Enable Bit

Common Options : Enabled, Disabled

Quick Review

This BIOS feature allows you to determine if the system controller will perform ISA aliasing to prevent conflicts between ISA devices.

The default setting of Enabled forces the system controller to alias ISA addresses using address bits [15:10]. This restricts all 16-bit addressing devices to a maximum contiguous I/O space of 256 bytes.

When disabled, the system controller will not perform any ISA aliasing and all 16 address lines can be used for I/O address space decoding. This gives 16-bit addressing devices access to the full 64KB I/O space.

It is recommended that you disable ISA Enable Bit for optimal AGP (and PCI) performance. It will also prevent your AGP or PCI cards from conflicting with your ISA cards. Enable it only if you have ISA devices that are conflicting with each other.

No such option exists.

Thanks

stephenw10

If you boot into Ubuntu and issue the workaround does that enable all the ports? I'm strugling to see how it could.
Looking into this there are other similar threads about e.g.:
http://forums.freebsd.org/showthread.php?t=27611

If pciconf doesn't work for some reason, it didn't appear to for you earlier, you can install lspci and setpci:

pkg_add -r http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/8.1-RELEASE/packages/Latest/pciutils.tbz
rehash

I can't see how this would make any difference but maybe worth a shot.

Steve

stephenw10

Just spotted a pretty fundamental typo! :-[
[quote author=stephenw10 link=topic=54066.msg289161#msg289161 date=1348669111]
Could be no difference but it looks like you have missed a : in your pciconf test. It should be:

pciconf -rb pci0:0:2:0: 0x3e

The command should actually be:

pciconf -rb pci0:2:0:0: 0x3e

None of us spotted that 2 in the wrong place. See if you can read the correct bit with that before anything else.

Steve

stephenw10

Here is another user struggling with the same problem. This thread (translated from Czech) has a possible workaround:
http://translate.google.co.uk/translate?hl=en&sl=cs&u=http://www.freebsd.cz/listserv/archive/users-l/2011q4/026764.html

Workaround is to modify the pci-pci driver to reset the noisa bit when it loads. However can that be loaded as a kernel module? Otherwise you would have to recompile the kernel. :-\

But as a hack that there is not such a problem FIXME:

Driver PCI-PCI bridge is sys / dev / pci / pci_pci.c

In it is a key feature pcib_attach_common ()

Well, you have to put it in a suitable place data sequence:

pci_write_config (dev, PCIR_BRIDGECTL_1,
sc-> bridgectl PCIB_BCR_ISA_ENABLE & ~, 2);
sc-> bridgectl pci_read_config = (dev, PCIR_BRIDGECTL_1, 2);

What is the "proper place" is concerned - there you will find the "case" escaped note
"Quirk handling" where he solves individual specifics chip
(Identified by devid).

Steve

jmserrano

@stephenw10:

Just spotted a pretty fundamental typo! :-[
[quote author=stephenw10 link=topic=54066.msg289161#msg289161 date=1348669111]
Could be no difference but it looks like you have missed a : in your pciconf test. It should be:
pciconf -rb pci0:0:2:0: 0x3e

The command should actually be:

pciconf -rb pci0:2:0:0: 0x3e

None of us spotted that 2 in the wrong place. See if you can read the correct bit with that before anything else.

Steve

Corrected the typo, this is the result

pciconf -rb pci0:2:0:0: 0x3e

returns 04

I can change (i suppose disable NOISA bit) with

pciconf -wb pci0:2:0:0: 0x3e 0

and

pciconf -rb pci0:2:0:0: 0x3e

now returns 00

pciconf works perfectly (in thread http://forums.freebsd.org/showthread.php?t=27611, there's an error in the address, it must be hexadecimal)

In ubuntu, after the hack, all ports are enabled (i need to reload sundance module).

Now, the question is:

can i do such opperation in pfsense (reload ste module)? or
can i do pciconf -wb … at boot time?

thanks in advance

stephenw10

As mentioned in the Czech freebsd thread it's not possible to unload/reload the ste driver because it's compiled into the kernel.
You would have to compile a new kernel without it. Probably not worth it!
It is possible to load some kernel modules at boot to override the in kernel version. However I've not seen the pci_pci driver as a module, I don't know if it's possible.

Steve

jmserrano

@stephenw10:

As mentioned in the Czech freebsd thread it's not possible to unload/reload the ste driver because it's compiled into the kernel.
You would have to compile a new kernel without it. Probably not worth it!
It is possible to load some kernel modules at boot to override the in kernel version. However I've not seen the pci_pci driver as a module, I don't know if it's possible.

Steve

Last chance.

Is there anyway to run the command "pciconf -wb …" at boot time.

Thanks again

stephenw10

Nope. You couldn't run that until after the bridge has been detected and then it's too late. You have a few options as I see it:
You could try to reload the ste(4) driver somehow though I don't know how.
You could try to patch the pci_pci driver and load it as a kernel module.
You could hack the BIOS to stop it setting the noisa bit. Not as difficult as you might think but risky.

Steve

wallabybob

Another option is to report the problem to Intel and ask for a BIOS update.

I suspect, not having written any kernel modules to behave this way, that it might be possible to write a kernel module to be loaded at boot time, run before the PCI device tree is walked and fixup the bridge register. Send me a message if you would like to discuss incentives that would persuade me to attempt it.

stephenw10

@wallabybob:

…discuss incentives that would persuade me to attempt it.

:D Ha.
I'd be interested to know how you might go about this.
From my own limited perspective it seems that since the pci_pci driver is not a standard kernel module it would require some fairly high level tinkering to make it into one. Can it even be done?
Perhaps it would be easier to include the workaround code in the ste(4) driver such that it ran before the driver attached?
Were you thinking of something completely different?

Steve

wallabybob

@stephenw10:

I'd be interested to know how you might go about this.

The kernel module declaration includes a numeric code giving the order in system startup in which the module should have its initialisation call (e.g. hard disk scan occurs after device driver initialisation and before root mount is attempted).

I was thinking of a small kernel module loaded by the boot loader because of a suitable entry in /boot/loader.conf. This module would be "initialised" before device drivers, would correct the bridge control register and then "exit". When device drivers run the bridge on the plugin card allows access to i/o registers on ALL the ste NICs.

This wouldn't require any changes to pfSense and would probably be fairly robust over normal FreeBSD version changes.

It would need a careful investigation to see if the kernel modules are sufficiently favourably segmented for there to be "room" to do this before PCI device drivers initialise.

stephenw10

Sounds promising.
Has nobody else done something like this before?
Have to get jmserrano working on those incentives…. ;)

Steve

stephenw10

Hmm, here's something that looks like it might work. It's a nasty workaround and you'd have to come up with the right way of running it at boot but….
Load the ste(4) driver as a kernel module at boot. This will override the in kernel driver with an identical driver.
To do this copy if_ste.ko into /boot/modules and add the line: if_ste_load='yes' to /boot/loader.conf.local
This allows you unload the driver using: kldunload if_ste
Issue the command to reset the NoISA bit on the bridge.
Reload the driver with: kldload if_ste
Punch the air or the wall depending on the result! ;)

Steve

jmserrano

@stephenw10:

Hmm, here's something that looks like it might work. It's a nasty workaround and you'd have to come up with the right way of running it at boot but….
Load the ste(4) driver as a kernel module at boot. This will override the in kernel driver with an identical driver.
To do this copy if_ste.ko into /boot/modules and add the line: if_ste_load='yes' to /boot/loader.conf.local
This allows you unload the driver using: kldunload if_ste
Issue the command to reset the NoISA bit on the bridge.
Reload the driver with: kldload if_ste
Punch the air or the wall depending on the result! ;)

Steve

Thanks Steve, you are great

It works now like a charm.

stephenw10

Nice. :)
When I tried it, with different NICs, I found that after reloading the driver all the interfaces were disabled in pfSense. I didn't experiment further but I imagine this would be resolved by reloading the interfaces or by reloading the driver during boot before the pfSense interfaces are loaded. Did you create a script to do this? Where/when did you run it?
It would be good to get all the details down for anyone else having similar trouble.

Steve

jmserrano

Hi
After a little standby, i can document the solution (excuse the little accuracy, i'm not a hardware expert).

This is the hardware configuration:

PFSense Version : 2.01 - i386
MotherBoard : Intel D2500HN
Multiport Ethernet PCI Board : D-Link DFE-580TX 4 port Server Adapter
Issue : PFSense can only detect 2 of the 4 ports.
Cause: I think it's a bogus BIOS setting incorrectly the noisa bit in the pci-to-pci bridge

Hack (thanks to Steve):
–---
1- Get if_ste kernel module for the correct version of the base freebsd (in this case 8.1) from here http://files.pfsense.org/jimp/ko-8.1/i386/if_ste.ko.
2- Copy it in /boot/modules
3- Add the line: if_ste_load='yes' to /boot/loader.conf.local.
This step failed, and the only way i have found to load the kernel module is to directly modify the file /boot/defaults/loader.conf, setting the line if_ste_load to "yes"
4- After reboot, the kernel module is loades correctly
5- Install package 'Shellcmd'
6- And configure the next 3 commands:

Command Type
–------- -----
kldunload if_ste earlyshellcmd
pciconf -wb pci0:2:0:0 0x3e 0 earlyshellcmd
kldload if_ste earlyshellcmd

From here, PFSense detects 4 of the 4 ports.

I must perform some test, but have all the ports assigned.

Again thanks to all

@stephenw10:

Nice. :)
When I tried it, with different NICs, I found that after reloading the driver all the interfaces were disabled in pfSense. I didn't experiment further but I imagine this would be resolved by reloading the interfaces or by reloading the driver during boot before the pfSense interfaces are loaded. Did you create a script to do this? Where/when did you run it?
It would be good to get all the details down for anyone else having similar trouble.

Steve

stephenw10

Excellent. :)
It's probably worth noting here for anyone replicating this that the pciconf command is system specific. Don't just copy it without reading through this whole thread.

Steve