Unable to use both integrated and PCI expansion NIC

darmok

Goal

I am trying to run pfsense 2.2.5-RELEASE on a small Atom desktop system. The motherboard I have has only one integrated NIC, and one PCI expansion slot. I purchased an Intel Pro/1000 GT Desktop Adapter (82541PI) to install in that slot and serve as the second NIC. I wish to have two separate NICs to assign to WAN and LAN.

Problem

pfsense will automatically detect and use the integrated NIC (a RealTek 8111) as re0, until the Intel NIC is installed. After the card is installed, pfsense can identify and name both the integrated NIC and the Intel NIC, but does not bring either of them up automatically. Also with both NICs installed, I can't seem to bring the interfaces up manually with ifconfig. Thus, my problem is two-fold:

I can't get the RealTek NIC to work with the Intel NIC installed.
I can't get the Intel NIC to work.

Things I've Tried

Research indicates that FreeBSD supports the Intel NIC I purchased using the em driver [1]. This is supported by the output from dmesg that I see discussing both the RealTek and the Intel NICs:

	re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0x2000-0x20ff irq 16 at device 0.0 on pci1
	re0: Using 1 MSI-X message
	re0: Chip rev. 0x28000000
	re0: MAC rev. 0x00100000
	re0: PHY read failed
	re0: attaching PHYs failed
	device_attach: re0 attach returned 6
	em0: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.6="">port 0x1000-0x103f irq 21 at device 0.0 on pci5
	em0: Setup of Shared code failed
	device_attach: em0 attach returned 6</intel(r)></realtek>

The dmesg output above shows all lines with "re0" or "em0" in them when the Intel NIC is installed. Note that both NICs result in a line reading "device_attach: re0 attach returned 6".
When the Intel NIC is removed, the dmesg has the following to say about the integrated RealTek NIC:

re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0x1000-0x10ff irq 16 at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: Chip rev. 0x28000000
re0: MAC rev. 0x00100000
miibus0: <mii bus="">on re0
re0: Ethernet address: my:et:he:er:ne:tt</mii></realtek>

The re0 interface is then available for use as WAN or LAN.
Bring re0 and em0 up manually using /sbin/ifconfig <interface>up. This fails with an error, saying re0 and em0 do not exist.</interface>
Increasing the number of mbufs available to the NICs using the procedure listed in pfsense documentation [2]. This had no effect in allowing me to use the Intel NIC as an interface.
Forcing the loading of the em driver by specifying "if_em_load="YES"" in loader.conf. According to the em man page [3], this should load the driver at boot time. I'm not sure whether this is a problem, though, as I see pfsense trying to handle the Intel NIC as em0 in dmesg output. I made sure adjust /boot/loader.rc as directed in the loader.conf man page [4], to make sure that loader.conf gets called, but still no effect.
Searched these forums. A lot of people have reported similar issues, but still others have said they've had no problem with the 82541PI.

Questions

Has anyone else had similar trouble with two different NICs on pfsense?
What other information can I get about my situation that would help me understand it better? (e.g. command output, diagnostics)
Does anyone have an idea why the integrated NIC is usable when the Intel NIC isn't present, but unavailable when present?

Thanks for reading!

Citations

https://www.freebsd.org/releases/10.1R/hardware.html
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
http://www.unix.com/man-page/freebsd/4/em/
http://www.unix.com/man-page/freebsd/5/loader.conf/

System Specs

Intel Atom D510
4GB RAM
Realtek 8111DL integrated ethernet NIC
Intel BOXD510MO motherboard
Intel Pro/1000 GT Desktop Adapter (82541PI) (PCI)
pfsense 2.2.5

Has anyone else had similar trouble with two different NICs on pfsense?

No. But what kind of pfSense you are using or running?

NanoBSD or full install?

I would suggest to insert the PCI card right into the slot and then do a fresh full install
on a medium likes a HDD/SSD, mSATA or M.2 drive. To be sure to get it right recognized.

What other information can I get about my situation that would help me understand it better? (e.g. command output, diagnostics)

system logfiles
dmesg output

One view into the manual would be mostly also bringing up some light about the slot and his usage
together with the LAN port.

Does anyone have an idea why the integrated NIC is usable when the Intel NIC isn't present, but unavailable when present?

Interrupt collisions
a jumper for using both on the mainboard must be set
A BIOS setting is false
The NIC card is defect
the PCI Slot is defect
the PCI slot is a shared slot and the other shared medium is up and running
PCI card was inserted bad or dust is between the slot and the card
the gold finger from the card are broken or scratched or broken
PCI slot is not 3,3V but more 5,5V but the card is 3,3V or vice versa

darmok

BlueKobold,

Thank you very much for your suggestions. I've made the following progress.

The NIC card is defect

the PCI Slot is defect

PCI card was inserted bad or dust is between the slot and the card

the gold finger from the card are broken or scratched or broken

PCI slot is not 3,3V but more 5,5V but the card is 3,3V or vice versa

I would suggest to insert the PCI card right into the slot and then do a fresh full install
on a medium likes a HDD/SSD, mSATA or M.2 drive. To be sure to get it right recognized.

To test this, I booted the system off of a Fedora 23 live image. Fedora 23 detected and brought up both the integrated NIC and the Intel NIC. Testing each NIC independently, I was able to browse the web and send and receive other Internet traffic. Diagnostics under Fedora 23 show (a) the Intel NIC is brought up without error, (b) the Intel NIC is supported at full duplex, and its vendor and model information is available:

From Fedora 23's dmesg:


[    5.200758] e1000 0000:05:00.0 enp5s0: renamed from eth0
[   30.618834] IPv6: ADDRCONF(NETDEV_UP): enp5s0: link is not ready
[   30.619804] IPv6: ADDRCONF(NETDEV_UP): enp5s0: link is not ready
[   30.619814] 8021q: adding VLAN 0 to HW filter on device enp5s0
[   30.642005] e1000: enp5s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[   30.642247] IPv6: ADDRCONF(NETDEV_CHANGE): enp5s0: link becomes ready

Output from Fedora 23's lspci:


01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 03)
05:00.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)

Output from Fedora 23's ifconfig:


enp5s0: flags=4163<up,broadcast,running,multicast>mtu 1500
        inet 192.168.1.7  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::20e:4ff:fe05:737  prefixlen 64  scopeid 0x20
        ether 00:0e:04:05:07:37  txqueuelen 1000  (Ethernet)
        RX packets 9277  bytes 9893469 (9.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4259  bytes 493365 (481.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0</up,broadcast,running,multicast>

Thus, I think the problem is definitely one of software and not hardware.

Follow-up to Your Questions and Suggestions

But what kind of pfSense you are using or running?

NanoBSD or full install?

I'm using a full install of pfsense 2.2.5-RELEASE for the AMD64 architecture. I'm not aware of the NanoBSD options.

system logfiles

dmesg output

I've examined output from dmesg and ifconfig, but am still looking for logfiles that would be useful. I'm not familiar enough with FreeBSD to know where to look at the moment, but am researching this.

One view into the manual would be mostly also bringing up some light about the slot and his usage
together with the LAN port.

This is a great point; I'm so accustomed to them that I ignored the lights. When I boot pfsense, the lights on board both the integrated NIC and the Intel NIC are lit, indicating that they have network connectivity. The integrated NIC shows it operating at 1000BASE-T, while the Intel NIC is only operating at 100BASE-T. These lights are the same when both NICs are operating under Fedora 23.

Interrupt collisions

a jumper for using both on the mainboard must be set

A BIOS setting is false

Thank you for telling me about these possibilities. I will look into them. The product documentation actually suggested I do the same, so my apologies for your having to mention them.

the PCI slot is a shared slot and the other shared medium is up and running

I was not aware that this was possible. I will look into this, thank you.

Mit Gruss,
darmok

robi

Also try a 32-bit version of pfSense. Read this: https://forum.pfsense.org/index.php?topic=84679.msg481942#msg481942

darmok

robi,

Thanks for the suggestion to try the 32-bit version of pfsense. I will try that next.

Update

I've tried booting the system off a live image of the latest snapshot of FreeBSD 10.2-STABLE (FreeBSD-10.2-STABLE-amd64-20151217-r292403), and had the same problem of the OS failing to recognize both NICs as interfaces. Output from dmesg:


re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0x2000-0x20ff irq 16 at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: Chip rev. 0x28000000
re0: MAC rev. 0x00100000
re0: PHY read failed
re0: attaching PHYs failed
re0: attaching PHYs failed
device_attach: re0 attach returned 6
em0: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.6="">port 0x1000-0x103f irq 21 at device 0.0 on pci5
em0: Setup of Shared code failed
device_attach: em0 attach returned 6</intel(r)></realtek>

The fact that the latest stable snapshot is affected points further toward a software issue.

cmb

Don't bother trying 32 bit, it's not likely to change and if you have something that doesn't work on 64 bit it should be fixed rather than downgrading to something that's a dead end.

The "Setup of shared code failed" message has happened to some with bad firmware version on Intel NICs. If that's one that's upgradeable, that's the first thing to try.

If the system has a BIOS update available, install that. Weird issues like that sometimes are attributable to BIOS bugs fixed in later revisions.

I'd also try a Linux distro of some sort, as hardware problems or bugs or combinations of hardware that just don't work at all together may be the root cause. If Linux works reliably, then you can eliminate the possibility that the combination of hardware is just non-functional.

Then if it works on Linux, take your results with stock FreeBSD 10.2-STABLE and report to freebsd-net list. Someone will hopefully be willing to assist there who can track down and fix the root problem.

darmok

CMB,

Thanks for your suggestions. A colleague of mine who's quite involved with the FreeBSD project recommended taking this to the FreeBSD list as well, so that will be my next step. I've been able to get the NICs working both under Fedora 23 (latest release) and Debian, so I'm pretty sure this is a software issue rather than a firmware one. Your point about the BIOS is well-taken, though, and I'll look into it.

For the record, the 32-bit version of pfSense 2.2.6-RELEASE didn't work. I had the same dmesg output as under 64-bit.

cmb

Sounds like you've definitely confirmed an issue in FreeBSD there. That's probably best for the freebsd-net list, hopefully one of the driver guys will be able to assist there.

UltramaticOrange

Thought I'd chime in and say that I've hit the same issue. The onboard realtek NIC (Asus N3150I-C) worked fine up until I added the 4 port NIC.

Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
4 port Gb HP PCIe(x4) NIC using Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)

pfsense 2.2.6 64bit

everything tests fine under Linux.

UltramaticOrange

Just quick follow-up for anyone else who finds this thread because of the same issue: I installed yesterday's daily build of the 2.3 beta (FreeBSD 10.3 pre-release) and it brought up all interfaces.

For reference, the specific build I'm using:
pfSense-memstick-2.3-BETA-amd64-20160205-0028