Unable to use both integrated and PCI expansion NIC



  • Goal

    I am trying to run pfsense 2.2.5-RELEASE on a small Atom desktop system. The motherboard I have has only one integrated NIC, and one PCI expansion slot. I purchased an Intel Pro/1000 GT Desktop Adapter (82541PI) to install in that slot and serve as the second NIC. I wish to have two separate NICs to assign to WAN and LAN.

    Problem

    pfsense will automatically detect and use the integrated NIC (a RealTek 8111) as re0, until the Intel NIC is installed. After the card is installed, pfsense can identify and name both the integrated NIC and the Intel NIC, but does not bring either of them up automatically. Also with both NICs installed, I can't seem to bring the interfaces up manually with ifconfig. Thus, my problem is two-fold:

    • I can't get the RealTek NIC to work with the Intel NIC installed.

    • I can't get the Intel NIC to work.

    Things I've Tried

    • Research indicates that FreeBSD supports the Intel NIC I purchased using the em driver [1]. This is supported by the output from dmesg that I see discussing both the RealTek and the Intel NICs:
    
    	re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0x2000-0x20ff irq 16 at device 0.0 on pci1
    	re0: Using 1 MSI-X message
    	re0: Chip rev. 0x28000000
    	re0: MAC rev. 0x00100000
    	re0: PHY read failed
    	re0: attaching PHYs failed
    	device_attach: re0 attach returned 6
    	em0: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.6="">port 0x1000-0x103f irq 21 at device 0.0 on pci5
    	em0: Setup of Shared code failed
    	device_attach: em0 attach returned 6</intel(r)></realtek> 
    
    • The dmesg output above shows all lines with "re0" or "em0" in them when the Intel NIC is installed. Note that both NICs result in a line reading "device_attach: re0 attach returned 6".

    • When the Intel NIC is removed, the dmesg has the following to say about the integrated RealTek NIC:

    ```

    re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0x1000-0x10ff irq 16 at device 0.0 on pci1
    re0: Using 1 MSI-X message
    re0: Chip rev. 0x28000000
    re0: MAC rev. 0x00100000
    miibus0: <mii bus="">on re0
    re0: Ethernet address: my:et:he:er:ne:tt</mii></realtek> 
    
    
    *   The re0 interface is then available for use as WAN or LAN.
    
    *   Bring re0 and em0 up manually using /sbin/ifconfig <interface>up. This fails with an error, saying re0 and em0 do not exist.</interface>
    
    *   Increasing the number of mbufs available to the NICs using the procedure listed in pfsense documentation [2]. This had no effect in allowing me to use the Intel NIC as an interface.
    
    *   Forcing the loading of the em driver by specifying "if_em_load="YES"" in loader.conf. According to the em man page [3], this should load the driver at boot time. I'm not sure whether this is a problem, though, as I see pfsense trying to handle the Intel NIC as em0 in dmesg output. I made sure adjust /boot/loader.rc as directed in the loader.conf man page [4], to make sure that loader.conf gets called, but still no effect.
    
    *   Searched these forums. A lot of people have reported similar issues, but still others have said they've had no problem with the 82541PI.
    
    **Questions**
    
    *   Has anyone else had similar trouble with two different NICs on pfsense?
    
    *   What other information can I get about my situation that would help me understand it better? (e.g. command output, diagnostics)
    
    *   Does anyone have an idea why the integrated NIC is usable when the Intel NIC isn't present, but unavailable when present?
    
    Thanks for reading!
    
    **Citations**
    
    *   https://www.freebsd.org/releases/10.1R/hardware.html
    
    *   https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
    
    *   http://www.unix.com/man-page/freebsd/4/em/
    
    *   http://www.unix.com/man-page/freebsd/5/loader.conf/
    
    **System Specs**
    
    *   Intel Atom D510
    
    *   4GB RAM
    
    *   Realtek 8111DL integrated ethernet NIC
    
    *   Intel BOXD510MO motherboard
    
    *   Intel Pro/1000 GT Desktop Adapter (82541PI) (PCI)
    
    *   pfsense 2.2.5


  • Has anyone else had similar trouble with two different NICs on pfsense?

    No. But what kind of pfSense you are using or running?

    • NanoBSD or full install?

    I would suggest to insert the PCI card right into the slot and then do a fresh full install
    on a medium likes a HDD/SSD, mSATA or M.2 drive. To be sure to get it right recognized.

    What other information can I get about my situation that would help me understand it better? (e.g. command output, diagnostics)

    • system logfiles
    • dmesg output

    One view into the manual would be mostly also bringing up some light about the slot and his usage
    together with the LAN port.

    Does anyone have an idea why the integrated NIC is usable when the Intel NIC isn't present, but unavailable when present?

    • Interrupt collisions
    • a jumper for using both on the mainboard must be set
    • A BIOS setting is false
    • The NIC card is defect
    • the PCI Slot is defect
    • the PCI slot is a shared slot and the other shared medium is up and running
    • PCI card was inserted bad or dust is between the slot and the card
    • the gold finger from the card are broken or scratched or broken
    • PCI slot is not 3,3V but more 5,5V but the card is 3,3V or vice versa


  • BlueKobold,

    Thank you very much for your suggestions. I've made the following progress.

    • The NIC card is defect
    • the PCI Slot is defect
    • PCI card was inserted bad or dust is between the slot and the card
    • the gold finger from the card are broken or scratched or broken
    • PCI slot is not 3,3V but more 5,5V but the card is 3,3V or vice versa

    I would suggest to insert the PCI card right into the slot and then do a fresh full install
    on a medium likes a HDD/SSD, mSATA or M.2 drive. To be sure to get it right recognized.

    To test this, I booted the system off of a Fedora 23 live image. Fedora 23 detected and brought up both the integrated NIC and the Intel NIC. Testing each NIC independently, I was able to browse the web and send and receive other Internet traffic. Diagnostics under Fedora 23 show (a) the Intel NIC is brought up without error, (b) the Intel NIC is supported at full duplex, and © its vendor and model information is available:

    From Fedora 23's dmesg:

    
    [    5.200758] e1000 0000:05:00.0 enp5s0: renamed from eth0
    [   30.618834] IPv6: ADDRCONF(NETDEV_UP): enp5s0: link is not ready
    [   30.619804] IPv6: ADDRCONF(NETDEV_UP): enp5s0: link is not ready
    [   30.619814] 8021q: adding VLAN 0 to HW filter on device enp5s0
    [   30.642005] e1000: enp5s0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
    [   30.642247] IPv6: ADDRCONF(NETDEV_CHANGE): enp5s0: link becomes ready
    
    

    Output from Fedora 23's lspci:

    
    01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 03)
    05:00.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
    
    

    Output from Fedora 23's ifconfig:

    
    enp5s0: flags=4163<up,broadcast,running,multicast>mtu 1500
            inet 192.168.1.7  netmask 255.255.255.0  broadcast 192.168.1.255
            inet6 fe80::20e:4ff:fe05:737  prefixlen 64  scopeid 0x20
            ether 00:0e:04:05:07:37  txqueuelen 1000  (Ethernet)
            RX packets 9277  bytes 9893469 (9.4 MiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 4259  bytes 493365 (481.8 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0</up,broadcast,running,multicast> 
    

    Thus, I think the problem is definitely one of software and not hardware.

    Follow-up to Your Questions and Suggestions

    But what kind of pfSense you are using or running?

    • NanoBSD or full install?

    I'm using a full install of pfsense 2.2.5-RELEASE for the AMD64 architecture. I'm not aware of the NanoBSD options.

    • system logfiles
    • dmesg output

    I've examined output from dmesg and ifconfig, but am still looking for logfiles that would be useful. I'm not familiar enough with FreeBSD to know where to look at the moment, but am researching this.

    One view into the manual would be mostly also bringing up some light about the slot and his usage
    together with the LAN port.

    This is a great point; I'm so accustomed to them that I ignored the lights. When I boot pfsense, the lights on board both the integrated NIC and the Intel NIC are lit, indicating that they have network connectivity. The integrated NIC shows it operating at 1000BASE-T, while the Intel NIC is only operating at 100BASE-T. These lights are the same when both NICs are operating under Fedora 23.

    • Interrupt collisions
    • a jumper for using both on the mainboard must be set
    • A BIOS setting is false

    Thank you for telling me about these possibilities. I will look into them. The product documentation actually suggested I do the same, so my apologies for your having to mention them.

    • the PCI slot is a shared slot and the other shared medium is up and running

    I was not aware that this was possible. I will look into this, thank you.

    Mit Gruss,
    darmok



  • Also try a 32-bit version of pfSense. Read this: https://forum.pfsense.org/index.php?topic=84679.msg481942#msg481942



  • robi,

    Thanks for the suggestion to try the 32-bit version of pfsense. I will try that next.

    Update

    I've tried booting the system off a live image of the latest snapshot of FreeBSD 10.2-STABLE (FreeBSD-10.2-STABLE-amd64-20151217-r292403), and had the same problem of the OS failing to recognize both NICs as interfaces. Output from dmesg:

    
    re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" g="" pcie="" gigabit="" ethernet="">port 0x2000-0x20ff irq 16 at device 0.0 on pci1
    re0: Using 1 MSI-X message
    re0: Chip rev. 0x28000000
    re0: MAC rev. 0x00100000
    re0: PHY read failed
    re0: attaching PHYs failed
    re0: attaching PHYs failed
    device_attach: re0 attach returned 6
    em0: <intel(r) 1000="" pro="" legacy="" network="" connection="" 1.0.6="">port 0x1000-0x103f irq 21 at device 0.0 on pci5
    em0: Setup of Shared code failed
    device_attach: em0 attach returned 6</intel(r)></realtek> 
    

    The fact that the latest stable snapshot is affected points further toward a software issue.



  • Don't bother trying 32 bit, it's not likely to change and if you have something that doesn't work on 64 bit it should be fixed rather than downgrading to something that's a dead end.

    The "Setup of shared code failed" message has happened to some with bad firmware version on Intel NICs. If that's one that's upgradeable, that's the first thing to try.

    If the system has a BIOS update available, install that. Weird issues like that sometimes are attributable to BIOS bugs fixed in later revisions.

    I'd also try a Linux distro of some sort, as hardware problems or bugs or combinations of hardware that just don't work at all together may be the root cause. If Linux works reliably, then you can eliminate the possibility that the combination of hardware is just non-functional.

    Then if it works on Linux, take your results with stock FreeBSD 10.2-STABLE and report to freebsd-net list. Someone will hopefully be willing to assist there who can track down and fix the root problem.



  • CMB,

    Thanks for your suggestions. A colleague of mine who's quite involved with the FreeBSD project recommended taking this to the FreeBSD list as well, so that will be my next step. I've been able to get the NICs working both under Fedora 23 (latest release) and Debian, so I'm pretty sure this is a software issue rather than a firmware one. Your point about the BIOS is well-taken, though, and I'll look into it.

    For the record, the 32-bit version of pfSense 2.2.6-RELEASE didn't work. I had the same dmesg output as under 64-bit.



  • Sounds like you've definitely confirmed an issue in FreeBSD there. That's probably best for the freebsd-net list, hopefully one of the driver guys will be able to assist there.



  • Thought I'd chime in and say that I've hit the same issue. The onboard realtek NIC (Asus N3150I-C) worked fine up until I added the 4 port NIC.

    Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
    4 port Gb HP PCIe(x4) NIC using Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)

    pfsense 2.2.6 64bit

    everything tests fine under Linux.



  • Just quick follow-up for anyone else who finds this thread because of the same issue: I installed yesterday's daily build of the 2.3 beta (FreeBSD 10.3 pre-release) and it brought up all interfaces.

    For reference, the specific build I'm using:
    pfSense-memstick-2.3-BETA-amd64-20160205-0028