PCI express TP-LINK

rosch

Hi,

I am trying to get pfsense working on an Fujitsu Esprimo P400 with two physical ethernet NICs: one integrated and one additional card.
The board has PCI Express x1 and x16 slots.
I used one x1 slot for a TP-LINK TG-3468 ethernet card which comes with the Realtek r8169 driver sources.

The onboard NIC is also a Realtek chip.

The issue I am having is neither pfsense 2.1 beta1 nor Ubuntu 12.04 is seeing the PCIe NIC.
No issues for the onboard NIC.
BIOS is up-to-date.
I tried different BIOS setting without success.

Has anybody had success with PCIe NICs?
Could this be a hardware problem?
Two of the four LEDs are on (green, not blinking as no traffic).

The PCIe NIC from TP-LINK:
http://www.tp-link.us/products/details/?model=TG-3468

cheers

ptt

I'm using one of these PCI-e TP-Link adapters on my pfSense 2.0.x (at my home) and it works OK, so seems that you have some HW issue


re1: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" pcie="" gigabit="" ethernet="">port 0xe800-0xe8ff mem 0xfebff000-0xfebfffff irq 17 at device 0.0 on pci2
re1: Using 1 MSI messages
re1: Chip rev. 0x38000000
re1: MAC rev. 0x00000000
miibus1: <mii bus="">on re1
rgephy0: <rtl8169s 8110s="" 8211b="" media="" interface="">PHY 1 on miibus1
rgephy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto</rtl8169s></mii></realtek>

rosch

Thanks for your quick reply.
Can I ask what onboard NIC you have?

ptt

RealTek, too, but it is an 10/100…

The Motherboard is an ECS TIGT-I (Atom D510)


re0: <realtek 10="" 8101e="" 8102e="" 8102el="" 8103e="" pcie="" 100basetx="">port 0xd800-0xd8ff mem 0xfdfff000-0xfdffffff,0xfdfe0000-0xfdfeffff irq 16 at device 0.0 on pci1
re0: Using 1 MSI messages
re0: Chip rev. 0x24800000
re0: MAC rev. 0x00400000
miibus0: <mii bus="">on re0
rlphy0: <rtl8201l 10="" 100="" media="" interface="">PHY 1 on miibus0
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto</rtl8201l></mii></realtek>

I also have used 2 of these on an Asus M2N32-SLI Deluxe, running pfsense 2.0.1 AMD64, for more than a year without problem….

rosch

Tried on another machine having an Intel on-board NIC. There all is well. (for some reason the box wanted to boot forcefully from the card but once I got past that I saw the two interfaces)

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)
	Subsystem: Realtek Semiconductor Co., Ltd. TEG-ECTX Gigabit PCI-E Adapter [Trendnet]
	Flags: bus master, fast devsel, latency 0, IRQ 45
	I/O ports at e000 [size]
	Memory at fe420000 (64-bit, non-prefetchable) [size]
	Expansion ROM at fe400000 [disabled] [size]
	Capabilities: [40] Power Management version 2
	Capabilities: [48] Vital Product Data
	Capabilities: [50] MSI: Enable+ Count=1/2 Maskable- 64bit+
	Capabilities: [60] Express Endpoint, MSI 00
	Capabilities: [84] Vendor Specific Information: Len=4c 
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [12c] Virtual Channel
	Capabilities: [148] Device Serial Number 06-1a-00-00-68-4c-e0-00
	Capabilities: [154] Power Budgeting 
	Kernel driver in use: r8169
	Kernel modules: r8169

It looks like on the P400 the on-board NIC is not visible any more as the PCIe card is "taking over".[/size][/size][/size]

rosch

This is how the on-board card is reported in Ubuntu (unplugged TP-LINK PCIe card):

description: Ethernet interface
               product: RTL8111/8168B PCI Express Gigabit Ethernet controller
               vendor: Realtek Semiconductor Co., Ltd.
               physical id: 0
               bus info: pci@0000:02:00.0
               logical name: eth0
               version: 07
               serial: 00:19:99:eb:80:1b
               size: 10Mbit/s
               capacity: 1Gbit/s
               width: 64 bits
               clock: 33MHz
               capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
               configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=2.3LK-NAPI duplex=half firmware=rtl8168e-3_0.0.4 03/27/12 latency=0 link=no multicast=yes port=MII speed=10Mbit/s
               resources: irq:41 ioport:e000(size=256) memory:f0004000-f0004fff memory:f0000000-f0003fff

Is it healthy that it is seen as a PCI Express card although it is on-board?

wallabybob

@rosch:

The issue I am having is neither pfsense 2.1 beta1 nor Ubuntu 12.04 is seeing the PCIe NIC.
No issues for the onboard NIC.

My guess is that the BIOS is disabling the on-board NIC when it sees another NIC. That might be because the box is targetted to a particular application.

To test my theory I suggest you post the output of the Ubuntu shell command```
lspci -n -n ; lspci -t

rosch

@wallabybob:

To test my theory I suggest you post the output of the Ubuntu shell command```
lspci -n -n ; lspci -t

No sure what you mean with the labelling..here are the lspci (concatenated) outputs:

TP-LINK PCIex1 plugged in

00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0100] (rev 09)
00:02.0 VGA compatible controller [0300]: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller [8086:0102] (rev 09)
00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev 04)
00:1b.0 Audio device [0403]: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller [8086:1c20] (rev 04)
00:1c.0 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 [8086:1c10] (rev b4)
00:1c.4 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 [8086:1c18] (rev b4)
00:1c.5 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 [8086:1c1a] (rev b4)
00:1d.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 [8086:1c26] (rev 04)
00:1f.0 ISA bridge [0601]: Intel Corporation H61 Express Chipset Family LPC Controller [8086:1c5c] (rev 04)
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 04)
00:1f.3 SMBus [0c05]: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller [8086:1c22] (rev 04)
03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller [10ec:8168] (rev 07)
-[0000:00]-+-00.0
           +-02.0
           +-1a.0
           +-1b.0
           +-1c.0-[01]--
           +-1c.4-[02]--
           +-1c.5-[03]----00.0
           +-1d.0
           +-1f.0
           +-1f.2
           \-1f.3

TP-LINK PCIex1 NOT plugged in

00:00.0 Host bridge [0600]: Intel Corporation 2nd Generation Core Processor Family DRAM Controller [8086:0100] (rev 09)
00:02.0 VGA compatible controller [0300]: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller [8086:0102] (rev 09)
00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev 04)
00:1b.0 Audio device [0403]: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller [8086:1c20] (rev 04)
00:1c.0 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 [8086:1c10] (rev b4)
00:1c.5 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 [8086:1c1a] (rev b4)
00:1d.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 [8086:1c26] (rev 04)
00:1f.0 ISA bridge [0601]: Intel Corporation H61 Express Chipset Family LPC Controller [8086:1c5c] (rev 04)
00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 04)
00:1f.3 SMBus [0c05]: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller [8086:1c22] (rev 04)
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller [10ec:8168] (rev 07)
-[0000:00]-+-00.0
           +-02.0
           +-1a.0
           +-1b.0
           +-1c.0-[01]--
           +-1c.5-[02]----00.0
           +-1d.0
           +-1f.0
           +-1f.2
           \-1f.3

Update: the reported Ethernet controller is the on-board one, both times. I checked by plugging a network cable.

wallabybob

@rosch:

TP-LINK PCIex1 plugged in


-[0000:00]-+-00.0
           +-02.0
           +-1a.0
           +-1b.0
           +-1c.0-[01]--
           +-1c.4-[02]--
           +-1c.5-[03]----00.0
           +-1d.0
           +-1f.0
           +-1f.2
           \-1f.3

TP-LINK PCIex1 NOT plugged in


-[0000:00]-+-00.0
           +-02.0
           +-1a.0
           +-1b.0
           +-1c.0-[01]--
           +-1c.5-[02]----00.0
           +-1d.0
           +-1f.0
           +-1f.2
           \-1f.3

Note that with the card plugged in lspci reports a PCI bridge at PCI bus 00, slot 1c function 4 and when the card is not plugged in lspci doesn't report a bridge there.

The BIOS is not fully making the expansion card visible. A similar sort has been reported in the pfSense forums on a SuperMicro motherboard. I'm not sure of the details since it was a while ago, but I think there was a PCI-E x16 slot that was advertised as supporting a graphics card but the BIOS wouldn't report a quad port NIC card that was plugged into the slot. I think you will have to take this up with Fujitsu support.

You might be able to get different behaviour by tweaking BIOS options. For example, if the onboard LAN is disabled does the plugin NIC become visible?

rosch

Thanks, I had already called Fujitsu but got an incompetent person. The second one was unfriendly but I got it sorted out:
I finally see both cards in the BIOS and pfsense. So some success :)
I had to change the "Slot Link Speed" to Gen1 instead of Auto under the BIOS PCI settings.

But now after installing pfsense to the harddisk I get:

random reboots (things like Fatal trap 12: page fault while in kernel mode have occurred)
inability to ping the Lan interface from another machine. Hence also unable to connect via the web interface.

I tried different BIOS settings including setting SATA to IDE and booting without APIC and/or ACPI.
No luck so far.
I wish I never bought this low-end machine.

edit: the installed version is beta1 from today, March 14.

wallabybob

@rosch:

I wish I never bought this low-end machine.

Sometimes what you save in cash you spend in hours.

@rosch:

inability to ping the Lan interface from another machine. Hence also unable to connect via the web interface.

Please post the ping command and response. Perhaps you are attempting the ping from an inappropriately configured system.

Please post the output of the pfSense shell command```
/etc/rc.banner ; ifconfig


The pfSense LAN interface must be UP and RUNNING before you will a response to any stimulus directed at it.

If you can get communication with the pfSense LAN interface then you should be able to capture a crash report which could help identify the cause of the crashes.

rosch

@wallabybob:

Sometimes what you save in cash you spend in hours.

This machine has become a very expensive one over the last two weeks.. :-)

@wallabybob:

Please post the ping command and response. Perhaps you are attempting the ping from an inappropriately configured system.

I am able to get an IP address over dhcp now..and hence LAN pinging is working fine too.

@wallabybob:

Please post the output of the pfSense shell command```
/etc/rc.banner ; ifconfig


WAN(wan) -> pppoe1 ->
LAN(lan) -> re0 -> v:4 192.168.5.211/24

re0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 90:f6:52:05:11:f0
	inet 192.168.5.211 netmask 0xffffff00 broadcast 192.168.5.255
	inet6 fe80::92f6:52ff:fe05:11f0%re0 prefixlen 64 scopeid 0x1 
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
re1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 00:19:99:eb:80:1b
	inet6 fe80::219:99ff:feeb:801b%re1 prefixlen 64 scopeid 0x2 
	nd6 options=1 <performnud>media: Ethernet autoselect (100baseTX <full-duplex>)
	status: active
enc0: flags=0<> metric 0 mtu 1536
pfsync0: flags=0<> metric 0 mtu 1460
	syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
	options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5 
	nd6 options=3 <performnud,accept_rtadv>pflog0: flags=100 <promisc>metric 0 mtu 33664
ovpns2: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
	options=80000 <linkstate>inet6 fe80::92f6:52ff:fe05:11f0%ovpns2 prefixlen 64 scopeid 0x8 
	inet 192.168.10.1 --> 192.168.10.2 netmask 0xffffffff 
	nd6 options=3 <performnud,accept_rtadv>Opened by PID 12199
pppoe2: flags=88d1 <up,pointopoint,running,noarp,simplex,multicast>metric 0 mtu 1500
	inet6 fe80::92f6:52ff:fe05:11f0%pppoe2 prefixlen 64 scopeid 0x7 
	nd6 options=3<performnud,accept_rtadv></performnud,accept_rtadv></up,pointopoint,running,noarp,simplex,multicast></performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></promisc></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast>

@wallabybob:

If you can get communication with the pfSense LAN interface then you should be able to capture a crash report which could help identify the cause of the crashes.

Since I got an IP I could access the web interface and restore my old configuration.
I also saw a crash report there and copied it, see the attached file.
I did not get any new crashes..which is good news.

The only missing part is the WAN connection..for some reason I am unable to get an IP from my ISP.
Tail of the ppp log:

Mar 17 19:48:49 	ppp: [wan_link0] Link: DOWN event
Mar 17 19:48:49 	ppp: [wan_link0] LCP: Down event
Mar 17 19:48:49 	ppp: [wan_link0] Link: reconnection attempt 26 in 4 seconds
Mar 17 19:48:53 	ppp: [wan_link0] Link: reconnection attempt 26
Mar 17 19:48:53 	ppp: [wan_link0] PPPoE: Connecting to ''
Mar 17 19:49:02 	ppp: [wan_link0] PPPoE connection timeout after 9 seconds
Mar 17 19:49:02 	ppp: [wan_link0] Link: DOWN event
Mar 17 19:49:02 	ppp: [wan_link0] LCP: Down event
Mar 17 19:49:02 	ppp: [wan_link0] Link: reconnection attempt 27 in 4 seconds
Mar 17 19:49:06 	ppp: [wan_link0] Link: reconnection attempt 27
Mar 17 19:49:06 	ppp: [wan_link0] PPPoE: Connecting to ''
Mar 17 19:49:15 	ppp: [wan_link0] PPPoE connection timeout after 9 seconds
Mar 17 19:49:15 	ppp: [wan_link0] Link: DOWN event
Mar 17 19:49:15 	ppp: [wan_link0] LCP: Down event
Mar 17 19:49:15 	ppp: [wan_link0] Link: reconnection attempt 28 in 4 seconds
Mar 17 19:49:19 	ppp: [wan_link0] Link: reconnection attempt 28
Mar 17 19:49:19 	ppp: [wan_link0] PPPoE: Connecting to ''

pfsense_crash_report.txt

wallabybob

The amd64 crash reports seem rather less informative than the i386 crash reports (for example the register contents aren't shown and the crash cause, presumably some sort of machine fault, isn't provided). If you keep having regular crashes I suggest you switch over to i386 builds.

The PPP log says the PPP code isn't seeing a response from the ISP end. I suggest you take a packet capture on the underlying physical interface (re1) to see what traffic is getting to the physical interface. Since you know your LAN interface is working it could be useful to reconfigure so that the current LAN interface becomes the PPPoE interface for WAN and vice versa to see if the fault moves.

rosch

Before getting the current configuration to work (mostly the LAN part), I tried switching the interfaces..and often I did not get a successful LAN connection. It looks like both cards don't want to work together.
Will try capturing the WAN interface traffic and report back.

thanks again.

rosch

Finally some news. I tested 2 configurations, conf1 and conf2 where I switched both interface cables.

conf1

wan: TP-LINK
lan: onboard
lan ok (web interface ok)
packet capture on WAN interface: no packets
switch to conf2, then back to conf1:
lan: no
wan: no
ping fw
PING fw (192.168.2.1) 56(84) bytes of data.
From L530.local (192.168.2.99) icmp_seq=1 Destination Host Unreachable
From L530.local (192.168.2.99) icmp_seq=2 Destination Host Unreachable
^C
--- fw ping statistics ---
21 packets transmitted, 0 received, +18 errors, 100% packet loss, time 20112ms

conf2

wan: onboard
lan: TP-LINK
lan network: very slow (lossy ping, web interface: unable to connect)

PING fw (192.168.2.1) 56(84) bytes of data.
64 bytes from fw (192.168.2.1): icmp_req=15 ttl=64 time=0.174 ms
64 bytes from fw (192.168.2.1): icmp_req=25 ttl=64 time=0.106 ms
64 bytes from fw (192.168.2.1): icmp_req=40 ttl=64 time=0.117 ms
64 bytes from fw (192.168.2.1): icmp_req=45 ttl=64 time=0.107 ms
^C
--- fw ping statistics ---
61 packets transmitted, 4 received, 93% packet loss, time 60447ms
rtt min/avg/max/mdev = 0.106/0.126/0.174/0.028 ms

With conf1 I was able to start a packet capture on the WAN interface. I captured for around 3-5 minutes which should be enough. On a working system getting a WAN IP takes less than 30s. Here no packets at all.

So none of both configurations was successfully getting an IP on both interfaces.
It might be that the driver is not handling well both interfaces.
I don't think this to be a BIOS configuration issue.
Mabe I get lucky with the final release of pfSense 2.1.
Other ideas welcome :)

rosch

I more or less gave up on this one and got an Alix box.
It works like a charm.