Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Jetway JNF9D-2550 + Jetway AD3RTLANG 3 x Gigabit LAN Port Daughter Board

    Hardware
    9
    35
    20.6k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      seedrs
      last edited by

      I have compiled the driver for x64 systems.  You can download it here http://forum.pfsense.org/index.php/topic,58150.msg311249.html.  I have not tested it since my router is currently running i386 build.

      1 Reply Last reply Reply Quote 0
      • I
        IuZ
        last edited by

        Really, thank you very much I'll try.
        Thanks

        1 Reply Last reply Reply Quote 0
        • I
          IuZ
          last edited by

          Update: I think that your x64 drivers works… 'cause from the pfSense interface I can see my other 2 NICs!! Thanks again! :)

          1 Reply Last reply Reply Quote 0
          • S
            seedrs
            last edited by

            @IuZ:

            Update: I think that your x64 drivers works… 'cause from the pfSense interface I can see my other 2 NICs!! Thanks again! :)

            You are welcome, I'm glad it worked.

            1 Reply Last reply Reply Quote 0
            • E
              elgo
              last edited by

              Hi,
              I'm running the exact same hardware setup (well, CPU frequency appart, it's a NF9D-2700+AD3RTLANG), and I'm experiencing very nasty things with both onboard Realtek 8111EVL NICs (let's call them 8111EVIL). When network activity goes up, they completly go mad and error message shows "reX: watchdog timeout" with link going up and down continuously.

              I first check cables and all, and checked that nor BIOS nor hardware was in cause: I can reproduce the problem with 100% success on pfsense beta 2.1 (so with the re driver from FreeBSD 8.3) on i386 and amd64. OpenWRT (linux 3.3.8 kernel) doesn't seem to show the symptom (although the test slightly differed, the pfsense box was recieving traffic, not the desktop):

                +-----pfsense---+
  |		  |
8111EVL		8110SC
  |		  |
 DMZ		 LAN
  |		  |
server	   switch GS108Tv2
		  |	
	       Desktop

              I generate traffic from the server to the desktop box (using nc on both machine). This traffic saturate the gigabit link, and immediatly the DMZ NIC (so the 8111EVIL interface) goes "watchdog timeout".
              If I do generate traffic from desktop to server, it won't trigger it (same commands).

              It's not only a matter of raw throughtput (as for packet per second, I don't know) because my WAN interface which is the other 8111EVIL NIC and is connected to a 100Mbps fiber transciever suffers from this too (and traffic can vary from 20Mbps to 60Mbps in both way, "watchdog" symptom seems really random).

              What I tried:
              Disabling MSI/MSI-X: no impact.
              Disabling all "hardware accelerated" stuff: no impact.

              My question is: what can I try next? What data can I gather if I can help for a freebsd re driver fix?

              1 Reply Last reply Reply Quote 0
              • D
                drewy
                last edited by

                Not terribly constructive I know but I paid the extra and got the intel daughter card. May be a cost effective fix, for 3 decent ports.

                1 Reply Last reply Reply Quote 0
                • E
                  elgo
                  last edited by

                  @drewy:

                  Not terribly constructive I know but I paid the extra and got the intel daughter card. May be a cost effective fix, for 3 decent ports.

                  The 3 realtek ports on daughter-board are fine (8110SC). Used them for years (had them on a previous jetway mobo that "died").
                  The 2 realtek ports on the mother board are causing trouble (8111EVL).

                  1 Reply Last reply Reply Quote 0
                  • W
                    wallabybob
                    last edited by

                    @elgo:

                    I'm experiencing very nasty things with both onboard Realtek 8111EVL NICs (let's call them 8111EVIL). When network activity goes up, they completly go mad and error message shows "reX: watchdog timeout" with link going up and down continuously.

                    I SUSPECT your 8111EVL devices are PCI-Express devices and the devices on the daughter card are PCI devices sharing a single PCI bus. It is likely then that the 8111EVL devices can accept data at a much higher rate than it leave the box out the daughter card devices which are limited by the PCI bus speed. Consequently data going out the daughter card devices (under heavy load) will "bank up" on the transmit queue of the daughter card device and (if the driver is dumb about this) the transmit watchdog timer will go off because it takes "too long" for the transmit queue to empty.

                    Data from the DMZ to the internet could suffer a similar problem, not because of bus speed limitations but because the channel speed to the internet is much slower than the channel speed to the server.

                    Are you interested in doing some experiments to attempt to tune things a bit better? I suggest reassigning your interfaces so the daughter card provides the pfSense WAN interface and the two onboard devices are used for LAN and DMZ. Then try your test again from DMZ to LAN and see if any watchdog timeouts are reported and note the interface reporting them.

                    Please also post the output of pfSense shell commands:```
                    dmesg
                    pciconf -l
                    sysctl -a | grep re
                    /etc/rc.banner

                    The re driver MIGHT provide some tunables that could be adjusted to attempt to stop the messages.

@elgo:

> I generate traffic from the server to the desktop box (using nc on both machine). This traffic saturate the gigabit link,

What sort of traffic is this? I presume some sort of datagram traffic (UDP?) rather than TCP.

@elgo:

> If I do generate traffic from desktop to server, it won't trigger it (same commands).

In that direction incoming data rate is limited by the PCI bus and will be lower than the rate at which traffic can leave the system over the PCI Express bus so is unlikely to back up on the transmit queue.
                    1 Reply Last reply Reply Quote 0
                    • ?
                      Guest
                      last edited by

                      I don't know if this will help you.
                      I have run the iperf on both the onboard and daughter network ports, as a server, I was maxing out at about 500 Mbps.
                      The CPU was almost at 100%, I assume that is where the slowdown was.

                      Have you tried an internal iperf test from a PC on a gigabit network?

                      1 Reply Last reply Reply Quote 0
                      • E
                        elgo
                        last edited by

                        Thank you for your interest, gentlemen.
                        @wallabybob: you are totally right about PCIe/8111EVL and PCI/8110SC.
                        As for re driver tunable, I tried MSI/MSIX and intr_filter but not hw.re.prefer_iomap which I don't understand. No luck. Though I see a dev.re.%d.int_rx_mod I didn't test… well, can be worth the try.
                        As for the shell commands, I'll post their output soon, as for now the pfsense box has been temporary taken out of "the production network" :).

                        Yesterday I simplified my test case (which may be what you thought about, JoeMcJoe):

                          +-----pfsense---+
  |		  
8111EVL		
  |		  
 DMZ		 
  |		 
server

                        I generate traffic from the server (TCP traffic or UDP with -u option, it doesn't affect results: cat /dev/zero | nc -vv -u pfsense_IP pfsense_port) and recieve it on pfsense box (nc -l other_options_i_dont_remember > /dev/null).
                        "reX: watchdog timeout" visible on dmesg output on pfsense box and on its VGA console.
                        Same test with pfsense generating traffic and server recieving it won't trigger this error message.
                        During this test, no CPU core on pfsense box was used at 100% (maybe a core at 70% max by NIC related "intr" thread. Remember it's a 4 cores as it's a hyperthreaded dual atom).

                        So the PCIe/PCI rate problem was a smart guess, but It's not involved in this test case.

                        1 Reply Last reply Reply Quote 0
                        • E
                          elgo
                          last edited by

                          @wallabybob:

                          Please also post the output of pfSense shell commands:```
                          dmesg
                          pciconf -l
                          sysctl -a | grep re
                          /etc/rc.banner

                          I filtered the output of sysctl to what seems really related to re.

                          banner.txt
                          dmesg.txt
                          pciconf.txt
                          sysctl_re.txt

                          1 Reply Last reply Reply Quote 0
                          • W
                            wallabybob
                            last edited by

                            Thanks for the additional information.

                            When you run the "nc" test which pfsense interface is the server connected to? DMZ (re1)?

                            Which interfaces are reported in the "rex: watchdog timeout" messages? (The driver includes the comment Tx completion interrupt which seems to be lost on PCIe based controllers under certain situations. but the watchdog function also looks for received frames.)

                            Interface re0 runs a bunch of VLANs including (apparently) a VLAN for the PPPoE WAN interface. Correct?

                            Which interfaces are members of the bridge interface?

                            Are you using jumbo frames?

                            1 Reply Last reply Reply Quote 0
                            • E
                              elgo
                              last edited by

                              @wallabybob:

                              Thanks for the additional information.

                              Thank you taking time helping me :)

                              @wallabybob:

                              When you run the "nc" test which pfsense interface is the server connected to? DMZ (re1)?

                              Which interfaces are reported in the "rex: watchdog timeout" messages? (The driver includes the comment Tx completion interrupt which seems to be lost on PCIe based controllers under certain situations. but the watchdog function also looks for received frames.)

                              Yes, re1 is the DMZ interface with the server at the end, and it's the one reporting watchdog timeout in my test case. During normal operations, when the "WAN" happened to complain, it's re0 (which is the physical interface it is based on finally) that's reported.

                              @wallabybob:

                              Interface re0 runs a bunch of VLANs including (apparently) a VLAN for the PPPoE WAN interface. Correct?

                              Yes, "The Internet, by Orange". The pfense WAN interface is PPPoE over a VLAN over a physical link (re0). And it's supposed to be an optical fiber access commercial offer… innovative technologies and all (ppp).

                              @wallabybob:

                              Which interfaces are members of the bridge interface?

                              Erk, I'll detail this, it's not trivial, but I'm not sure it will be really useful, as the bridge is not involved in the last simplified test case.

                              BRIDGE_TV 
	|- ORANGE_VLAN_838 --> re0
	|- ORANGE_VLAN_839 --> re0
	|- ORANGE_VLAN_840 --> re0
	|- ORANGE_VLAN_841 --> re0
	|_ TRUNK_TV (vlan 1000) --> re4

                              Actually, there is barelly any traffic there (IGMP/multicast announces sometimes). No IP address configured on any of these brigded interfaces nor on the bridge itself.

                              @wallabybob:

                              Are you using jumbo frames?

                              Not on re0 or re1 (WAN/DMZ) which are the 2 EVIL NICs. re2 (LAN) and re3 (DMZ2, unused for now) have jumbo frames enabled (7k, due to their chip design).

                              To sum it up:

                              • re0 - 8111EVL -> …. -> WAN

                              • re1 - 8111EVL -> DMZ

                              • re2 - 8110SC -> LAN

                              • re3 - 8110SC -> DMZ2 (unused)

                              • re4 - 8110SC -> … -> bridge

                              1 Reply Last reply Reply Quote 0
                              • N
                                n1ko
                                last edited by

                                I'm seeing 100% similar issues with my Realtek card. Loading the server from the client (aka running iperf -s on the pfsense machine and iperf -c on a desktop) results in link flapping. If I test it the other way around (iperf -s on a desktop, iperf -c on the pfsense) it works fine.

                                I'm running a single physical card (integrated) with two vlans. Connected to a HP 1810G v2 switch over a 1Gbps link. Cables tested, no jumbos (although i have tried with jumbos too). All offloading features etc have been turned on and off for testing, no help.

                                One difference though, i'm not seeing any watchdog timeouts anywhere. Only link flapping.

                                dmesg:
                                re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" pcie="" gigabit="" ethernet="">port 0xe800-0xe8ff mem 0xfdffb000-0xfdffbfff,0xfdffc000-0xfdffffff irq 19 at device 0.0 on pci4
                                re0: Chip rev. 0x2c800000
                                re0: MAC rev. 0x00000000
                                pciconfig:
                                re0@pci0:4:0:0: class=0x020000 card=0x31251019 chip=0x816810ec rev=0x06 hdr=0x00
                                uname:
                                FreeBSD pfsense.koti 8.3-RELEASE-p6 FreeBSD 8.3-RELEASE-p6 #0: Tue Mar 19 06:22:08 EDT 2013    root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8  amd64</realtek>

                                1 Reply Last reply Reply Quote 0
                                • E
                                  elgo
                                  last edited by

                                  Yeah, it really seems as a driver issu (don't get me wrong, I know what OSS is :)) or "broken hardware by design" issue (which can be verified by trying another OS. I'll give a shot to linux based OpenWRT AA in a couple of weeks on this hardware, and test it thoroughly).

                                  Additionnally I took a look at the differences between re driver in freebsd 8.3 (actual pfsense 2.1 beta) and 9.1, but I don't see any hope to be expected there (hardly any difference "features side").

                                  So either it's a matter that is to be reported upstream (how? what data are to be provided?) or it's… hopeless. I don't even know the technical differences between 8111E and 8111E-VL chips (they seem different as they weren't supported by re driver at the same time according to driver's changelog).
                                  Would it be useful to create a new dedicated topic to issues with realtek 8111E & other variants?

                                  --
                                  edit:
                                  freebsd 8.3 based freeNAS users have the same issue (people that don't have simple cable issues, of course).

                                  1 Reply Last reply Reply Quote 0
                                  • W
                                    wallabybob
                                    last edited by

                                    @elgo:

                                    So either it's a matter that is to be reported upstream (how? what data are to be provided?)

                                    FreeBSD problem reports can be lodged at http://www.freebsd.org/send-pr.html

                                    Reports so far suggest the problem might be related to heavy receive traffic. PERHAPS the device experiences receive overflow and the driver doesn't really know how to recover.

                                    I haven't found any way of setting "flow control" in FreeBSD. (Receiver sends XOFF when it wants transmitter to stop, sends XON when transmitter can resume.) Maybe it needs to be set on these devices because they have minimal buffering and the driver doesn't set it. Maybe it can't be set. Maybe it is set and the other end is ignoring it.

                                    It could be interesting to see what numbers come out of the following test:
                                    While iperf (or similar) test is running and before it terminates, give shell commandsvmstat -i ; sleep 10 ; vmstat -iand```
                                    netstat -i ; sleep 10; netstat -i

                                    
@elgo:

> or it's… hopeless. I don't even know the technical differences between 8111E and 8111E-VL chips

It might be a simple as a different PCI ID. It might be a complete redesign.
                                    1 Reply Last reply Reply Quote 0
                                    • N
                                      n2qcn
                                      last edited by

                                      the pciconf output confirms wallabybob's theory that the two internal (re0, re1) are each on their own pci express lane and the three others (re2, re3, re4) share one pci express lane (to a shared PCI rev.2.3, 32-bit, 33/66MHz) and hence have much lower potential throughput. funny how the slower interfaces aren't an issue :-)

                                      differences between 8111E and 8111EVL chips

                                      the 8111EVL is older than the 8111C, and uses pci express 1.0a rather than the  1.1a interface used by RTL8111C, RTL8111E and Intel 82574L. Not that it should make any difference, just pointing out its a very old design.

                                      1 Reply Last reply Reply Quote 0
                                      • E
                                        elgo
                                        last edited by

                                        @wallabybob:

                                        Reports so far suggest the problem might be related to heavy receive traffic. PERHAPS the device experiences receive overflow and the driver doesn't really know how to recover.

                                        I haven't found any way of setting "flow control" in FreeBSD. (Receiver sends XOFF when it wants transmitter to stop, sends XON when transmitter can resume.) Maybe it needs to be set on these devices because they have minimal buffering and the driver doesn't set it. Maybe it can't be set. Maybe it is set and the other end is ignoring it.

                                        @wallabybob:

                                        It could be interesting to see what numbers come out of the following test:
                                        While iperf (or similar) test is running and before it terminates, give shell commandsvmstat -i ; sleep 10 ; vmstat -iand```
                                        netstat -i ; sleep 10; netstat -i

                                        Following this idea, I changed the test case slightly: indeed, I noticed that when the pfsense box and the server where directly connected (without any switch), the server reported "Flow control is off for TX and off for RX" (broadcom BCM5723 with tigon3 linux driver). That's when the issue was easy to trigger.

                                        pfsense
  |		  
8111EVL	
  |	
  |		  
BCM5723	 
  |		 
server

                                        I now have a "switch" (some 450G cheap crap from mikrotik, but working (slowly but) perfectly now it runs openwrt) inserted between the pfsense box and the server. Server now proudly report "Flow control is on for TX and on for RX". Is there a way to see that flow control state from pfsense, I cant' see it from the switch device right now. I'd like to confirm if it's Realtek 8111-EVL that doesn't support flow control or if it's some sort of failed negiciation/capabilities advertisment.

                                        pfsense
  |		  
8111EVL	
  |
  |
switch	
  |
  |		  
BCM5723	 
  |		 
server

                                        And guess what? I can't reproduce the issue with this setup. I can get over 13k-18k interrupts/s (mesured with vmstat -w 5) with UDP traffic induced by the same nc commands than before. Got 80+ MB/s with UDP in both direction without a hick.

                                        So…  Is there some flow control guru around? ;)

                                        1 Reply Last reply Reply Quote 0
                                        • E
                                          elgo
                                          last edited by

                                          Found this: Flow control in FreeBSD

                                          Running the commande when I can't reproduce the issue (that is, with the swith):

                                          ifconfig -m re1
re1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
	options=2098 <vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>capabilities=39db <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_ucast,wol_mcast,wol_magic>ether 00:30:18:a3:2b:b3
	inet 192.168.10.200 netmask 0xffffff00 broadcast 192.168.10.255
	inet6 fe80::230:18ff:fea3:2bb3%re1 prefixlen 64 scopeid 0x2
	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex,master>)
	status: active
	supported media:
		media autoselect mediaopt flowcontrol
		media autoselect
		media 1000baseT mediaopt full-duplex,flowcontrol,master
		media 1000baseT mediaopt full-duplex,flowcontrol
		media 1000baseT mediaopt full-duplex,master
		media 1000baseT mediaopt full-duplex
		media 1000baseT mediaopt master
		media 1000baseT
		media 100baseTX mediaopt full-duplex,flowcontrol
		media 100baseTX mediaopt full-duplex
		media 100baseTX
		media 10baseT/UTP mediaopt full-duplex,flowcontrol
		media 10baseT/UTP mediaopt full-duplex
		media 10baseT/UTP
		media none</full-duplex,master></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_ucast,wol_mcast,wol_magic></vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast>

                                          Errr, so no flow control?

                                          –
                                          edit:
                                          indeed, no flow control.

                                          ifconfig re1 media autoselect mediaopt flowcontrol
ifconfig -m re1
re1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
[...]
	media: Ethernet autoselect <flowcontrol> (1000baseT <full-duplex,flowcontrol,rxpause,txpause>)</full-duplex,flowcontrol,rxpause,txpause></flowcontrol></up,broadcast,running,simplex,multicast>

                                          Now I have flow control.
                                          I'll run some more tests without the switch in a couple of days, playing with this flow control feature.

                                          1 Reply Last reply Reply Quote 0
                                          • W
                                            wallabybob
                                            last edited by

                                            Good work discovering the flow control mediaopt. It is not documented in either the ifconfig man page or the re man page.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.