Jetway JNF9D-2550 + Jetway AD3RTLANG 3 x Gigabit LAN Port Daughter Board



  • I built a system that includes a Jetway JNF9D-2550 which has 2 x Realtek RTL8111EVL PCI-E Gigabit Ethernet ports.
    Plus the Jetway AD3RTLANG 3 x Gigabit LAN Port Daughter Board.

    The system only sees the ports that are on the daughter board.
    pfsense names the ports: re2, re3, re4, which seems to indicate that it knows about the other 2 ports which I assume would be named re0, re1.
    When I don't have the daughter board installed, the system complains that there are no interfaces.

    When I plug in an active Ethernet cable to the 2 onboard ports, the LEDs so flash for a while and indicate a link on a switch.

    Currently version 2.0.2 i386 is installed.

    Looking in this forum, other boards that use the RTL8111EVL chipset seem to work with build 2.1.

    edit: I just did the update and it does show all of the 5 re0 to re04 Ethernet ports available, plus 5 USB ports for some reason.

    If I install the latest 2.1 snapshot and I find issues with other parts of the install, is it easy to step back to 2.0.2, using the firmware update option?



  • @JoeMcJoe:

    edit: I just did the update and it does show all of the 5 re0 to re04 Ethernet ports available, plus 5 USB ports for some reason.

    Update to a pfSense 2.1 snapshot build?

    @JoeMcJoe:

    If I install the latest 2.1 snapshot and I find issues with other parts of the install, is it easy to step back to 2.0.2, using the firmware update option?

    You might lose access to ports re0 and re1 if you go back. Does that matter?

    I suspect the re driver in pfSense 2.0.2 (based on FreeBSD 8.1) doesn't get the response it expects when the driver is attempting to initialise them but the driver in pfSense 2.1 (based on FreeBSD 8.3) does. It might be informative to compare the startup output of the two versions. The pfSense shell command dmesg will display the startup output of the most recently loaded operating system.



  • yes, I updated to this snapshot build.
    2.1-BETA1 (i386) - built on Fri Dec 21 04:09:13 EST 2012 - FreeBSD router.tetch.com 8.3-RELEASE-p5 FreeBSD 8.3-RELEASE-p5 #1: Fri Dec 21 04:35:58 EST 2012 root@snapshots-8_3-i386.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8 i386

    I only need 3 ports at the moment, but its good to know that the other main two re0/re1 are recognised by the new version of the OS.
    I didn't keep a copy of the older dmesg output.

    thanks very much for your help



  • Is it ok to run the 64 bit version of Pfsense on this system?
    Intel shows the cpu being used as 64 bit capable.

    http://www.jetwaycomputer.com/NF9D.html
    http://ark.intel.com/products/65470/Intel-Atom-Processor-D2550-(1M-Cache-1_86-GHz)

    When installing the software, the pfsense menu graphics were all messed up, only half the characters were readable sometimes, made it tricky.


  • Netgate Administrator

    Hmm I thought a fix had gone in for that.
    http://redmine.pfsense.org/issues/2595

    Steve



  • I did the original install with 2.0.1, looks like that was fixed in the 2.1 beta release,  from reading that report.



  • Hello,
    me too I have the same problem on pfSense 2.0.2.

    I saw the link http://redmine.pfsense.org/issues/2595 and the other http://freshbsd.org/commit/freebsd/r237203

    My question is:
    Can it work if I change the file head/sys/dev/fb/fbreg.h, as in the link above, on my pfSense 2.0.2?
    Thanks, bye.



  • See http://forum.pfsense.org/index.php/topic,58150.0.html to fix the issue with the onboard NICs not showing up in 2.0.2.  The solution is for i386 only



  • I saw that link (thank you the same), but I have x64 version. :(



  • I have compiled the driver for x64 systems.  You can download it here http://forum.pfsense.org/index.php/topic,58150.msg311249.html.  I have not tested it since my router is currently running i386 build.



  • Really, thank you very much I'll try.
    Thanks



  • Update: I think that your x64 drivers works… 'cause from the pfSense interface I can see my other 2 NICs!! Thanks again! :)



  • @IuZ:

    Update: I think that your x64 drivers works… 'cause from the pfSense interface I can see my other 2 NICs!! Thanks again! :)

    You are welcome, I'm glad it worked.



  • Hi,
    I'm running the exact same hardware setup (well, CPU frequency appart, it's a NF9D-2700+AD3RTLANG), and I'm experiencing very nasty things with both onboard Realtek 8111EVL NICs (let's call them 8111EVIL). When network activity goes up, they completly go mad and error message shows "reX: watchdog timeout" with link going up and down continuously.

    I first check cables and all, and checked that nor BIOS nor hardware was in cause: I can reproduce the problem with 100% success on pfsense beta 2.1 (so with the re driver from FreeBSD 8.3) on i386 and amd64. OpenWRT (linux 3.3.8 kernel) doesn't seem to show the symptom (although the test slightly differed, the pfsense box was recieving traffic, not the desktop):

      +-----pfsense---+
      |		  |
    8111EVL		8110SC
      |		  |
     DMZ		 LAN
      |		  |
    server	   switch GS108Tv2
    		  |	
    	       Desktop
    

    I generate traffic from the server to the desktop box (using nc on both machine). This traffic saturate the gigabit link, and immediatly the DMZ NIC (so the 8111EVIL interface) goes "watchdog timeout".
    If I do generate traffic from desktop to server, it won't trigger it (same commands).

    It's not only a matter of raw throughtput (as for packet per second, I don't know) because my WAN interface which is the other 8111EVIL NIC and is connected to a 100Mbps fiber transciever suffers from this too (and traffic can vary from 20Mbps to 60Mbps in both way, "watchdog" symptom seems really random).

    What I tried:
    Disabling MSI/MSI-X: no impact.
    Disabling all "hardware accelerated" stuff: no impact.

    My question is: what can I try next? What data can I gather if I can help for a freebsd re driver fix?



  • Not terribly constructive I know but I paid the extra and got the intel daughter card. May be a cost effective fix, for 3 decent ports.



  • @drewy:

    Not terribly constructive I know but I paid the extra and got the intel daughter card. May be a cost effective fix, for 3 decent ports.

    The 3 realtek ports on daughter-board are fine (8110SC). Used them for years (had them on a previous jetway mobo that "died").
    The 2 realtek ports on the mother board are causing trouble (8111EVL).



  • @elgo:

    I'm experiencing very nasty things with both onboard Realtek 8111EVL NICs (let's call them 8111EVIL). When network activity goes up, they completly go mad and error message shows "reX: watchdog timeout" with link going up and down continuously.

    I SUSPECT your 8111EVL devices are PCI-Express devices and the devices on the daughter card are PCI devices sharing a single PCI bus. It is likely then that the 8111EVL devices can accept data at a much higher rate than it leave the box out the daughter card devices which are limited by the PCI bus speed. Consequently data going out the daughter card devices (under heavy load) will "bank up" on the transmit queue of the daughter card device and (if the driver is dumb about this) the transmit watchdog timer will go off because it takes "too long" for the transmit queue to empty.

    Data from the DMZ to the internet could suffer a similar problem, not because of bus speed limitations but because the channel speed to the internet is much slower than the channel speed to the server.

    Are you interested in doing some experiments to attempt to tune things a bit better? I suggest reassigning your interfaces so the daughter card provides the pfSense WAN interface and the two onboard devices are used for LAN and DMZ. Then try your test again from DMZ to LAN and see if any watchdog timeouts are reported and note the interface reporting them.

    Please also post the output of pfSense shell commands:```
    dmesg
    pciconf -l
    sysctl -a | grep re
    /etc/rc.banner

    The re driver MIGHT provide some tunables that could be adjusted to attempt to stop the messages.
    
    @elgo:
    
    > I generate traffic from the server to the desktop box (using nc on both machine). This traffic saturate the gigabit link,
    
    What sort of traffic is this? I presume some sort of datagram traffic (UDP?) rather than TCP.
    
    @elgo:
    
    > If I do generate traffic from desktop to server, it won't trigger it (same commands).
    
    In that direction incoming data rate is limited by the PCI bus and will be lower than the rate at which traffic can leave the system over the PCI Express bus so is unlikely to back up on the transmit queue.


  • I don't know if this will help you.
    I have run the iperf on both the onboard and daughter network ports, as a server, I was maxing out at about 500 Mbps.
    The CPU was almost at 100%, I assume that is where the slowdown was.

    Have you tried an internal iperf test from a PC on a gigabit network?



  • Thank you for your interest, gentlemen.
    @wallabybob: you are totally right about PCIe/8111EVL and PCI/8110SC.
    As for re driver tunable, I tried MSI/MSIX and intr_filter but not hw.re.prefer_iomap which I don't understand. No luck. Though I see a dev.re.%d.int_rx_mod I didn't test… well, can be worth the try.
    As for the shell commands, I'll post their output soon, as for now the pfsense box has been temporary taken out of "the production network" :).

    Yesterday I simplified my test case (which may be what you thought about, JoeMcJoe):

      +-----pfsense---+
      |		  
    8111EVL		
      |		  
     DMZ		 
      |		 
    server
    

    I generate traffic from the server (TCP traffic or UDP with -u option, it doesn't affect results: cat /dev/zero | nc -vv -u pfsense_IP pfsense_port) and recieve it on pfsense box (nc -l other_options_i_dont_remember > /dev/null).
    "reX: watchdog timeout" visible on dmesg output on pfsense box and on its VGA console.
    Same test with pfsense generating traffic and server recieving it won't trigger this error message.
    During this test, no CPU core on pfsense box was used at 100% (maybe a core at 70% max by NIC related "intr" thread. Remember it's a 4 cores as it's a hyperthreaded dual atom).

    So the PCIe/PCI rate problem was a smart guess, but It's not involved in this test case.



  • @wallabybob:

    Please also post the output of pfSense shell commands:```
    dmesg
    pciconf -l
    sysctl -a | grep re
    /etc/rc.banner

    I filtered the output of sysctl to what seems really related to re.

    banner.txt
    dmesg.txt
    pciconf.txt
    sysctl_re.txt



  • Thanks for the additional information.

    When you run the "nc" test which pfsense interface is the server connected to? DMZ (re1)?

    Which interfaces are reported in the "rex: watchdog timeout" messages? (The driver includes the comment Tx completion interrupt which seems to be lost on PCIe based controllers under certain situations. but the watchdog function also looks for received frames.)

    Interface re0 runs a bunch of VLANs including (apparently) a VLAN for the PPPoE WAN interface. Correct?

    Which interfaces are members of the bridge interface?

    Are you using jumbo frames?



  • @wallabybob:

    Thanks for the additional information.

    Thank you taking time helping me :)

    @wallabybob:

    When you run the "nc" test which pfsense interface is the server connected to? DMZ (re1)?

    Which interfaces are reported in the "rex: watchdog timeout" messages? (The driver includes the comment Tx completion interrupt which seems to be lost on PCIe based controllers under certain situations. but the watchdog function also looks for received frames.)

    Yes, re1 is the DMZ interface with the server at the end, and it's the one reporting watchdog timeout in my test case. During normal operations, when the "WAN" happened to complain, it's re0 (which is the physical interface it is based on finally) that's reported.

    @wallabybob:

    Interface re0 runs a bunch of VLANs including (apparently) a VLAN for the PPPoE WAN interface. Correct?

    Yes, "The Internet, by Orange". The pfense WAN interface is PPPoE over a VLAN over a physical link (re0). And it's supposed to be an optical fiber access commercial offer… innovative technologies and all (ppp).

    @wallabybob:

    Which interfaces are members of the bridge interface?

    Erk, I'll detail this, it's not trivial, but I'm not sure it will be really useful, as the bridge is not involved in the last simplified test case.

    BRIDGE_TV 
    	|- ORANGE_VLAN_838 --> re0
    	|- ORANGE_VLAN_839 --> re0
    	|- ORANGE_VLAN_840 --> re0
    	|- ORANGE_VLAN_841 --> re0
    	|_ TRUNK_TV (vlan 1000) --> re4
    

    Actually, there is barelly any traffic there (IGMP/multicast announces sometimes). No IP address configured on any of these brigded interfaces nor on the bridge itself.

    @wallabybob:

    Are you using jumbo frames?

    Not on re0 or re1 (WAN/DMZ) which are the 2 EVIL NICs. re2 (LAN) and re3 (DMZ2, unused for now) have jumbo frames enabled (7k, due to their chip design).

    To sum it up:

    • re0 - 8111EVL -> …. -> WAN

    • re1 - 8111EVL -> DMZ

    • re2 - 8110SC -> LAN

    • re3 - 8110SC -> DMZ2 (unused)

    • re4 - 8110SC -> … -> bridge



  • I'm seeing 100% similar issues with my Realtek card. Loading the server from the client (aka running iperf -s on the pfsense machine and iperf -c on a desktop) results in link flapping. If I test it the other way around (iperf -s on a desktop, iperf -c on the pfsense) it works fine.

    I'm running a single physical card (integrated) with two vlans. Connected to a HP 1810G v2 switch over a 1Gbps link. Cables tested, no jumbos (although i have tried with jumbos too). All offloading features etc have been turned on and off for testing, no help.

    One difference though, i'm not seeing any watchdog timeouts anywhere. Only link flapping.

    dmesg:
    re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" pcie="" gigabit="" ethernet="">port 0xe800-0xe8ff mem 0xfdffb000-0xfdffbfff,0xfdffc000-0xfdffffff irq 19 at device 0.0 on pci4
    re0: Chip rev. 0x2c800000
    re0: MAC rev. 0x00000000
    pciconfig:
    re0@pci0:4:0:0: class=0x020000 card=0x31251019 chip=0x816810ec rev=0x06 hdr=0x00
    uname:
    FreeBSD pfsense.koti 8.3-RELEASE-p6 FreeBSD 8.3-RELEASE-p6 #0: Tue Mar 19 06:22:08 EDT 2013    root@snapshots-8_3-amd64.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8  amd64</realtek>



  • Yeah, it really seems as a driver issu (don't get me wrong, I know what OSS is :)) or "broken hardware by design" issue (which can be verified by trying another OS. I'll give a shot to linux based OpenWRT AA in a couple of weeks on this hardware, and test it thoroughly).

    Additionnally I took a look at the differences between re driver in freebsd 8.3 (actual pfsense 2.1 beta) and 9.1, but I don't see any hope to be expected there (hardly any difference "features side").

    So either it's a matter that is to be reported upstream (how? what data are to be provided?) or it's… hopeless. I don't even know the technical differences between 8111E and 8111E-VL chips (they seem different as they weren't supported by re driver at the same time according to driver's changelog).
    Would it be useful to create a new dedicated topic to issues with realtek 8111E & other variants?

    --
    edit:
    freebsd 8.3 based freeNAS users have the same issue (people that don't have simple cable issues, of course).



  • @elgo:

    So either it's a matter that is to be reported upstream (how? what data are to be provided?)

    FreeBSD problem reports can be lodged at http://www.freebsd.org/send-pr.html

    Reports so far suggest the problem might be related to heavy receive traffic. PERHAPS the device experiences receive overflow and the driver doesn't really know how to recover.

    I haven't found any way of setting "flow control" in FreeBSD. (Receiver sends XOFF when it wants transmitter to stop, sends XON when transmitter can resume.) Maybe it needs to be set on these devices because they have minimal buffering and the driver doesn't set it. Maybe it can't be set. Maybe it is set and the other end is ignoring it.

    It could be interesting to see what numbers come out of the following test:
    While iperf (or similar) test is running and before it terminates, give shell commandsvmstat -i ; sleep 10 ; vmstat -iand```
    netstat -i ; sleep 10; netstat -i

    
    @elgo:
    
    > or it's… hopeless. I don't even know the technical differences between 8111E and 8111E-VL chips
    
    It might be a simple as a different PCI ID. It might be a complete redesign.


  • the pciconf output confirms wallabybob's theory that the two internal (re0, re1) are each on their own pci express lane and the three others (re2, re3, re4) share one pci express lane (to a shared PCI rev.2.3, 32-bit, 33/66MHz) and hence have much lower potential throughput. funny how the slower interfaces aren't an issue :-)

    differences between 8111E and 8111EVL chips

    the 8111EVL is older than the 8111C, and uses pci express 1.0a rather than the  1.1a interface used by RTL8111C, RTL8111E and Intel 82574L. Not that it should make any difference, just pointing out its a very old design.



  • @wallabybob:

    Reports so far suggest the problem might be related to heavy receive traffic. PERHAPS the device experiences receive overflow and the driver doesn't really know how to recover.

    I haven't found any way of setting "flow control" in FreeBSD. (Receiver sends XOFF when it wants transmitter to stop, sends XON when transmitter can resume.) Maybe it needs to be set on these devices because they have minimal buffering and the driver doesn't set it. Maybe it can't be set. Maybe it is set and the other end is ignoring it.

    @wallabybob:

    It could be interesting to see what numbers come out of the following test:
    While iperf (or similar) test is running and before it terminates, give shell commandsvmstat -i ; sleep 10 ; vmstat -iand```
    netstat -i ; sleep 10; netstat -i

    Following this idea, I changed the test case slightly: indeed, I noticed that when the pfsense box and the server where directly connected (without any switch), the server reported "Flow control is off for TX and off for RX" (broadcom BCM5723 with tigon3 linux driver). That's when the issue was easy to trigger.

    pfsense
      |		  
    8111EVL	
      |	
      |		  
    BCM5723	 
      |		 
    server
    

    I now have a "switch" (some 450G cheap crap from mikrotik, but working (slowly but) perfectly now it runs openwrt) inserted between the pfsense box and the server. Server now proudly report "Flow control is on for TX and on for RX". Is there a way to see that flow control state from pfsense, I cant' see it from the switch device right now. I'd like to confirm if it's Realtek 8111-EVL that doesn't support flow control or if it's some sort of failed negiciation/capabilities advertisment.

    pfsense
      |		  
    8111EVL	
      |
      |
    switch	
      |
      |		  
    BCM5723	 
      |		 
    server
    

    And guess what? I can't reproduce the issue with this setup. I can get over 13k-18k interrupts/s (mesured with vmstat -w 5) with UDP traffic induced by the same nc commands than before. Got 80+ MB/s with UDP in both direction without a hick.

    So…  Is there some flow control guru around? ;)



  • Found this: Flow control in FreeBSD

    Running the commande when I can't reproduce the issue (that is, with the swith):

    ifconfig -m re1
    re1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    	options=2098 <vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>capabilities=39db <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_ucast,wol_mcast,wol_magic>ether 00:30:18:a3:2b:b3
    	inet 192.168.10.200 netmask 0xffffff00 broadcast 192.168.10.255
    	inet6 fe80::230:18ff:fea3:2bb3%re1 prefixlen 64 scopeid 0x2
    	nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex,master>)
    	status: active
    	supported media:
    		media autoselect mediaopt flowcontrol
    		media autoselect
    		media 1000baseT mediaopt full-duplex,flowcontrol,master
    		media 1000baseT mediaopt full-duplex,flowcontrol
    		media 1000baseT mediaopt full-duplex,master
    		media 1000baseT mediaopt full-duplex
    		media 1000baseT mediaopt master
    		media 1000baseT
    		media 100baseTX mediaopt full-duplex,flowcontrol
    		media 100baseTX mediaopt full-duplex
    		media 100baseTX
    		media 10baseT/UTP mediaopt full-duplex,flowcontrol
    		media 10baseT/UTP mediaopt full-duplex
    		media 10baseT/UTP
    		media none</full-duplex,master></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,polling,vlan_hwcsum,tso4,wol_ucast,wol_mcast,wol_magic></vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast>
    

    Errr, so no flow control?


    edit:
    indeed, no flow control.

    ifconfig re1 media autoselect mediaopt flowcontrol
    ifconfig -m re1
    re1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
    [...]
    	media: Ethernet autoselect <flowcontrol> (1000baseT <full-duplex,flowcontrol,rxpause,txpause>)</full-duplex,flowcontrol,rxpause,txpause></flowcontrol></up,broadcast,running,simplex,multicast>
    

    Now I have flow control.
    I'll run some more tests without the switch in a couple of days, playing with this flow control feature.



  • Good work discovering the flow control mediaopt. It is not documented in either the ifconfig man page or the re man page.



  • If you'd like me to run any of your tests let me know.
    All 5 ports on my system are the realtec ones as described in the first post.

    I played with the flow control setting, the only different I saw it make was that this was posted in the system log when flow control was enabled by using your command line.
    'kernel: interrupt storm detected on "irq256:"; throttling interrupt source'

    This is when running the iperf server on the motherboard's interface. It seems to max out at about 300 Mbps.

    What is the advantage to enable flowcontrol in this case? I had always assumed that it was always used.



  • @JoeMcJoe:

    If you'd like me to run any of your tests let me know.
    All 5 ports on my system are the realtec ones as described in the first post.

    Sure, if you can do the very same test with nc which allow to involve only one box and the pfsense one on the 2 EVIL (onboard) NICs, that's really valuable :)
    Being sure that my issue is not due to a failing motherboard sample would priceless.

    To describe the method I use:
    On the box generating traffic, from example under linux:

    cat /dev/zero | nc <ip_pfsense_box> <a_reachable_port></a_reachable_port></ip_pfsense_box>
    

    On the pfsense box to recieve traffic:

    nc -l <same_ip_pfsense_box_previous_command> <same_port_previous_command> > /dev/null</same_port_previous_command></same_ip_pfsense_box_previous_command>
    

    To generate more throughtput, add "-u" to both nc commands to use UDP mode instead of TCP mode. Of course, you can do the opposite and generate traffic from the pfsense box with similar commands (or do both if you are in a nasty mood).
    To see current interrupt rate (int/s average for 5 sec), look at "int" row from the output of the following command:

    vmstat -w 5
    

    The fact that both box are directly connected (no switch) or not, seems to matter too. So if you can, please specify what is your exact hardware test setup.

    @JoeMcJoe:

    I played with the flow control setting, the only different I saw it make was that this was posted in the system log when flow control was enabled by using your command line.
    'kernel: interrupt storm detected on "irq256:"; throttling interrupt source'

    This is when running the iperf server on the motherboard's interface. It seems to max out at about 300 Mbps.

    What is the advantage to enable flowcontrol in this case? I had always assumed that it was always used.

    That new message is great too. That means that playing with the last driver tunable I didn't test is worthy, in your case: according to the re documentation:
    @re:

    dev.re.%d.int_rx_mod
        Maximum amount of time to delay receive interrupt processing in
        units of 1us.  The accepted range is 0 to 65, the default is
        65(65us). Value 0 completely disables the interrupt moderation.
        The interface need to be brought down and up again before a
        change takes effect.

    Of course, I would try to set it to 0 to see if it breaks something else that could be meaningfull :)

    I always assumed flox control was negociated and likely to be on by default too, but proof is in my case that it may not :) I'll work on that probably monday.



  • Will do this weekend, I have to setup a client linux system, unless there is a command in windows I can use. Will let you know.



  • I can turn on and off flowcontrol on my switch and don't see any media changes with seedrs's amd64 re driver for 2.0.2.

    Using the```

    one % nc -l 5555 > /dev/null &
    one % ssh two
    two % cat /dev/zero | nc one 5555

    test, my pfsense box can generate traffic and still have enough cpu remaining to let pings pass through it.```
    
    procs      memory      page                    disks     faults         cpu
     r b w     avm    fre   flt  re  pi  po    fr  sr ad4 da0   in   sy   cs us sy id
     0 0 0   2187M  5806M   485   0   0   1   489   0   0   0  212   16  880 15  6 79
     0 0 0   2187M  5806M     1   0   0   1     1   0   3   0   75  311 2489  0  0 100
     0 0 0   2187M  5806M   310   0   0   0   319   0   3   0   75  844 2426  0  0 100
     0 0 0   2187M  5806M     2   0   0   0     1   0   0   0   67  255 2382  0  0 100
     0 0 0   2187M  5806M     0   0   0   0     0   0   0   0   69  297 2387  0  0 100
     1 0 0   2201M  5805M    40   0   0   0     9   0   0   0 20248 334214 81340  6 34 60
     1 0 0   2201M  5805M  2347   0   0   0  2405   0   3   0 22065 364382 88527  6 38 56
     1 0 0   2201M  5805M     3   0   0   0     1   0   0   0 22521 363193 88808  5 37 58
     1 0 0   2201M  5805M     1   0   0   3     1   0  11   0 21971 362757 87897  6 38 56
     1 0 0   2201M  5805M     1   0   0   0     1   0   0   0 22463 362896 88939  6 37 57
     0 0 0   2201M  5805M     5   0   0   0     0   0   0   0 22081 362817 88033  6 38 56
     0 0 0   2201M  5805M     5   0   0   0     1   0   0   0 21558 362825 86785  7 38 55
     0 0 0   2187M  5806M     2   0   0   0    27   0   2   0 1602 25553 8316  0  3 97
     0 0 0   2187M  5806M   309   0   0   0   318   0   0   0   70  552 2419  0  0 100
     0 0 0   2187M  5806M     2   0   0   1     2   0   8   0  120  383 2630  0  0 100
     0 0 0   2187M  5806M     1   0   0   0     1   0   0   0   62  124 2350  0  0 100
     0 0 0   2187M  5806M     2   0   0   0     0   0   0   0   77  315 2424  0  0 100
    
                input        (Total)           output
       packets  errs idrops      bytes    packets  errs      bytes colls
          6.3K     0     0       609K       166K     0        72M     0
          6.5K     0     0       749K       167K     0        72M     0
          6.4K     0     0       693K       166K     0        72M     0
          6.5K     0     0       782K       166K     0        72M     0
          6.6K     0     0       791K       166K     0        72M     0
          6.5K     0     0       729K       166K     0        72M     0
          6.4K     0     0       733K       166K     0        72M     0
          6.5K     0     0       813K       166K     0        72M     0
          6.6K     0     0       909K       166K     0        72M     0
          6.1K     0     0       696K       156K     0        68M     0
           356     0     0       342K        757     0       349K     0
           488     0     0       474K       1.0K     0       486K     0
           525     0     0       539K       1.1K     0       551K     0
           395     0     0       378K        780     0       384K     0
    
    

    but when my iMac generates the traffic, one core of my pfsense i3-3220 is completely saturated and it drops pings passing through pfsense.

    
    procs      memory      page                    disks     faults         cpu
     r b w     avm    fre   flt  re  pi  po    fr  sr ad4 da0   in   sy   cs us sy id
     0 0 0   2187M  5806M   485   0   0   1   489   0   0   0  212    4  878 15  6 79
     0 0 0   2187M  5806M    69   0   0   0     1   0   0   0   73  584 2401  0  0 100
     0 0 0   2187M  5806M     1   0   0  24     0   0  43   0  138  287 2716  0  0 100
     0 0 0   2187M  5806M   312   0   0   0   319   0   0   0  105  657 2539  0  0 100
     0 0 0   2194M  5805M    18   0   0   0     4   0   0   0   78  292 2415  0  0 100
     0 0 0   2194M  5805M     1   0   0   0     1   0   0   0   39  242 2281  0  0 100
     2 0 0   2194M  5805M     1   0   0   0     1   0   6   0  302 619572 151937 15 44 41
     2 0 0   2194M  5805M     3   0   0   0     1   0   0   0  933 757011 129951 13 53 34
     0 0 0   2194M  5805M  2345   0   0   1  2383   0   6   0  976 708221 142489 15 51 33
     2 0 0   2194M  5805M     2   0   0   0     1   0   0   0  398 738376 145782 13 51 35
     2 0 0   2194M  5805M     2   0   0   0     0   0   0   0 1101 700710 142078 16 50 34
     2 0 0   2194M  5805M     3   0   0   0     1   0   0   0  284 696816 163463 16 49 35
     2 0 0   2194M  5805M     2   0   0   0     1   0   5   0  889 700871 148974 16 50 35
     2 0 0   2194M  5805M     2   0   0   0     1   0   0   0  364 723701 154982 13 50 37
     2 0 0   2194M  5805M   309   0   0   3   314   0   7   0 1772 729066 118540 15 53 33
     0 0 0   2194M  5805M     3   0   0   0     1   0   0   0  497 439803 92152  9 31 60
     0 0 0   2194M  5805M     1   0   0   0     1   0   3   0  102  485 2505  0  0 100
     0 0 0   2194M  5805M     2   0   0   0     1   0   0   0   47  247 2320  0  0 100
     0 0 0   2194M  5805M     0   0   0   0     2   0   6   0   38  193 2277  0  0 100
    
                input        (Total)           output
       packets  errs idrops      bytes    packets  errs      bytes colls
           285     0     0       258K        531     0       262K     0
           212     0     0       169K        349     0       166K     0
           408     0     0       395K        864     0       403K     0
           372     0     0       365K        828     0       368K     0
           390     0     0       369K        806     0       378K     0
            80     0     0        25K         50     0        22K     0
           51K     0     0        72M        44K     0       3.0M     0
           79K     0     0        95M        51K     0       3.3M     0
           79K     0     0        99M        57K     0       3.6M     0
           79K     0     0       110M        71K     0       4.8M     0
           79K     0     0        95M        50K     0       3.3M     0
           94K     0     0       111M        59K     0       3.8M     0
           91K     0     0       110M        61K     0       3.9M     0
    
    

    I don't see any re errors in dmesg. I'm not sure what this shows other than my mac is faster than my pfsense box. :-)



  • Enabling flow control has no impact on my issue :'(
    (what follows has been tested with and without flow control enabled on 8111 EVL. I verified that the linux server on the other end of the link validated symetric flow control if enabled)

    Precise test case: pfsense box directly connected to a linux server, generating UDP from toward pfsense box.

    linux server: cat /dev/zero | nc pfsense 10000 -u
    pfsense: nc -u -l psense 10000 > /dev/null
    

    It won't suffice to trigger the watchdog timeout, it goes to 15k int/s.

    I add then some TCP traffic from pfsense box:

    linux server: nc -l -p 10000 > /dev/null
    pfsense: cat /dev/zero | nc linux_server 10000
    

    And bam, interrupts skyrocket to 26k int/s, and "watchdog timeout"

    re1: watchdog timeout
    re1: link state changed to DOWN
    re1: link state changed to UP
    

    Ok, now if I want to verify if it's an int/s related issue, I disable interrupts rate limiting (flow control is on, even if we don't care now…)

    sysctl dev.re.1.int_rx_mod=0
    dev.re.1.int_rx_mod: 65 -> 0
    

    Generating only UDP from toward pfsense box now gives 55k int/s… without watchdog barking. Damn.
    Generating TCP from pfsense to linux server... brings int/s down?? I stop, retry, get 176k int/s... and watchdog. Ok, so nothing interrupts related at first sight.
    But this time, the link won't come back after terminating frame generating processes. I try from the pfsense box:

    ping linux_server
    PING linux_server (192.168.10.1): 56 data bytes
    ping: sendto: No buffer space available
    ping: sendto: No buffer space available
    ^C
    
    

    Woot! Something new when I finally got nothing left to look at.

    Soooo, this time I don't know a thing about network related buffers on freebsd (I'll give it an eye though). Someone to the rescue? :)


    edit:
    letting the pfsense box "as is" for a couple of minutes, without rebooting. Link is still "lost", despite no network activity at all:

    re1: watchdog timeout
    re1: link state changed to DOWN
    re1: link state changed to UP
    re1: watchdog timeout
    re1: link state changed to DOWN
    re1: link state changed to UP
    ...
    
    

    And now no buffer related message anymore:

    ping linux_server
    PING linux_server (192.168.10.1): 56 data bytes
    ^C
    --- linux_server ping statistics ---
    47 packets transmitted, 0 packets received, 100.0% packet loss
    
    


  • Ok, I give up.
    I gave a shot to OpenWRT AA (so linux 3.3.8), and FYI, I had to include additionnal firmware (rtl8168e-3.fw), that I didn't needed before owning this RTL chip. It is related to these 8111EVL as they are seen as RTL8168evl/8111evl whereas older NICs are seen as 8169/8110. I don't know if it is required for some offloading, or if it is a "fix" to harware issues.
    Used the exact same commands without reproducing the issue.
    Btw, linux driver seems to have had problem with this chip before being fixed: https://bugzilla.kernel.org/show_bug.cgi?id=14962

    I'm definitly thinking about bsd re driver having an issue with this chip (more likely the other way around, but you get the idea :)).


Log in to reply