UPG 2.1 -> 2.1.1.: extremely high latency & pakage loss Intel IGB
-
As in so many times already; I am in debt for help in this complicated matter. Extremely thank you very much for helping me out ;D
-
And this is all using 2.1.2 64bit full install?
Weird doesn't cut it, utterly bizarre is more like. How can the em driver attach to the pro/1000 VT NIC? ???Could you give us the output of:
: pciconf -lv | grep 20000Steve
-
And this is all using 2.1.2 64bit full install?
Weird doesn't cut it, utterly bizarre is more like. How can the em driver attach to the pro/1000 VT NIC? ???Could you give us the output of:
: pciconf -lv | grep 20000Steve
Hi Steve ;D
Yes, this is a fresh install of 2.1 AMD64 (like said, upgrades have never worked for me), and then upgrading that using the GUI to 2.1.1 and consequently to 2.1.2 in the short time frame in which they came available.
Your command on the Dell, with the mysterious assignment of the NICs, gave:
[2.1.2-RELEASE][root@dell.workgroup]/root(1): pciconf -lv | grep 20000 em0@pci0:4:0:0: class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00 em1@pci0:4:0:1: class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00 em2@pci0:5:0:0: class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00 em3@pci0:5:0:1: class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00 bge0@pci0:6:0:0: class=0x020000 card=0x023c1028 chip=0x165914e4 rev=0x21 hdr=0x00 bge1@pci0:7:0:0: class=0x020000 card=0x023c1028 chip=0x165914e4 rev=0x21 hdr=0x00 [2.1.2-RELEASE][root@dell.workgroup]/root(2):The same command on the pfSense1, the mini-ITX, which still has the high latencies despite the Dell NIC now being removed, gives:
[2.1.2-RELEASE][root@ids.workgroup]/root(1): pciconf -lv | grep 20000 em0@pci0:0:25:0: class=0x020000 card=0x20368086 chip=0x15028086 rev=0x04 hdr=0x00 em1@pci0:2:0:0: class=0x020000 card=0x20368086 chip=0x10d38086 rev=0x00 hdr=0x00 [2.1.2-RELEASE][root@ids.workgroup]/root(2):(Note: this pfSense1, the mini-ITX, also is a fresh install of 2.1 -> 2.1.1 -> 2.1.2. I've used this for over a year, but given upgrades never worked I freshly installed 2.1 after having been on 2.0 for a year. So no leftovers from previous trials and errors.)
So, do you agree with me that the - still remaining - high RTT/latency on pfSense1 appears to be a problem with 2.1.2 with my hardware (after all, the Dell NIC is not inside in it anymore yet the high latency remains)?
And of course, is there a simple way for me to tell the pfsense2 (R200) to use the IGB's it originally correctly installed? Or will I have to do a fresh install every time I switch switch- and VDSL/CABLE ethernet cable from pfsense1 to pfsense2? (That will be horrible).
Thank you again Steve; in debt as always ;D
-
Yep, I agree that the latency remains without the Dell card in place.
So the interfaces on the Dell card appear as PCI Vendor ID: 8086 (Intel), PCI Device ID: 10BC. Consulting the list of hardware supported by Intel Gigabit drivers from FreeBSD 8.3 we see that this is:
@http://svnweb.freebsd.org/base/release/8.3.0/sys/dev/e1000/e1000_hw.h?revision=234063&view=markup:#define E1000_DEV_ID_82571EB_QUAD_COPPER_LP 0x10BC
O.K. so that looks right, we know it's a quad copper NIC. However that chip is supported, in FreeBSD 8.3, by the em(4) driver not igb. Also checking the other source files it's still supported by the em driver in 8.1(2.0.X) and in10 (2.2). The actual driver used in 2.1.2 is not the FreeBSD 8.3 release version but a backport. I haven't got around to signing up to the tools repo access yet so I can't check exactly but since it hasn't changed in 10 I think it's very unlikely to be anything other than em.
The same is true for the two on-board NICs in system 1.
Why the were those NICs ever attched to the igb(4) driver? :-\ Were they returning a different PCI Dev ID before perhaps for some reason? It doesn't seem to be a Pro/1000 VT card as those are using 827575GB controllers.
Anyway since they appear to be correctly using the em driver I suggest you try some of the loader variables with em instead of igb.
I'll try and re-read this thread because it seems like I may have misunderstood/forgotten something.
Steve
-
Yep, I agree that the latency remains without the Dell card in place.
So the interfaces on the Dell card appear as PCI Vendor ID: 8086 (Intel), PCI Device ID: 10BC. Consulting the list of hardware supported by Intel Gigabit drivers from FreeBSD 8.3 we see that this is:
@http://svnweb.freebsd.org/base/release/8.3.0/sys/dev/e1000/e1000_hw.h?revision=234063&view=markup:#define E1000_DEV_ID_82571EB_QUAD_COPPER_LP 0x10BC
O.K. so that looks right, we know it's a quad copper NIC. However that chip is supported, in FreeBSD 8.3, by the em(4) driver not igb. Also checking the other source files it's still supported by the em driver in 8.1(2.0.X) and in10 (2.2). The actual driver used in 2.1.2 is not the FreeBSD 8.3 release version but a backport. I haven't got around to signing up to the tools repo access yet so I can't check exactly but since it hasn't changed in 10 I think it's very unlikely to be anything other than em.
The same is true for the two on-board NICs in system 1.
Why the were those NICs ever attched to the igb(4) driver? :-\ Were they returning a different PCI Dev ID before perhaps for some reason? It doesn't seem to be a Pro/1000 VT card as those are using 827575GB controllers.
Anyway since they appear to be correctly using the em driver I suggest you try some of the loader variables with em instead of igb.
I'll try and re-read this thread because it seems like I may have misunderstood/forgotten something.
Steve
That is some real British Sherlock Holmes work you have done, Steve; thank you very much ;D
So now it appears not to be the VT? But then what(?)
I don't know what has happened, because:
1. The IBM card didn't work. So I had only one other card to try, the Dell.
2. This one single card from Dell I installed in both machines; first in the R200, then in the mini-ITX.
3. When I was finished installing the R200 - to be stashed away as a backup machine - I took note of which of the 6 ports was for what (WAN, WAN2, VLAN, LAN), removed the cables and shut down the Dell, to go work on my fresh re-install of the mini-ITX, in which I also put the Dell quad NIC.
4. Both machines assigned the IGB-driver to the Dell card on their first install (the mini-ITX still has all the IGB-interfaces after I removed the card yesterday).
5. The Dell worked perfectly on the IGB-driver the whole week I tested it before shutting it down and storing it as a backup. So: with the IGB-driver.
6. Only yesterday did it suddenly decide to assign em-drivers to them; and it started crashing immediately.I will of course put the em-variables in /boot/loader.conf.local as you suggest, but it strikes me: point 5.
I can also reinstall the Dell, but I am sure it will assign the IGB again (I recall doing that every time, since I had to reinstall the Dell 3 times before I got it to work).
So, while I will do the em-variables, my questions are:
1. What could I do about the mini-ITX? (This has a little high priority above the Dell, since I can not return cards after this week, should I need to do so).
2. Suppose I add the em-variables in the Dell (and perhaps also in the mini-ITX), aren't things supposed to run into a mess since I am sure the references to IGB are spread through config parts in different kinds of the system?Thank you once again for your help, Sir Steve ;D
-
I assume that the mini-ITX board is using the em driver for it's on-board interfaces? Do you have any em tunables loading? Try playing with them if not. Check for errors in Status: Interfaces: and the logs. The latency could be some sort of excessive buffering or huge error rate (are you seeing packet loss?)
If you put the card back in some other box and it appears as igb interfaces run the pciconf command again and see if it's reporting a different device ID. There are only a couple of things I could imagine changing the device ID: the card firmware has been updated. I would expect it to be some manual process with multiple 'are you sure?'s ;) but I coul;d just about imagine the Dell box talks to their card differently or that an Intel board updates the firmware somehow. Very odd.Also if you reinstall for any reason I suggest you use a 2.1.2 CD directly to eliminate any upgrade issues you may have coming from 2.1.
Steve
-
I assume that the mini-ITX board is using the em driver for it's on-board interfaces? Do you have any em tunables loading? Try playing with them if not. Check for errors in Status: Interfaces: and the logs. The latency could be some sort of excessive buffering or huge error rate (are you seeing packet loss?)
If you put the card back in some other box and it appears as igb interfaces run the pciconf command again and see if it's reporting a different device ID. There are only a couple of things I could imagine changing the device ID: the card firmware has been updated. I would expect it to be some manual process with multiple 'are you sure?'s ;) but I coul;d just about imagine the Dell box talks to their card differently or that an Intel board updates the firmware somehow. Very odd.Also if you reinstall for any reason I suggest you use a 2.1.2 CD directly to eliminate any upgrade issues you may have coming from 2.1.
Steve
Thanks Steve ;D
Update: adding the em-settings to the mini-ITX does nothing good, but one thing bad: I can not ping to it anymore, or access the GUI. I had to go over to the console and manually edit these settings out again:
kern.ipc.nmbclusters="131072" hw.em.num_queues=1 #hw.em.rxd=4096 #hw.em.txd=4096The mini-ITX for it's onboard indeed is em0 and em1. Status/interfaces is clean, no in/out errors, no collissions, on none of the interfaces. Package loss occassionaly happens in the GUI, especially when the RTT goes above around 100.
I did not update any firmware; neither on the nic or the mobo of the Dell, nor on the mini-ITX. I am assuming these boards don't update themselves over the internets without me knowing anything about it ( :-[ ).
I will now add the em-settings to the Dell and see if that crashes again within minutes.
Thanks for your help Steve ;D
-
If you are still using 4096 for the Tx and Rx buffers I would definitely trt 2048 instead. Reading through the docs it appears it could well be 4096 total, though it's not very clear. Remind me why you're setting those?
The Dell box especially may be susceptible to mbuf exhaustion. It has both Broadcom and Intel NICs and multiples of both and is running 64bit. I would expect the advice in the NIC tuning docs page to hold true. However even without any tuning the mbufs would not be exhausted immediately and the symptoms of that happing are loss of all traffic. Not latency issues.
Another place you can look for problems are the extensive sysctl counters provided by Intel. For example:
[2.1.2-RELEASE][root@pfsense.fire.box]/root(1): sysctl hw.em hw.em.num_queues: 0 hw.em.eee_setting: 1 hw.em.rx_process_limit: 100 hw.em.enable_msix: 1 hw.em.sbp: 0 hw.em.smart_pwr_down: 0 hw.em.txd: 1024 hw.em.rxd: 1024 hw.em.rx_abs_int_delay: 66 hw.em.tx_abs_int_delay: 66 hw.em.rx_int_delay: 0 hw.em.tx_int_delay: 66 [2.1.2-RELEASE][root@pfsense.fire.box]/root(2): sysctl dev.em.0 dev.em.0.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.0.6 dev.em.0.%driver: em dev.em.0.%location: slot=1 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x1075 subvendor=0x8086 subdevice=0x1075 class=0x020000 dev.em.0.%parent: pci2 dev.em.0.nvm: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.itr: 488 dev.em.0.rx_processing_limit: 100 dev.em.0.flow_control: 3 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 0 dev.em.0.tx_desc_fail1: 0 dev.em.0.tx_desc_fail2: 0 dev.em.0.rx_overruns: 0 dev.em.0.watchdog_timeouts: 0 dev.em.0.device_control: 1077674561 dev.em.0.rx_control: 32770 dev.em.0.fc_high_water: 28672 dev.em.0.fc_low_water: 27172 dev.em.0.fifo_workaround: 0 dev.em.0.fifo_reset: 0 dev.em.0.txd_head: 149 dev.em.0.txd_tail: 149 dev.em.0.rxd_head: 173 dev.em.0.rxd_tail: 172 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 0 dev.em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 0 dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.mac_stats.total_pkts_recvd: 2207149 dev.em.0.mac_stats.good_pkts_recvd: 2207149 dev.em.0.mac_stats.bcast_pkts_recvd: 6288 dev.em.0.mac_stats.mcast_pkts_recvd: 0 dev.em.0.mac_stats.rx_frames_64: 1002865 dev.em.0.mac_stats.rx_frames_65_127: 1166547 dev.em.0.mac_stats.rx_frames_128_255: 8643 dev.em.0.mac_stats.rx_frames_256_511: 11258 dev.em.0.mac_stats.rx_frames_512_1023: 12989 dev.em.0.mac_stats.rx_frames_1024_1522: 4847 dev.em.0.mac_stats.good_octets_recvd: 169744603 dev.em.0.mac_stats.good_octets_txd: 5512131147 dev.em.0.mac_stats.total_pkts_txd: 3914121 dev.em.0.mac_stats.good_pkts_txd: 3914121 dev.em.0.mac_stats.bcast_pkts_txd: 1707 dev.em.0.mac_stats.mcast_pkts_txd: 5 dev.em.0.mac_stats.tx_frames_64: 18655 dev.em.0.mac_stats.tx_frames_65_127: 64577 dev.em.0.mac_stats.tx_frames_128_255: 25093 dev.em.0.mac_stats.tx_frames_256_511: 24830 dev.em.0.mac_stats.tx_frames_512_1023: 23427 dev.em.0.mac_stats.tx_frames_1024_1522: 3757539 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tso_ctx_fail: 0Steve
-
Thanks Steve :D
If you are still using 4096 for the Tx and Rx buffers I would definitely trt 2048 instead. Reading through the docs it appears it could well be 4096 total, though it's not very clear. Remind me why you're setting those?
I was told to do this for the quad NIC. Somewhere on this forum, I don't recall (I think by now I have the whole forum bookmarked ;D ;D ;D ).
The Dell box especially may be susceptible to mbuf exhaustion. It has both Broadcom and Intel NICs and multiples of both and is running 64bit. I would expect the advice in the NIC tuning docs page to hold true. However even without any tuning the mbufs would not be exhausted immediately and the symptoms of that happing are loss of all traffic. Not latency issues.
At least in the GUI the mbuf's weren't exhausted :(
Another place you can look for problems are the extensive sysctl counters provided by Intel. For example:
[2.1.2-RELEASE][root@pfsense.fire.box]/root(1): sysctl hw.em hw.em.num_queues: 0 hw.em.eee_setting: 1 hw.em.rx_process_limit: 100 hw.em.enable_msix: 1 hw.em.sbp: 0 hw.em.smart_pwr_down: 0 hw.em.txd: 1024 hw.em.rxd: 1024 hw.em.rx_abs_int_delay: 66 hw.em.tx_abs_int_delay: 66 hw.em.rx_int_delay: 0 hw.em.tx_int_delay: 66 [2.1.2-RELEASE][root@pfsense.fire.box]/root(2): sysctl dev.em.0 dev.em.0.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.0.6 dev.em.0.%driver: em dev.em.0.%location: slot=1 function=0 dev.em.0.%pnpinfo: vendor=0x8086 device=0x1075 subvendor=0x8086 subdevice=0x1075 class=0x020000 dev.em.0.%parent: pci2 dev.em.0.nvm: -1 dev.em.0.rx_int_delay: 0 dev.em.0.tx_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_abs_int_delay: 66 dev.em.0.itr: 488 dev.em.0.rx_processing_limit: 100 dev.em.0.flow_control: 3 dev.em.0.mbuf_alloc_fail: 0 dev.em.0.cluster_alloc_fail: 0 dev.em.0.dropped: 0 dev.em.0.tx_dma_fail: 0 dev.em.0.tx_desc_fail1: 0 dev.em.0.tx_desc_fail2: 0 dev.em.0.rx_overruns: 0 dev.em.0.watchdog_timeouts: 0 dev.em.0.device_control: 1077674561 dev.em.0.rx_control: 32770 dev.em.0.fc_high_water: 28672 dev.em.0.fc_low_water: 27172 dev.em.0.fifo_workaround: 0 dev.em.0.fifo_reset: 0 dev.em.0.txd_head: 149 dev.em.0.txd_tail: 149 dev.em.0.rxd_head: 173 dev.em.0.rxd_tail: 172 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.missed_packets: 0 dev.em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.xon_recvd: 0 dev.em.0.mac_stats.xon_txd: 0 dev.em.0.mac_stats.xoff_recvd: 0 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.mac_stats.total_pkts_recvd: 2207149 dev.em.0.mac_stats.good_pkts_recvd: 2207149 dev.em.0.mac_stats.bcast_pkts_recvd: 6288 dev.em.0.mac_stats.mcast_pkts_recvd: 0 dev.em.0.mac_stats.rx_frames_64: 1002865 dev.em.0.mac_stats.rx_frames_65_127: 1166547 dev.em.0.mac_stats.rx_frames_128_255: 8643 dev.em.0.mac_stats.rx_frames_256_511: 11258 dev.em.0.mac_stats.rx_frames_512_1023: 12989 dev.em.0.mac_stats.rx_frames_1024_1522: 4847 dev.em.0.mac_stats.good_octets_recvd: 169744603 dev.em.0.mac_stats.good_octets_txd: 5512131147 dev.em.0.mac_stats.total_pkts_txd: 3914121 dev.em.0.mac_stats.good_pkts_txd: 3914121 dev.em.0.mac_stats.bcast_pkts_txd: 1707 dev.em.0.mac_stats.mcast_pkts_txd: 5 dev.em.0.mac_stats.tx_frames_64: 18655 dev.em.0.mac_stats.tx_frames_65_127: 64577 dev.em.0.mac_stats.tx_frames_128_255: 25093 dev.em.0.mac_stats.tx_frames_256_511: 24830 dev.em.0.mac_stats.tx_frames_512_1023: 23427 dev.em.0.mac_stats.tx_frames_1024_1522: 3757539 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tso_ctx_fail: 0Steve
That is an extreme list :o
But what I am to do? As you know I am an economist; having to understand what all these settings mean would easily require a multi-year network study on some school, I guess. If I am to try these one by one this will be my magnum opus.
But more over: on the mini-ITX, it was running ok on 2.0 and 2.1 without the quad nic. It isn't running ok on 2.1.2 with the quad nic, but it also isn't running ok without the quad nic. With- or without the quoted settings in loader.conf.local, it doesn't make a difference. So something is wrong with 2.1.2, at least: if it was running fine before that would be my assumption.
Update, to make matters even more insane: adding the em-settings on the Dell, on which I am currently typing this, may have done something. So far it hasn't crashed, and it is on for 45 minutes now (pic).
I have to go now: I have to drive to the store to buy me a new Zyxel advanced soho modem/router/firewall ;D
(I hate this mess :-[ :-\ :'( :( )
Thank you Sir Steve
-
@Hollander:
Update, to make matters even more insane: adding the em-settings on the Dell, on which I am currently typing this, may have done something. So far it hasn't crashed, and it is on for 45 minutes now (pic).
Update to the update; while returning from the store, after 1 hour, the Dell again had crashed hard. I am going to give up :'(
-
By the way, I have to add, I was looking for a way to find something like logs or so, and google brought me to man crash:
http://www.freebsd.org/cgi/man.cgi?query=crash&sektion=8&n=1
But what is described there is not what happened here. No 'kernel panick messages', with automatic reboots or something like that. No simply, at the console, the LCD screen is filled with strange signs, and a long row of 'SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS', and then system hangs completely. And the red/orange light at the front of Dell, which signals when something bad happened, is flashing.
-
@Hollander:
By the way, I have to add, I was looking for a way to find something like logs or so, and google brought me to man crash:
http://www.freebsd.org/cgi/man.cgi?query=crash&sektion=8&n=1
But what is described there is not what happened here. No 'kernel panick messages', with automatic reboots or something like that. No simply, at the console, the LCD screen is filled with strange signs, and a long row of 'SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS', and then system hangs completely. And the red/orange light at the front of Dell, which signals when something bad happened, is flashing.
That's a good indication of a hardware problem that is caused by either overheating (CPU/GPU or some other component that requires cooling) or voltages fluctuating because the power supply can not keep up with the requested load.
-
@kpa:
@Hollander:
By the way, I have to add, I was looking for a way to find something like logs or so, and google brought me to man crash:
http://www.freebsd.org/cgi/man.cgi?query=crash&sektion=8&n=1
But what is described there is not what happened here. No 'kernel panick messages', with automatic reboots or something like that. No simply, at the console, the LCD screen is filled with strange signs, and a long row of 'SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS', and then system hangs completely. And the red/orange light at the front of Dell, which signals when something bad happened, is flashing.
That's a good indication of a hardware problem that is caused by either overheating (CPU/GPU or some other component that requires cooling) or voltages fluctuating because the power supply can not keep up with the requested load.
Thanks Kpa ;D
It was working fine without the quad NIC, and it was working fine for the first week with the quad nic. I am writing the Dell diagnostics disc to a USB, and will then have that test the Dell both with, and without, the quad nic to see what gives.
-
@Hollander:
That is an extreme list :o
It is. I have no idea what most of those mean. Look at my values compare them to your or to other posts on the forum. If something looks different or some value with the word 'error' in title is >0 Google it. :)
Steve
