Interface errors, missed packets and rec overruns
-
I'm slowly racking up interfaced errors on my CE 2.7.2 home box and I'm not sure why. It's an old firewall appliance box, a Lanner FW-7541C. It has an Intel D525 CPU, 1 Intel 82574L and 5 Intel 82583V interfaces, a SanDisk SSD storage device and 4GB of RAM. I keep missing a small number of packets. Like the majority are getting through fine and I'm not seeing any trouble, but the errors keep slowly ticking up. I have hardware offload enabled, with large send/recv disabled per the documentation recommendations. I know the D525 has all the processing power of a wet noodle, but I'm not trying to do anything processor intensive. Basically just basic firewall duty and that's it. I've tried turning hyper-threading on and off. I've tried moving the WAN from a direct line to the pfsense to passing through a switch. I've tried the 82583V's instead of the 84574L. I just don't get it, my internet is only like 300-350Mbit, I think it should be able to keep up with that, right? Is there any tweak I should try?
sysctl dev.em.0 output
dev.em.0.interrupts.rx_overrun: 3 dev.em.0.interrupts.rx_desc_min_thresh: 0 dev.em.0.interrupts.tx_queue_min_thresh: 1 dev.em.0.interrupts.tx_queue_empty: 0 dev.em.0.interrupts.tx_abs_timer: 2 dev.em.0.interrupts.tx_pkt_timer: 1 dev.em.0.interrupts.rx_abs_timer: 0 dev.em.0.interrupts.rx_pkt_timer: 32 dev.em.0.interrupts.asserts: 118670 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tx_frames_1024_1522: 181167932 dev.em.0.mac_stats.tx_frames_512_1023: 7667118 dev.em.0.mac_stats.tx_frames_256_511: 7806568 dev.em.0.mac_stats.tx_frames_128_255: 14078839 dev.em.0.mac_stats.tx_frames_65_127: 54260108 dev.em.0.mac_stats.tx_frames_64: 48427909 dev.em.0.mac_stats.mcast_pkts_txd: 1884495 dev.em.0.mac_stats.bcast_pkts_txd: 45661 dev.em.0.mac_stats.good_pkts_txd: 313408474 dev.em.0.mac_stats.total_pkts_txd: 313436934 dev.em.0.mac_stats.good_octets_txd: 285602451945 dev.em.0.mac_stats.good_octets_recvd: 287772957030 dev.em.0.mac_stats.rx_frames_1024_1522: 182980173 dev.em.0.mac_stats.rx_frames_512_1023: 7219728 dev.em.0.mac_stats.rx_frames_256_511: 7331488 dev.em.0.mac_stats.rx_frames_128_255: 12598599 dev.em.0.mac_stats.rx_frames_65_127: 98395148 dev.em.0.mac_stats.rx_frames_64: 4681295 dev.em.0.mac_stats.mcast_pkts_recvd: 286677 dev.em.0.mac_stats.bcast_pkts_recvd: 2006878 dev.em.0.mac_stats.good_pkts_recvd: 313206431 dev.em.0.mac_stats.total_pkts_recvd: 313284437 dev.em.0.mac_stats.xoff_txd: 25803 dev.em.0.mac_stats.xoff_recvd: 22665 dev.em.0.mac_stats.xon_txd: 2657 dev.em.0.mac_stats.xon_recvd: 22305 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_no_buff: 20674810 dev.em.0.mac_stats.missed_packets: 23148 dev.em.0.mac_stats.defer_count: 0 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.queue_rx_1.rx_irq: 0 dev.em.0.queue_rx_1.rxd_tail: 802 dev.em.0.queue_rx_1.rxd_head: 803 dev.em.0.queue_rx_0.rx_irq: 0 dev.em.0.queue_rx_0.rxd_tail: 616 dev.em.0.queue_rx_0.rxd_head: 617 dev.em.0.queue_tx_1.tx_irq: 0 dev.em.0.queue_tx_1.txd_tail: 507 dev.em.0.queue_tx_1.txd_head: 510 dev.em.0.queue_tx_0.tx_irq: 0 dev.em.0.queue_tx_0.txd_tail: 543 dev.em.0.queue_tx_0.txd_head: 543 dev.em.0.fc_low_water: 16932 dev.em.0.fc_high_water: 18432 dev.em.0.rx_control: 67403778 dev.em.0.device_control: 1477444168 dev.em.0.watchdog_timeouts: 0 dev.em.0.rx_overruns: 81652 dev.em.0.link_irq: 123229 dev.em.0.dropped: 0 dev.em.0.eee_control: 1 dev.em.0.itr: 488 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_int_delay: 66 dev.em.0.rx_int_delay: 0 dev.em.0.rs_dump: 0 dev.em.0.reg_dump: General Registers CTRL 58100248 STATUS 00080783 CTRL_EXT 80580000 Interrupt Registers ICR 80000001 RX Registers RCTL 04048002 RDLEN 00004000 RDH 00000269 RDT 00000268 RXDCTL 01050420 RDBAL 0461a000 RDBAH 00000000 TX Registers TCTL 3103f0fa TDBAL 04600000 TDBAH 00000000 TDLEN 00004000 TDH 0000021f TDT 0000021f TXDCTL 0341011f TDFH 00000b28 TDFT 00000b28 TDFHS 00000b28 TDFPC 00000000 dev.em.0.fc: 3 dev.em.0.debug: -1 dev.em.0.fw_version: EEPROM V1.9-0 dev.em.0.nvm: -1 dev.em.0.iflib.rxq1.rxq_fl0.buf_size: 2048 dev.em.0.iflib.rxq1.rxq_fl0.credits: 1023 dev.em.0.iflib.rxq1.rxq_fl0.cidx: 818 dev.em.0.iflib.rxq1.rxq_fl0.pidx: 817 dev.em.0.iflib.rxq1.cpu: 2 dev.em.0.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.em.0.iflib.rxq0.rxq_fl0.credits: 1023 dev.em.0.iflib.rxq0.rxq_fl0.cidx: 617 dev.em.0.iflib.rxq0.rxq_fl0.pidx: 616 dev.em.0.iflib.rxq0.cpu: 0 dev.em.0.iflib.txq1.r_abdications: 0 dev.em.0.iflib.txq1.r_restarts: 0 dev.em.0.iflib.txq1.r_stalls: 0 dev.em.0.iflib.txq1.r_starts: 148037804 dev.em.0.iflib.txq1.r_drops: 0 dev.em.0.iflib.txq1.r_enqueues: 152658558 dev.em.0.iflib.txq1.ring_state: pidx_head: 0640 pidx_tail: 0640 cidx: 0640 state: IDLE dev.em.0.iflib.txq1.txq_cleaned: 171150878 dev.em.0.iflib.txq1.txq_processed: 171150918 dev.em.0.iflib.txq1.txq_in_use: 45 dev.em.0.iflib.txq1.txq_cidx_processed: 587 dev.em.0.iflib.txq1.txq_cidx: 550 dev.em.0.iflib.txq1.txq_pidx: 592 dev.em.0.iflib.txq1.no_tx_dma_setup: 0 dev.em.0.iflib.txq1.txd_encap_efbig: 0 dev.em.0.iflib.txq1.tx_map_failed: 0 dev.em.0.iflib.txq1.no_desc_avail: 0 dev.em.0.iflib.txq1.mbuf_defrag_failed: 0 dev.em.0.iflib.txq1.m_pullups: 12029099 dev.em.0.iflib.txq1.mbuf_defrag: 0 dev.em.0.iflib.txq1.cpu: 2 dev.em.0.iflib.txq0.r_abdications: 0 dev.em.0.iflib.txq0.r_restarts: 91 dev.em.0.iflib.txq0.r_stalls: 91 dev.em.0.iflib.txq0.r_starts: 156570070 dev.em.0.iflib.txq0.r_drops: 0 dev.em.0.iflib.txq0.r_enqueues: 160858354 dev.em.0.iflib.txq0.ring_state: pidx_head: 0242 pidx_tail: 0242 cidx: 0242 state: IDLE dev.em.0.iflib.txq0.txq_cleaned: 177709558 dev.em.0.iflib.txq0.txq_processed: 177709598 dev.em.0.iflib.txq0.txq_in_use: 41 dev.em.0.iflib.txq0.txq_cidx_processed: 542 dev.em.0.iflib.txq0.txq_cidx: 502 dev.em.0.iflib.txq0.txq_pidx: 543 dev.em.0.iflib.txq0.no_tx_dma_setup: 0 dev.em.0.iflib.txq0.txd_encap_efbig: 0 dev.em.0.iflib.txq0.tx_map_failed: 0 dev.em.0.iflib.txq0.no_desc_avail: 0 dev.em.0.iflib.txq0.mbuf_defrag_failed: 0 dev.em.0.iflib.txq0.m_pullups: 12129578 dev.em.0.iflib.txq0.mbuf_defrag: 0 dev.em.0.iflib.txq0.cpu: 0 dev.em.0.iflib.override_nrxds: 0 dev.em.0.iflib.override_ntxds: 0 dev.em.0.iflib.use_logical_cores: 0 dev.em.0.iflib.separate_txrx: 0 dev.em.0.iflib.core_offset: 0 dev.em.0.iflib.tx_abdicate: 0 dev.em.0.iflib.rx_budget: 0 dev.em.0.iflib.disable_msix: 0 dev.em.0.iflib.override_qs_enable: 0 dev.em.0.iflib.override_nrxqs: 0 dev.em.0.iflib.override_ntxqs: 0 dev.em.0.iflib.driver_version: 7.7.8-fbsd dev.em.0.%parent: pci1 dev.em.0.%pnpinfo: vendor=0x8086 device=0x10d3 subvendor=0x8086 subdevice=0x0000 class=0x020000 dev.em.0.%location: slot=0 function=0 dbsf=pci0:2:0:0 dev.em.0.%driver: em dev.em.0.%desc: Intel(R) Gigabit CT 82574L -
Oh, I forgot possibly the most interesting tidbit. My brother is experiencing the same behavior and his system has a Ryzen 3 2200G. CPU to spare on that machine with an old quad NIC Intel 82571. It only seems to happen on the WAN side. I had the same experience when my WAN was separate. (I currently have all my traffic VLANed into a single physical NIC).
Brother's sysctl dev.em.0 output
dev.em.0.interrupts.rx_overrun: 0 dev.em.0.interrupts.rx_desc_min_thresh: 0 dev.em.0.interrupts.tx_queue_min_thresh: 24 dev.em.0.interrupts.tx_queue_empty: 0 dev.em.0.interrupts.tx_abs_timer: 9824 dev.em.0.interrupts.tx_pkt_timer: 7967 dev.em.0.interrupts.rx_abs_timer: 0 dev.em.0.interrupts.rx_pkt_timer: 80527 dev.em.0.interrupts.asserts: 484449432 dev.em.0.mac_stats.tso_ctx_fail: 0 dev.em.0.mac_stats.tso_txd: 0 dev.em.0.mac_stats.tx_frames_1024_1522: 83478008 dev.em.0.mac_stats.tx_frames_512_1023: 619494980 dev.em.0.mac_stats.tx_frames_256_511: 4447683 dev.em.0.mac_stats.tx_frames_128_255: 26350772 dev.em.0.mac_stats.tx_frames_65_127: 90891554 dev.em.0.mac_stats.tx_frames_64: 42590904 dev.em.0.mac_stats.mcast_pkts_txd: 1192 dev.em.0.mac_stats.bcast_pkts_txd: 3346 dev.em.0.mac_stats.good_pkts_txd: 867253901 dev.em.0.mac_stats.total_pkts_txd: 867253901 dev.em.0.mac_stats.good_octets_txd: 564113139828 dev.em.0.mac_stats.good_octets_recvd: 600386896379 dev.em.0.mac_stats.rx_frames_1024_1522: 395679755 dev.em.0.mac_stats.rx_frames_512_1023: 17668892 dev.em.0.mac_stats.rx_frames_256_511: 7229265 dev.em.0.mac_stats.rx_frames_128_255: 158134130 dev.em.0.mac_stats.rx_frames_65_127: 48923871 dev.em.0.mac_stats.rx_frames_64: 0 dev.em.0.mac_stats.mcast_pkts_recvd: 6626 dev.em.0.mac_stats.bcast_pkts_recvd: 0 dev.em.0.mac_stats.good_pkts_recvd: 627635913 dev.em.0.mac_stats.total_pkts_recvd: 629974672 dev.em.0.mac_stats.xoff_txd: 0 dev.em.0.mac_stats.xoff_recvd: 1165746 dev.em.0.mac_stats.xon_txd: 0 dev.em.0.mac_stats.xon_recvd: 1165746 dev.em.0.mac_stats.coll_ext_errs: 0 dev.em.0.mac_stats.alignment_errs: 0 dev.em.0.mac_stats.crc_errs: 0 dev.em.0.mac_stats.recv_errs: 0 dev.em.0.mac_stats.recv_jabber: 0 dev.em.0.mac_stats.recv_oversize: 0 dev.em.0.mac_stats.recv_fragmented: 0 dev.em.0.mac_stats.recv_undersize: 0 dev.em.0.mac_stats.recv_no_buff: 0 dev.em.0.mac_stats.missed_packets: 7310 dev.em.0.mac_stats.defer_count: 1161488 dev.em.0.mac_stats.sequence_errors: 0 dev.em.0.mac_stats.symbol_errors: 0 dev.em.0.mac_stats.collision_count: 0 dev.em.0.mac_stats.late_coll: 0 dev.em.0.mac_stats.multiple_coll: 0 dev.em.0.mac_stats.single_coll: 0 dev.em.0.mac_stats.excess_coll: 0 dev.em.0.queue_rx_0.rx_irq: 0 dev.em.0.queue_rx_0.rxd_tail: 518 dev.em.0.queue_rx_0.rxd_head: 519 dev.em.0.queue_tx_0.tx_irq: 0 dev.em.0.queue_tx_0.txd_tail: 164 dev.em.0.queue_tx_0.txd_head: 164 dev.em.0.fc_low_water: 29220 dev.em.0.fc_high_water: 30720 dev.em.0.rx_control: 67403778 dev.em.0.device_control: 1209795137 dev.em.0.watchdog_timeouts: 0 dev.em.0.rx_overruns: 111 dev.em.0.link_irq: 0 dev.em.0.dropped: 0 dev.em.0.eee_control: 1 dev.em.0.itr: 488 dev.em.0.tx_abs_int_delay: 66 dev.em.0.rx_abs_int_delay: 66 dev.em.0.tx_int_delay: 66 dev.em.0.rx_int_delay: 0 dev.em.0.rs_dump: 0 dev.em.0.reg_dump: General Registers CTRL 481c0241 STATUS 00080387 CTRL_EXT 101400c0 Interrupt Registers ICR 00000000 RX Registers RCTL 04048002 RDLEN 00004000 RDH 00000207 RDT 00000206 RXDCTL 00010000 RDBAL b0034000 RDBAH 00000000 TX Registers TCTL 3103f0fa TDBAL b002c000 TDBAH 00000000 TDLEN 00004000 TDH 000000aa TDT 000000aa TXDCTL 0341011f TDFH 00001296 TDFT 00001298 TDFHS 00001296 TDFPC 00000000 dev.em.0.fc: 3 dev.em.0.debug: -1 dev.em.0.fw_version: EEPROM V5.12-2 dev.em.0.nvm: -1 dev.em.0.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.em.0.iflib.rxq0.rxq_fl0.credits: 1023 dev.em.0.iflib.rxq0.rxq_fl0.cidx: 519 dev.em.0.iflib.rxq0.rxq_fl0.pidx: 518 dev.em.0.iflib.rxq0.cpu: 3 dev.em.0.iflib.txq0.r_abdications: 324 dev.em.0.iflib.txq0.r_restarts: 56590 dev.em.0.iflib.txq0.r_stalls: 56590 dev.em.0.iflib.txq0.r_starts: 858032115 dev.em.0.iflib.txq0.r_drops: 1072 dev.em.0.iflib.txq0.r_enqueues: 868117756 dev.em.0.iflib.txq0.ring_state: pidx_head: 1279 pidx_tail: 1279 cidx: 1279 state: IDLE dev.em.0.iflib.txq0.txq_cleaned: 1474843782 dev.em.0.iflib.txq0.txq_processed: 1474843822 dev.em.0.iflib.txq0.txq_in_use: 44 dev.em.0.iflib.txq0.txq_cidx_processed: 174 dev.em.0.iflib.txq0.txq_cidx: 134 dev.em.0.iflib.txq0.txq_pidx: 178 dev.em.0.iflib.txq0.no_tx_dma_setup: 0 dev.em.0.iflib.txq0.txd_encap_efbig: 0 dev.em.0.iflib.txq0.tx_map_failed: 0 dev.em.0.iflib.txq0.no_desc_avail: 0 dev.em.0.iflib.txq0.mbuf_defrag_failed: 0 dev.em.0.iflib.txq0.m_pullups: 602346602 dev.em.0.iflib.txq0.mbuf_defrag: 0 dev.em.0.iflib.txq0.cpu: 2 dev.em.0.iflib.override_nrxds: 0 dev.em.0.iflib.override_ntxds: 0 dev.em.0.iflib.use_logical_cores: 0 dev.em.0.iflib.separate_txrx: 0 dev.em.0.iflib.core_offset: 0 dev.em.0.iflib.tx_abdicate: 0 dev.em.0.iflib.rx_budget: 0 dev.em.0.iflib.disable_msix: 1 dev.em.0.iflib.override_qs_enable: 0 dev.em.0.iflib.override_nrxqs: 0 dev.em.0.iflib.override_ntxqs: 0 dev.em.0.iflib.driver_version: 7.7.8-fbsd dev.em.0.%parent: pci3 dev.em.0.%pnpinfo: vendor=0x8086 device=0x10bc subvendor=0x103c subdevice=0x704b class=0x020000 dev.em.0.%location: slot=0 function=0 dbsf=pci0:18:0:0 dev.em.0.%driver: em dev.em.0.%desc: Intel(R) PRO/1000 PT 82571EB/82571GB (Quad Copper) -
Looks like slightly different cause though. recv_no_buff vs defer_count
The first thing I'd try is reassigning the NICs to use a different one as WAN and see if the issue follows it.
Next I'd try a different flow-control setting and check the current negotiated value. That should prevent the other side overloading the receive buffers if both ends support it.
Steve
-
@stephenw10 said in Interface errors, missed packets and rec overruns:
Looks like slightly different cause though. recv_no_buff vs defer_count
I noticed that too, but I'm not sure what "defer" means here. I mean I'm also not sure what "recv_no_buff" means, but my educated guess is it received a packet but had no buffer space available to place it in.
@stephenw10 said in Interface errors, missed packets and rec overruns:
The first thing I'd try is reassigning the NICs to use a different one as WAN and see if the issue follows it.
I tried that already, the 82583V NICs all do the same thing.
@stephenw10 said in Interface errors, missed packets and rec overruns:
Next I'd try a different flow-control setting and check the current negotiated value.
So it's currently set to 3 for flow control. How do I check the negotiated value?
-
AFAIK
recv_no_buffimplies there are no available receive buffers. So potentially you could increase the buffers.I will say that I see some no_buff failures on a box here and don't see any connection issues:
[2.7.2-RELEASE][admin@xtm5.stevew.lan]/root: sysctl dev.em.0 | grep buf dev.em.0.mac_stats.recv_no_buff: 36 dev.em.0.iflib.rxq1.rxq_fl0.buf_size: 2048 dev.em.0.iflib.rxq0.rxq_fl0.buf_size: 2048 dev.em.0.iflib.txq1.mbuf_defrag_failed: 0 dev.em.0.iflib.txq1.mbuf_defrag: 0 dev.em.0.iflib.txq0.mbuf_defrag_failed: 0 dev.em.0.iflib.txq0.mbuf_defrag: 0Good question about seeing how it's linked though! em doesn't appear to report that. I'd try setting fc to 0 and see if that changes anything.
-
OK, I found some interesting things. Turns out old Intel datasheets have really thorough descriptions of what all these counters mean. Intel 82583V datasheet
Defer Count: This register counts defer events. A defer event occurs when the transmitter cannot immediately send a packet due to the medium being busy either because:
• Another device is transmitting
• The IPG timer has not expired
• Half-duplex deferral events
• Reception of XOFF frames
• The link is not up
This register only increments if transmits are enabled. The behavior of this counter is slightly different in the 82583V relative to previous devices. For the 82583V, this counter does not increment for streaming transmits that are deferred due to TX IPG.Receive No Buffers Count: This register counts the number of times that frames were received when there were no available buffers in host memory to store those frames (receive descriptor head and tail pointers were equal). The packet is still received if there is space in the FIFO. This register only increments if receives are enabled. This register does not increment when flow control packets are received.
Missed Packets Count: Counts the number of missed packets. Packets are missed when the receive FIFO has insufficient space to store the incoming packet. This could be caused because of too few buffers allocated, or because there is insufficient bandwidth on the IO bus. Events setting this counter cause RXO, the receiver overrun interrupt, to be set. This register does not increment if receives are not enabled. Note that these packets are also counted in the Total Packets Received register as well as in the Total Octets Received register.
-
So Defers are specifically transmits and won't ever be Missed packets. And Recv_No_Buff is the NIC has a frame, but the CPU has no buffer for it. Recv_no_buff can become missed_packets if the situation persists long enough.
I reconfigured things. Instead of shoving everything into em.0 in a router on a stick config, I changed em.0 to be the only WAN and all the LAN VLANs on em.1. This was weird as I can literally just watch the recv_no_buff incrementing on em.0, but em.1 which is seeing the same number of packets is not having any trouble.
Then I reconfigured things again, this time put the WAN on em.5 and the LAN VLANs on em.4. This setup has no immediate issues, but in the past it slowly accumulates errors just the same.
My brother's stats confuse me. Defers seem like they should mostly not happen on a full-duplex link, but maybe this is older hardware that increments this when XOFF is received. His defers count is very similar to the XOFF received count. More confusing is that he's getting missed packets, but without recv_no_buff. That doesn't seem like it should be possible.
-
Hmm, interesting. Are those all the exact same NIC chip? Different PCIe bus maybe?
-
@stephenw10 em.0 is a 82574L, but em.1-5 are 82583V's. According to pciconf I think they all have a direct x1 lane to the CPU.
My brother is using an old HP 4 port nic with 82571 chips. I'm starting to think the older 82571 just doesn't have the recv_no_buff register.
-
Actually I just found a bit in the 2014 errata update for the 82571EB that might explain my brother's missed packets. It might just be old crap.
- Missed RX Packets
Problem:
When the device operates with multiple-requests or Large Send enabled, there could be receive packet loss. When the Tx FIFO is full, the Tx flow may block the host DMA interface of the device. When the transmission of packets is prevented for a long time, due to capture effect or very long backoff in half-duplex, the transmit FIFO is filled and the fetch of Rx descriptors is prevented also. This will prevent the release of the packets from the Rx FIFO to the host, causing the Rx buffer to overflow and the loss of incoming packets. This is a temporary state that will be released once the transmit side is be able to empty the Tx packet buffer.Implication:
There could be some packet loss in the Rx path if the transmission of packets is prevented for a long time. Normally, if this occurs, these packets will be re-transmitted by upper-layer protocols.Workaround:
None -
Hmm the 82574 is extremely common. I would have expected that to work more reliably if anything. But there is a difference there so whatever is causing a problem on em0 the 82583V apparently doesn't suffer from it.
-
Of the 2 options I have on this box it's supposed to be the "better" one. It has MSI-X and dual tx/rx queues to the 82583V's MSI and single tx/rx queue.
Also, I definitely had the em.5/WAN em.4/LAN setup in the past and it would miss packets over time, but this time it's all good.
Only I've reconfigured this so many times and it's never worked as well as it finally is. Computers man, what the hell.
