Traffic stops

Jonb

I swapped from RL nics to ET and ever since I have been having traffic flow stopping or full kernal panics.

On further debugging I have found the following:

et0: DMA error(0x00000010)
et0: WARNING!! ring 0, buf_idx 74, rr_idx 55

To try and debug the issue I have edited /boot/loader.conf.local

kern.ipc.nmbclusters="131072"
hw.et.num_queues=1
hw.pci.enable_msix=0

I have also turned off:
hardware checksum offload
hardware TCP segmentation offload
hardware large receive offload

I am running v2.1 buld.

Could anyone point me in the right direction.

Jonb

looking at the source for the nic I guess I am triggering

2128 rx_ring = &sc->sc_rx_ring[ring_idx];
if (buf_idx != rx_ring->rr_index) {
if_printf(ifp,
"WARNING!! ring %d, buf_idx %d, rr_idx %d\n",
ring_idx, buf_idx, rx_ring->rr_index);
}

I am guessing this is a driver issue but does anyone know how to resolve this.

Jonb

Would this get solved in 2.2 release?

stephenw10

Unlikely if it's a driver issue. There doesn't seem to have been much work on the et(4) driver since 8.3 was released (2 years ago):
http://svnweb.freebsd.org/base/head/sys/dev/et/if_et.c?view=log
Nothing that looks like it might fix the issue you're having.
Though some of it was sponsored by Netflix so if they're using them you can bet there's no ongoing issues when combined with whatever hardware they're using.

When does that DMA error appear?

Steve

Jonb

It appears when pushing under load.

stephenw10

Hmm, so no errors at boot then?

Steve

Jonb

No nothing at boot and if there is no load it runs for days. As soon as you push 20Mb or so it goes, so when I say load more then tick over.

stephenw10

You could try booting 'verbose' to see if any more clues are given.
It looks like you've already tried evrything I might suggest. :-\

Steve

Jonb

Bummer. Guess only thing left is hack the driver :(

stephenw10

Since you first asked about this it looks like some dma related fixes have gone into the driver. Probably worth trying though it might only compile against FreeBSD 10.

Steve

Jonb

sweet so we should be good on v2.2

stephenw10

Maybe. 2.2 is tracking 10-stable (or was until recently at least) so any bug fixes should be included. Only testing will tell if they address your issue though. ;)

Steve

Jonb

Thanks will update

stephenw10

Unfortunately it doesn't look like the patch made it into 10.1, and hence pfSense 2.2. In fact it's not in 10-stable either only in head:
https://svnweb.freebsd.org/base/head/sys/dev/et/if_et.c?view=log

You might have to compile it yourself if you want to try it.

Steve

Jonb

Your right because after testing I didn't have any joys. I am tempted to move it to ESXi anyway which would solve this. A 1GHz via just isn't cutting it anymore now I am running squid and dansgardian.