New Alix board for 2013

GruensFroeschli

I have somewhat different results:


    10.0.8.3                10.0.8.1 (LAN - re0)
    ----------              ----------
    |   PC1  |--------------|  APU   |
    ----------              ----------
                                 |   10.0.42.237 (WAN - re2)
                                 |
                                 |
                          -----------
                          |   PC2   | 10.0.42.2
                          -----------

pfSense 2.1
portforward from WAN to PC1
50 concurrent iperf connections from PC1 to PC2
50 concurrent iperf connections from PC2 to PC1
Quite constant 440~445 Mbit/s (see quotes below)

@PC1:

TCP window size: 85.3 KByte (default)
[SUM] 2230.0-2240.0 sec 259 MBytes 217 Mbits/sec
[SUM] 2240.0-2250.0 sec 262 MBytes 219 Mbits/sec
[SUM] 2250.0-2260.0 sec 257 MBytes 216 Mbits/sec
[SUM] 2260.0-2270.0 sec 259 MBytes 217 Mbits/sec
[SUM] 2270.0-2280.0 sec 257 MBytes 215 Mbits/sec
[SUM] 2280.0-2290.0 sec 260 MBytes 219 Mbits/sec
[SUM] 2290.0-2300.0 sec 256 MBytes 215 Mbits/sec
[SUM] 2300.0-2310.0 sec 258 MBytes 217 Mbits/sec
[SUM] 2310.0-2320.0 sec 259 MBytes 217 Mbits/sec
[SUM] 2320.0-2330.0 sec 259 MBytes 217 Mbits/sec
[SUM] 2330.0-2340.0 sec 260 MBytes 218 Mbits/sec
[SUM] 2340.0-2350.0 sec 260 MBytes 218 Mbits/sec
[SUM] 2350.0-2360.0 sec 259 MBytes 217 Mbits/sec
[SUM] 2360.0-2370.0 sec 262 MBytes 219 Mbits/sec
[SUM] 2370.0-2380.0 sec 257 MBytes 216 Mbits/sec
[SUM] 2380.0-2390.0 sec 260 MBytes 218 Mbits/sec
[SUM] 2390.0-2400.0 sec 260 MBytes 218 Mbits/sec
[SUM] 2400.0-2410.0 sec 258 MBytes 217 Mbits/sec
[SUM] 2410.0-2420.0 sec 260 MBytes 218 Mbits/sec
[SUM] 2420.0-2430.0 sec 263 MBytes 220 Mbits/sec

@PC2:

TCP window size: 85.3 KByte (default)
[SUM] 2230.0-2240.0 sec 271 MBytes 227 Mbits/sec
[SUM] 2240.0-2250.0 sec 270 MBytes 226 Mbits/sec
[SUM] 2250.0-2260.0 sec 270 MBytes 227 Mbits/sec
[SUM] 2260.0-2270.0 sec 268 MBytes 225 Mbits/sec
[SUM] 2270.0-2280.0 sec 268 MBytes 225 Mbits/sec
[SUM] 2280.0-2290.0 sec 269 MBytes 226 Mbits/sec
[SUM] 2290.0-2300.0 sec 270 MBytes 226 Mbits/sec
[SUM] 2300.0-2310.0 sec 270 MBytes 226 Mbits/sec
[SUM] 2310.0-2320.0 sec 271 MBytes 227 Mbits/sec
[SUM] 2320.0-2330.0 sec 269 MBytes 226 Mbits/sec
[SUM] 2330.0-2340.0 sec 273 MBytes 229 Mbits/sec
[SUM] 2340.0-2350.0 sec 271 MBytes 227 Mbits/sec
[SUM] 2350.0-2360.0 sec 268 MBytes 225 Mbits/sec
[SUM] 2360.0-2370.0 sec 268 MBytes 225 Mbits/sec
[SUM] 2370.0-2380.0 sec 270 MBytes 227 Mbits/sec
[SUM] 2380.0-2390.0 sec 270 MBytes 226 Mbits/sec
[SUM] 2390.0-2400.0 sec 271 MBytes 227 Mbits/sec
[SUM] 2400.0-2410.0 sec 269 MBytes 226 Mbits/sec
[SUM] 2410.0-2420.0 sec 269 MBytes 225 Mbits/sec
[SUM] 2420.0-2430.0 sec 269 MBytes 226 Mbits/sec

GruensFroeschli

After pushing traffic for 3 hours i took a look at the board with our thermal camera.

IR000609: From top, closed case.

IR000611: From bottom, closed case.

IR000613: From top, open case.

IR000609.JPG_thumb

IR000611.JPG_thumb

IR000613.JPG_thumb

watercooled

Some great images there, cheers!

What colour case do you have BTW (I'm thinking about emissivity)?

GruensFroeschli

It's the black case.

forbiddenlake

@tirsojrp:

MiniPCI Express not just MiniPCI

Use Google

Oops :-[

Google tells me there are no Mini-PCIe crypto accelerators, so I guess it's moot anyway.

tmueko

@GruensFroeschli:

I have somewhat different results:


    10.0.8.3                10.0.8.1 (LAN - em0)
    ----------              ----------
    |   PC1  |--------------|  APU   |
    ----------              ----------
                                 |   10.0.42.237 (WAN - em2)
                                 |
                                 |
                          -----------
                          |   PC2   | 10.0.42.2
                          -----------

You em0 .. em2 on your APU? Is that the 4GB Device? Can you post a dmesg from boot?

stephenw10

Nice, love the thermal images. :)

You surely mean re0-1 though rather than em?

Steve

GruensFroeschli

Wooops.
Yes i meant re.
I have a dual miniPCIe gbit card which is (em).
Going to test the board with them tomorrow.

gkri

I have received my board today :)

I will be using this board to upgrade an Alix system.

The fit of the board in the Alix case is no good :(
EDIT: Netgate told me that my case is an old Netgate made case (not PCEngine), and do not work with APU board :(

The board is curved and it's impossible to align the leds an reset button with the case holes.
EDIT: Netgate told me that my case is an old Netgate made case (not PCEngine), and do not work with APU board :(

I hope Netgate get the new cases soon.
(I just order 2 correct cases)

GruensFroeschli

One noteworthy point of the test i did.
During the test the board is not responsive anymore.
E.g:

serial console is dead
webinterface doesn't load anymore

As soon as the test-traffic is disabled, everything works immediately again.

When i stopped the test i saw lots of these messages in the syslog:

ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=21459663
ad4: WARNING - WRITE_DMA requeued due to channel reset LBA=21459599
ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=21459515
ad4: WARNING - WRITE_DMA requeued due to channel reset LBA=21459663
ata2: FAILURE - already active DMA on this device
ata2: setting up DMA failed
ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=21459523
g_vfs_done():ad4s1a[WRITE(offset=10987307008, length=32768)]error = 5
ad4: WARNING - WRITE_DMA requeued due to channel reset LBA=21459515
ata2: FAILURE - already active DMA on this device
ata2: setting up DMA failed
ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=2145953g5
_vfs_done():ad4s1a[WRITE(offset=10987231232, length=2048)]error = 5
ad4: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=21459523
ad4: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=21459535
ad4: FAILURE - WRITE_DMA timed out LBA=21459535
g_vfs_done():ad4s1a[WRITE(offset=10987241472, length=32768)]error = 5
ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=1152271
.
.
.
etc.

i guess this are failed writes of the RRD graph (or is there anything else writing regularly to the disk?).
Without investigating this further for now, my bet is that the realtek cards generate an interrupt storm that stops the rest of the system from working correctly.
Let's see how everything behaves with the intel cards :)

GruensFroeschli

Ok tests done.

Setup same as before, but this time with em instead of re.
Used card: http://www.computex.biz/cervoz/default.aspx?com_id=6417&pdt_id=57237&PageType=ProductDetail&ContentTab=Introduction
(It seems this card is no longer available and there is a successor: http://www.cervoz.com/products/products_04.php?id=1815 )

Quite constant traffic around 660~680 Mbit/s (compared to the 440Mbit/s of the onboard interfaces)
Same unresponsiveness of the board under load
Same DMA errors

@PC1:

[SUM] 1400.0-1410.0 sec 404 MBytes 339 Mbits/sec
[SUM] 1410.0-1420.0 sec 399 MBytes 334 Mbits/sec
[SUM] 1420.0-1430.0 sec 405 MBytes 340 Mbits/sec
[SUM] 1430.0-1440.0 sec 400 MBytes 335 Mbits/sec
[SUM] 1440.0-1450.0 sec 409 MBytes 343 Mbits/sec
[SUM] 1450.0-1460.0 sec 394 MBytes 331 Mbits/sec
[SUM] 1460.0-1470.0 sec 397 MBytes 333 Mbits/sec
[SUM] 1470.0-1480.0 sec 399 MBytes 335 Mbits/sec
[SUM] 1480.0-1490.0 sec 397 MBytes 333 Mbits/sec
[SUM] 1490.0-1500.0 sec 406 MBytes 341 Mbits/sec
[SUM] 1500.0-1510.0 sec 416 MBytes 349 Mbits/sec
[SUM] 1510.0-1520.0 sec 416 MBytes 349 Mbits/sec
[SUM] 1520.0-1530.0 sec 419 MBytes 351 Mbits/sec
[SUM] 1530.0-1540.0 sec 419 MBytes 352 Mbits/sec
[SUM] 1540.0-1550.0 sec 418 MBytes 351 Mbits/sec
[SUM] 1550.0-1560.0 sec 397 MBytes 333 Mbits/sec
[SUM] 1560.0-1570.0 sec 403 MBytes 338 Mbits/sec
[SUM] 1570.0-1580.0 sec 399 MBytes 335 Mbits/sec
[SUM] 1580.0-1590.0 sec 405 MBytes 340 Mbits/sec
[SUM] 1590.0-1600.0 sec 407 MBytes 342 Mbits/sec

@PC2:

[SUM] 1400.0-1410.0 sec 400 MBytes 336 Mbits/sec
[SUM] 1410.0-1420.0 sec 394 MBytes 330 Mbits/sec
[SUM] 1420.0-1430.0 sec 399 MBytes 335 Mbits/sec
[SUM] 1430.0-1440.0 sec 392 MBytes 329 Mbits/sec
[SUM] 1440.0-1450.0 sec 401 MBytes 336 Mbits/sec
[SUM] 1450.0-1460.0 sec 386 MBytes 324 Mbits/sec
[SUM] 1460.0-1470.0 sec 396 MBytes 332 Mbits/sec
[SUM] 1470.0-1480.0 sec 404 MBytes 339 Mbits/sec
[SUM] 1480.0-1490.0 sec 390 MBytes 327 Mbits/sec
[SUM] 1490.0-1500.0 sec 397 MBytes 333 Mbits/sec
[SUM] 1500.0-1510.0 sec 399 MBytes 335 Mbits/sec
[SUM] 1510.0-1520.0 sec 395 MBytes 331 Mbits/sec
[SUM] 1520.0-1530.0 sec 395 MBytes 331 Mbits/sec
[SUM] 1530.0-1540.0 sec 395 MBytes 331 Mbits/sec
[SUM] 1540.0-1550.0 sec 396 MBytes 332 Mbits/sec
[SUM] 1550.0-1560.0 sec 391 MBytes 328 Mbits/sec
[SUM] 1560.0-1570.0 sec 403 MBytes 338 Mbits/sec
[SUM] 1570.0-1580.0 sec 396 MBytes 332 Mbits/sec
[SUM] 1580.0-1590.0 sec 399 MBytes 334 Mbits/sec
[SUM] 1590.0-1600.0 sec 397 MBytes 333 Mbits/sec

tohil

@tohil:

@tmueko:

@tohil:

hi

i tried to install the embedded version on a 4GB Kingston SD Card. but it doesn't fit, so i used the 2GB nanobsd edition.
but when i try to boot, i got errors about mounting the root parition, wrong geometry aborting startup

can i fix this by this
https://doc.pfsense.org/index.php/Soekris_6501_USB_Flash_installation#Root_Mount_error

or the USB Boot Problem article?

Yes, that are the steps, you have to go, but 'set kern.cam.boot_delay="5000"' will be enough.

Thanks for your reply. Will this maybe added to the nanobsd image as default? also the automatic mapping of the new APU interfaces re0 re1 re2?

I used now the normal installer from a usb stick. i installed the embedded kernel and set a ramdisk for /tmp and /var in the webconfigurator. is this enough, that my SD card will not die in a few weeks or have i better to reinstall the whole boxes with the embedded edition?

regards angelo

Can somebody give me an answer on this?

regards

jimp

@GruensFroeschli:

When i stopped the test i saw lots of these messages in the syslog:

ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=21459663
ad4: WARNING - WRITE_DMA requeued due to channel reset LBA=21459599
ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=21459515
ad4: WARNING - WRITE_DMA requeued due to channel reset LBA=21459663
ata2: FAILURE - already active DMA on this device
ata2: setting up DMA failed
ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=21459523
g_vfs_done():ad4s1a[WRITE(offset=10987307008, length=32768)]error = 5
ad4: WARNING - WRITE_DMA requeued due to channel reset LBA=21459515
ata2: FAILURE - already active DMA on this device
ata2: setting up DMA failed
ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=2145953g5
_vfs_done():ad4s1a[WRITE(offset=10987231232, length=2048)]error = 5
ad4: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=21459523
ad4: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=21459535
ad4: FAILURE - WRITE_DMA timed out LBA=21459535
g_vfs_done():ad4s1a[WRITE(offset=10987241472, length=32768)]error = 5
ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=1152271
.
.
.
etc.

i guess this are failed writes of the RRD graph (or is there anything else writing regularly to the disk?).

Are you running a plain NanoBSD install on there? That really sounds like what happens when you need to make these changes:
https://doc.pfsense.org/index.php/Boot_Troubleshooting#NanoBSD_on_Newer_Hardware

@tohil:

Thanks for your reply. Will this maybe added to the nanobsd image as default? also the automatic mapping of the new APU interfaces re0 re1 re2?

Unlikely, as it isn't needed for most. Not sure on interface mappings, but the general default may eventually change, though I'm not sure if the APU interfaces would be the new default or em0/em1 as those are increasingly more common.

@tohil:

I used now the normal installer from a usb stick. i installed the embedded kernel and set a ramdisk for /tmp and /var in the webconfigurator. is this enough, that my SD card will not die in a few weeks or have i better to reinstall the whole boxes with the embedded edition?

If you have a RAM disk setup for /var and /tmp using a full install should be OK on there so long as you aren't going crazy with packages that would write to the disk.

I'm running a full install on an mSATA in my APU and it's good so far. Not sure I'll bother with NanoBSD.

When Netgate starts shipping them in assembled form, they'll likely be full installs on mSATA not SD.

GruensFroeschli

@jimp:

@GruensFroeschli:

When i stopped the test i saw lots of these messages in the syslog:

ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=21459663
ad4: WARNING - WRITE_DMA requeued due to channel reset LBA=21459599
ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=21459515
ad4: WARNING - WRITE_DMA requeued due to channel reset LBA=21459663
ata2: FAILURE - already active DMA on this device
ata2: setting up DMA failed
ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=21459523
g_vfs_done():ad4s1a[WRITE(offset=10987307008, length=32768)]error = 5
ad4: WARNING - WRITE_DMA requeued due to channel reset LBA=21459515
ata2: FAILURE - already active DMA on this device
ata2: setting up DMA failed
ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=2145953g5
_vfs_done():ad4s1a[WRITE(offset=10987231232, length=2048)]error = 5
ad4: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=21459523
ad4: TIMEOUT - WRITE_DMA retrying (0 retries left) LBA=21459535
ad4: FAILURE - WRITE_DMA timed out LBA=21459535
g_vfs_done():ad4s1a[WRITE(offset=10987241472, length=32768)]error = 5
ad4: TIMEOUT - WRITE_DMA retrying (1 retry left) LBA=1152271
.
.
.
etc.

i guess this are failed writes of the RRD graph (or is there anything else writing regularly to the disk?).

Are you running a plain NanoBSD install on there? That really sounds like what happens when you need to make these changes:
https://doc.pfsense.org/index.php/Boot_Troubleshooting#NanoBSD_on_Newer_Hardware

No this is a memstick install on the SSD.
However with the embedded kernel to get the serial output.
I've added these settings. and am testing again.

One thing i've noticed is that with the intel cards a don't get any "DMA failed" messages anymore.
Only the retrying messages.

jimp

@GruensFroeschli:

No this is a memstick install on the SSD.
However with the embedded kernel to get the serial output.
I've added these settings. and am testing again.

One thing i've noticed is that with the intel cards a don't get any "DMA failed" messages anymore.
Only the retrying messages.

Difference could be that I'm using AHCI (Netgate image) and not the old ATA stack. Seems to work fine for me with the embedded kernel that way.

GruensFroeschli

Tested again with the onboard NICs.
Getting lots of DMA errors under load again.

Which image for netgate do you mean?
I could try with this one too.

For now i'm going to test 2.2 :)

jimp

@GruensFroeschli:

Tested again with the onboard NICs.
Getting lots of DMA errors under load again.

Which image for netgate do you mean?
I could try with this one too.

For now i'm going to test 2.2 :)

It's not a public image, it's the one that will be loaded onto the APU when it's released by Netgate. It's Netgate's 2.1.1 version tailored for the hardware, though some things are still being tweaked for it yet.

2.2 would have AHCI by default (no old ATA stack) so it might not be a bad test, though the current 2.2 snap has a pf rule issue the next one should be better (or pick an older one)

@gkri:

I hope Netgate get the new cases soon.
(I just order 2 correct cases)

The cases in the Netgate store are made by PC Engines.

A couple years back we were making out own (due to supply issues with PC Engines). These fit the Alix boards correctly.

After that, PC Engines revised their case, and it is this revised case that they can get adequate volumes of, and that fits the APU.

@jimp:

@GruensFroeschli:

Tested again with the onboard NICs.
Getting lots of DMA errors under load again.

Which image for netgate do you mean?
I could try with this one too.

For now i'm going to test 2.2 :)

It's not a public image, it's the one that will be loaded onto the APU when it's released by Netgate. It's Netgate's 2.1.1 version tailored for the hardware, though some things are still being tweaked for it yet.

2.2 would have AHCI by default (no old ATA stack) so it might not be a bad test, though the current 2.2 snap has a pf rule issue the next one should be better (or pick an older one)

This "Netgate image" is also the image that will be on APUs sold through the pfSense Store.

The problem is not just AHCI, .vs ATA. There are a lot of variables to get right in order to use that SSD with TRIM support enabled and have a stable system. Patches to the system (which are not in 2.2), patches to the installer (also not yet in 2.2), etc. are all required.

The issue is complex enough (and occurs on other OSes, such as Linux) that PC Engines has pulled the msata16 for now.

theIntangible

Noob question here:

I just got my board yesterday and have tried just about every pfsense build (nano, memstick, i386, etc) but cannot get the device to boot from the usb/sd. All I get is:

booting from Hard Disk…
Booting from 0000:7c00

Then it just sits there indefinitely.

I read that the ALIX required you to boot into the BIOS to change the baud rate to 9600 to load pfsense. I have tried every key combination I could think of and nothing gets me into the BIOS. Can someone point me in the right direction for what to do to get something running?

Thanks!