Supermicro PDSMi+



  • I have 4 pfsense boxes in our DC running on Supermicro 5015M-MR+ systems. With PDSMi+ Boards.
    Intel 3000 chipsets.
    C2D 2.4Ghz
    1G ram

    These boxes have are rock solid with 1.0.1.  When I bring any of them up to 1.2 (Have tested RC1 2 and 3) they wont last 1/2 an hour. The system crashed with a page fault and reboots. The boxes are stable with Freebsd 6.2 on them, under pretty heavy load. I am leaning twords this being caused by a sysctl setting that changed between 1.0.1 and 1.2… That or the SMP kernel.

    Anyone have any thoughts on this?

    ================================================

    Copyright (c) 1992-2006 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD 6.1-RELEASE-p10 #0: Sun Oct 29 01:06:20 UTC 2006
        sullrich@builder.livebsd.com:/usr/obj.pfSense/usr/src/sys/pfSense.6
    Timecounter "i8254" frequency 1193182 Hz quality 0
    CPU: Intel(R) Core(TM)2 CPU          6600  @ 2.40GHz (2394.01-MHz 686-class CPU)
      Origin = "GenuineIntel"  Id = 0x6f6  Stepping = 6
      Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0xe3bd<sse3,rsvd2,mon,ds_cpl,vmx,est,tm2,<b9>,CX16,<b14>,<b15>>
      AMD Features=0x20100000 <nx,lm>AMD Features2=0x1 <lahf>Cores per package: 2
    real memory  = 1072562176 (1022 MB)
    avail memory = 1040416768 (992 MB)
    ACPI APIC Table: <ptltd   ="" apic ="">FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
    cpu0 (BSP): APIC ID:  0
    cpu1 (AP): APIC ID:  1
    ioapic0 <version 2.0="">irqs 0-23 on motherboard
    ioapic1 <version 2.0="">irqs 24-47 on motherboard
    wlan: mac acl policy registered
    kbd1 at kbdmux0
    ath_hal: 0.9.16.16 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
    acpi0: <ptltd  rsdt="">on motherboard
    acpi0: Power Button (fixed)
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
    cpu0: <acpi cpu="">on acpi0
    cpu1: <acpi cpu="">on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    pcib1: <acpi pci-pci="" bridge="">irq 16 at device 1.0 on pci0
    pci1: <acpi pci="" bus="">on pcib1
    pcib2: <acpi pci-pci="" bridge="">irq 17 at device 28.0 on pci0
    pci9: <acpi pci="" bus="">on pcib2
    pcib3: <acpi pci-pci="" bridge="">at device 0.0 on pci9
    pci10: <acpi pci="" bus="">on pcib3
    em0: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 3.2.18="">port 0x4000-0x403f mem 0xe0240000-0xe025ffff,0xe0200000-0xe023ffff irq 24 at device 1.0 on pci10
    em0: Ethernet address: 00:1b:21:08:38:26
    em1: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 3.2.18="">port 0x4040-0x407f mem 0xe0260000-0xe027ffff,0xe0280000-0xe02bffff irq 25 at device 1.1 on pci10
    em1: Ethernet address: 00:1b:21:08:38:27
    pci9: <base peripheral,="" interrupt="" controller=""> at device 0.1 (no driver attached)
    pcib4: <acpi pci-pci="" bridge="">irq 17 at device 28.4 on pci0
    pci13: <acpi pci="" bus="">on pcib4
    em2: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 3.2.18="">port 0x5000-0x501f mem 0xe0300000-0xe031ffff irq 16 at device 0.0 on pci13
    em2: Ethernet address: 00:30:48:8e:df:fa
    pcib5: <acpi pci-pci="" bridge="">irq 16 at device 28.5 on pci0
    pci14: <acpi pci="" bus="">on pcib5
    em3: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 3.2.18="">port 0x6000-0x601f mem 0xe0400000-0xe041ffff irq 17 at device 0.0 on pci14
    em3: Ethernet address: 00:30:48:8e:df:fb
    pcib6: <acpi pci-pci="" bridge="">at device 30.0 on pci0
    pci15: <acpi pci="" bus="">on pcib6
    pci15: <display, vga="">at device 0.0 (no driver attached)
    isab0: <pci-isa bridge="">at device 31.0 on pci0
    isa0: <isa bus="">on isab0
    atapci0: <intel ich7="" sata300="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x3030-0x303f at device 31.2 on pci0
    atapci0: failed to enable memory mapping!
    ata0: <ata 0="" channel="">on atapci0
    ata1: <ata 1="" channel="">on atapci0
    pci0: <serial bus,="" smbus="">at device 31.3 (no driver attached)
    acpi_button0: <power button="">on acpi0
    speaker0: <pc speaker="">port 0x61 on acpi0
    pmtimer0 on isa0
    orm0: <isa option="" roms="">at iomem 0xc0000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0
    atkbdc0: <keyboard controller="" (i8042)="">at port 0x60,0x64 on isa0
    atkbd0: <at keyboard="">irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    ppc0: parallel port not found.
    sc0: <system console="">at flags 0x100 on isa0
    sc0: VGA <16 virtual consoles, flags=0x300>
    sio0: configured irq 4 not in bitmap of probed irqs 0
    sio0: port may not be enabled
    sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
    sio0: type 8250 or not responding
    sio1: configured irq 3 not in bitmap of probed irqs 0
    sio1: port may not be enabled
    vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    Timecounters tick every 1.000 msec
    Fast IPsec: Initialized Security Association Processing.
    ad0: 76319MB <seagate st380815as="" 3.aad="">at ata0-master SATA150
    acd0: CDROM <cd-224e-n 1.aa="">at ata1-master UDMA33
    SMP: AP CPU #1 Launched!
    Trying to mount root from ufs:/dev/ad0s1a
    WARNING: / was not properly dismounted
    WARNING: R/W mount of / denied.  Filesystem is not clean - run fsck
    WARNING: R/W mount of / denied.  Filesystem is not clean - run fsck
    em2: promiscuous mode enabled
    em1: promiscuous mode enabled
    pflog0: promiscuous mode enabled
    em1: link state changed to UP
    em3: link state changed to UP

    ===================================================================</cd-224e-n></seagate></generic></system></at></keyboard></isa></pc></power></serial></ata></ata></intel></isa></pci-isa></display,></acpi></acpi></intel(r)></acpi></acpi></intel(r)></acpi></acpi></intel(r)></intel(r)></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></ptltd ></version></version></ptltd ></lahf></nx,lm></b15></b14></sse3,rsvd2,mon,ds_cpl,vmx,est,tm2,<b9></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>



  • Highly unlikely it's a sysctl difference. A stock FreeBSD 6.2 install runs perfectly fine on this exact same hardware? Could be one of our kernel patches or differences.



  • It also appears that 1.3 is stable as well. (Although I notice the that load surges a bit.)

    I have a 100K PPS udp flow going through a 1.3 box on this hardware. It has been running for 2 days now and it is stable. But the CPU is bouncing between 45% and 90% utilization.
    It seams there is something specific about the 1.2 series that this hardware doesn't like.



  • OK,
    RC3 installed from CD seams to be stable. But installing RC2 and upgrading to RC3 is NOT stable.

    But 1.2 with the current RC3 ISO has been up for over a week with 800Mb/s and 65,000 states.

    We are running this on 4 identical systems.

    Supermicro barebones system "5015M-MR+"
    Intel Core2Duo 6600
    1G (2x512M) Crucial DDR2 ECC.
    Intel 2port PRO 1000MT PCI-X card

    This combo consistantly gets us over 1Gb/s combined throughput with almost any combo of packets.
    We have these fronting web clusters in our facility. I have set up FTP via this post:
    http://forum.pfsense.org/index.php/topic,6861.0.html

    Overall a good system. We have in the area of 60 Pix 515Es. We are replacing them with this combo.
    Very much hope someone can get the functionality I am running via the above post into the 1.2 final release.



  • vantage,

    I am interested in your setup. Couple of questions
    The on-board NICS did not work for you with pfSense? If they are not compatible, can you post the part number for your Intel NIC.

    Any reason you used the 6600 over a Xeon processor?

    Thanks,

    Issac



  • The onboard NICs work great. I needed 4 GigE nics. Only 2 on the board so I needed 2 more.
    I use the Intel Pro1000MT 2 port NIC as the add on.

    I used the 6600 over the Xeon due to price. They are a good deal cheaper and I can still push over 1Gbps. It met my goals and was the least costly way to achieve them in what I thought would be a reliable build.
    I have several of these in production at the moment and am VERY happy. 1.2RC3 is very happy with them (But do not upgrade to it, Upgrades to RC3 are unstable but installs from the ISO have been rock solid.)

    In my heaviest loaded config:
    With 1G of Ram I have over 60K states and 1 interface for pfsync and 3 interfaces passing traffic.
    1 is WAN and internaly I have 2 interfaces customer facing. I have 9 tagged vlan interfaces at the moment and regularly burst up above 800Mb/s.

    1 customer has over 1000 Lotus notes and Domino connections, 1 has about 10Mbps of SIP traffic. Most of the actual bandwidth is going to several Web hosts.

    Overall I have been very happy,



  • vantage,

    For CPU cooling did you use the CPU heatsink supplied with the case? if yes, was it sufficient?



  • Yes, I have been using the ones supplied. They have worked fine so far.
    I bench tested this setup in our office. The temp was ~74'. I ran it there for about 2 weeks prior to deployment. So temp doesn't seam to be an issue.


Log in to reply