I have 4 pfsense boxes in our DC running on Supermicro 5015M-MR+ systems. With PDSMi+ Boards.
Intel 3000 chipsets.
These boxes have are rock solid with 1.0.1. When I bring any of them up to 1.2 (Have tested RC1 2 and 3) they wont last 1/2 an hour. The system crashed with a page fault and reboots. The boxes are stable with Freebsd 6.2 on them, under pretty heavy load. I am leaning twords this being caused by a sysctl setting that changed between 1.0.1 and 1.2… That or the SMP kernel.
Anyone have any thoughts on this?
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD 6.1-RELEASE-p10 #0: Sun Oct 29 01:06:20 UTC 2006
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz (2394.01-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
AMD Features=0x20100000 <nx,lm>AMD Features2=0x1 <lahf>Cores per package: 2
real memory = 1072562176 (1022 MB)
avail memory = 1040416768 (992 MB)
ACPI APIC Table: <ptltd ="" apic ="">FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0 <version 2.0="">irqs 0-23 on motherboard
ioapic1 <version 2.0="">irqs 24-47 on motherboard
wlan: mac acl policy registered
kbd1 at kbdmux0
ath_hal: 0.9.16.16 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <ptltd rsdt="">on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <acpi cpu="">on acpi0
cpu1: <acpi cpu="">on acpi0
pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus="">on pcib0
pcib1: <acpi pci-pci="" bridge="">irq 16 at device 1.0 on pci0
pci1: <acpi pci="" bus="">on pcib1
pcib2: <acpi pci-pci="" bridge="">irq 17 at device 28.0 on pci0
pci9: <acpi pci="" bus="">on pcib2
pcib3: <acpi pci-pci="" bridge="">at device 0.0 on pci9
pci10: <acpi pci="" bus="">on pcib3
em0: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 3.2.18="">port 0x4000-0x403f mem 0xe0240000-0xe025ffff,0xe0200000-0xe023ffff irq 24 at device 1.0 on pci10
em0: Ethernet address: 00:1b:21:08:38:26
em1: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 3.2.18="">port 0x4040-0x407f mem 0xe0260000-0xe027ffff,0xe0280000-0xe02bffff irq 25 at device 1.1 on pci10
em1: Ethernet address: 00:1b:21:08:38:27
pci9: <base peripheral,="" interrupt="" controller=""> at device 0.1 (no driver attached)
pcib4: <acpi pci-pci="" bridge="">irq 17 at device 28.4 on pci0
pci13: <acpi pci="" bus="">on pcib4
em2: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 3.2.18="">port 0x5000-0x501f mem 0xe0300000-0xe031ffff irq 16 at device 0.0 on pci13
em2: Ethernet address: 00:30:48:8e:df:fa
pcib5: <acpi pci-pci="" bridge="">irq 16 at device 28.5 on pci0
pci14: <acpi pci="" bus="">on pcib5
em3: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 3.2.18="">port 0x6000-0x601f mem 0xe0400000-0xe041ffff irq 17 at device 0.0 on pci14
em3: Ethernet address: 00:30:48:8e:df:fb
pcib6: <acpi pci-pci="" bridge="">at device 30.0 on pci0
pci15: <acpi pci="" bus="">on pcib6
pci15: <display, vga="">at device 0.0 (no driver attached)
isab0: <pci-isa bridge="">at device 31.0 on pci0
isa0: <isa bus="">on isab0
atapci0: <intel ich7="" sata300="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x3030-0x303f at device 31.2 on pci0
atapci0: failed to enable memory mapping!
ata0: <ata 0="" channel="">on atapci0
ata1: <ata 1="" channel="">on atapci0
pci0: <serial bus,="" smbus="">at device 31.3 (no driver attached)
acpi_button0: <power button="">on acpi0
speaker0: <pc speaker="">port 0x61 on acpi0
pmtimer0 on isa0
orm0: <isa option="" roms="">at iomem 0xc0000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0
atkbdc0: <keyboard controller="" (i8042)="">at port 0x60,0x64 on isa0
atkbd0: <at keyboard="">irq 1 on atkbdc0
kbd0 at atkbd0
ppc0: parallel port not found.
sc0: <system console="">at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250 or not responding
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
Fast IPsec: Initialized Security Association Processing.
ad0: 76319MB <seagate st380815as="" 3.aad="">at ata0-master SATA150
acd0: CDROM <cd-224e-n 1.aa="">at ata1-master UDMA33
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted
WARNING: R/W mount of / denied. Filesystem is not clean - run fsck
WARNING: R/W mount of / denied. Filesystem is not clean - run fsck
em2: promiscuous mode enabled
em1: promiscuous mode enabled
pflog0: promiscuous mode enabled
em1: link state changed to UP
em3: link state changed to UP
===================================================================</cd-224e-n></seagate></generic></system></at></keyboard></isa></pc></power></serial></ata></ata></intel></isa></pci-isa></display,></acpi></acpi></intel(r)></acpi></acpi></intel(r)></acpi></acpi></intel(r)></intel(r)></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></ptltd ></version></version></ptltd ></lahf></nx,lm></b15></b14></sse3,rsvd2,mon,ds_cpl,vmx,est,tm2,<b9></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>
Highly unlikely it's a sysctl difference. A stock FreeBSD 6.2 install runs perfectly fine on this exact same hardware? Could be one of our kernel patches or differences.
It also appears that 1.3 is stable as well. (Although I notice the that load surges a bit.)
I have a 100K PPS udp flow going through a 1.3 box on this hardware. It has been running for 2 days now and it is stable. But the CPU is bouncing between 45% and 90% utilization.
It seams there is something specific about the 1.2 series that this hardware doesn't like.
RC3 installed from CD seams to be stable. But installing RC2 and upgrading to RC3 is NOT stable.
But 1.2 with the current RC3 ISO has been up for over a week with 800Mb/s and 65,000 states.
We are running this on 4 identical systems.
Supermicro barebones system "5015M-MR+"
Intel Core2Duo 6600
1G (2x512M) Crucial DDR2 ECC.
Intel 2port PRO 1000MT PCI-X card
This combo consistantly gets us over 1Gb/s combined throughput with almost any combo of packets.
We have these fronting web clusters in our facility. I have set up FTP via this post:
Overall a good system. We have in the area of 60 Pix 515Es. We are replacing them with this combo.
Very much hope someone can get the functionality I am running via the above post into the 1.2 final release.
I am interested in your setup. Couple of questions
The on-board NICS did not work for you with pfSense? If they are not compatible, can you post the part number for your Intel NIC.
Any reason you used the 6600 over a Xeon processor?
The onboard NICs work great. I needed 4 GigE nics. Only 2 on the board so I needed 2 more.
I use the Intel Pro1000MT 2 port NIC as the add on.
I used the 6600 over the Xeon due to price. They are a good deal cheaper and I can still push over 1Gbps. It met my goals and was the least costly way to achieve them in what I thought would be a reliable build.
I have several of these in production at the moment and am VERY happy. 1.2RC3 is very happy with them (But do not upgrade to it, Upgrades to RC3 are unstable but installs from the ISO have been rock solid.)
In my heaviest loaded config:
With 1G of Ram I have over 60K states and 1 interface for pfsync and 3 interfaces passing traffic.
1 is WAN and internaly I have 2 interfaces customer facing. I have 9 tagged vlan interfaces at the moment and regularly burst up above 800Mb/s.
1 customer has over 1000 Lotus notes and Domino connections, 1 has about 10Mbps of SIP traffic. Most of the actual bandwidth is going to several Web hosts.
Overall I have been very happy,
For CPU cooling did you use the CPU heatsink supplied with the case? if yes, was it sufficient?
Yes, I have been using the ones supplied. They have worked fine so far.
I bench tested this setup in our office. The temp was ~74'. I ran it there for about 2 weeks prior to deployment. So temp doesn't seam to be an issue.