Help needed with performance tuning with Verizon FIOS..
-
Some basic info. about the setup
-
I am paying for a 20Mbit/5Mbit connection with Verizon.
-
The ONT is configured to use only the ethernet port (ONT -> pfSense -> LAN)
-
There is a short (1 meter) run of CAT5 between the ONT and the pfSense box.
-
The LAN connection feed into a 3COM switch, CAT5 for distribution network.
-
The NICs are set to auto configure. They both latch at 100bTX full duplex.
-
The pfSense box is configured to Grub multiboot; pfSense or Ubuntu 10.x or Windows 7
The problem is stated simply as; When we directly connect the ONT to a PC while it's not running pfSense we can use either the Verizon sppedtest or speedtest.net and get pretty close to the 20Mbits down and 5Mbits up speeds. When we reboot into the pfSense partition we get about 5-6Mbit/4Mbit performance of both of those tests.
We aren't seeing many errors at the NICs..so we are wondering if there is some sort of default overhead in the pfSense kernel or module that is causing us this problem. Any ideas?
Netstat data;
netstat -I msk0
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
msk0 1500 <link#2> 00:1d:ba:8e:83:87 39986 0 0 40213 0 0
msk0 1500 fe80:2::21d:b fe80:2::21d:baff: 0 - - 0 - -
msk0 1500 96.233.38.0 pool-96-233-38-12 3373 - - 418 - -[2.0.2-RELEASE][admin@pfSense.localdomain]/root(3): netstat -I ue0
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
ue0 1500 <link#9> 00:10:60:dd:cc:0a 45306 1952 0 40413 0 0
ue0 1500 fe80:9::210:6 fe80:9::210:60ff: 0 - - 1 - -
ue0 1500 192.168.1.0 pfSense 6407 - - 3981 - -ifconfig data;
ue0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=80000 <linkstate>ether 00:10:60:dd:cc:0a
inet6 fe80::210:60ff:fedd:cc0a%ue0 prefixlen 64 scopeid 0x9
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
status: activemsk0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=c001a <txcsum,vlan_mtu,vlan_hwtagging,vlan_hwtso,linkstate>ether 00:1d:ba:8e:83:87
inet6 fe80::21d:baff:fe8e:8387%msk0 prefixlen 64 scopeid 0x2
inet 96.233.38.12 netmask 0xffffff00 broadcast 96.233.38.255
nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex,flag0,flag1>)
status: active</full-duplex,flag0,flag1></performnud,accept_rtadv></txcsum,vlan_mtu,vlan_hwtagging,vlan_hwtso,linkstate></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></linkstate></up,broadcast,running,simplex,multicast></link#9></link#2> -
-
If ue0 is a 12Mbps USB device that will severely limit throughput.
-
We aren't seeing many errors at the NICs..
[2.0.2-RELEASE][admin@pfSense.localdomain]/root(3): netstat -I ue0
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
ue0 1500 <link#9>00:10:60:dd:cc:0a 45306 1952 0 40413 0 0</link#9>I disagree. You seem to be showing quite a lot of errors on ue0. Stats from one of my NICs here:
[2.0.2-RELEASE][root@pfsense.fire.box]/root(2): netstat -I em1 Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll em1 1500 <link#2>00:90:7f:32:4f:ee 6090864 0 0 8029860 0 0 em1 1500 192.168.2.0 192.168.2.1 60062 - - 710736 - - em1 1500 fe80:2::290:7 fe80:2::290:7fff: 0 - - 0 - -</link#2>USB NICs are almost always bad performers.
Steve
-
Hmm..I didn't really think that was a large enough qty for concern.. Ok Stephen, Thanks!
We chose that particular device due to the fact it was on the supported list:
http://www.freebsd.org/releases/8.1R/hardware.html#ETHERNETI've just learned that "supported" doesn't mean the same thing to me as it does to pfSense. ;-)
I don't mind changing NICs, I just wish I didn't have to play trial & error with my wallet.
Any ideas on where I can get a listing of truly supported NIC's and some performance stats from users? It'd be a pretty handy list to maintain on these forums I would think..given so many postings expressing the same frustrations.
-
I'm sorry about that. :(
Had you asked I'm sure people would have advised you not get a USB NIC. I know there are people using them with no problems though so you could just be unlucky. You could try a different USB port, some older hardware has a mix or USB1 and USB2 ports. I really wouldn't expect any errors on a wired NIC, pehaps a few if the cable is removed. Try a different cable.
Here are the stats for my wifi NIC where there are many more errors, to be expected on wifi in a congested area:[2.0.2-RELEASE][root@pfsense.fire.box]/root(1): netstat -I ath0 Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll ath0 2290 <link#4>00:11:f5:ee:42:9c 3829362 285387 0 4331435 6 0</link#4>In general Intel NICs are best supported under FreeBSD and hence pfSense. What is the rest of your hardware?
Steve
-
Any ideas on where I can get a listing of truly supported NIC's and some performance stats from users? It'd be a pretty handy list to maintain on these forums I would think..given so many postings expressing the same frustrations.
What are your constraints? What expansion ports do you have available?
What do mean by truly supported? You haven't responded to my point about USB speed so the following example may not be relevant. "Supported" can never mean "performs the impossible" such as sucking 20Mbps through a 12Mbps pipe.
-
Stephen -
Meh..no need to be sorry, just trying to find the right fit for our network and firewall. DD-WRT is "ok", but their cheap NICs built into ~$75 routers don't hold up well. And whatever eeprom devices they use for image storage don't seem to have very many read cycles in them either. I have a small pile of them to prove it.
Most of my pfSense hardware has been pieced together from sundry spare parts, an old motherboard for example. I will try several PCI Intel cards and one PCI-X card that I can borrow from a friend on Monday. I was simply giving what I had easily accessible on hand a try. If the Intel cards work out I may either buy them from him or search out the same controllers online.
It'd still be good to know what cards routinely have performed the best for people in their builds and what kind of throughput they are pushing through their cards. A moderated wiki-fied matrix might be useful..just a thought. It would certainly be a great resource for people trying pfSense for the first time..and probably avoid a lot of "what's up with my performance?" type questions to the forums. :-)
-
There is this which basically says what I did:
http://doc.pfsense.org/index.php/Hardware_requirements#Network_Card_Selection
The hardware examples given there are a bit outdated now though.Perhaps unsurprisingly a number of people have suggested some sort of wiki. It's tough to get firm numbers without a single person (or group) doing some identical testing. No one has volunteered to do that yet. ;)
Steve
-
Hi Steve,
So here's some validation for you.. ;-)
While I was watching the Patriots cream the Texans my friend with the NIC cards stopped by. Since he lives nearby I agreed to share the ribs I made in trade for the Intel cards he had. I was just joking..but he was off and 10 mins later brought the cards to me. A somewhat different result with the PCI and PCI-X cards..
em0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether 00:1d:ba:8e:83:87
inet6 fe80::21b:21ff:fe6c:2eb%em0 prefixlen 64 scopeid 0x1
inet 96.233.38.12 netmask 0xffffff00 broadcast 96.233.38.255
nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
em2: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 00:1b:21:6e:aa:5f
inet6 fe80::21b:21ff:fe6e:aa5f%em2 prefixlen 64 scopeid 0x2
inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
nd6 options=43 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
status: activenetstat -I em0
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
em0 1500 <link#1> 00:1d:ba:8e:83:87 20671 0 0 11975 0 0
em0 1500 fe80:1::21b:2 fe80:1::21b:21ff: 0 - - 1 - -
em0 1500 96.233.38.0 pool-96-233-38-12 973 - - 623 - -netstat -I em2
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
em2 1500 <link#2> 00:1b:21:6e:aa:5f 23312 0 0 29764 0 0
em2 1500 fe80:2::21b:2 fe80:2::21b:21ff: 0 - - 1 - -
em2 1500 192.168.1.0 pfSense 10492 - - 8101 - -The speedtest.net aren't exactly 20M/5M..but it's a heck of a lot better now.
</link#2></link#1></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,simplex,multicast> -
Good result.
I hope you gave him some ribs! ;)Steve
-
Steve,
Sure did..he ate them up like there was no tomorrow.
Invited himself over for the Ravens v Patriots game too..so long as I make more ribs. :-)
So how do I go about getting the pfSense guys to host that wiki we were talking about..Ideas?
-Warren
-
I would suggest the first thing is to agree on a test procedure that will give a useful result. It has to be relatively easy to test to get a good number of people producing numbers. It also has to give results that are meaningful to users (not 'I can get 800Mbps if I use jumbo frames across a 4 NIC LAGG').
Then setup a forum thread where people can post results. Finally ask for edit rights on the wiki and setup a page describing the above. ;)A good example of some great testing is here: http://forum.pfsense.org/index.php/topic,27780.0.html
Speak to Jim (jimp) and/or Chris (cmb) who will have a better idea about this than me!
Steve
