HYPERVISOR performance testing
-
Thank you SO MUCH for this thread, it is extremely useful for me! I've been running ipfire under kvm as an alternative to pfsense due to the horrendous performance virtio in pfsense/bsd used to have a few years ago. I look forward to switching back to running pfsense now that it seems kvm/virtio support under pfsense is finally able to push gigabit. I noticed you gave it 8 cores to achieve that, I really hope it doesn't actually need all 8?
Hopefully performance continues to improve, as virtio under ipfire is able to saturate gigabit easily with a single core on ancient nehalem era hardware. I'm going to be throwing part of an e5-2680v3 at pfsense this time around…
-
Hopefully performance continues to improve, as virtio under ipfire is able to saturate gigabit easily with a single core on ancient nehalem era hardware. I'm going to be throwing part of an e5-2680v3 at pfsense this time around…
obtaining gigabit line speed did not take all 8 cores, tho it did take more than one would like, 4-6, native linux firewall appears faster in a linux hypervisor, there is alot more going on in these tests im doing than 1 giant download, it is 40+ downloads that attempt to go as fast as they can, so states do get more excited than a couple browser downloads, the cpus only gets near maxed out when doing high packet per second thruput testing.
i just finished doing some testing today with 2 centos hypervisors, 2 pfsense 2.2.4 guest running in a carp failover. they are working better than ive ever seen before using virtio! no kpanics or anything!
the backup was about 100mbit slower (850mbit or so), i did not install adm-tune on the centos hypervisor, that may be the reason why.
i tested a linux firewall guest on this setup also, heres the downside of a linux firewall. the centos router guest got 930mbit, but i could not get it to go anyfaster with the bonding, even tho i added 2 more intertfaces to the guest to try and send traffic out.
the centos router was getting 222megabytes from vlan on its wan(which it saw as a 1g interface), but could only send 118megabytes no matterwhat i tried bond 4,5 etc. the test wasnt comparable to the pfsense tests as i had the linux firewalld off, and i did not turn on ipmasq(nat) (pfsense with pfctl -d) pf off, runs crazy fast too, its really a shame there isnt a way to get it running that fast with the firewall on!
pfsense negotiates its virtio interfaces at 10gigbit even if you only have 2 bonded 1g nics, because linux firewalls are "sortofatthispoint" para-virtualized in its drivers, they perform WAY faster, but they give you 1g even if you bonded bridge, so the native linux driver support is a blessing its also a curse, unless of course you have 10g hardware across the board. then linux firewall might be a faster option for you in a linux hypervisor.
im glad to see pfsense 2.2 is doing virtio flawlessly without kpanics, now we need performance!
-
@jstar1:
@jstar1:
[just forget about ms all together as a production env os.
[/quote]you do that in your reality, while the rest of us are stuck in this reality ;)
hey, ive got my share of prod windows servers like everyone else, everyday is another opportunity for me phase them out / move user interaction away from them ;)
ill be over here in my nice soft padded reality, just remember, microsoft wants to be an ASP and grab market share, every time you pay them for software, you are paying your competitor to allow you to compete with them, if you arent providing products that would suggest a conflict of interest, at a minimum you are stuck supporting a monopoly.
i found a Hyper-V Server 2012 R2 Evaluations | Unlimited, i might give that a try if i get some freetime, although it sounds like a major waste of time
http://www.microsoft.com/en-us/evalcenter/evaluate-hyper-v-server-2012-r2?i=1
https://technet.microsoft.com/en-us/library/dn792027.aspx
claims 2012hyperv supports freebsd, might have some interesting test results
apparently hyper-v has no UI to speak of, and to manage it remotely i would need a 2012 install with hyper-v mmc along with a ton of other nonsense i read about here.
http://pc-addicts.com/12-steps-to-remotely-manage-hyper-v-server-2012-core/i think ill give up on testing this nightmare for now.
Or you do the managment the easy way :)
download the free edition of 5nine manager from their web http://www.5nine.com/products.aspx and install it directly on the hyper-v host -
You shouldn't need a 3rd party tool to make it easy to manage sorry- that's not cool.
-
another update on this front, related to 10gb chelsio cards.
we put Chelsio-T520-SO-CR into our c2758 test lab, baremetal performance was amazing, we actually couldnt push the connection in the test lab past 2gb due to the limits on our clients, although the hping floods were handled amazingly well, only minor bursts of 100-300ms ping spikes, rather than majors fireballs like we saw before on bonded 1gb. on the baremetal setup we were pushing 2gb in/2gbs out the interrupts were at 33%.
we then attempted the same tests in centos 7 with kvm and virtio,
in a bridged setup the traffic would not go over 1gb, ethtool showed the link up at 10gbit, so did the switch. under full iperf load, the pfsense was at 6.6 out of 8 load. so it appears we were mostly cpu maxed.attempted to adjust the txqueuelen of the hypervisor 5000 and 10000, it did make the connection go about 5% faster.
attempted to assign the chelsio cards directly to the guest but dont support vt-d. so it doesnt work.
looks like cheslio+c2758 is baremetal is really the best option.
while bonding the built in intel 1g nics works similarly well in both centos 7 and baremetal
-
hi, just stumbled on this topic as I'm having trooble achieving the same performances on similar hardware
Hypervisor : Proxmox (KVM) on a Atom C2750 supermicro board (A1SAM), 4 intel GB nic.
VM : up-to-date pfsense 2.2.5 with 4 cores assigned, 2 virtio NIC with 4 queues enabled on both interfaces, 2048Gb memory
MTU is 1500 everywheretarget iperf cmdline : iperf3 -s
input iperf cmdline : iperf3 -c 192.168.50.10 -P 20 -t 30Direct switching (without passing through pfsense nor the hypervisor at all)
INPUT IPERF –-> switch ---> TARGET iperf
941 Mbits/sec
=> input iperf, target iperf, switch and cables are able to sustain 941mbps.4 core, 4queues, through pfsense with pf disabled (pfctl -d)
INPUT IPERF –-> switch ---> pfsenseNic0 ---> pfsenseNic1 ---> TARGET iperf
935-941Mbits/s
nearly 100% interrupts on 2 cpu cores
no significative system load on cpu (somewhat 2-5%)
=> the two nics and virtualised pfsense are able to sustain nearly the same bandwitdth (a bit less actually, maybe some kvm overhead ?)4 core, 4queues, through pfsense with pf enabled (pfctl -e)
751-811Mbits/s
100% interrupt on 1 cpu core, 75% interrupts on 1 cpu core
30 + 20% system cpu load on the 2 last cores- why am I missing 130mbps on this last test
- why do I "only" have 75% interupt load on the second core (considering I had 100% on 2 cores during previous test) ?
I already tried giving 8 cores to the vm, either with 4 or 8 virtio queues, doesn't change anything.
also tried playing with numa/taskset in order to lock the kvm process to the same 4 coresAny idea ?
[edit] pfsense baremetal perf: 941mbps.
-
i know a few people tried my setup with other flavors of linux and had similar issues to what you are seeing, i would consider retesting with centos 7.1 or later and see if that resolves it, your interrupts should be maxed out during your tests.
the tunable that i mentioned for centos may not exist in prox, ive never used prox.
see if you can force all your cores to max performance, and tune the operating system sysctls to be a hypervisor like i mentioned in prior posts
see my prior posts about setting up centos to see if you can apply similar tunables to prox.
-
the c2758 i am using says it uses VMDQ for its network controllers? does your motherboard support this?
Network Controllers
C2000 SoC I354 Quad GbE Controller (MACs)
Virtual Machine Device Queues reduce I/O overhead
Supports 10BASE-T, 100BASE-TX, and 1000BASE-T, RJ45 outputhttp://www.supermicro.com/products/motherboard/Atom/X10/A1SRi-2758F.cfm
i think theres a good chance if your only limited with the pfctl -e you are maxing out your cores or atleast maxing out what prox will give you for cores, try to disable any limiters in prox or try centos?
-
The c2750 (and my supermicro A1SAM motherboard) also provides VMDQ.
Tunning sysctl according to centos/rhel tuned-adm made me gain only a few mbps.
If your test lab is still up and running, can you please post your kvm invocation line ? I'd like to compare it to the one generated by proxmox
-
What a great thread, I hope to setup a pfsense system one day and i will probably just over build and go with a hypervisor setup like you have done here.
I will perform some similar tests but will probably only report if there are differences to your findings.
Will be my first time messing with CentOS, I will probably start with esxi and hyper-v as those are what i am most familiar with.