Pfsense with XEN



  • Does pfsense run with Xen as DomU (Guest), does it need any modification to the kernel or anything like that? sullrich wrote some day that maybe with 6.2 (which pfsense later snapshots are based on) will support it, so does anyone have some information on that?



  • FreeBSD 6.2 does not have xen support as of yet.  It's not possible unfortunately without kernel modifications, etc.  And honestly I have never even touched Xen so I wouldn't be of much help.



  • Okay i see. Is there any way to (para)virtualize pfsense and make it run besides other operation systems on the same hardware? Its just that the idea of having a firewall running with dedicated nics on the same machine as some server services do is pretty amazing. But i dont want to switch to ipcop or some other linux based firewall for that.



  • @Dortje:

    Okay i see. Is there any way to (para)virtualize pfsense and make it run besides other operation systems on the same hardware? Its just that the idea of having a firewall running with dedicated nics on the same machine as some server services do is pretty amazing. But i dont want to switch to ipcop or some other linux based firewall for that.

    vmware, parallels, xen on modern hardware that supports virtualization in the processor, ms virtual pc (although there are comments that it doesn't work quite so well)

    –Bill



  • I have tried running FreeBSD 7 as a domU on top of a Fedora Core dom0.  Theoretically, pfSense can be run as a domU using a xenofreebsd kernel.  As it is still experimental, we can't expect it to be stable enough for production use.

    AFAIK, xenofreebsd only runs on i386; there's no amd64 port yet.  For more details, kindly check http://www.fsmware.com/.

    [ simon.cpu ]



  • pfsense and bsd distros don't work on hvm full virtualization domains, the BTX loader fails booting from virtual cdrom (device or iso) or virutal hard disk; howover i tested that it works fine on a standar qemu on a normal machine, i'm still testing methos for make them boot properly or import an hard disk image with a different boot manager, i'll make you know if i make any advance.

    Regards.



  • Great to hear. But qemu wouldnt be of any advantage compared to VMware, right? When it comes to performance they should be equal as far as i know. XEN would be much better. I will also try to get more into this this weekend.



  • @Psymon:

    pfsense and bsd distros don't work on hvm full virtualization domains, the BTX loader fails booting from virtual cdrom (device or iso) or virutal hard disk; howover i tested that it works fine on a standar qemu on a normal machine, i'm still testing methos for make them boot properly or import an hard disk image with a different boot manager, i'll make you know if i make any advance.

    Regards.

    I have googled for info on HVM, and it seems that some people have success with it.  My amd64 workstation boxes don't support SVM though, so I'll have to wait for our test servers to become available before I can experiment… :)

    [ simon.cpu ]



  • Okay, want to give a small update for this Topic.
    According to a previos post which states that FreeBSD does not run on HVM Domains I disagree. Right now i finished installing a Pfsense-1.2RC2 installation in HVM Domain. The networdcars are reported es ed0 and ed1. Will keep you updated. Of course you need an VT oder Pacifica enabled CPU.
    Cheers



  • Any details on how you did the install?
    Was it a straight forward install like any other HVM domain, or did you have to pull any tricks to get it to install?
    Did you pass the network cards directly to the HVM domain, or are they virtualized nics?



  • Install:
    created Xen-HVM Domain. The importand thing here is, that you first two Network Bridges. These Bridges are then passed to the HVM Domain.
    Then install the HVM Domain from a downloaded ISO File.
    Right now it is not possible to pass directly the NICs to HVM-Domain it works only with PVM if the Kernel in PVM-Domain supports the PCI-Backhide Function.  But what you can do is, that you assign in youd Dom-0 the appropriate Nics exclusive to the Bridges. Oh before I forgot, right now there is the limitation of 3 virtual Interfaces per DomU.
    Hope this Helps
    Regards Marco



  • That's what I was thinking… Just wanted to ask to make sure I was on the right path.
    I am getting a quad nic card in the next few days, and will be trying this out. I'm running xen 3.1, so should have no problem with 4 network interfaces.

    As for the limitation of 3 virtual network interfaces, you should upgrade to XEN 3.1. The limit has been increased to 8 network interfaces.
    http://wiki.xensource.com/xenwiki/XenFaq#head-9896478cf65a16f43ab4fb066f74c0e0d67a16ac



  • I was also very interested in this setup (pfSense is just great!), unfortunally I do get the BTX error that psymon already mentioned. Since I did read someone never finished vmxassist on the Intel, my question is what hardware platform are you running (AMD I guess?) a HVM FreeBSD?

    • Joris


  • Try checking "Use grub" on the installer bootblocks screen during installation.



  • I don't get the option to do anything. I cannot seem to get the ISO file started.
    Can anyone help me on a disk image that uses the grub bootloader? Is vmware the way to go and transfer the disk image?
    Already many thanks in advance…



  • I don't know about the other guys, but my server is an Athlon64 X2 4000+ with 3 gigs ram running a SuSE10.3 Dom0, with Xen 3.1.0_15042
    I chose "Other" as my environment in the setup wizard ('virt-manager'), and was able to install pfSense as a HVM (full virtualization) without a problem. No error messages came up, and the iso booted and installed on the virtual HD just fine. I created 2 virtual network cards (both of which were bridged to the since real nic) and assigned them as WAN and LAN without problems.
    I now have pfSense running and am able to play with it.

    What version of XEN are you running? What dom0 OS? What architecture (Intel I presume?)

    @Joris:

    I was also very interested in this setup (pfSense is just great!), unfortunally I do get the BTX error that psymon already mentioned. Since I did read someone never finished vmxassist on the Intel, my question is what hardware platform are you running (AMD I guess?) a HVM FreeBSD?

    • Joris


  • BTW…

    Intel and AMD doesn´t have the same architecture....

    Intel designed VT in a strange way, such that only protected mode instructions are allowed or be virtualized.
    Anything done in real mode must be emulated...every instruction. The Xen and KVM folks have emulated enough to get things working, but have not handled every instruction, including apparently some of the fancy (VBE?) graphics isolinux (Ubuntu CD´s for example), and in my tests FreeBSD's btx loader also has problems with Intel VT.

    For reference, AMD's SVM analogue does, in fact, virtualize real mode instructions on the processor, and I'm able to boot all install CD's just fine on an AMD machine. Maybe, this is an Intel VT problem.....



  • My system is an Intel Core 2 Duo E6750 with 8 GB, with Gentoo 2007.0 running Xen 3.1.2, all AMD64 software.
    Indeed, I think the differences between AMD SVM and Intel VT-x are the main problem in my case. Unfortunally most of the world has problems with FreeBSD on Xen (on Intels I believe). It seems that the emulation is called vmxassist (or vmx_assist) and was broken at some point, but the author is not really interested into fixing that any more (probably has other priorities).

    Hence I like to give Grub a try, but unfortunally the ISO files use the BTX loader and I cannot get them running. I'll try to get it installed on vmware with grub and hope to get it working. If I have some success, I will let you know…



  • Hi all,

    After a long search for Linux compatible hardware I ordered a new system for my new CentOS / XEN server. It is Intel based (P35 chipset with ICH9R and Core 2 Duo E6750 processor). I chose this because I needed good support for my SATA drives (fast access for Mail & File server in guest domains). In another guest domain pfsense was planned.

    From this tread I make up that Intel is not the right hardware voor a XEN server. I still have a change to cancel or change my order. Can anyone who got pfsense running in a guest domain tell me their hardware configuration. The components I am interesting in are mainly the motherboard and CPU. Moreover I like to know if the onboard SATA controller, graphics card and nic where recognized by Linux.

    Hope to hear from you, I like to have a good XEN server with a cool firewall for the comming 3 years.
    Joost.



  • I have the same processor and it seems Intels have broken support for some HVM emulation. Windows runs fine, nevertheless. I use Linux with paravirtualization only.
    I bought a system with a G33 (onboard video) and ICH-9R. I use Linux RAID instead of Intel's, but I put the controller in ACHI mode. This allows hotplugging of disks. My board (gigabyte) has 8 sata ports. I did need a recent linux version, like 2.6.20, to get my hard disks recognized (hence Gentoo). I believe Debian Etch has a recent version in backports but did not try. Don't know about RHEL/CentOS or SUSe.
    I never got FreeBSD to run on the box, unfortunally. vmWare on Xen is also out of the question. Didn't get qemu to compile too. Trying to get FreeBSD running with Grub at this point…



  • Since I've had no problems with Xen yet, I can tell you my exact configuration:
    Mobo: Asus M2N-E
    CPU: Athlon64 X2 4000+

    The on-board nic, sound card, usb, and SATA controler were detected by SuSE10.3 with no issues (SuSE 10.1 and 10.2 also detected everything with no problems)
    There were no issues with detecting ANY of the hardware in dom0 setup.
    The Mobo is using the nForce chipset, which is well supported in linux, which I persume is why everything worked out of the box.

    hope that helps.

    @jhavers:

    Hi all,

    From this tread I make up that Intel is not the right hardware voor a XEN server. I still have a change to cancel or change my order. Can anyone who got pfsense running in a guest domain tell me their hardware configuration. The components I am interesting in are mainly the motherboard and CPU. Moreover I like to know if the onboard SATA controller, graphics card and nic where recognized by Linux.

    Hope to hear from you, I like to have a good XEN server with a cool firewall for the comming 3 years.
    Joost.



  • Hi I a running Xen 3.1 on Debian Etch.
    3GB Ram, AMD Athlon64 X2 BE-2350 EE; MSI K9AG Neo2-Digital RS690G
    There reason why I am running only 3Gb of Ram is because there are some Problems with the 690Chipset an 64Bit Addressing. Therefore I have chosen the safe path.
    I could install Pfsense in HVM without any Problems.

    Regarding the 3Network Interfaces,  seems that I had some old INfos.
    Thanks for the updated Information!



  • Hi, thanks for the reactions.

    I just removed the Intel hardware from my order and replaced it by the following AMD hardware:
    Asus Moederbord M2N-SLI Deluxe
    AMD Athlon 64 X2 5200+ 65 Watt
    Asus VGA GeForce EN7200GS/HTD 128 MB

    Its a slower configuration (with less headroom to expand) than the Intel hardware, but since virtualisation has to work with Linux, FreeBSD and Windows guests I see no other way. I will report back on my findings when I get everything…

    Regards, Joost.



  • One short comment on your hardware: If you are going for a pure server, remove the video card and take on-board. It doesn't give any performance hit and saves money on purchase and on your power bill (arround 20+ Watts for those cards).
    Its the reason I selected a G33 chipset, but I'm a little disapointed with getting FreeBSD on Xen running.



  • Joris, thanks for your comment.

    I really would like to have onboard video, because I don't care for fancy graphics. But the problem is that the Nvidia nforce 570 SLI MCP chipset works well with AMD and Linux, but doesn't have onboard video. If you have another suggestion for an AMD and Linux compatible chipset, I really like to know. I didn't know a simple videocard like that consumes 20W.

    Regards, Joost.



  • It seems grub only allows chainloading the FreeBSD, so it seems BTX is the only way to go. So choose AMD if you desire to run FreeBSD on top of Xen.

    @Joost: If your from Holland (name suggests that), look at www.alternate.nl. Their site is very good and makes it quite easy to find an board that suits you needs. But maybe you don't get the peripherals & chipset you desire as a nice single package. If you look for power consumption, toms hardware includes this in their benchmarks.



  • I think most if not all nforce chipsets work well with linux, since nvidia is quite supportive of the linux movement, and they even provide their own nforce driver.
    Anyway, as for a 20W video card, why don't you look at getting an old PCI video card. Some 2D card like a Mtrox millenium or something like that… Those things couldn't have sucked up very much power.
    Just a thought.

    @jhavers:

    Joris, thanks for your comment.

    I really would like to have onboard video, because I don't care for fancy graphics. But the problem is that the Nvidia nforce 570 SLI MCP chipset works well with AMD and Linux, but doesn't have onboard video. If you have another suggestion for an AMD and Linux compatible chipset, I really like to know. I didn't know a simple videocard like that consumes 20W.

    Regards, Joost.



  • Hi all,

    Last week I got the harware for my new server:
    Asus Moederbord M2N-SLI Deluxe
    AMD Athlon 64 X2 5200+ 65 Watt
    Asus VGA GeForce EN7200GS/HTD 128 MB
    Samsung Spinpoint F1 750 GB (RAID 1)

    CentOS 5 installed without problems and everything was detected. The write performance on the motherboard/drives is good also (85 MB/s average during mirror rebuild). The only thing is that the power consumption during installation was a nice 80W, but after reboot it constantly stays on 110W. I think it is the XEN kernel that does not allow for frequency scaling. And about the video card's power consumption that isn't to bad either, less than 9W.

    But what is really inportant to this thread: pfsense work on this configuration as an HVM guest!!!

    I just asked the core team to add a howto about Xen and pfsense to their site. When that is possible I will write down the steps to make pfsense working on your (RHEL/CentOS) machine.

    In the mean time I will play around with the guest and try to get it configured properly. Another thing I have to look after is the stability of the pfsense guest. I already ran into an unresponsive pfsense guest twice. But this is not reproducible, and the last 2 days there were no problems. If anybody else has experience with pfsense & XEN stability I really like to know.

    I will report back when the howto is ready or when I find out more about the stability (the later takes a while  :)).

    Regards,
    Joost.



  • As of this past weekend, I am now using pfSense as my main router running in a HVM machine.
    I bought a 4 port NIC, and after a bit of setup (to get the hardware nics assigned as virtual nics for all VM machines to see) all is running smoothly.
    I just have to learn how to use pfSense now.

    I was debating about how I should set the LAN side of things. My two options are:

    • Use the internal bridged interface (which connects all the VM's together) as the LAN port
    • Use one of the ports on the 4nic card as a LAN, and connect that to my switch, to which the server (and all the VMs) are hooked up to through.

    I decided to use the spare nic port as the LAN, since I had the option.
    I don't know if this was the best choice.

    Any thoughts?



  • Hi all,

    After a lot of stability problems, I finally got it right by creating a completely new XEN configuration file. The only thing that the pfsense HVM guest can't handle is when I start copying large (>2GB) files on the host system. Then the guest crashes and I have to destroy it before I can start it again. The weird thing is that my Windows XP HVM guest can handle this and is always stable just like the para-virtuelized guests.

    Outsidre, the best option is to delegate the WAN interface to the pfsense guest alone. This way the WAN interface only connects pfsense to the WAN. HOWEVER, right now this can only be done for para-virtualized guests (so we have to wait for HVM support and use a bridge for now).
    For your LAN interface you can use a bridge. This way you can also connect the other guests on the pfsense host to the LAN. In addition you can (physically) attach a switch and other computers.
    I also have a few questions for you. Which OS is installed on your XEN host? Do you also have problems copying large files?

    Regards,
    Joost.



  • @jhavers:

    I also have a few questions for you. Which OS is installed on your XEN host? Do you also have problems copying large files?

    The dom0 is OpenSuse 10.3.
    The guests are: pfSense (HVM), FreeNAS (HVM), and 2 openSuSE10.3 (para) installs. I have XP installed as an HVM, but don't use it much.

    I have had some problems similar to what you describe. If I do heavy file access + CPU access on the host (dom0) machine, then the system crashes. From what I've been able to find on the net, this is a problem experienced by others as well, and is a xen 3.1 problem.
    I know it's an XEN problem, cuz if I boot the system with the regular kernel (no xen installed) I can do as much file access as I want with no problems at all.
    My workaround this has been to do all my file access over NFS, in another VM. Of course it's slower, but it doesn't crash my system.



  • jhavers You might try vmware instead of xen. Since your only using the full virtualization with the windows and pfsense guest your not taking advantage of the what xen offers over vmware. I've never had an issue running pfsense in vmware. It has been rock solid.

    Keep in mind that you'll need to reboot into the non xen kernel as vmware and xen can't coexist in dom0.



  • Outsidre, to bad you experience the same. My work around is to shutdown the pfsense guest, since that is the only system that is crashing under heavy file access. Therefore I think that it has to do with the combination XEN & pfsense, since my HVM Windows XP guest and the para-virtualized guest are not crashing under heavy file access.

    What i still want to try (when I have the time) is to install pfsense on its own partition instead of in a file. Mayby that makes the difference.

    rsw686, I can't use vmware since I also have 3 para-virtualized Linux guests. Their performance is outstanding, little processor load (1/20 of that of a HVM) and rock solid.



  • @jhavers:

    rsw686, I can't use vmware since I also have 3 para-virtualized Linux guests. Their performance is outstanding, little processor load (1/20 of that of a HVM) and rock solid.

    Gotcha. It really seems like you need two systems to have a rock stable setup. Xen excels with para virtualization and vmware excels with full virtualization. Maybe Xen will fix this issue along with the incompatibilities with Intel VT chips in the later releases.



  • Got this from the http://www.xen.org/

    Welcome to the Xen 3.2 download page! This release contains architectural improvements and new user-visible features including:
        * Xen Security Modules (XSM)
        * ACPI S3 suspend-to-RAM support for the host system
        *** Preliminary PCI pass-through support (using appropriate Intel or AMD I/O-virtualization hardware)**
        * Preliminary support for a wider range of bootloaders in fully virtualized (HVM)guests, using full emulation of x86 real mode
        * Faster emulation of standard (non-super) VGA modes for HVM guests
        * Configurable timer modes for HVM guests, depending on how the guest OS manages time-keeping
        * Many other changes and enhancements across all supported machine architectures

    Now I just wait until Redhat implements Xen 3.2 in RHEL 5 and then I can give pfsense a dedicated WAN interface.

    Joost



  • i have pfsense on xen 3.2 running excellent



  • I need to get PF Sense runn on XEN.  I just install 5.0.0 and can't get it to install yet.  How big is your image?  And would you be will to let me get a copy of a working image from you.

    If not how did you tweak the settings so that it will run.
    RC



  • I am now up and running on XEN.  I have replaced my permanent firewall with a virtual server instance.  It is running really well.  So far everything is working really well.  I have down graded from 1.2.1 to 1.2 production release.

    I am getting ready to shut my virtual server down and make a backup of the instance.  This will let me recover very quickly.  I get the specification of my server out in the forum in the next few days.  I going to setup a 1.3 instance and so I can shut my production instance down and then bring the other up for quick testing.

    XEN offers a great deal of flexibility.

    RC



  • ;D  I now am running on XEN 5.0.0.  Here is the hardware that I am running on.

    AMD Quad Phenom 2.5 processor
    MSN-SLI motherboard
    6 GB of ram
    512 mb SLI video adapter
    Broadcom 1 GB adapter
    On board ethernet adapter
    dual 100 mb Intel ethernet adapter
    3 x 320 GB HD's (non raid array currently)
    550 watt  power supply

    Replacing the following servers:
    DELL SC1500
    DELL SC 440
    COMPAQ DL360
    COMPAQ DL380

    It's a mix of production and test boxes.  For hosting servers and devices that you will access it's great.  It's working awesome.
    RC


Locked