[HELP WITH NEW SETUP] Setup new server on XEN/UBUNTU with pfSence as main router

  • Hello, (moderators if I am in the wrong place feel free to move this to proper spot).

    I am not a Linux guy, but if pushed to I try to figure out the solution.
    I just got a off-lease  server and want to make it a XEN  with Ubuntu Dom0 based VM lab.

    with pfSence VM to replace my hardware router.

    the hardware is good

    Motherboard: H8DME-2
    Procs: 2 AMD Opteron Hex Core 2431 @ 2.4Ghz for total of 12 cores
    RAM: 40GB DDR2 PC-5300 @ 667mHz
    HDD: 1TB sata drive for OS and VM store
    option to add 20 HDD with 3 SAS cards (have the cards, no drives yet.)
    2 built in NICs @ 1000

    Netgear 24 port Gigabit unmanaged switch

    what I need is some help from someone who know Linux networking and Xen especially.
    let say my nics are such:
    eth0 – on board
    eth1 -- on board
    eth2 -- intell
    eth3 -- intell

    I am thinking to keep the onboard  NICs for Xen and internal network.
    in config 
    LAN==>  eth0 --> xen internal  :: connect server to LAN for Xen internet conectivity and management (let say IP

    LAN==>eth1 --> bridge for VLAN :: connect server to LAN to provide network access to all VMs (let say IP

    and use the Intel for pfsence WAN as
    WAN==>eth2-->pfsenceVM -->eth3==>LAN
    :: connect WAN from my cable provider to NIC for pfSence
    than let pfs do the routing/nat/firewall whatever
    and output it to LAN

    how would I go in do such configuration?
    should I use Xen openswitch or bridged connection?

    is this config crazy and I am crazy to even thinking of trying this?

  • Let me make a diagram to make sure I got everything right.
    There are a couple of ways to go about it wiith xen, you can either use PCI passthrough for eth2 and eth3 and they will ONLY be available to pfSense or you can use the pv drivers and use xenbr0 (eth2) and xenbr1 (eth3) with the interfaces.

     This a setup with just virtual drivers. Note: eth2 and vif0.0 are part of xenbr0 and so on.
    			Dom0				pfSense VM
    	┌───────────────┐	┌────────────┐
    	│					│	│				 │
    	│	[eth2]↔[vif0.0] <──────────>	[xn0] <──────> WAN
    	│					│	│				 │
    	│					│	│				 │
    	│	[eth3]↔[vif1.0] <──────────>	[xn1] <──────> LAN (Switch)
    	│					│	│				 │		↑
    	│					│	└────────────┘		│
    	│					│							│
    	│	[eth0]	<────────────────────────────┤
    	│					│							│    
    	│		   ┌>[vif2.0]<┐	│							│
    	│	[eth1]<┼>[vif2.1]<┼───────────────────────┘
    	│		   └>[vif2.X]<┘	│
    eth0 provides Dom0 access to LAN
    eth1 via xenbr2 provides DomUs access to LAN

    This is actually a pretty simple set up as you don't need any crazy bridges or anything.

    If you are using the setup above then just have the following config for pfSense VM
    I would use mac address of old network cards that you are not using at least for the WAN interface because unless you are using a router for WAN you might run into trouble using xen generated MAC with your ISP.

    When setting up /etc/network/interfaces you should (and probably have to) use an ip less bridge at least for xenbr0.

    auto xenbr0
    iface xenbr0 inet manual
            pre-up brctl addbr xenbr0
            up ip link set xenbr0 up
            bridge_waitport 0
            bridge_fd 0
            bridge_ports eth2
            down ip link set xenbr0 down
            post-down brctl delbr xenbr0
    auto eth2
    iface eth2 inet manual

    As for the DomUs a simple vif=['bridge=xenbr2']

    You can alternatively also set xenbr2 to a static IP and it can be used both by the Dom0 and by DomUs. This way you have an extra interface for whatever else you fancy. In the Dom0 I would recommend only using static IP because when you restart the Dom0, your pfSense will not be up and running for DHCP and Dom0 might steal the WAN ip if you are running dhclient.

    Personally we use pci passthrough for the WAN interface because I don't really trust xenstore to be very secure when given a WAN side bridge and a separate interface/subnet for the Dom0. We have been using a similiar setup for the last year and a half with no problems. Even added another pfSense VM for CARP.

  • yes , the main plan is to use PCI passthrough.
    however,  I am not sure my hardware supports it, so trying to make a contingency plan :-)

    also, except for (eth2)WAN port and VM interface port(eth1), all of my ports will have static IPs
    I like my server(s) to be on static ip for easy SSH/RDM thus eth0 will be static
    eth3 will belong to pfSence
    and eth0 is Xen.

    I am assuming pfSence have all the bells and whistles of normal router and more
    so I can run DHCP on it as well as NAT if need arise.

  • vl1969 Im pretty certain others on another forum said pci passthru will not work with the hardware, however I am currently looking to get the same setup you have so please do let me and others know how your setup works out and what you did so that others can search and find this useful information.

  • @JamesBondJr:

    vl1969 Im pretty certain others on another forum said pci passthru will not work with the hardware, however I am currently looking to get the same setup you have so please do let me and others know how your setup works out and what you did so that others can search and find this useful information.

    hi JamesbondJr.
    well #1 it might not be 100% true " pci passthru will not work with the hardware"
    it might not work in ESXi but I heard that it works some what with Hyper-V
    can not confirm it but there are rumors about some one able to make it work on this hardware
    with Hyper-V. so I still have some hope.

    unfortunately I am still in process, as in not even close yet :-(
    I am trying to setup the latest,  Ubuntu 13.04 with Xen 4.3 (there are several help posts on the thread below that help with compiling the lates kernel for both ubuntu and xen.)
    I have been at it for several days now, have some issues (most of them are do to my own fault )
    but as of yesterday I have this setup running Custom built Ubuntu 13.10.5 kernel
    with Xen 4.3 .

    also, not sure about Intel chassis, but AMD BIOS have several configs that need to be enabled.
    #1 there is an IOMMU option in the north bridge section I think, that I have set to 128M
    and there is an SVM(secure virtual mode) under CPU config that need to be enabled.

    should not be long now, or so I hope.

    you can follow the process and get some help here
    my main post on that forum is "http://lime-technology.com/forum/index.php?topic=28671.0"
    here I will put all of my mods and progress. as well as links to most helpful threads.

    and the help thread is here  "http://lime-technology.com/forum/index.php?topic=28665.msg258096#msg258096"
    which you will also find links to other help posts/threads

  • Just an FYI here.
    it is confirmed that a PCI pass-through is possible on
    AMD based chassis from TAMs based on supermicro H8DME - 2 MB
    using Ubuntu Server 12.04 and Xen 4.2

    also keep in mind that Ubuntu server 13.04 with Xen 4.3 is a no-go
    it might be something in the kernel I am not sure but I was able to get things working on 12.04 but not on 13.04
    since I am still playing with the setup I might try getting things work on 13.04+Xen 4.3  , but not sure about this yet…

    ESXi is a no-go
    XenServer 6.2 is a  no-go

    there are some suggestions that an ArchLinux can work but as a noob I can not /will not go there :-)

Log in to reply