[HELP WITH NEW SETUP] Setup new server on XEN/UBUNTU with pfSence as main router

[vl1969]

Hello, (moderators if I am in the wrong place feel free to move this to proper spot).

I am not a Linux guy, but if pushed to I try to figure out the solution.
I just got a off-lease  server and want to make it a XEN  with Ubuntu Dom0 based VM lab.

with pfSence VM to replace my hardware router.

the hardware is good

Motherboard: H8DME-2
Procs: 2 AMD Opteron Hex Core 2431 @ 2.4Ghz for total of 12 cores
RAM: 40GB DDR2 PC-5300 @ 667mHz
HDD: 1TB sata drive for OS and VM store
option to add 20 HDD with 3 SAS cards (have the cards, no drives yet.)
2 built in NICs @ 1000

Netgear 24 port Gigabit unmanaged switch

what I need is some help from someone who know Linux networking and Xen especially.
let say my nics are such:
eth0 – on board
eth1 -- on board
eth2 -- intell
eth3 -- intell

I am thinking to keep the onboard  NICs for Xen and internal network.
in config 
LAN==>  eth0 --> xen internal  :: connect server to LAN for Xen internet conectivity and management (let say IP 192.168.1.20)

LAN==>eth1 --> bridge for VLAN :: connect server to LAN to provide network access to all VMs (let say IP 192.168.1.21)

and use the Intel for pfsence WAN as
WAN==>eth2-->pfsenceVM -->eth3==>LAN
:: connect WAN from my cable provider to NIC for pfSence
than let pfs do the routing/nat/firewall whatever
and output it to LAN

how would I go in do such configuration?
should I use Xen openswitch or bridged connection?

is this config crazy and I am crazy to even thinking of trying this?

[Sabrewarrior]

Let me make a diagram to make sure I got everything right.
There are a couple of ways to go about it wiith xen, you can either use PCI passthrough for eth2 and eth3 and they will ONLY be available to pfSense or you can use the pv drivers and use xenbr0 (eth2) and xenbr1 (eth3) with the interfaces.

 This a setup with just virtual drivers. Note: eth2 and vif0.0 are part of xenbr0 and so on.
			Dom0				pfSense VM
	┌───────────────┐	┌────────────┐
	│					│	│				 │
	│	[eth2]↔[vif0.0] <──────────>	[xn0] <──────> WAN
	│					│	│				 │
	│					│	│				 │
	│	[eth3]↔[vif1.0] <──────────>	[xn1] <──────> LAN (Switch)
	│					│	│				 │		↑
	│					│	└────────────┘		│
	│					│							│
	│	[eth0]	<────────────────────────────┤
	│					│							│    
	│		   ┌>[vif2.0]<┐	│							│
	│	[eth1]<┼>[vif2.1]<┼───────────────────────┘
	│		   └>[vif2.X]<┘	│
	└───────────────┘
eth0 provides Dom0 access to LAN
eth1 via xenbr2 provides DomUs access to LAN

This is actually a pretty simple set up as you don't need any crazy bridges or anything.

If you are using the setup above then just have the following config for pfSense VM
vif=['bridge=xenbr0,mac=aa:bb:cc:dd:ee:ff','bridge=xenbr1']
I would use mac address of old network cards that you are not using at least for the WAN interface because unless you are using a router for WAN you might run into trouble using xen generated MAC with your ISP.

When setting up /etc/network/interfaces you should (and probably have to) use an ip less bridge at least for xenbr0.

auto xenbr0
iface xenbr0 inet manual
        pre-up brctl addbr xenbr0
        up ip link set xenbr0 up
        bridge_waitport 0
        bridge_fd 0
        bridge_ports eth2
        down ip link set xenbr0 down
        post-down brctl delbr xenbr0

auto eth2
iface eth2 inet manual

As for the DomUs a simple vif=['bridge=xenbr2']

You can alternatively also set xenbr2 to a static IP and it can be used both by the Dom0 and by DomUs. This way you have an extra interface for whatever else you fancy. In the Dom0 I would recommend only using static IP because when you restart the Dom0, your pfSense will not be up and running for DHCP and Dom0 might steal the WAN ip if you are running dhclient.

Personally we use pci passthrough for the WAN interface because I don't really trust xenstore to be very secure when given a WAN side bridge and a separate interface/subnet for the Dom0. We have been using a similiar setup for the last year and a half with no problems. Even added another pfSense VM for CARP.

[vl1969]

yes , the main plan is to use PCI passthrough.
however,  I am not sure my hardware supports it, so trying to make a contingency plan :-)

also, except for (eth2)WAN port and VM interface port(eth1), all of my ports will have static IPs
I like my server(s) to be on static ip for easy SSH/RDM thus eth0 will be static
eth3 will belong to pfSence
and eth0 is Xen.

I am assuming pfSence have all the bells and whistles of normal router and more
so I can run DHCP on it as well as NAT if need arise.

[JamesBondJr]

vl1969 Im pretty certain others on another forum said pci passthru will not work with the hardware, however I am currently looking to get the same setup you have so please do let me and others know how your setup works out and what you did so that others can search and find this useful information.

[vl1969]

@JamesBondJr:

vl1969 Im pretty certain others on another forum said pci passthru will not work with the hardware, however I am currently looking to get the same setup you have so please do let me and others know how your setup works out and what you did so that others can search and find this useful information.

hi JamesbondJr.
well #1 it might not be 100% true " pci passthru will not work with the hardware"
it might not work in ESXi but I heard that it works some what with Hyper-V
can not confirm it but there are rumors about some one able to make it work on this hardware
with Hyper-V. so I still have some hope.

unfortunately I am still in process, as in not even close yet :-(
I am trying to setup the latest,  Ubuntu 13.04 with Xen 4.3 (there are several help posts on the thread below that help with compiling the lates kernel for both ubuntu and xen.)
I have been at it for several days now, have some issues (most of them are do to my own fault )
but as of yesterday I have this setup running Custom built Ubuntu 13.10.5 kernel
with Xen 4.3 .

also, not sure about Intel chassis, but AMD BIOS have several configs that need to be enabled.
#1 there is an IOMMU option in the north bridge section I think, that I have set to 128M
and there is an SVM(secure virtual mode) under CPU config that need to be enabled.

should not be long now, or so I hope.

you can follow the process and get some help here
my main post on that forum is "http://lime-technology.com/forum/index.php?topic=28671.0"
here I will put all of my mods and progress. as well as links to most helpful threads.

and the help thread is here  "http://lime-technology.com/forum/index.php?topic=28665.msg258096#msg258096"
which you will also find links to other help posts/threads

[vl1969]

Just an FYI here.
it is confirmed that a PCI pass-through is possible on
AMD based chassis from TAMs based on supermicro H8DME - 2 MB
using Ubuntu Server 12.04 and Xen 4.2

also keep in mind that Ubuntu server 13.04 with Xen 4.3 is a no-go
it might be something in the kernel I am not sure but I was able to get things working on 12.04 but not on 13.04
since I am still playing with the setup I might try getting things work on 13.04+Xen 4.3  , but not sure about this yet…

ESXi is a no-go
XenServer 6.2 is a  no-go

there are some suggestions that an ArchLinux can work but as a noob I can not /will not go there :-)