[HowTo] Install pfSense 2.1 RC0 amd64 on Xen 4.3 as PV HVM

Sabrewarrior

I have been using a HVM pfSense 2.0.1 on Xen 4.1 for the last year or so with no problems and have absolutely loved it, I tried a high end router for 200+ clients and found it was outgunned within 10 minutes, pfSense on the other hand has turned out to be cheaper, VERY stable and with Xen I can use my server for other stuff as well. With the recent releases of Xen 4.3, Release Candidate of 2.1 with the new captive portal upgrades and its move to FreeBSD 8.3 I decided it was time to upgrade stuff.

After almost 3 weeks of tinkering with http://devwiki.pfsense.org/DevelopersBootStrapAndDevIso I have finally been able to get a stable pfSense 2.1 RC0 build up and running. Since I had documented most of it anyway I might as well share it with you guys. The only 2 things I have found that don't work are a keyboard on VNC and being able to connect to my cable company using a virtual interface. I can DHCP to a local router but if I connect it to the cable modem it doesn't work, if someone figures out how to do either please let me know. I tried to get an IP on all 5 NICs and I was able to get an IP on my non RealTek NIC (VIA Rhine) so it might have something to do with that.

Give your builder DomU plenty of space, RAM and vcpu power to hurry things up. You will need around 5.5GB of space.
Making hvm requires following in builder.cfg


firmware="/usr/local/lib/xen/boot/hvmloader"
builder='hvm'

Install FreeBSD 8.3 (express/minimal you are going to get latest ports anyway)
If you are new to FreeBSD, to get ssh up and running make sure you select minimal and not express.
When it asks you to set up a user account make sure to add them to other groups = wheel
Enabling sshd is the same business (uncomment listen port in /etc/ssh/sshd_config and then /etc/init.d/sshd start)

1. Start out by following the instructions on the devwiki. However change https:/ to git:/ for freesbie2

su
echo "WITHOUT_X11=yo" >> /etc/make.conf
echo "OPTIONS_UNSET=X11" >> /etc/make.conf
echo "BATCH=yo" >> /etc/make.conf
mkdir -p /home/pfsense/pfSenseGITREPO /usr/pfSensesrc

portsnap fetch extract 
cd /usr/ports/textproc/expat2 && make depends install
cd /usr/ports/devel/git && make depends install
cd /usr/ports/sysutils/fastest_cvsup/ && make depends install
# tcsh requires reloading available binaries - if you run this command and it fails -- don't worry.
rehash

cd /home/pfsense && git clone git://github.com/pfsense/pfsense-tools.git tools
cd /home/pfsense && git clone git://github.com/pfsense/freesbie2.git freesbie2
cd /home/pfsense/tools/builder_scripts && chmod a+rx *.sh

csup -h `fastest_cvsup -c tld -q` /usr/share/examples/cvsup/standard-supfile

2. You will also have to edit the BSDinstaller script to look in git instead of https.
I would not recommend installing any other editors, I have had build problems every time I installed anything. So stick with vi editor.
vi being one of the most un-intuitive editors a cheat sheet helps.

vi scripts/get_bsdinstaller.sh
# Find https:/ - You can type /string to find a string in vi
# Change it to git:/ - Press escape and press x to delete character. Press i to insert.
exit vi 
# Press escape and type :x or :x! for read-only files

3. This fixes a problem you might get later when building an iso. Which is quite frustrating since its after almost an hour and a half of build time.
Gives dfuife_curses missing error if not fixed.

vi scripts/rebuild_bsdinstaller.sh
# Add the following line after: mv /etc/make.conf /tmp/
	echo "MAKE_JOBS_UNSAFE=yes" >> /etc/make.conf
exit vi

4. RELENG_2_1 just got branched. So HEAD now refers to RELENG_2_2 (to make 2.2 ALPHA which is quite unstable)
We have to fix set_version.sh to get the right branch.

cd /home/pfsense/tools/builder_scripts
vi set_version.sh
# find RELENG_8_3)
# In that function:
# Change PFSENSETAG=HEAD to PFSENSETAG=RELENG_2_1
# Can change PFSENSE_VERSION=2.1-BETA to PFSENSE_VERSION=2.1-RC0 (looks like its only symbolic, doesn't really change any scripts)
exit vi

5. The following enables the serial console on ttyu0, required for xl console.

vi builder_common.sh
# find LOADER_CONF_CUSTOM
# Insert the following line after: install_extra_loader_conf_options() {
	sed -i "" -Ee "s/^ttyu0.*$/ttyu0	\"\/usr\/libexec\/getty bootupcli\"	cons25	on	secure/" ${TTYS}
exit vi

vi loader.conf.local
boot_multicons="YES"
boot_serial="YES"
console="comconsole,vidconsole"
exit vi

6. Add Xen stuff to kernel config

cd conf/kernel
# Add the following lines to pfSense*.8* (depending on what version your FreeBSD is, if you are not sure just add that to all files with pfSense and 8 in them)
	# virtualization stuff
	# adaptive locks probably don't work right with virtualization. Disable them.
	options         NO_ADAPTIVE_MUTEXES
	options         NO_ADAPTIVE_RWLOCKS # I read somewhere Xen 4.3 has support for this. Not 100% sure though. Also not sure what it even does.
	options         NO_ADAPTIVE_SX

	# Xen stuff
	options         XENHVM
	device          xenpci

7. Now we can get back to the devwiki and finish building iso.

./menu.sh # Select Set Version -> Select RELENG_8_3\. everything else is just default.
exit menu

vi pfbuild-local.conf
# Add the following:
	export LOADER_CONF_CUSTOM="/usr/home/pfsense/tools/builder_scripts/loader.conf.local"
exit vi

./menu.sh
# Select Apply kernel patches
# Select Rebuild BSDInstaller
# Select Build ISO (selecting developers ISO does nothing as far as I can tell.)

Once the build iso is completed you will have the iso at /tmp/pfSense/ and the kernel at /tmp/kernels/

Extra. Since I can't get my vif to work with cable modem I need regular drivers for that interface.
If you are using any Realtek other than a Realtek 8139+ you should probably do the following things.

vi /sys/amd64/conf/GENERIC
# find the following line and press dd. you can probably search for 8111 to make life easier.
	device		re
# Personally I also removed device rl, just incase.
exit vi

# Go to [url]ftp://209.222.7.36/cn/nic/[/url] and search for rtl_bsd_drv
# I have had success with the v180 drivers on pfSense 2.01\. 
# My stock kernel RealTek drivers definitely dont work with 2.1 RC0\. and I am going to try the v183 now.

# The readme provided is a bit confusing but I found it easiest to just extract the files
rm -rf /usr/src/sys/dev/re
mkdir /usr/src/sys/dev/re 
cp if_re* /usr/src/sys/dev/re/
rm -rf /usr/src/sys/modules/re
mkdir /usr/src/sys/moduels/re
cp Makefile /usr/src/sys/modules/re/
cd /usr/src/sys/modules/re/
make clean
make
cp if_re.ko /mnt/v180 # To a usb drive or something
# If you only have remote access you can probably also add the files to builder_scripts/conf/copylist/copy.list.RELENG_2_1.
# I would just copy all versions to the usb drive and then try them one after the other.

Proof that it is running under PVHVM


dmesg | grep | "xn"
xn0: <virtual network="" interface="">at device/vif/0 on xenbusb_front0
xn1: <virtual network="" interface="">at device/vif/1 on xenbusb_front0
xn0: backend features: feature-sg feature-gso-tcp4
xn1: backend features: feature-sg feature-gso-tcp4
xn0: 2 link states coalesced
xn0: link state changed to UP
xn1: 2 link states coalesced
xn1: link state changed to UP

# I can also use xl shutdown pfSense in Dom0 which doesn't work on HVM.</virtual></virtual>

Speed wise iperf tests have not seen a massive improvement over HVM. Throughput of 540 Mbps on HVM vs 590 Mbps on PVHVM. Although I would have to check CPU loads as well to see how much better it actually is.

Let me know how it works out for you guys, if there is other stuff not working and if I could improve anything. Thanks!

Sabrewarrior

Well I think I might have found the single biggest flaw of running pfsense under PVHVM…you can't use traffic shaping because the xen interface drivers do not support ALTQ and I am not smart enough to add that by myself. I am giving it a shot, but I lack some serious knowledge of ALTQ itself to go about it. Just copying other drivers where ALTQ is concerned and hope for the best haha. Anyway just using PCI passthrough now for most interfaces except CARP. If anyone is interesting in adding ALTQ support, take a look at the man page for it, seems to be pretty helpful on what needs to be changed.

Also some of the steps have changed with 2.1-RELEASE
Step 2 - no longer needed
Step 4 - no longer needed
Step 7 - pfbuild-local.conf is now called pfsense-build.conf

KarboN

I've been trying to make PVHVM (or just PV) work for a long time, and finally got through it with your procedure! I've done all the steps but 2 and 4 since I built the stable version and everything but ALTQ works fine for now! I've noticed a slight improvement in latency between the LAN and the PfSense VM and the web browsing feels a bit snappier!

It's a shame though that ALTQ is not implemented in the xennet drivers as traffic shaping is one of the big features of pfSense. I am no expert, but I might do a bit of research just to see what could be done.

Thanks a lot!

ren22

Hi

i also try to make an pfsense PVHVM running on an intel atom cpu without vt/x-d … so it needs PV.

Sabrewarrior is it possible to provide your config's and may a small compressed image of your PV-pfsense ?

i am trying since days without succes atm ....

best regards
ren22

Sabrewarrior

Your welcome KarboN, good luck with the ALTQ. I haven't had much time to mess around with it yet, might start working on in a week or 2.

Hello Ren22, theres a difference between PVHVM and PV. I haven't tried to make a pure i386 PV pfSense. I can upload an amd64 PVHVM iso of the 2.1 Release later on today though.

Also if you are trying to make a PVHVM you should easily be able to with these instructions, that way you have more control over what you want to trim down on as far as the Kernel goes.

A Former User

Thanks for posting the howto Sabrewarrior!

I'm still trying to get a paravirt kernel built and working, but as a fall back, PVHVM will still make a huge difference to cpu usage (I have NAS4free running PVHVM with my LSI card - the difference between emulated e1000 and xen network driver is huge).

A Former User

Great, the howto worked for me, building 2-1 on BSD 8.3, many thanks!

The only gotcha I encoutnered was that the iso image didn't build - a quick look at the logs and it turns out that I needed to install the cdrtools package.

cd /usr/ports/sysutils/cdrtools && make depends install

After that the iso image built just fine, and I was able to install from the iso image using a simple HVM xen config, with the extra line "pty='serial'" so that I could use the xen console to do the install.

In the console text during boot I saw this, confirming its running in PVHVM mode:

xenbusb_front0: <xen frontend="" devices="">on xenstore0
xenbusb_add_device: Device device/suspend/event-channel ignored. State 6
xn0: <virtual network="" interface="">at device/vif/0 on xenbusb_front0
xenbusb_back0: <xen backend="" devices="">on xenstore0
xctrl0: <xen control="" device="">on xenstore0
GEOM: ad0s1: geometry does not match label (255h,63s != 16h,63s).
xn0: backend features: feature-sg feature-gso-tcp4
xbd0: 5120MB <virtual block="" device="">at device/vbd/51712 on xenbusb_front0
SMP: AP CPU #1 Launched!
Root mount waiting for: usbus0</virtual></xen></xen></virtual></xen>

I did notice that its still got the re0 device (HVM emulated realtek) showing up alongside the xn0 one, so I'll need to disable that in rc.local.

I also just did PCI passthrough for an Intel 82574L Gigabit ethernet card (dedicated NIC is for WAN connection, on separate physical subnet - I'm using the Xen network interface xn0 for internal LAN), although I haven't done more than boot the VM with the PCI device at this stage (it let me configure it).

Now I've got a working install, I'll have a play with making a paravirt domU, although in my experience paravirt performance is largely the same as PVHVM, with the only difference being the ability to pass through PCI devices without needing IOMMU/VT-d.

(FYI my dom0 is Centos 6.4 using standard Centos-supported xen-4.2 and kernel-3.4.61)

Sabrewarrior

That's odd that its showing up twice. I would go over the vm config file or do a brctl show in the Dom0 to see whats going on.

A Former User

Yes - not sure why its showing up twice.

Here's the output from bridge control:

]# brctl show
bridge name bridge id STP enabled interfaces
xenbr0 8000.0025900d41b6 no eth0
vif1.0
vif5.0
vif5.0-emu

The domU config file is pretty simple - nothing out of the ordinary:

name = 'pfsense21'
builder = 'hvm'
memory = 1024
vcpus=2
acpi=1
vif = [ 'bridge=xenbr0, mac=00:16:3e:14:01:97' ]
disk = [ 'phy:/dev/vg_ssd/pfsense21,xvda,w' ]
boot='c'
serial='pty'
pci = [ '0000:05:00.0' ]

An ifconfig on the domU shows both the xn0 and re0 interfaces (along with others including the pci-passthrough Intel em0 one:

re0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:16:3e:14:01:97
media: Ethernet autoselect (100baseTX <full-duplex,flowcontrol,rxpause,txpause>)
status: active
xn0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=3 <rxcsum,txcsum>ether 00:16:3e:14:01:97
inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255
inet6 fe80::216:3eff:fe14:197%xn0 prefixlen 64 scopeid 0x8
nd6 options=1 <performnud>media: Ethernet manual
status: active

Note in the above, both interfaces have the same MAC address. Interestingly, a check on the dom0 shows:

~]# xl network-list pfsense21
Idx BE Mac Addr. handle state evt-ch tx-/rx-ring-ref BE-path
0 0 00:16:3e:14:01:97 0 4 7 769/770 /local/domain/0/backend/vif/5/0

I'd leave it alone, except that I have an issue with networking which seems to be affecting some network traffic via random high latency. First thing to try is rebuilding my PVHVM kernel with a clean build environment.</performnud></rxcsum,txcsum></up,broadcast,running,simplex,multicast></full-duplex,flowcontrol,rxpause,txpause></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></broadcast,simplex,multicast>

A Former User

I also just switched the active LAN device to the re0 one.. it appears to work:

re0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:16:3e:14:01:97
inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255
inet6 fe80::216:3eff:fe14:197%re0 prefixlen 64 scopeid 0x1
nd6 options=1 <performnud>media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
xn0: flags=8802 <broadcast,simplex,multicast>metric 0 mtu 1500
options=503 <rxcsum,txcsum,tso4,lro>ether 00:16:3e:14:01:97
media: Ethernet manual
status: active

Definitely time to try rebuilding my kernel and see how it goes :)</rxcsum,txcsum,tso4,lro></broadcast,simplex,multicast></full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast>

A Former User

@Sabrewarrior:

Hello Ren22, theres a difference between PVHVM and PV. I haven't tried to make a pure i386 PV pfSense. I can upload an amd64 PVHVM iso of the 2.1 Release later on today though.

Hi Sabrewarrior - any chance you could upload the PVHVM amd64 image? And ideally, the contents of your "conf/kernels" directory?

I built the i386 32-bit pfSense iso ok and it boots PVHVM, but as noted it has both the re0 and xn0 interfaces - I checked a vanilla XENHVM freeBSD8.3 kernel and it has the same symptoms and appears to be a bug in the 32-bit HVM code, so I decided to have a go at a 64-bit kernel seeing as how you've had success at it.

I built a vanilla amd64 freeBSD8.3 kernel, and it works just fine without duplicate network interfaces, so I figured a pfSense kernel should be easy. However, I'm getting a Fatal trap 12 error booting the pfSense XENHVM kernel - which means there has to be an issue in the pfSense code somewhere, unless there's something I'm missing in the options/devices.

If you could post/upload your amd64 PVHVM iso for me to try, it'll at least give me an indication whether the issue is my build process, or whether it doesn't like my hardware combo (Supermicro X8SIL and Xeon X3450 cpu).

Any help appreciated - its getting frustrating after about 8 iso builds in the last 24 hours!

Sabrewarrior

Hello xlot,
Sorry for the delay. I am honestly at a loss to explain the 2 interfaces since brctl is only showing 1 interface being shared, assuming vif1.0 is for a different DomU. Also both your interfaces seem to have the same mac address which is a very bad thing and might be the cause of your high latency.

As promised I have uploaded the amd64 PVHVM image to my blog which is in my signature (don't worry no ads!). I have attached the config file I used here though. My sever is running AMD not Intel. I have not tested this image so let me know how it works. I will be working on a 32 bit PV image now.

pfSense_SMP.8.txt

A Former User

Many thanks! The dual-interface problem appears to be something to do with the xenpci driver for the i386 platform.

As suspected, your kernel conf file is identical to one of the variations I've tried (note - the same outcome can be achieved by just replacing the import statement to use the "XENHVM" bsd kernel conf instead of "GENERIC" - saves adding the extra lines).

I just downloaded your image and will have a try and report back here. Thanks for your help - using a known-good build will tell me whether the problem lies with my hardware combo, or possibly dom0 (I'm suddenly wondering as I type this whether I should try the old xm toolset using xend, instead of the newer xl/xenstored one).

A Former User

Ok, your image gives the same error I've been getting on the 64-bit platform. The trace shows that its the same problem as found here:

http://lists.freebsd.org/pipermail/freebsd-xen/2013-June/001613.html which in turn links to this post that includes a patch:

http://lists.freebsd.org/pipermail/freebsd-xen/2012-September/001359.html

Here's my trace. Its the same issue and looks like it might be a problem with xenstored on my dom0 (xen-4.2.3 on kernel 3.4.61 - stock centos packages).


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xffffff010294c2ef
fault code              = supervisor write data, page not present
instruction pointer     = 0x20:0xffffffff80a56a68
stack pointer           = 0x28:0xffffffff816b19f0
frame pointer           = 0x28:0xffffffff816b1a30
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = trace trap, interrupt enabled, resume, IOPL = 0
current process         = 0 (swapper)

A Former User

I just posted to the Development section - patching the xenstore.c file with the fix found on the freebsd-xen mailing list has worked and I've got a pfSense VM running using the Xen-native block device drivers (xn0 and xbd0).

Info here:

http://forum.pfsense.org/index.php/topic,69546.0.html

Thanks for your help sabrewarrior.

Incidentally, it seems that there's a problem with my pci-passthrough of an Intel Corporation 82574L NIC - I was getting irq errors and the interface wasn't working, but that was fixed by forcing use of MSI interrupts rather than MSIX, by adding these lines to loader.conf:


hw.pci.enable_msi=1
hw.pci.enable_msix=0

So finally, I've got it all working as wanted, and I can start to actually test the new setup.. woohoo!

Sabrewarrior

Yeah, I am using ubuntu with xen 4.3 so that might be one of the patches that fixes your problem. I get irq errors with my vr interface, I wonder if its a related problem to MSI interrupts.

Sabrewarrior

Well never thought that I could make a pure PV work but after some ridiculous amount of time spent, I have managed to get a working pfSense i386 PV iso. I am gonna make another one with a bit more clean up to make it straightforward and documentation so I can post the how to.

Issues so far:
SMP is broken. If you use more than 1 vcpu it will not work. Issue is with FreeBSD Xen drivers again. First function that gives an error can be found in src under sys/xen/interface/xen.h at line 80: #define __HYPERVISOR_vcpu_op 24

It gives the following panic at vcpu=2:
SMP: Added CPU 0 (BSP)
SMP: Added CPU 1 (AP)
gdtpfn=83ec38 pdptpfn=89c17
panic: HYPERVISOR_vcpu_op(VCPUOP_initialise, cpu, &ctxt): /usr/pfSensesrc/src/sys/i386/xen/mp_machdep.c:930
cpuid = 0
KDB: enter: panic

A Former User

I posted some info before in another thread here:

http://forum.pfsense.org/index.php/topic,67866.msg375470.html#msg375470

The problems with pure-PV on freeBSD are described here on a post from the freebsd-xen list:

http://freebsd.1045724.n5.nabble.com/Paravirt-domU-and-PCI-Passthrough-td5858307.html

IIRC, there's also a memory limit for PV on i386 - various old posts on the freebsd-xen list mentioned somewhere around 700 megs.

I've been running pfSense as my gateway for a week for one small subnet - and working through reading docs.. PVHVM is seeming workable (I need PCI passthrough - I prefer to run the pfSense VM with a separate physical NIC/subnet for the external interface.

archedraft

Would you mind posting the contents of you pfsense.cfg file?