ESXi Won't boot PfSense /w passthrough
-
My build is as follows:
Supermicro X8DTi-F
Corsair CX500 500W PSU
(2) Intel Xeon W5590 @3.3GHZ
24GB G.SKILL DDR3 1333 (6 x 4GB)
(1) 2TB WD HDD connected to onboard controller (Datastore to hold VMs)
(5) 3TB Seagate ST3000DM001s connected to LSI 9220-8i cross flashed to a LSI9211-8i in IT mode
(1)Intel EXPI9402PT Dual Port Gigabit PCI-ECurrently I have a Windows Server Standard 2012 R2 VM that has access to the (5) Seagate drives through Physical RDM. Trying to get a PfSesnse VM running. Plan was to passthrough the Intel NIC and use one port for WAN and one for LAN. However, I'm having trouble getting it to boot/passthrough the NIC.
Event tab shows error:
VMware ESX unrecoverable error: (vcpu-0) PCIPassthruChangeIntrSettings: 02:00.0 failed to register interrupt (error code 195887110)
other errors:
2014-10-09T03:13:13.416Z cpu14:35318)VMKPCIPassthru: 1790: BDF=02:00.0 intrType = 2 numVectors: 1
2014-10-09T03:13:13.416Z cpu14:35318)VMKPCIPassthru: 1577: Action already registered for vector 0x31After ESXi host reboot ,if PfSesne is first VM turned on(no other running) it boots and PfSense can see Intel NIC and I can ping it and access webconfig from another computer.
If I turn on the Win Server 2012 VM things work ok for a bit but then if I try to reboot PfSense it results in the previously mentioned error.
Once it refuses to boot the only thing that will let it boot again is restarting the entire ESXi host. However if add pciPassthru0.msiEnabled = "FALSE" and pciPassthru1.msiEnabled = "FALSE" to my .VMX (per this it will boot, but then PfSense sees no interfaces.
Would love any help or guidance any one can provide. Already tried posting on vmware and Hardforum with no luck.
-
So the host only has 1 dual port nic? How is your 2k12 VM going to access the real world? You would need to create a vswitch with pfsense having another leg and your 2k12 box connected to this vswitch and then route traffic through pfsense to get to your lan. Is this what your after?
I don't understand the fascination with passthru on the nics.
Let esxi see the nics, then connect the wan one to a vswitch that pfsense vm has a leg in and other end to your wan network(modem). Then use the other port as lan leg, tied to your real world lan switch. Then there will be your lan vswitch, and your vmkern port group tied to this nic as well.
You then connect whatever other vms you have on the host to the lan vswitch and your ready to rock. You can either just use e1000 vnics, or vmx3net works fine on pfsense as well once you install the tools.
I run this setup with 4 physical nics on esxi host. 1 onboard nic, 1 single port intel and 1 dual port intel.
Your just making for a more complex setup doing passthru - that buys you what exactly? If you want to isolate your 2k12 VM from the real world lan nic and vswitch you can still do that. But it is much easier to just let esxi control the hardware nics. What your going to connect your vmkern to if pfsense has control over the hardware nics?
-
I don't understand the fascination with passthru on the nics.
Same here. I just don't see any real advantage, but lots of drawbacks. I was running MS ISA (don't ask) virtually, now pfSense virtually. No passthrough required.
-
First let me same thank you for responding with some useful information. Never had so much trouble finding help on a tech issue in my life. You mention passthrough issues with ESXi and no one has anything to say but passthrough is bad and rarely works right.
So the host only has 1 dual port nic? How is your 2k12 VM going to access the real world? You would need to create a vswitch with pfsense having another leg and your 2k12 box connected to this vswitch and then route traffic through pfsense to get to your lan. Is this what your after?
I'm thinking I failed to not only to explain my setup, but also how noob I am with VMware in general :P. My motherboard has 1 onboard dual port NICs(Intel 82576 Dual-Port Gigabit Ethernet Controller w/ VMDq support) and an add in Intel EXPI9402PT Dual Port Gigabit on PCI-E(failed passthrough one). The onboard one is currently using only 1 port which is used by both ESXi management interface( and the vm network interface)
I don't understand the fascination with passthru on the nics.
I was told it was more secure and higher performance(latency)
Let esxi see the nics, then connect the wan one to a vswitch that pfsense vm has a leg in and other end to your wan network(modem). Then use the other port as lan leg, tied to your real world lan switch. Then there will be your lan vswitch, and your vmkern port group tied to this nic as well.
You then connect whatever other vms you have on the host to the lan vswitch and your ready to rock. You can either just use e1000 vnics, or vmx3net works fine on pfsense as well once you install the tools.
I'm not sure I understood you completely, but from what I did understand I have now stopped attempting passthrough and configured my network as seen in the following picture.
If I'm correct I still have the security of using a separate NIC/VM network.
All that to say this. Now my VM has no trouble booting/rebooting. But after it has been running for a while it will lose WAN access (IP appears empty in console, LAN IP still shown as 192.168.1.1)and no longer respond to pings on 192.168.1.1 interface (computer manually assigned ip 192.168.1.5). I can reboot VM but upon reboot same thing. Wont grab WAN IP from modem, cant ping computer from pfsense and computer can ping LAN interface. Now if I reboot the ESXi host upon boot of PfSense VM everything is fine again-WAN works , webconfig works, etc. Also, while everything is working I can reboot the VM and it will still work fine. Only after it has been running for 30-60 minutes does it drop WAN IP and wont respond to pings, etc. until a host and VM reboot
Open-VMtools installed , 64bit PfSense 2.1.5
Any advice on my new issues :/
-
well from your picture your pfsense wan vmnic2 is not connected.. So that is going to be an issue, same with vmnic3
How told you it was more secure and less latency? The basis of the whole concept is to let esxi control the physical nics..
So here is my esxi network. See attached
See how pfsense (pfsense-vm) has interface in the 4 segments. Wan, Lan, WLAN and DMZ (does not have physical connection only VMs)
Pfsense handles the routing between segments and firewall rules between them. So off the wan vswitch you see it connect to vmnic3 which is connected to my modem only. Off the Lan vswitch I have vmnic2 which is connected to my real world physical switch this allows both VMs and real machines to talk to each other on this segment. Off the wlan vswitch is vmnic0 which is connected to different switch that has my AP and printer (ease of wireless to access the printer via airprint, etc.) and my wireless controller - that unifi-linux vm.
I then have my vmkern (esxi managment interface) connected to vmnic1, which is also connected to my lan physical switch. I broke this out on its on physical nic because for 1 I had a spare nic :) And 2 vmkern takes a bit of a performance hit when sharing nic with other vms. My transfers to and from the datastore are now faster - but this is not really a big deal and can go either way - if you have spare nic then sure break it out. if you want to be more secure put it on its own managment segment where you have your managment station, etc.
Then there is the DMZ vswitch - this is not connected to the real world, and only VMs are use here - internet and other segments can talk to them via pfsense routing and firewall.
Why do you have yoru windows VMs on their own switch and physical nic - this should be your lan, and pfsense should have an interface it. And your physical network should be connected to this. Along with your vmkern.
In your setup how do you get to windows vms without being physically connected to that vmnic?
-
I was just going to run ethernet from ESXi management interface to a real life switch (also connected to PfSense LAN ), although I admit your way makes more sense now that I see/understand it.
It shows as disconnected in my picture b/c nothing is currently connected. During all my testing it was connected.
So any idea at all why all my interfaces stop responding after 30-60 minutes time???? Until i fix this not much else matters :(
-
so the physical stops working? Your setup is not a viable option as you posted so its hard for me to speculate what could be the problem
So your picture pfsense doesn't do anything.. Is there a physical switch that connects vmnic0 and 3? What does vmnic2 connect too?
-
I think your misunderstanding me. My set-up works fine…..for 30-60 minutes. Right now I'm just trying to get PfSense working at all. So for testing purposes PfSense's LAN port connects directly to a dell laptop (192.168.1.5), WAN port connects directly to cable modem.
Whats not viable about it? I will definitely switch things up once I get it working and have it take over routing from my Asus router.
my picture doesnt show that I plan on connectting a gigabit switch to PfSense's LAN port (and then a wifi access point to the switch). This is how I plan to connect all my stuff to the internet. I realise right now none of my VM's are routing through PfSense. This is b/c I can't keep PfSense working and they need internet. So they are hooked into my asus router for now.
vmnic3 will eventually connect to real switch, right now dell laptop is connected. Right now vmnic2 unused.
-
your vms are not connected to the pfsense lan network - unless you have some physical switch connecting those physical nics? Makes no sense to go from virtual to physical, back in physical to virtual again. Connect your VMs to the pfsense lan vswitch.
So maybe this helps I added part of my physical network and how it connects to the esxi host nics
-
your vms are not connected to the pfsense lan network - unless you have some physical switch connecting those physical nics?
So maybe this helps I added part of my physical network and how it connects to the esxi host nics
I think you might have missed my previous post were I explained my setup a little better.
@mysongranhills:I think your misunderstanding me. My set-up works fine…..for 30-60 minutes. Right now I'm just trying to get PfSense working at all. So for testing purposes PfSense's LAN port connects directly to a dell laptop (192.168.1.5), WAN port connects directly to cable modem.
Whats not viable about it? I will definitely switch things up once I get it working and have it take over routing from my Asus router.
my picture doesnt show that I plan on connectting a gigabit switch to PfSense's LAN port (and then a wifi access point to the switch). This is how I plan to connect all my stuff to the internet. I realise right now none of my VM's are routing through PfSense. This is b/c I can't keep PfSense working and they need internet. So they are hooked into my asus router for now.
vmnic3 will eventually connect to real switch, right now dell laptop is connected. Right now vmnic2 unused.
Does that make sense?
-
ah ok – so it works for 30 60 mins and then what errors do you get? You just loose your wan connection?
-
ah ok – so it works for 30 60 mins and then what errors do you get? You just loose your wan connection?
Pretty much. No errors in PfSense VM console still shows LAN IP as 192.168.1.1 WAN shows no ip and pings from 192.168.1.5 to 192.168.1.1 come back as timed out or destination unreachable. When 5 minutes earlier WAN had IP pings to LAN int. came back <1ms. Nothing done in PfSense other then boot it up.
-
Does any one know what logs I can check and what I should be looking for? Is this problem I'm having not that common?
-
you can look in pfsense system log, clearly it would report loss of wan IP.. But seems more like problem with your esxi box if you can not ping the lan IP? Does esxi show the nics active.
-
I haven't checked PfSense logs yet b/c I now believe it to be a ESXi issue. Once all the connections dropped I checked the vSwitch and it shows the physical NICs as disconnected for LAN and WAN vSwitches. NIC lights still blinking and laptop still shows connected on ethernet.
Any idea what could be causing this?
-
This is suppose to be only have upgrade adding, etc.. but you could try this
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2019871
What specific build of esxi are you running. Current build is 2143827 and was just released the other day.
I would also check your logs, the logs are in /var/log – enable shell on your esxi host and check them out
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2032076
Location of ESXi 5.1 and 5.5 log files (2032076) -
This is suppose to be only have upgrade adding, etc.. but you could try this
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2019871
What specific build of esxi are you running. Current build is 2143827 and was just released the other day.
I would also check your logs, the logs are in /var/log – enable shell on your esxi host and check them out
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2032076
Location of ESXi 5.1 and 5.5 log files (2032076)Thx. I think I'm on build 16xxxxx. Will update to latest build tonight when I get home and try moving NIC to another PCI-E slot. Could this be a bad NIC ?
Worst case scenario can I use my unused onboard NIC port for WAN and just add use existing VM network for Pfsense LAN?
-
Build 1623387? That is Update 1, way back in March.. Yeah I would update lots of changes and fixes and drivers updated, etc.
