Pfsense 2.1.3 + ESXi 5.5 = reboot after every shutdown of pfsense needed
-
@KOM:
Perhaps DHCP server on LAN may not work if WAN is undefined? I don't know, but it might be a safety mechanism so that DHCP doesn't give out IP addresses in what might end up being a LAN range? No idea for sure and I'm just guessing. Anything in your DHCP logs under Status - System Logs - DHCP tab?
OK, so you're moving from pfSense on a desktop-class machine to an instance under ESXi? I'm still trying to figure out the big picture what you're really trying to do
I don't think a mechanism would be in a pro-like software like pfsense… and honestly I don't want to make a static IP where it should not be.. I mean, would you do it?
The big picture is to get everything under the same roof because of enough powerful NICs and a powerful CPU..
But let's get back on the subject. Still CAN NOT believe I'm the only one with this problem and still noone knows how to fix it. Cmon, I'm doing passthrough which is kinda same as baremetal! I'm not doing rocket science!
-
I am running pfsense 2.1.3 on esxi 5.5u1 and not having any issues. I reboot my host all the time, has multiple vms on it that autostart with the host. I do a shutdown of guests when shutdown hosts, and then the vms restart after the host has been up for a couple of minutes. Autostart setting in esxi.
I am curious what performance your looking to get that you think you need to go passthru? How much to you need pfsense to handle as far as bandwidth and what is the connections?
So your saying that even if you shutdown pfsense (guest) and then start the guest it doesn't get an IP on wan and lan is not even up? Then you reboot the guest and it then works? Or you have to reboot the host.
So does pfsense think the nic is working - what does pfsense show for the nic? Nothing in the log on pfsense? It just doesn't pass any traffic?
Are you running vmtools on pfsense? Is pfsense 32 or 64bit version?
-
I am running pfsense 2.1.3 on esxi 5.5u1 and not having any issues. I reboot my host all the time, has multiple vms on it that autostart with the host. I do a shutdown of guests when shutdown hosts, and then the vms restart after the host has been up for a couple of minutes. Autostart setting in esxi.
I am curious what performance your looking to get that you think you need to go passthru? How much to you need pfsense to handle as far as bandwidth and what is the connections?
First its not about the bandwith. Its the pure thinking that I have not ruined anything in any way by going from bare-metal to fully virtualized.
My WAN is 120down with somewhat heavy usage from many devices.So your saying that even if you shutdown pfsense (guest) and then start the guest it doesn't get an IP on wan and lan is not even up?
When I autostart ESXi from power loss or just simply reboot it, it autostarts pfsense. It does not get/give any IP WAN/LAN-side even though in the menu it says LAN:192.168.1.1/24.
The interfaces are up I think, because I can normally assign in the menu igb1/igb0.
So yes.Then you reboot the guest and it then works? Or you have to reboot the host.
****If I now reboot the guest it works as it should (WAN from ISP, LAN DHCP server).
So does pfsense think the nic is working - what does pfsense show for the nic? Nothing in the log on pfsense? It just doesn't pass any traffic?
Are you running vmtools on pfsense? Is pfsense 32 or 64bit version?
Where and how to check logs? Its pfSense 2.1.3 64 with openVM-tools installed under ESXi 5.5 u1 with 1.6.2014 latest patch.
-
You may not be the only one in the world.
This thread was about what seems to be the same problem with similar hardware.
Unfortunately, no resolution posted.
-
perhaps there is an issue with passthrough & freebsd8.3 & esxi.
Since current development is focussed on freebsd 10, I wouldn't get my hopes up, that this gets fixed soon.
Freebsd has para-virtualized driver support builtin, and thus would no longer require the use of legacy stuff. Performance should go up dramatically.on that hardware getting around 1gbit/s with the legacy virtual drivers should not be a problem. ( i run around 10 boxes on similar hardware)
my advice: stop using the passthrough and go legacy-virtual ;) -
While we're hopefully still searching for an answer… Out of the subject is pfsense2.2 in its current state considered stable and secure as 2.1.3?
-
Yeah you only have 120mbit connection - there is not going to be an issue using that with virtual nic vs passthru.. Your thinking is wrong on the performance.. If your worried about virtual performance - then you shouldn't be running virtual at all. Since you can not get over the mindset of using it the way its designed.
My file storage is box is VM, its nic (vmxnet3) is vm connected to the vswitch that connects to the physical world with cheap nics, on a N40L box.. And I get great performance to and from the real world network.
here is my workstation to my VM storage box.
–----------------------------------------------------------
Client connecting to storage.local.lan, TCP port 5001
TCP window size: 256 KByte[344] local 192.168.1.100 port 52507 connected with 192.168.1.8 port 5001
[ ID] Interval Transfer Bandwidth
[344] 0.0-10.0 sec 1.06 GBytes 912 Mbits/secWhy should I be worried about performance on that??
phy box - switch – phy nic (N40L) -- vswitch - vm nic (storage vm)
-
Yeah you only have 120mbit connection - there is not going to be an issue using that with virtual nic vs passthru.. Your thinking is wrong on the performance.. If your worried about virtual performance - then you shouldn't be running virtual at all. Since you can not get over the mindset of using it the way its designed.
My file storage is box is VM, its nic (vmxnet3) is vm connected to the vswitch that connects to the physical world with cheap nics, on a N40L box.. And I get great performance to and from the real world network.
here is my workstation to my VM storage box.
–----------------------------------------------------------
Client connecting to storage.local.lan, TCP port 5001
TCP window size: 256 KByte[344] local 192.168.1.100 port 52507 connected with 192.168.1.8 port 5001
[ ID] Interval Transfer Bandwidth
[344] 0.0-10.0 sec 1.06 GBytes 912 Mbits/secWhy should I be worried about performance on that??
phy box - switch – phy nic (N40L) -- vswitch - vm nic (storage vm)
Again its not about the throughput only, its about LATENCY and MANY connections at the same time from many devices where latencies play a role.
Stick to the subject please. I want and I need passthrough.
So now this is a "known" bug or something. Maybe driver fault? Can I update drivers somehow?
Would something in System>Advanced>Networking solve it?
Any other suggestion?
-
Latency really on a freaking LAN what could the pos be .001 seconds ? Your nuts if u think latency going to be an issue phy vs virt. Your causing yourself grief for no reason
-
Hopefully found the solution:
System > Advanced > Networking
Check Enable device polling
Uncheck everything below
RebootSurvived for 3 host reboots already
2x Intel i210AT on Supermicro X10SL7-F -
It isn't quite solved yet. Always have atleast 50-51% CPU inside pfsense. That means one of the cores is always maxed out. Anyone got any idea in this area what these switches do and why they kind of fixed this?
EDIT: And that is on an idle connection and pfsense idling!
-
It isn't quite solved yet. Always have atleast 50-51% CPU inside pfsense. That means one of the cores is always maxed out. Anyone got any idea in this area what these switches do and why they kind of fixed this?
EDIT: And that is on an idle connection and pfsense idling!
You are sure your reading the graph right? On the performance tab of vSphere? It can be confusing looking at the left side it says percent, on the right side it say MHz. The graph varies by the maximum usage, if your max was 200 MHz and you are using 100MHz right now it will say 50%? My current usage is 62 MHz with Max usage at 1291 MHz. The Max is so high on mine because ever hour a 2 minute script runs that uses a lot of processor.
-
I have the same problem with pfSense 2.1.5 and esxi 5.5 with Intel I350 Nics.
Works when you reboot the VM -On latest pfsense 2.2 Snapshot it seems to be working correctly without the need to reboot.
-
http://christopher-technicalmusings.blogspot.com/2012/12/passthrough-pcie-devices-from-esxi-to.html
Try that. it seems to be a BSD/ESXi issue. Found it while dealing w/ a similar problem
-
Did you ever get farther on this?
I'm going to try device polling before next reboot and see if that helps me. The tickboxes below… I can't see how those would impact no connectivity post "first" reboot. It's 100% reliable- every SECOND reboot is fine. I am sure all vmware and passed through intel nics support polling fine.
I agree 100% you need to passthrough, virt NICs are just not good enough for replacing baremetal intelligently. Even when I only had 200MB in, I could see a huge loss on the ESX nic....Even played with the 3 different driver options you can pass to hack toward pfsense, always lossy. Can't happen with voice and other stuff.
The dual reboot hing makes me wonder if its a slice thing- I know the flash installs to two slices...And seem to remember reading they alternate at every reboot. Any comments on that?
Next week I will have one wan on gigabit/300 and the other at 200/30. Of course you need a good intel card for those, and to be smart to even see the throughput behind pfsense.
I think I ordered a quad ET 82576, my dual ET plus single 82574 CT pass through fine and dandy to the two wans which still are just 300/300 and 200/30.
ESX 5.1 is on x9scm-f e3-1230 32GB running lots of PCI passthrough to other stuff too.
pfsense 2.2's limiters are bustd. so 2.1.5 is best for me for now.
I may get around to trying a 2.2 pfsense and see if reboot works. Last time I tried the upgrade it broke everything, which I later found out was just because 2.2 busted limiters.