LAN issues with Hyper-V
-
Here's what I have so far:
–Windows Server 2012R2 running Hyper-V with pfSense 2.2.4 as a VM
--2 onboard Intel 82574L adapters set up as virtual switches:
----"NIC1" is physically connected to an Arris SB6190 cable modem & set up in Hyper-V as an external virtual switch without host OS access. In other words, this is my WAN port, and the only thing that gets to use it is pfSense.
–--"NIC2" is physically connected to an HP Procurve 4000M managed switch, which will be the interface for the rest of the LAN. The NIC is set up in Hyper-V as an external virtual switch with host OS access.I have gotten pfSense installed and configured to the point of being able to receive DHCP assignment from the modem (ISP: Cox Cable), run the web configurator, and ping the switch, host physical PC, & another PC connected to the switch. The WAN is set up to receive DHCP from the ISP, and it seems to be doing that. The LAN side is set up in pfSense with DHCP server enabled, and if I log in to each device on the network, I can see that they are receiving an IP address dynamically.
The objective is to use this VM installation as a router & firewall.
I guess my biggest problems are:
1. The host machine has no network access, other than the ability to open pfSense in the web browser. I suspect this has something to do with the network adapter configuration in both Win Server and Hyper-V.
2. Other than ping, no device can "see" any other devices on the LAN. Even pfSense does not show them in its status pages. But I know they are communicating in some fashion, as they are receiving DHCP assignments and pinging just fine.
3. Nothing can reach the internet, including ping and the pfSense VM.This is complicated by the fact that I have to drive 10 minutes to gain access to any internet service in order to check these forums, etc.
-
1. Have you configured the right NIC on your host? You must NOT change the configuration of the NIC itself as it's now managed by Hyper-V, configure the virtual NIC instead (should be called vEthernet or something similar).
2 and 3 might be related as if you change the configuration of the NIC on the host then pfSense cannot use it properly as it will affect the Hyper-V virtual switch.
If that's not it 2 and 3 may be an issue with your pfSense firewall rules. In your firewall rules config window, on your LAN interface, make sure there's a rule that allows IPv4 and IPv6, all protocols, source: your local LAN net, destination all/any and everything else to all/any. The line should look like this:
IPv4+6 * LAN net * * * * none Default allow LAN to any WAN rule
-
1. Have you configured the right NIC on your host? You must NOT change the configuration of the NIC itself as it's now managed by Hyper-V, configure the virtual NIC instead (should be called vEthernet or something similar).
2 and 3 might be related as if you change the configuration of the NIC on the host then pfSense cannot use it properly as it will affect the Hyper-V virtual switch.
If that's not it 2 and 3 may be an issue with your pfSense firewall rules. In your firewall rules config window, on your LAN interface, make sure there's a rule that allows IPv4 and IPv6, all protocols, source: your local LAN net, destination all/any and everything else to all/any. The line should look like this:
IPv4+6 * LAN net * * * * none Default allow LAN to any WAN rule
Are you saying that if I make any changes to the adapter settings in Windows, those do not get passed to the virtual switch or the VM? That seems like a major design flaw. I have been fiddling with the adapters all night long trying out fixed IP and gateway combinations.
The Windows firewall is totally disabled, and the pfSense firewall rules have been temporarily set to permit everything.
A correction to the original information I gave:
I have NOT been able to ping or lookup any external IP address/URL. I can only ping all the computers on my network, the switch, and the modem.
-
You didn't specify a gateway for LAN, per chance?
-
Are you saying that if I make any changes to the adapter settings in Windows, those do not get passed to the virtual switch or the VM? That seems like a major design flaw. I have been fiddling with the adapters all night long trying out fixed IP and gateway combinations.
It's not a design flaw, there's no other way it could work. If all of the VMs connected to a virtual switch tied to a NIC port ended up with the same configuration inherited from the host they could never talk to each other because they'd all have the same IP address. You have to configure your VM's virtual interfaces from within the VMs.
In Hyper-V manager, when you create a virtual switch, you have three options: External, Internal, and Private.
An External v-switch must be tied into one of your hosts adapters. At that point the configuration of the physical adapter in your hosts network config is taken over by Hyper-V and you must never change its configuration again. The physical port on your computer becomes just another port on the virtual switch, managed by Hyper-V. If you check the "Allow management operating system to share this adapter" option then a new virtual adapter is created on your host and it's virtually plugged in that v-switch. That's the adapter whose configuration you can change to have your host talk with the other interfaces connected to that v-switch (including the physical port on your computer).
An Internal v-switch is the same except it's not tied in to any of your hosts physical adapters, and it automatically creates a virtual adapter on your host so it can connect to the v-switch. A Private v-switch can only be used between your VMs, it doesn't create a virtual adapter for your host to connect to it.
Every time you go into your VM's properties and Add a network adapter, you have to tell it which v-switch to connect to. At that point a new virtual adapter is created in your VM and connected to the v-switch, and you configure the properties of that adapter from within the VM.
I hope this helps…
-
I thought I gave a pretty descriptive account of my configuration in the OP. I think the source of the problem lies with my virtual networking adapters. Maybe someone can spot the problem. Keep in mind that I have never used Hyper-V until a few days ago, so I am still learning.
My concept of operations is that pfSense will run as a VM and be the sole router and firewall for the whole network.
Physical topology: Cable Modem –> NIC 1 ---> 2012R2 Host (+ pfSense VM) --> NIC 2 --> HP Switch --> rest of LAN
Virtual topology: [PHY]NIC 1 –> [OS]vEthernet1 –> [Hyper-V]Virtual Switch 1* –> [pfSense]WAN ….. [pfSense]LAN –> [Hyper-V]Virtual Switch 2** –> [OS]vEthernet2 –> [PHY]NIC 2
*External, no access by host OS, SR-IOV enabled, no extensions enabled
**Same as above, but host OS is allowed access
I have also disabled all offloading features in the NIC hardware settings, as per some troubleshooting guides on this site.Am I wrong to believe that my host OS can use the vEthernet2 device to join the network behind the firewall?
I will remove all virtual switches and adapters and create them again from scratch, then reconfigure pfSense.
-
Now I seem to have taken several steps backward. I can no longer access the webconfigurator, even after rebuilding the vNICs/vSwitches, reverting pfSense to defaults, and rebooting.
If I had the money, I'd have bought a COTS router by now. :-\
-
Let's make things easier and remove possible sources of misconfiguration. First ignore your cable modem and pfsense VM. Let's try to get your host to talk to the rest of your local network through the vSwitch.
So remove all of your vSwitches in Hyper-V Manager and ensure your hosts network configuration only has your two physical interfaces listed. Once you're there, disable NIC1 and create a new External vSwitch tied to NIC2 and share it with your host (call this vSwitch "LAN"). Once that's done, configure the new vInterface that just showed up in your hosts network configuration (should be called vEthernet (LAN)) to use a static IP address on your local LAN subnet and verify if you have connectivity with the rest of your local LAN. This should absolutely not require your pfSense VM to be running as, remember, all you're doing is connecting your host to your local LAN through the vSwitch and your physical HP switch. Your local LAN devices will also need a static IP for this part as you won't have the DHCP server in pfSense running.
Once you got this to work then we can look at your WAN. Re-enable NIC1 in your host and create a new vSwitch called "WAN" in Hyper-V Manager, choosing NOT to share it with the management OS. At this point your hosts Network configuration will still have three NICs listed, your two physical NICs and the virtual one. In Hyper-V Manager create yourself a new pfSense VM and assign two network adapters to it (regular, not legacy), WAN first, then LAN. Boot up from the pfSense installation image, assigning adapters hn0 to WAN and hn1 to LAN. Make sure WAN is set to DHCP and you put a static address from your LAN subnet on the LAN interface within the pfSense VM. Once you're done installing your WAN interface should be getting a DHCP address from your cable provider and you should be able to ping any client on your LAN from the pfSense VM. Rebuilding the pfSense from scratch will remove the possibility that some change you made while troubleshooting somehow made pfSense not work anymore.
Once your pfSense VM has access both to the internet and your local LAN then take a snapshot of it in Hyper-V Manager and start configuring the rest of it, like the DHCP server, etc. At any point if it stops working, go back to that snapshot and start the configuration from scratch. The only configuration that should be changed on your host from that point on is the configuration of the vEthernet (LAN) interface, which may go from static IP to DHCP if you want your host to get its IP address from the pfSense VM, which I would recommend against since your host reboots before the VM and you wouldn't get access to your local network if there's somehow a problem with your pfSense VM.
Good luck.
-
I was already in the process of starting over from scratch. I deleted all the virtual adapters, deleted the pfSense VM. I downloaded the newest ISO file in order to create a new VM, and now I can't for the life of me remember the workaround I had to use in order to get the installer to create/use a VHD. There is some glitch with the installer that requires a specific number of cylinders/blocks on the VHD. I have searched for hours and I can't find the workaround I used before. Another step backward…..
I did have success finding the LAN from my host machine using the first steps in your helpful post, though.
-
It shouldn't matter what your VHD settings are. I've installed it by booting from the ISO image on VMs many times on a lot of VHDs with different settings and it was never an issue.
-
Ok, I finally got it to install after trying progressively smaller VHD sizes. There is something about FreeBSD that does not like large partitions.
Following your instructions, I was able to achieve the following:
1. Connection to LAN, as evidenced by pinging devices and accessing shared folders
2. Connection to WAN, but only in the form of pinging the modem and 8.8.8.8. Windows automatically switched the network status icon to "Internet Access" as well. However, I could not access pfSense via browser or SSH terminal on PuTTY. I also could not access the modem's "console" page via browser.Per your instructions, I clicked "Save State" (my best translation of your term "snapshot"; should I have used "checkpoint" instead?) in Hyper-V, and it basically pulled the virtual power cord on pfSense, causing it to reboot. Once it booted back up, I got the broken network connection tray icon and no longer had a connection to the LAN. My vNIC properties window showed a an IP address 168.25.x.x with a 255.0.0.0 mask and no gateway.
-
Yes, sorry, it is Checkpoint in the current version.
Rebooting through "save state" shouldn't break anything though. You do have to hit "start" after "save state" though, as "save state" stops the VM from running (but doesn't actually shuts it down, it just suspends it). Checkpoint saves the state of your VM but keeps it running, recording any further changes to a different vhd, which allows you to go back to the state that existed when you created the checkpoint and ignore and changes that happened after that.
The IP address you describe is what happens when there's no DHCP server. It's an address that's assigned by the host itself. Like I said you really should be running a static IP on your host. Use another device connected to your LAN to test DHCP.
When you try to access the pfSense web interface, you have to connect to the LAN address (which is also your subnet's gateway address). IIRC you have to connect in a secure session (https://…) as non encrypted connections are disabled by default in pfSense.
-
After rereading my last post, it occurred to me that I had somehow removed the static IP from the host after the initial test of the LAN only. Once I reinstated it, everything started falling into place. I now have web access from any computer on the LAN. My Remote Desktop shortcuts are all broken, but I can access any shared folders that were previously set up.
Here's the only remaining problem: enabling DHCP for the LAN breaks the host machine's connection. The only way I found to repair it is to rebind the LAN interface in the Hyper-V terminal window. I don't have a problem manually configuring static IPs in all my clients, but there are wireless users (my housemates) who are probably going to need me to do this for them on a daily basis when they blindly allow their devices to return to defaults. I'm not even sure if Android or iOS devices can store unique static IPs for specific networks, so fixing them here may break them everywhere else. I may have to set up multiple subnets or a VLAN to keep my host machine isolated from DHCP, or implement a DHCP server on the WAP (which is a repurposed older router). God, I hate wireless and the people who love it!
Thanks, Gimli, for your patient and methodical help. I really needed such an approach after getting stressed out and sleep-deprived trying to make this work. If you are running a similar configuration, I'd love to know some best-practice settings and tweaks.
-
You're very welcome. We all need someone to take our hand and guide us sometimes ;D
As far as the DHCP thing goes I would say just don't use the one provided by pfSense. I have the DHCP server installed on my AD Domain Controller and all you have to do in pfSense is to configure the DHCP Relay service to point to the DC's IP address and any DHCP request that hits your pfSense box will be forwarded to your Windows DHCP server. The DHCP server in pfSense is pretty basic and I was more comfortable using the Windows one.
As for configuration recommendations beyond that it's very difficult to make any as every environment is different. What I would suggest though is to make a checkpoint of your pfSense VM before you try anything new so you can go back to a known good state if you somehow screw something up. God knows that's saved me many times. Oh and thoroughly research whatever options/services you plan to change in these forums and on Google in general because pfSense has a way of tricking you into not fully understanding the impact of every configuration item.
One thing I had issues with: by default pfSense will change the outboud port that your clients use to communicate with Internet servers, just in case two clients tried to go out on the same port at the same time. This broke a few applications where the client initiates the communication but the server always responds on a pre-determined port. In those cases, because pfSense changed the outbound port from (for example) 501 to 50001 and the server always tried to connect back on port 501, the connection would time out. Unfortunately I can't remember for the life of me the setting I changed to fix that and tell pfSense not to change the outbound port for client-originated requests. I think it might be the "Insert a stronger id into IP header of packets passing through the filter." under System -> Firewall/NAT but I'm not 100% sure. Anyway you probably don't have to play with this unless it becomes a problem.
Another hint regarding checkpoints is to delete them all once you've attained a good, stable state. Checkpoints are great but if you keep too many or you keep them too long the delta vhd that are created with each checkpoint become very big and your VM's performance may suffer in the longer term. It's a great troubleshooting tool though. Also I don't know how you do your backups but I simply have Windows Server Backup backup Hyper-V and its VMs and that works very well too. That has saved me when I have forgotten to make a checkpoint before going in blind and changing stuff just for kicks :D