Handle all traffic from within virtual environment



  • I'm running a Proxmox environment (with mostly default settings), and I'd like to have a pfSense instance running as a VM that all other VMs talk through to access the internet.

    When I boot a fresh pfSense install, I get an error during setup reading "No link-up detected." I've tried all the interface names I've seen suggested elsewhere: xl0, rl0, re0, re1, etc, but I'm still getting nothing. I currently have an IPv4 /30 and an IPv6 /64 I'd like pfSense to be able to handle. Is there something simple I'm missing, or is this a problem with the network settings on the hypervisor?



  • Which hypervisor are you using?



  • I have absolutely no experience with Proxmox but judging by the web page it appears to be based on KVM.



  • I have zero experience with KVM, but if it's complaining about no link, perhaps pfSense isn't detecting the virtual network adapters that you have specified.  I'm assuming yo specified two virtual adapters?  KVM appears to provide an Intel E1000 and a Realtek 8139, both of which are supported by FreeBSD 8.3.



  • Proxmox supports KVM and OpenVZ, but in the case of pfSense I've got it running in a KVM environment.

    For some reason, I didn't think to add the second network device to the VM. This should help illustrate how many times I've set up something like this before. sigh.

    Anyway, hooking up a second network interface at least gets me past the initial error message. Back in Proxmox, these are the network devices I have available:

    For the pfSense VM, I gave it net0 pointing to vmbr0 and net1 pointing to vmbr1. During configuration, I set em1 to be used for VLAN (which, as I understand it, is how I can set up an internal network for other VMs to communicate with each other without Internet access – correct me if I'm wrong), and WAN uses em0.

    Now when I try giving it a small block of addresses to work with, I get another cryptic error that Google doesn't seem to help with. Here's the message I got from the datacenter when they allocated the block:

    We have just assigned a block of 4 IP addresses that will be routed via 192.99.10.135:
    192.99.198.148/30
    Network IP: 192.99.198.148
    Gateway IP: 192.99.198.150 (not usable)
    Broadcast IP: 192.99.198.151

    192.99.10.135 is the IP this server started with when it was new, so this seems straightforward enough. However, when I choose option 2 (set interface IP address), say no to DHCP, and supply "192.99.198.148" as my network address with a bit count of 30, I get an error stating "You cannot set network address to an interface".

    Thoughts?



  • Try using .149.  A /30 only has 2 usable addresses.  I'm not a subnetting guru by any stretch, but I thought you can't use the network IP or the broadcast IP, so that leaves you with .149 and .150.  They have the gateway mapped to .150, so does that mean you only have .149 to play with?



  • Whatever the case may be, using .149 made the error go away. I got the "success" message saying I could access the webConfigurator at http://192.99.198.149/, but visiting that IP in my browser turns up nothing. I can't even ping that IP (from the hypervisor or from my home PC).

    This could be a misconfiguration in pfSense, in Proxmox, or at the datacenter. I really don't know.



  • Try to get at it from the LAN IP address.

    What are your WAN settings again, exactly?



  • Right above the main menu it shows "WAN (wan)      -> em0        -> v4: 192.99.198.149/30"

    Is there a way to just dump all the relevant configuration information? Is it all saved in a single file somewhere?



  • What's your WAN gateway?  If you can get into the WebConfigurator via your pfSense LAN IP address, you can see all this on one screen.



  • I can't get to the webConfigurator screen at all. If I drop into a shell, I can't even ping out (No route to host).

    Here's the output of ifconfig:



  • I see no IP addresses on those interfaces.

    Set WAN to 10.10.10.1 and LAN to 192.168.1.1 (obviously ensure no collisions with your existing subnets), and attempt to access the LAN side with a virtual machine that's on the same vNetwork as the LAN interface of the pfSense. At least this will allow you to verify that there are no faults in your local network config.

    Note that you will NOT be allowed to accesss pfSense remotely from the WAN addres until such time when you log into it from the LAN side and add firewall rules to allow access from WAN (i.e. a firewall rule that makes HTTPS open to the WAN address). OR, you can type

    pfctl -d
    ```which allows the bypass of the firewall component from the shell to enable access from WAN side of a freshly installed pfSense. However when you do this be advised that you will be opening up your pfSense to the whole world. Also ensure that you quickly add a rule to allow webUi access when doing this, because I believe there is a cronjob that un-does this command (a safety precaution no doubt)


  • He appears to have a WAN IP but no LAN IP.  According to your KVM output (and remember I'm new to KVM), it appears that you have the second NIC piped to a dummynet private LAN, and it isn't linked to ETH1 with a LAN IP address.  You need the pfSense LAN IP address to be something that is reachable from your desktop so that you can get at the GUI.  That means it has to be on the same subnet.



  • It seems like that install was all borked, so I reset to factory defaults and tried again. I didn't get a single error message, and this is what ifconfig looks like now:

    This looks like progress, but I still can't ping out from (or in to) pfSense. However, other VMs on the network are now able to lease a local IP through DHCP, so I'm getting somewhere.

    I can also get to the webConfigurator page now, although the only OS image I have loaded currently is a headless Ubuntu 14.04 image, and with no network access I can't install a graphical environment to get a browser. I'll sleep on it and see where I can get in the morning. I think this is almost there.



  • Progress is good.



  • @KOM:

    Progress is good.

    Indeed.

    Anyway, I got a graphical environment set up in a VM and I'm able to log into the webConfigurator. Here's what that screen looks like:

    I don't have an easy way to copy the full text of the crash report mentioned in that message (the machine still can't get Internet access), but here's the beginning of it:

    If this looks like something that needs to be investigated further, I can get more details.

    In the meantime, I'm going to get in touch with the data center and see if there's possibly something on their end that's preventing me from using the IP block.



  • Can you show the screen for Interfaces - WAN?





  • You have your gateway set to the broadcast IP.  Set it to the gateway as provided by your ISP, 192.99.198.150.



  • So I did. Whoops. Anyway, changing it to .150 hasn't changed the symptoms.



  • From the webGUI, can you go to Diagnostics - Ping and see if pfSense can ping the gateway?



  • @KOM:

    From the webGUI, can you go to Diagnostics - Ping and see if pfSense can ping the gateway?

    Nope. I get "ping: sendto: Host is down"



  • I'm starting to run out of ideas.  I don't know KVM and I don't know Proxmox.  Your ISP making you use one of your IP addresses as the gateway is strange from my perspective.  Usually the gateway is outside your usable range.  You can't even ping the gateway from another IP on the same subnet.

    Hopefully someone else can chime in.



  • I agree it's strange. I'm still waiting to hear back from their support folks.



  • I heard back from the support folks, and they said that all 4 IPs should be usable, and gave me this link in relation to IPv4 settings..

    It turns out the host requires you to use a MAC address they specify. I added that to pfSense and changed the gateway to .254 as they suggested, but I still can't get any connectivity. Here's my WAN settings now:

    Also, when I try changing the address to .148/30 (to cover all 4 IPs), pfSense gives me the error "This IPv4 address is the network address and cannot be used". Does this mean the support person was incorrect when they said all 4 IPs are usable, or is this just a limitation of pfSense?



  • The MAC addy thing is not uncommon, but it would have been nice for them to let you in on the secret before you wasted hours trying to get it working.

    OK, first off I would disable IP6 to reduce the complexity.  From the WebGUI, can you once again go to Diagnostics - Ping and ping the gateway?



  • After much back-and-forth, it turns out the problem was… ARP tables in a router I don't control!

    pfSense still wasn't happy with the fact that the gateway was outside my net range, but I was able to take care of that:

    route add -net YOUR.DEDICATED.IP.254 -iface em0
    route add default YOUR.DEDICATED.IP.254
    route del -net YOUR.DEDICATED.IP.254 -iface em0
    

    …and now other VMs can lease IPs via DHCP from pfSense and then access the Internet through it! Amazing how much better things work when they're not broken.

    Anyway, thanks so much for all the help. The root of the problem may have been out of my control, but I still picked up a few handy things from this thread.



  • It seems to be common that network problems you can't seem to resolve end up having an external factor.