VMWare VM with 4 ethernet NICs, can it be done?



  • Hello,

    I manage a lan gaming center, and have been using a pfSense based router on a VMWare machine since last summer for internet connection. It has been working GREAT for months, but now I wanted to upgrade with a new connection and I am having trouble doing it with VMWare. I had until now 2 DSL internet connections, each one has its own router.

    I am running:

    VMWare Workstation 5.0 (server is a Windows XP Professional)
    pfSense 1.2-BETA-1 (30/Apr/2007 built)
    Computer has 2 physical NICs, but I am currently using only one

    VM has 3 ethernet cards, "Ethernet" "Ethernet 2" and "Ethernet 3", all of them bridged
    "Ethernet" has is LAN and has IP 192.168.1.112 and is the default gateway for all computers on the lan
    "Ethernet2" has is WAN and has IP 192.168.2.112, it connects to 192.168.2.111 which is the router for internet connection 1
    "Ethernet3" has is WAN2 and has IP 192.168.3.112, it connects to 192.168.3.111 which is the router for internet connection 2

    I put the internet routers on different subnets so computers would not be able to connect to them directly, and because pfSense sometimes was giving me warnings like "packet received on WAN that is for LAN", since they all share the same physical NIC. Using different IP ranges for each interface made those warnings go away.

    I am also using multiwan, so the traffic is routed to the WAN interface I want given the destination port. That way I can route the game ports I want on one internet connection (which will give gamers lower ping) while routing web traffic, IM, and unknown traffic on the other connection, which has more bandwidth but also latency. That way ping times on games won't be affected by downloads and file transfers.

    Now we have a third internet connection, and I wanted to add another interface on the VMWare router, but I found out that VMWare will not let me add a fourth Ethernet adapter. If I create a VM machine, it comes with one adapter by default. I can add a second and a third one, but when there are already 3 adapters on the machine it will not let me add a fourth ethernet adapter.

    So I tried editing the freebsd.vmx and add the 4th adapter by hand, and it kinda work; the 4th adapter was there and pfSense recognised it, but after configuring everything it didn't quite work well. It was like a non working ethernet adapter, which makes me think that it might be a limitation on VMWare virtual machines, 3 NICs might be the maximum :(

    I would like to know if anyone has been able to make a VM work with 4 adapters or more. I would not mind puting some more physical NICs on the computer, it has 2 NICs right now, but I am only using one. It has several free PCI slots and I Have plenty of old 3Coms 905C NICs. I am also considering using some older computer dedicated as a pfSense router, by just plugin into it 4 3Com 905C lan cards, and forgeting about the VM all toguether, but for about 9-10 months my experience with the pfSense VM has been great so far.

    Any comments will be welcome, thank you!



  • To add more than three nics you need to edit the VMX file manually (or upgrade VM Workstation)

    Add this to the end of you VMX file:
    ethernet1.present = "TRUE"
    ethernet1.wakeOnPcktRcv = "FALSE"

    Once you launch the VM aditional lines will be generated for you.  You can then modify the NIC in the GUI.



  • Hello Seth,

    That's what I tried already, and it did seem to work (in the sense that I did have 4 NICs, and pfSense recognised them and allowed me to configure them).

    I basically copied:

    ethernet2.present = "TRUE"
    ethernet2.addressType = "generated"
    ethernet2.generatedAddress = "00:0c:29:25:96:d0"
    ethernet2.generatedAddressOffset = "20"
    ethernet2.connectionType = "bridged"
    ethernet2.vnet = "VMnet2"

    To:

    ethernet3.present = "TRUE"
    ethernet3.addressType = "generated"
    ethernet3.generatedAddress = "00:0c:29:25:96:da"
    ethernet3.generatedAddressOffset = "30"
    ethernet3.connectionType = "bridged"
    ethernet3.vnet = "VMnet3"

    So I had NICs "00:0c:29:25:96:bc" "00:0c:29:25:96:c6" "00:0c:29:25:96:d0" and "00:0c:29:25:96:da" with generatedAddressOffset "0" "10" "20" and "30" (no clue on what is this generatedAddressOffset for).

    On VMWare I could see the 4 NICs, pfSense recognised them all. I could configure them and I added something like:
    "Ethernet4" has is WAN3 and has IP 192.168.4.112, it connects to 192.168.4.111 which is the router for internet connection 3

    But it just didn't work out that easily. I do not quite remember now the "symptoms" because I tried it like 4-5 weeks ago, but I will try it again asap and report back. It was like if traffic to that 4th NIC just wasn't going out. With 3 NICs I can use any two of the WANs with no problems, they are tested ok, as soon as I added a 4th NIC, problems started, so I thought that maybe VMWare wasn't prepared to work with 4 NICs, and that editing the VMX file just wouldn't cut it.

    I'll try it again when I have a couple of hours alone when I can take down the current pfSense router and give some more info, thanks!



  • generatedAddressOffset ensures that you have unique MAC addresses.

    At this point I would say that your FW Rules haven't been configured.

    or

    Your VM is also dual homed on the same subnet.
    eth2 and eth3 specify two different virtual networks, VMnet2 and VMnet3.  These are user specified.  VMnet0, VMnet1 and VMnet8 are created for you during install.  VMnet2 and VMnet3 cannot be on the same subnet in your case 192.168.4.0/24.  This may be the root of your problem.  Check the subnet in Virtual Network Editor.  Edit | Virtual Network Editor - Summary - Subnet.  You need to be a local admin to change setting here.



  • My 3 (or 4) virtual ethernet adapters are all bridged, connected to the physical network through the only physical NIC on that server that has a link to the rest of the network (switches), so they all belong to VMNet0. The other physical NIC is unused (integrated, Realtek 8129 I would say, I do not use it)

    I will get back when I have time to do the tests again and tell exactly what were the problems, and when I can report with some exact problem and screenshots. I think all my rules were ok, after all I only needed to set up a 3rd WAN connection, and route the traffic with destination port whatever to that connection to try. I usually just route web traffic to one or another connection, and visit some web page that will show me my IP and do a ctrl+F5 to see exactly what connection I am going out through, all 3 connections are on different ISPs, so I can tell instantly if I am going out on the connection I wanted to.

    Thank you, I'll try to go a couple hours earlier tomorrow to work and do the tests again with the 4th adapter.



  • Ok, so here is so far the outcome of my tests done this morning.

    Before:
    WAN* -> le0 -> 192.168.2.227
    LAN* -> le1 -> 192.168.1.112
    OPT1(Jazztel) -> le2 -> 192.168.3.226

    For adding the 4th NIC adapter, I edited freebsd.vmx and added just oneline at the end:
    ethernet3.present = "TRUE"

    After booting up the VM and shutting it down later, I noticed VMWare has also added the following lines:
    ethernet3.addressType = "generated"
    ethernet3.generatedAddress = "00:0c:29:25:96:da"
    ethernet3.generatedAddressOffset = "30"

    So the 4th NIC seems to be there and working correctly.

    After booting up and doing an "1) Assign Interfaces" from the console, I had the following:

    WAN* -> le0 -> 192.168.2.227
    LAN* -> le1 -> 192.168.1.112
    OPT1(OPT1) -> le2 -> NONE
    OPT2(OPT2) -> le3 -> NONE

    Then I went to the WebGUI and configured the OPT1 and OPT2 interfaces

    WAN* -> le0 -> 192.168.2.227
    LAN* -> le1 -> 192.168.1.112
    OPT1(Jazztel)* -> le2 -> 192.168.3.226
    OPT2(Telefonica)* -> le3 -> 192.168.4.228

    So I basically have:

    192.168.1.112 LAN
    192.168.2.227 WAN -> gateway 192.168.2.112
    192.168.3.226 OPT1 -> gateway 192.168.3.111
    192.168.4.228 OPT2 -> gateway 192.168.4.113

    192.168.2.112 , 192.168.3.111 and 192.168.4.113 are physical routers to 3 different internet connections

    Problem with this setup; it seems as if OPT2/le3 is not working correctly (even I would say it is configured 100% the same was as OPT1/le2). I have different rules for traffic, so I can basically choose what internet connection the traffic will go out through. In rules -> LAN I have the following 2 rules (and more that do not matter now):

    Proto Source Port Dest Port Gateway Desc

    ICMP LAN net * * * 192.168.3.111 ICMP
    TCP LAN net * * 80 (HTTP) 192.168.2.112 Web

    So, as it is configured now, HTTP traffic will go out on WAN interface (192.168.2.112) and ICMP traffic will go out on OPT1 interface (192.168.3.111)

    So far, so good, this is working since I am only using WAN and OPT1, this is how I had it working until now (and how it is working now).

    More on next post…



  • Ok, so with this setup I can basically change my rules (Firewall: Rules -> LAN) and choose what internet connection my web traffic or my ICMP traffic will go out. This way I can easilly just go to a web page like whatismyipaddress.com and see exactly what internet connection I am using, or I can open a CMD window and with a "tracert www.google.com" I can see what router I am going out through.

    So I can put 192.168.3.111 as gateway for the ICMP rule, and with a "tracert www.google.com" I will get:
    (this tests also apply to web traffic. On WAN and OPT1 I can do web browsing ok, on OPT2 web pages will not load)

    Traza a la dirección www.l.google.com [209.85.135.99]
    sobre un máximo de 30 saltos:
      1     2 ms     1 ms     2 ms  192.168.3.111
      2     6 ms     6 ms     6 ms  197.217.106.212.static.jazztel.es [212.106.217.197]
    

    And I can put 192.168.2.112 as gateway for the ICMP rule, and I will get:

    Traza a la dirección www.terra.es [213.4.130.210]
    sobre un máximo de 30 saltos:
      1     2 ms     2 ms     3 ms  192.168.2.112
      2    14 ms    18 ms    11 ms  static-10-0-235-87.ipcom.comunitel.net [87.235.0.10]
    

    But then, if I put 192.168.4.113 as gateway for the ICMP rule, I get:

    Traza a la dirección www.l.google.com [209.85.135.99]
    sobre un máximo de 30 saltos:
      1     *        *        *     Tiempo de espera agotado para esta solicitud.
    

    Doh, not working. But then, on the WebGUI if I go to Diagnostics -> Ping , and ping 192.168.4.113 (physical router), I do get an answer:

    PING 192.168.4.113 (192.168.4.113) from 192.168.4.228: 56 data bytes
    64 bytes from 192.168.4.113: icmp_seq=0 ttl=255 time=1.371 ms
    64 bytes from 192.168.4.113: icmp_seq=1 ttl=255 time=0.845 ms
    64 bytes from 192.168.4.113: icmp_seq=2 ttl=255 time=0.937 ms
    --- 192.168.4.113 ping statistics ---
    3 packets transmitted, 3 packets received, 0% packet loss
    round-trip min/avg/max/stddev = 0.845/1.051/1.371/0.229 ms
    

    If I switch around the interface IP/gateway for OPT1 and OPT2 (I put ISP Jazztel on OPT2, and ISP Telefonica on OPT1) results are the same, OPT1 works ok with Telefonica's router, and OPT2 still doesn't work, this time with Jazztel's router.

    Here I put Telefonica IP and gateway on OPT1, router ICMPs on 192.168.4.113, and it does work!

    Traza a la dirección www.l.google.com [64.233.183.99]
    sobre un máximo de 30 saltos:
      1     1 ms     1 ms     1 ms  192.168.4.113
      2    35 ms    35 ms    36 ms  192.168.153.1
      3    35 ms    36 ms    35 ms  145.Red-81-46-34.staticIP.rima-tde.net [81.46.34.145]
    

    NOTE. The 192.168.153.1 IP does not belong to my network

    So, from the first time I tried all this, it made me think that VMWare didn't get along with the 4th NIC card I had created, and that's why it was giving me problems. If it should be working ok with 4 virtual NICs, I guess then that my problem must be somewhere else. I was even about to just grab an old computer, plug 4 3com NICs on it, and try it out there…

    I'll try to post some screenshots on my current config right now, thanks for reading! :)



  • Do your physical routers to which your OPT2 and 3 go know the route back to where the request comes from?



  • Here is my LAN interface (sorry about the huge width of the images, I made them with a firefox extension and have no image editing program here):

    Here are my three WAN interfaces, WAN OPT1 and OPT2:


    Here are my main firewall rules. I basically choose what traffic to route on OPT1 by destination port, and the traffic that doesn't match any rule usually is routed on WAN.

    Rules that would route a given type of traffic to WAN I usually have them disabled, since traffic will go out on WAN anyway. I just leave the routes in there for testing purposes, like, web traffic will always be going out on WAN, but for testing the other connections I enable the rule and change the gateway, so I can see what connection my web traffic is going through.

    Thanks!
    Aitor



  • @GruensFroeschli:

    Do your physical routers to which your OPT2 and 3 go know the route back to where the request comes from?

    Hello,

    I guess so. They are just normal ADSL routers, one is a Comtrend, the other one is a Telsey. If I put on my notebook fixed IP 192.168.2.1 gateway 192.168.2.112 or IP 192.168.4.1 gateway 192.168.4.113 I can use those connections with no problem at all. If I configure OPT1 to use the 192.168.4.113 router it works ok, if I configure it on OPT2 it doesn't seem to work.

    Note that I have the 3 internet connections as WAN OPT1 and OPT2. I do not have OPT3.

    Thanks!
    Aitor



  • I assume you didnt add a static route to your LAN on this router, did you?
    In this case this router does not have a clue that your LAN-subnet even exists.
    –> It can not answer to anything you send over it.

    Add a static route on this router pointing to your OPT2 interface for your LAN-subnet.



  • @GruensFroeschli:

    I assume you didnt add a static route to your LAN on this router, did you?
    In this case this router does not have a clue that your LAN-subnet even exists.
    –> It can not answer to anything you send over it.

    Add a static route on this router pointing to your OPT2 interface for your LAN-subnet.

    Mmmh nope, I did not add anything, no static routes. I am not sure I understand though, I need to add a route on the 192.168.4.113 router? Or on pfSense? If I configure OPT1 interface to make use of the 192.168.4.113 router it does work ok, so I did not think I would have to modify anything on the router… I did not have to add anything on the other 2 routers for the other 2 connections when I originally setup pfSense, any hints on what exactly do I have to do will be appreciated.

    Thanks Gruens!



  • This is what I have on the 192.168.4.113 router (Comtrend) that relates to routing. Destination 192.168.4.0 traffic goes out on br0 interface, so I think that should be ok. Otherwise it would not be responding to pings from the pfSense router on 192.168.4.228?

    On Advanced Setup -> Routing -> Static Route I have no info.

    Thanks!



  • On your WAN you're NATing traffic from the LAN.
    Meaning for the "real" router it seems as if the traffic comes from an IP in a known subnet.
    On the OPT interface no traffic is NATed.
    So for the "real" router the traffic seems to come from an unknown subnet.

    You can either configure pfSense so it NATs traffic on the OPT interface,
    or add a static route on the "real" router (192.168.4.113) with as destination 192.168.4.228 for the subnet 192.168.1.0/24



  • @GruensFroeschli:

    On your WAN you're NATing traffic from the LAN.
    Meaning for the "real" router it seems as if the traffic comes from an IP in a known subnet.
    On the OPT interface no traffic is NATed.
    So for the "real" router the traffic seems to come from an unknown subnet.

    You can either configure pfSense so it NATs traffic on the OPT interface,
    or add a static route on the "real" router (192.168.4.113) with as destination 192.168.4.228 for the subnet 192.168.1.0/24

    Now I feel so stupid…

    You are completely right. I didn't have a clue that traffic on OPT interfaces was not NATed by default, and I was asuming that if it was working ok on OPT1, it would also work ok on OPT2 . What I was failing to remember was that I actually configured it that way when I set up pfSense multiwan months and months ago.

    So traffic was coming out of 192.168.4.113, but incoming traffic didn't know where to go and was being dropped by the physical router. He was geting traffic for 192.168.1.x IPs, and it was just being discarded.

    So I had to go to Firewall : NAT : Outbound, which I have configured for Manual Outbound NAT rule generation (Advanced Outbound NAT (AON)), and there was my solution. I only had rules for WAN and OPT1, and I just copied the rule for OPT1 and created a new one for OPT2.

    Changing the ICMP rule to go out on OPT2, tried a "tracert www.google.com", and there it is, working without problems now.

    Traza a la dirección www.l.google.com [66.249.93.99]
    sobre un máximo de 30 saltos:
      1     1 ms     1 ms    <1 ms  192.168.4.113
      2    45 ms    37 ms    37 ms  192.168.153.1
      3    35 ms    35 ms    36 ms  145.Red-81-46-34.staticIP.rima-tde.net [81.46.34.145]
    

    I will do some more testing on monday but I am sure that this will be it. I can't do too much testing now since it is friday evening, or otherwise if I mess up and have to reboot the router I will have 3 dozens of crazy kids yelling at me because they have been kicked out from WOW or lost the DOTA match they were about to win.

    Thanks!
    Aitor



  • btw: why are you running this in a VM?
    Wouldnt it be easier to just install pfSense on the hardware itself?



  • Mainly because it is not a dedicated computer, it's a multi purpose WinXP computer that has right now 3 VMs:

    • pfSense
    • Win98 SE with out custom cybercafe management software
    • Debian (file sharing, squid proxy)

    I also use often the XP part by using MSTSC, VNC, some web browsing, a couple of programs for file sync, …

    I actually gave it a thought on just building a dedicated machine for pfSense, but I am not sure if it would be worth it. It would mean one more computer on 24/7, and I doubt that the performance would be a lot better, ping times would go down but maybe 1ms..? I do not know if I would even notice the difference.

    With several old computers laying around I might give it a try just to see if it would be worth it. Makes me wonder if I could run it from a bootable pendrive, so at least I could save myself the HD and its noise/heat.

    I have also grown used to VMs over the years for many tasks. It makes it soooo much easy to just backup a full machine, move it to different hardware, and never bother installing too much crap on the main computer. I even have some VMs dedicated to things like Visual Basic 6, I still use it once in a while, and almost every time I needed it it meant reinstalling, so now I just have a VM with VB6 installed that I boot up when I need it, so I will never need to install VB6 on the main computer.



  • From a security-point it's not such a good idea.
    Search the forum, since there are quite a few threads why running a firewall in a VM is insecure.

    I've been running for quite some time a windows xp "server" too.
    Ok it wasnt a server but just a computer that was running 24/7.
    I noticed that about every 20-30 days you HAVE to restart. (windows behaving really really strange)
    I dont know if you run into the same problem.

    If you're running pfSense on dedicated hardware you could take some lowpower embedded system since you dont seem to use packages.
    (Like http://pcengines.ch/alix2c3.htm )

    Ping wise…
    I dont know. You could run a longtime test through your VM-pfSense.
    I think i just saw in cT (german computermagazone) a free software which is capable of displaying ping tests over long periods of time.

    If your customers are playing FPSs it would make a difference if they have 100 or 50 ms latency.



  • I am positive that everything would be a bit more secure on dedicated hardware than on virtual machines, but the data I might be keeping is not critical at all. My weakest security would probably be someone just walking to one of the servers since we open 7 days a week and I just won't always be there, the fact that pfSense or any other server/service is running on dedicated hardware or a VM won't make me lose my sleep :)

    There are several reasons that could make me switch to a dedicated hardware solution:

    • Ping times. I doubt the difference between VM or dedicated would be 100 or 50ms, but maybe more like 51/52 or 50ms. I haven't even checked the difference between direct connect to the DSL routers or through pfSense and from there to the routers, but if I had to guess I would say that pfSense in the middle doesn't add more than 1/2ms.
      If you know of any tool to do some tests on latency times I would love to know, since always I have done test on this they have been by hand (ping for x hours and make an avarage, use a gamer tool like HLSW to make me an avarage of ping times to a game server over a period of time, etc). If I had a scientific way to say "VM pfSense adds 5ms to my traffic, dedicated hardware pfSense adds only 1ms" that would be a good reason to make the switch.

    • Good hardware, I once looked into it but didn't find anything that seemed good for me. It is not easy to find this kind of hardware in Spain, I looked some ITX computers and things like that. The one you linked me too seems great, but they do not seem to have any with 4 lan ports which is what I think I would need (I could probably use just 3, yeah, or even one since that is what I am using now with pfSense on a VM). I see it at 98.20E + taxes here in Spain, another 12.20E + taxes for the box, it doesn't seem so bad. It would be about 128E and I would just need a CF card, and I think that the power suply, which doesn't seem to be included.

    • About two years ago I retired an old P2-266Mhz 128Mb 2Gb HD which was running 98SE with our own management software. I just moved it to a VM on this computer for several reasons; it was one more computer running 24/7, which added to more wasted electricity, more noise, more dust, more heat, … If I go the other way now with the pfSense router it would make me feel like going backwards. Yeah, I would do it if I had a good reason, better ping times, good hardware like the one above that has a low power usage/noise/heat disipation, etc.

    I would love to try more things with pfSense, but my time (and knowledge) is limited. I could try it on a physical computer to see if it improves, I would love to have failover so if one connection fails traffic would be router on another one, I would like to use it as a proxy with squidguard, or a file server... I just love breaking things that work ok just for the heck of fixing and improving them :)



  • Everything seems to be working just great over here!

    I have now the 3 internet connections working of a single pfSense router on a VM, and all with only one physical NIC on the computer. I have one connection for WOW, Warcraft3, Quake3, UT, Guild Wars, Warsow, and some more, another connection for BF2, Steam games, CS, TF2, and a third one with more bandwidth but worse latency for web, IM, and any kind of unknown traffic. I love it! :D

    When I have the time I will take a look if it is possible for me to implement failover, so if one connection is offline traffic can be redirected automatically to another one, and I would also like to try to use pfSense on a physical computer, to see if there are good improvements vs running it on VM.

    Many thanks!
    Aitor


Locked