Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Extremely strange network issue with virtualised pfsense

    Scheduled Pinned Locked Moved Virtualization
    19 Posts 5 Posters 9.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      Mike-Bracknell
      last edited by

      Hi all,

      First post here, be gentle :)

      I have a customer with a Dell Poweredge T110 running ESXi 4.1.0 - upon which there are 2 VMs: SBS2008 and pfsense

      They were running well for ages with just the one SBS2008 VM, but the Draytek router I had was giving them bad traffic shaping for their Grandstream VoIP handsets so I added a single Intel NIC and a pfsense VM.

      Pfsense was built into VM with two E1000 NICs specified, and a second virtual switch was created (called WAN) and the intel NIC attached to that.  The LAN leg of pfsense therefore shared the inbuilt Broadcom NIC with the SBS2008 VM.

      Everything worked fine for about a week, and then I had a report from the customer that 'nobody can use the network'.  I thought the pfsense VM might have died at that point, but I was able to use remote desktop from my house and get into it.  Since the server couldn't see the clients or the Grandstream handsets, and vice versa, then I diagnosed a broken switch and drove over to replace it.

      With it replaced, everything worked again.

      A couple more days passed, and I went to their site to talk to the manager on another matter.  Just as I turned up, they said "glad you've arrived, we're having problems with the internet".  This turned out to be sporadic lack of DNS resolution for some sites from desktop PCs (although they would work fine from the server).  I therefore thought "something's wrong with pfsense" and rebooted the VM.  I thought this had fixed things, but some lack of resolution and slow network performance continued.  However, the customer then had a perfectly good VoIP conversation with a customer via his desk phone, through the PBX (sat in the SBS2008 box) and onto the internet.

      The symptoms went away shortly afterwards, so I left site.  I was rung up an hour later saying "nobody can get to the network", so by now fearing the worst I grabbed a bunch of things (switches, an intel dual NIC card, etc) and rushed back to site.

      I found when I came back to site that the only person who could see anything was the manager who's PC wasn't piggybacked off the Grandstream desk phone, but everyone else was dead on the network.  I plugged my laptop in to the same switch as the server, and couldn't even PING the SBS server on the LAN.  Even changing switch ports, LAN cables was fruitless.

      I therefore decided to insert the Intel dual NIC card and split the functions.  The Intel card now does both WAN and LAN functions for pfsense, and the SBS VM is back on the inbuilt Broadcom NIC, and it appeared to work better for the rest of Friday (yesterday).

      I'm therefore currently at a loss as to why I couldn't even PING the SBS server from the LAN when the trouble occurred.

      My current choice of culprits are:

      • that the Broadcom NIC ran out of resources running 2 VMs at once with pfsense
      • that the Netgear switches have some sort of ARP (or other) issue with 2 VMs running on the same switch port which hit a DoS trigger on them and blocked traffic
      • that there's a bug in VMware 4.1.0 in regard to networking of some sort that would make this happen

      Does anyone else have any suggestions as to why it would affect things this badly?  I don't have any issues with running pfsense virtualised on VMware at my house on an HP Microserver with dual Intel NIC, and it's been up for ages.  They're both on the same version (indeed the same ISO built both), and the version details are:

      2.0.1-RELEASE (i386)
      built on Mon Dec 12 17:53:52 EST 2011
      FreeBSD 8.1-RELEASE-p6

      I can post any more info I can get, but you'll need to tell me how to get it if it's Linux (as i'm a Windows guy mainly)

      Cheers,
      Mike.

      1 Reply Last reply Reply Quote 0
      • M
        Mike-Bracknell
        last edited by

        Another update….

        Customer rang this morning to say that it's gone again.  I therefore moved everything off the Broadcom NIC and got the customer to unplug it - still no joy, so that rues out the Broadcom.

        1 Reply Last reply Reply Quote 0
        • B
          biggsy
          last edited by

          …if it's Linux (as i'm a Windows guy mainly)

          You didn't happen tell ESXi that pfSense was Linux when you built the VM?

          It would probably be better to run ESXi 5.1 now than 4.1 now.

          1 Reply Last reply Reply Quote 0
          • M
            Mike-Bracknell
            last edited by

            No, just checked (was going to kick myself if I had), but it's FreeBSD 64bit (identical settings and identical ISO that I built my HP Microserver pfsense from that i'm typing this behind).

            The only difference my HP Microserver ESXi has is that it's running 4.1.0 build 502767 versus build 348481 on the Dell.

            1 Reply Last reply Reply Quote 0
            • M
              Mike-Bracknell
              last edited by

              Hold on….......

              Does "2.0.1 release (i386)" mean it's a 32bit ISO in a 64bit VM??

              This might explain something..... (although it wouldn't explain why the HP Microserver at my home is working fine).

              1 Reply Last reply Reply Quote 0
              • S
                Supermule Banned
                last edited by

                348481 is 4.1U1 and the 502767 is 4.1U2

                So different ESXi :)

                1 Reply Last reply Reply Quote 0
                • B
                  biggsy
                  last edited by

                  …although it wouldn't explain why the HP Microserver at my home is working fine

                  Maybe your customer's machine is under bit more load than yours and some situation just doesn't get handled properly.

                  Not sure about that 32-bit image running in a 64-bit VM thing but it's gotta be suspect.

                  It's probably easier to go straight to 5.1 than it is to update 4.1u1 to u2 or u3.

                  1 Reply Last reply Reply Quote 0
                  • S
                    Supermule Banned
                    last edited by

                    5.1 has some issues relating to FreeBSD and I would suggest 4.1U3 until 5.1U1 is released.

                    1 Reply Last reply Reply Quote 0
                    • B
                      biggsy
                      last edited by

                      @Supermule:

                      5.1 has some issues relating to FreeBSD and I would suggest 4.1U3 until 5.1U1 is released.

                      What sort of issues?  I thought I'd checked pretty thoroughly for any reported problems before going to 5.1 about a week ago.

                      1 Reply Last reply Reply Quote 0
                      • johnpozJ
                        johnpoz LAYER 8 Global Moderator
                        last edited by

                        Been running my pfsense on 5.1 since the day it released and not issue one.. So what are these issues your going on about?

                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                        If you get confused: Listen to the Music Play
                        Please don't Chat/PM me for help, unless mod related
                        SG-4860 24.11 | Lab VMs 2.8, 24.11

                        1 Reply Last reply Reply Quote 0
                        • S
                          Supermule Banned
                          last edited by

                          http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2032803

                          Cant find the rest right now…

                          1 Reply Last reply Reply Quote 0
                          • M
                            Mike-Bracknell
                            last edited by

                            Well, whichever it is i'll be doing it in a week or so's time.  I've just built an identical pfsense Microserver here in the office so that we can try it out for a week, then at least I know there's likely to be a solution for my customer (despite me needing to migrate VMs across whilst I upgrade the server).

                            1 Reply Last reply Reply Quote 0
                            • S
                              Supermule Banned
                              last edited by

                              Are you using E1000 NIC's in both boxes?

                              1 Reply Last reply Reply Quote 0
                              • johnpozJ
                                johnpoz LAYER 8 Global Moderator
                                last edited by

                                "Enhanced VMXNET adapter cannot obtain a DHCP IP address with FreeBSD 8.2"

                                Ok not using vmxnet, nor are we using 8.2 of freebsd ;)

                                So that would explain why this has been non issue for me..  And would never have used the vmxnet2 in the first place, article states it works with vmxnet3.  But using e1000, there was a benchmark done a few threads back and they sure didn't notice any real performance increase with them, and I had some issues with access ipsec vpn on the outside of the pfsense box from clients inside when using them.  Went back to e1000 and no issues.

                                An intelligent man is sometimes forced to be drunk to spend time with his fools
                                If you get confused: Listen to the Music Play
                                Please don't Chat/PM me for help, unless mod related
                                SG-4860 24.11 | Lab VMs 2.8, 24.11

                                1 Reply Last reply Reply Quote 0
                                • P
                                  podilarius
                                  last edited by

                                  I have used pfsense on a dell r710 before and it worked … I was not firewalling though ... just a soft router. I also have 128GB of memory and 4 CPUs, so I was not hurting for resources. Even though I had upwards of 30 VMs running at one time. props to pfsense.

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    biggsy
                                    last edited by

                                    @Mike-Bracknell:

                                    Well, whichever it is i'll be doing it in a week or so's time.

                                    Please post again and let us know how it goes.

                                    1 Reply Last reply Reply Quote 0
                                    • M
                                      Mike-Bracknell
                                      last edited by

                                      @Supermule:

                                      Are you using E1000 NIC's in both boxes?

                                      Yes, I have no need to go to VMXNET-based NICs, and one nagging doubt I had was that there may have been E1000-based code in FreeBSD which didn't translate well to the Broadcom NIC, but I must say it was incredibly weird to find that other VMs on the server were affected by the pfsense VM.

                                      1 Reply Last reply Reply Quote 0
                                      • M
                                        Mike-Bracknell
                                        last edited by

                                        @biggsy:

                                        @Mike-Bracknell:

                                        Well, whichever it is i'll be doing it in a week or so's time.

                                        Please post again and let us know how it goes.

                                        Will do.  However, so far so good with the Microserver running a 32bit VM of a 32bit OS, and i'm sure my web developers will test pfsense a hell of a lot more than a training company do :)

                                        1 Reply Last reply Reply Quote 0
                                        • M
                                          Mike-Bracknell
                                          last edited by

                                          Just to close off on this, I have rebuilt the 32-bit PFSense in a 32-bit VM container, and it's been stable for a week now.  I think that must have been the issue.  Glad to have spotted that or it would have driven me round the bend!

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.