Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    New esxi 5 build with pfsense help with NICS

    Scheduled Pinned Locked Moved Virtualization
    17 Posts 6 Posters 9.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      esxiNOOB
      last edited by

      new esxi 5 build with pfsense help with NICS

      hi community,
      Thanks for all the post so far and the great information.  Although, I'm unable to find an answer to my question.  I have 5 nics on my server. 2 onboard, 2 intel cards, and one ipmi.

      i have three vswitchs set up.

      1 off the board (general use), 1 intel (for wan), and 1 intel (for lan).

      instead of taking a screen shot i'll just use this guy's image below.
      notice how the lan only say 100mb?  well mine is acting the same way. i get 1gbit to the wan, and only 100mb to the lan even though they are identical brand and model cards.  strangely inside pfsense webconfig shows both interfaces as 1000mb uplink.
      All my hardware indicators show the connection is truly in a 100mb state. the managed switch and everything else is set to auto negotiate.
      can someone PLEASE help?!

      extra info:
      driver set to e1000 on all nics
      32 bit version of pfsense 2.0
      no vmware tools installed yet.
      esxi v 5.0.0

      Please advise if i should scrap this VM and go with a different config.

      1 Reply Last reply Reply Quote 0
      • johnpozJ
        johnpoz LAYER 8 Global Moderator
        last edited by

        Do you have a bad cable that is not capable of gig?  Did you set the interface speed with esxi? What is the nic connected too?

        I run pfsense on esxi, and I get 1000 on both interfaces.

        Here is info on how to set it
        http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004089

        An intelligent man is sometimes forced to be drunk to spend time with his fools
        If you get confused: Listen to the Music Play
        Please don't Chat/PM me for help, unless mod related
        SG-4860 24.11 | Lab VMs 2.8, 24.11

        1 Reply Last reply Reply Quote 0
        • E
          esxiNOOB
          last edited by

          thanks for the reply.

          the nic goes to a netgear prosafe gigabit smart switch.

          as i said earlier everything is set to auto negotiate.

          so according to the link you sent me (picture taken below).  I have it set up in the recommended setting, since both devices are gigabit worthy and in auto negotiate state.  It's too bad the netgear won't allow me to set the port to gbit. You can force all settings but gigabit, for that it has to be auto.

          but what the hell, I'll go against the recommendations and force esxi 5 to put the card into gigabit. I'll report back when i get a chance.

          1 Reply Last reply Reply Quote 0
          • E
            esxiNOOB
            last edited by

            well i manually changed the settings to 1000mb full duplex and all the indicators are showing it's running in gigabit now.

            still stumped why the auto wouldn't work, and i'm having to do this non recommended setting.

            1 Reply Last reply Reply Quote 0
            • B
              biggsy
              last edited by

              strangely inside pfsense webconfig shows both interfaces as 1000mb uplink

              That's because pfSense is only seeing the virtual NIC, which is presenting as a Gb/s vNIC.

              notice how the lan only say 100mb?

              That actually looks like the guy's WAN, given that he's got it labeled PPPOE.  Probably his modem only does 100 Mb/s on its LAN interface.

              driver set to e1000 on all nics

              Again, this is talking to a vNIC not a real one.

              Sorry, all the above doesn't explain why the physical NIC doesn't auto-negotiate to Gb speed.  I would be with johnpoz on the bad cable idea.  I've had brand new (but cheap) supposedly Cat 6 cables only run up to 100Mb/s before.

              1 Reply Last reply Reply Quote 0
              • E
                esxiNOOB
                last edited by

                thanks for your knowledge on the subject biggsy.

                everything you said makes sense, except "That's because pfSense is only seeing the virtual NIC, which is presenting as a Gb/s vNIC."  How can it be presenting as a Gb/s vNIC when on the config page of the main host it shows 100mb?  is that because it's capable of Gb speeds?

                I swapped the cables and nics around.

                before:
                red cable: modem -> intel nic 1 = 1000mb/s
                green cable: intel nic 2 -> netgear smart switch = 100mb/s

                after:
                green cable: modem -> intel nic 1 = 1000 mb/s
                red cable: intel nic 2 -> netgear smart switch = 100mb/s

                switching the nics produced the same results in all test. (switched the config in pfsense em0/em1)

                ugh if i put a vm of win 7 on that nic. it goes to 1000 mb/s.

                it almost seems like pfsense is too slow in the auto negotiation. the 100mb light will blink on, then off for a sec, then back on with 100mb lit.
                with win 7 it blinks on with 1000 mb connection and never turns off.

                i haven't noticed any package losses. is there any way to test this connection i have going from switch to pfsense lan?

                at first i thought i could put the management network on this nic and test it, but when i do that it goes to 1000mb/s

                :-\

                forcing the nic to 1000 mb/s does produce the desired results visually. I just need a way to test it.

                1 Reply Last reply Reply Quote 0
                • johnpozJ
                  johnpoz LAYER 8 Global Moderator
                  last edited by

                  "it almost seems like pfsense is too slow in the auto negotiation."

                  What?

                  
                  em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
                          options=9b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum>ether 00:50:56:00:00:01
                          inet6 fe80::250:56ff:fe00:1%em1 prefixlen 64 scopeid 0x2
                          inet 24.13.xxx.xxx netmask 0xfffff800 broadcast 255.255.255.255
                          nd6 options=1 <performnud>media: Ethernet autoselect (1000baseT <full-duplex>)
                          status: active</full-duplex></performnud></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum></up,broadcast,running,simplex,multicast> 
                  

                  As you can see from above and attached pfsense is running 1000 fully autoselected.

                  pfsensegig.png
                  pfsensegig.png_thumb

                  An intelligent man is sometimes forced to be drunk to spend time with his fools
                  If you get confused: Listen to the Music Play
                  Please don't Chat/PM me for help, unless mod related
                  SG-4860 24.11 | Lab VMs 2.8, 24.11

                  1 Reply Last reply Reply Quote 0
                  • M
                    matguy
                    last edited by

                    The way VMWare virtualizes the networking the virtual NIC that pfSense sees ignores the speed your "real" NIC on your ESX(i) box is connected as.

                    In practice, internally in ESX(i) the "real" NIC is connected to a virtual switch.  The virtual NIC that pfSense sees is connected to that virtual switch.  Just like in a real switch, even though the uplink to the outside world may be only 100Mb, another connection to an end device may be Gigabit (or more) and the connection negotiation won't be limited down based on another port's link speed, but the traffic would still be limited.

                    This explanation doesn't fix your issue, but tries to explain why pfSense can still "link" at Gigabit while your "real" NIC is only linking at 100Mb.

                    I would still look at the cable, a faulty cable can still link at 100Mb and work fine a that speed, but not be able to auto negotiate for Gigabit if one (or multiple) of the non-transmit/receive pins/wires are bad.  At least test a replacement cable.

                    1 Reply Last reply Reply Quote 0
                    • E
                      esxiNOOB
                      last edited by

                      i get it now matguy. thanks for the explanation!!

                      1 Reply Last reply Reply Quote 0
                      • E
                        esxiNOOB
                        last edited by

                        i'll try yet another cable.. and even a trendnet unmanaged gigabit switch.
                        i'm betting the fault is in the netgear prosafe gs 108t

                        1 Reply Last reply Reply Quote 0
                        • M
                          matguy
                          last edited by

                          @esxiNOOB:

                          i'll try yet another cable.. and even a trendnet unmanaged gigabit switch.
                          i'm betting the fault is in the netgear prosafe gs 108t

                          Oh, I missed the part where you were going to swap the cables/nics around.  But, trying a different switch is a good troubleshooting step.  I've used the small Netgear switches with ESX and ESXi with good success, although not that model personally.  Certainly try the Trendnet, it'll be interesting if something in your chain there is thwarting the GigE auto-negotiation.

                          1 Reply Last reply Reply Quote 0
                          • B
                            biggsy
                            last edited by

                            Maybe a silly question but are you sure all the cables are straight-through (not cross-over) cables?

                            Maybe you just used "red", "green" an "blue" to identify the cables for your post but (and sorry if you already know this) a lot of places use different colored cables for different things.  For example, red for cross-over is fairly common.

                            If you inherited these cables from somewhere or just chose different colors when you bought them, it might be worth checking.

                            I have seen a Netgear switch accept a cross-over cable where a straight-through should have been used but it would only negotiate up to 100Mb/s.  There might even be something in the standards that says that's what should happen.

                            1 Reply Last reply Reply Quote 0
                            • P
                              pf123user
                              last edited by

                              @biggsy:

                              notice how the lan only say 100mb?

                              That actually looks like the guy's WAN, given that he's got it labeled PPPOE.  Probably his modem only does 100 Mb/s on its LAN interface.

                              I'd agree with biggsy. What's on the other end of that NIC displaying only 100mb? The NIC and the cable are probably fine. To test it just take a laptop or PC that is gigabit and plug it into whatever device is on the other end of that link. If the other machines also display 100mb then I'd bet biggsy is correct in guessing that your modem or whatever it is at the other end is only capable of 100mb.

                              OR

                              Do you have a physical gigabit router or switch? Plug that into the "vmnic1" interface and see if it can resolve to gigabit from a known good gigabit device on the other end.

                              1 Reply Last reply Reply Quote 0
                              • M
                                matguy
                                last edited by

                                @pf2.0nyc:

                                @biggsy:

                                notice how the lan only say 100mb?

                                That actually looks like the guy's WAN, given that he's got it labeled PPPOE.  Probably his modem only does 100 Mb/s on its LAN interface.

                                I'd agree with biggsy. What's on the other end of that NIC displaying only 100mb? The NIC and the cable are probably fine. To test it just take a laptop or PC that is gigabit and plug it into whatever device is on the other end of that link. If the other machines also display 100mb then I'd bet biggsy is correct in guessing that your modem or whatever it is at the other end is only capable of 100mb.

                                OR

                                Do you have a physical gigabit router or switch? Plug that into the "vmnic1" interface and see if it can resolve to gigabit from a known good gigabit device on the other end.

                                The picture was from someone else's setup to just show basically what he's seeing.  His is actually showing 100Mb to a Netgear Prosafe GS108T.  Gigabit to his WAN is fine.

                                1 Reply Last reply Reply Quote 0
                                • E
                                  esxiNOOB
                                  last edited by

                                  ya the wan is connected to a Motorola surfboard modem, docsis 3.0… forget the model... but showing a blue light on connection which = gbit.

                                  it is a valid question about the cables and the name i chose for this account "..noob" probably wasn't the best.. but all my cables are patch, aka straight through. i have only one custom made crossover that i hardly use and it's only cat 5.

                                  my problem has been solved with a longer cable and using the 5 port trendnet gbit switch.

                                  although i think i may use the unrecommended settings to connect straight to the netgear.

                                  next question without opening a new thread maybe you couple guys that have kindly helped me can tell me what this hit on the firewall is all about:

                                  WAN 10.20.64.1:67 destination 255.255.255.255:68 UDP

                                  i don't get the 255.255.255.255. it happens almost every min. what the hell is going on!

                                  in the end i think my problem was not esxi related but more netgear switch.

                                  1 Reply Last reply Reply Quote 0
                                  • H
                                    heper
                                    last edited by

                                    this is related to DHCP traffic … the 255.255.255.255 is the broadcast range it is sending afaik.

                                    it is odd that this happens every minute on your WAN interface. (should only happen when your dhcp lease expires and renews)
                                    http://www.linklogger.com/UDP67_68.htm

                                    1 Reply Last reply Reply Quote 0
                                    • B
                                      biggsy
                                      last edited by

                                      I'm surprised it only happens every minute.  This is DHCP traffic, as heper has said.  It's cable modems obtaining or renewing leases.

                                      If I leave on the Log packets blocked by the default rule I see the DHCP requests and replies for every cable modem on the same segment of cable.

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post
                                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.