Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Home Lab Questions

    Scheduled Pinned Locked Moved Routing and Multi WAN
    1 Posts 1 Posters 805 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B Offline
      bughatti
      last edited by

      Hello all, I have come to the forums for help as I have been working on a setup for a week now.  I will try my best to describe what I have so far and what my intent is.  I hope someone has a similar setup or can suggest a better way for me to get to my end goal.

      Current equipment and its purpose
      2 dell r610 servers (esxi 6.0 U2) hosting internal home network VMs and a nested environment (6 x esxi 6.0 U2)
      1 cisco 3560g with multiple vLans and trunk ports to physical esxi hosts
      1 netgear 24 port dumb switch for all internal home services
      1 physical synology server as a NFS server
      1 physical centos server as second NFS server
      1 firebox x550e with pfsense 2.3

      So my intent in all of this is a very extensive home lab to build out NSX, VDI, vRA and vRops.  I have this all setup at work fine, but enterprise equipment is not the same as cheap home lab equipment.

      sk0 - WAN port = Charter
      sk1 - LAN port = 10.66.66.0/24 DHCP and working/routing all internal home services fine
      sk2 - vLAN trunk, or what I can gather as much.  I have followed the guide to build out multiple VLANs in pfsense.  I just add another opt# for sk2, then go and add a VLAN and then go back to interfaces and set the opt# to that VLAN.  Go in and set that interface IP with no gateway and then go to dhcp server and build dhcp to it.  I make sure to go into firewall and allow all to all for that vlan id.
      sk3 - dumb netgear switch handling all internal home services

      Cisco 3560g

      interface GigabitEthernet0/2  –-- This is plugged into sk2
      switchport trunk encapsulation dot1q
      switchport mode trunk

      some configs I have tried to get working
      I have built out a LACP VDS in vsphere 6 following VMware guide.  Made sure promiscuous mode is on.  The 2 NICS in esxi assigned are below

      interface GigabitEthernet0/3
      switchport trunk encapsulation dot1q
      switchport mode trunk
      channel-group 1 mode active
      !
      interface GigabitEthernet0/4
      switchport trunk encapsulation dot1q
      switchport mode trunk
      channel-group 1 mode active
      !
      and port channel is
      interface Port-channel1
      switchport trunk encapsulation dot1q
      switchport mode trunk

      Group  Port-channel  Protocol    Ports
      ------+-------------+-----------+-----------------------------------------------
      1      Po1(SD)        LACP      Gi0/3(I)    Gi0/4(I)

      liquidsw#sh etherchannel detail
                      Channel-group listing:
                      ----------------------

      Group: 1

      Group state = L2
      Ports: 2  Maxports = 16
      Port-channels: 1 Max Port-channels = 16
      Protocol:  LACP
                      Ports in the group:
                      -------------------
      Port: Gi0/3

      Port state    = Up Sngl-port-Bndl Mstr Not-in-Bndl
      Channel group = 1          Mode = Active          Gcchange = -
      Port-channel  = null        GC  =  -            Pseudo port-channel = Po1
      Port index    = 0          Load = 0x00            Protocol =  LACP

      Flags:  S - Device is sending Slow LACPDUs  F - Device is sending fast LACPDUs.
              A - Device is in active mode.        P - Device is in passive mode.

      Local information:
                                  LACP port    Admin    Oper    Port        Port
      Port      Flags  State    Priority      Key      Key    Number      State
      Gi0/3    SA      indep    32768        0x1      0x1    0x3        0x7D

      Age of the port in the current state: 0d:00h:33m:39s

      Port: Gi0/4

      Port state    = Up Sngl-port-Bndl Mstr Not-in-Bndl
      Channel group = 1          Mode = Active          Gcchange = -
      Port-channel  = null        GC  =  -            Pseudo port-channel = Po1
      Port index    = 0          Load = 0x00            Protocol =  LACP

      Flags:  S - Device is sending Slow LACPDUs  F - Device is sending fast LACPDUs.
              A - Device is in active mode.        P - Device is in passive mode.

      Local information:
                                  LACP port    Admin    Oper    Port        Port
      Port      Flags  State    Priority      Key      Key    Number      State
      Gi0/4    SA      indep    32768        0x1      0x1    0x4        0x7D

      Age of the port in the current state: 0d:00h:34m:02s

      Port-channels in the group:
                      ---------------------------

      Port-channel: Po1    (Primary Aggregator)


      Age of the Port-channel  = 0d:01h:18m:31s
      Logical slot/port  = 2/1          Number of ports = 0
      HotStandBy port = null
      Port state          = Port-channel Ag-Not-Inuse
      Protocol            =  LACP
      Port security      = Disabled

      I assigned a Win 2012 R2 VM to this port on the VDS with no vlan id, with vlan trunk 1-4094 and also no vlan and I cannot ping the vlan ip assigned to vlan 16 in the pfsense box.  I made sure to turn off the firewall just in case.
      I have also not had the lacp setup on both ends and just had the Win 2012 assigned to a portgroup with vlan tagged 16 and made sure the vmnic# was set to 4095(all vlans) and the specific port on the cisco switch set to trunk, still no pinging.

      I also have 4 other cisco ports plugged into the esxi hosts that are trunked in cisco as above and vmnics set to 4095 (all vlans)

      each dell r610 has 5 ports,

      vmnic 0 on each host is set to the physical management of my home services network 10.66.66.0/24. Home service VMs, media servers and such assigned the this port group (no vlan tagging) on this nic and work fine

      vmnic 4 has a vmkernal assigned to a 10.55.55.0/24 and a port group that all servers run through and is strictly assigned to the storage and I have no issues with these (attached to the synology and centos NFS shares, even mapped all the way down to the nested esxi hosts)

      interface GigabitEthernet0/21
      switchport access vlan 55
      !
      interface GigabitEthernet0/22
      switchport access vlan 55
      !
      interface GigabitEthernet0/23
      switchport access vlan 55
      !
      interface GigabitEthernet0/24
      switchport access vlan 55

      interface Vlan55
      no ip address
      no ip route-cache

      vmnic 1 on each host are plugged into ports 3 and 4 on the cisco switch, intent was for this port to maintain the mgmt network of all nested esxi hosts and servers needed for vmware management services (172.28.16.0/24)

      vmnic 2 was suppose to be for a "tenant space" VDS that NSX will run over

      vmnic 3 was supposed to be for the vmotion network for the nested esxi hosts.

      I do not have a default gateway setup on my cisco 3560, the only ip on it is on vlan 3 which corresponds to vlan 3 on the pfsense just so I can ssh to it.  When I first setup the physical pfsense and built the first vlan and trunked port 2 to sk2 the vlans had been blocked on cisco due to vlan mismatch issues.  I looked up some steps and nothing I tried fixed that till I added vlan 1 to pfsense on sk2

      I cannot ping any IP on pfsense from the 3560 yet from pfsense i can ping 10.3.3.2 (cisco vlan ip) and in the pfsense arp table I can see a mac for vlan 3 on pfsense, vlan 3 on cisco and even the mac for the win 2012 nic assigned to 10.3.3.100.  I can also ping 10.3.3.100 from pfsense but not the opposite way.

      Here are all the routes I have in pfsense, dynamic not static

      Destination Gateway Flags Use Mtu Netif Expire
      default         x.x.x.x UGS 78263 1500 sk0
      10.1.1.0/24 link#11 U 0         1500 sk2_vlan1
      10.1.1.1         link#11 UHS 0 16384 lo0
      10.3.3.0/24 link#10 U 5736         1500 sk2_vlan3
      10.3.3.1         link#10 UHS 198 16384 lo0
      10.8.8.0/24 link#12 U 155844 1500 sk2_vlan8
      10.8.8.1         link#12 UHS 0 16384 lo0
      10.15.15.0/24 link#13 U 0         1500 sk2_vlan15
      10.15.15.1  link#13 UHS 0 16384 lo0
      10.66.66.0/24 link#2 U 3591268 1500 sk1
      10.66.66.1 link#2 UHS 42 16384 lo0
      x.x.x.x/20         link#1 U 182886 1500 sk0
      x.x.x.x         link#1 UHS 0 16384 lo0
      127.0.0.1         link#8 UH 484 16384 lo0
      172.28.16.0/24 link#14 U 0         1500 sk2_vlan16
      172.28.16.1 link#14 UHS 120 16384 lo0

      Sorry for the long post and any errors or missed info in it.  Like I said I have been working on this for a week and banging my head trying to get things to work.  I am open to any suggestions -buying any new gear

      Any help is greatly appreciated.

      1 Reply Last reply Reply Quote 0
      • First post
        Last post
      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.