Home Lab Questions



  • Hello all, I have come to the forums for help as I have been working on a setup for a week now.  I will try my best to describe what I have so far and what my intent is.  I hope someone has a similar setup or can suggest a better way for me to get to my end goal.

    Current equipment and its purpose
    2 dell r610 servers (esxi 6.0 U2) hosting internal home network VMs and a nested environment (6 x esxi 6.0 U2)
    1 cisco 3560g with multiple vLans and trunk ports to physical esxi hosts
    1 netgear 24 port dumb switch for all internal home services
    1 physical synology server as a NFS server
    1 physical centos server as second NFS server
    1 firebox x550e with pfsense 2.3

    So my intent in all of this is a very extensive home lab to build out NSX, VDI, vRA and vRops.  I have this all setup at work fine, but enterprise equipment is not the same as cheap home lab equipment.

    sk0 - WAN port = Charter
    sk1 - LAN port = 10.66.66.0/24 DHCP and working/routing all internal home services fine
    sk2 - vLAN trunk, or what I can gather as much.  I have followed the guide to build out multiple VLANs in pfsense.  I just add another opt# for sk2, then go and add a VLAN and then go back to interfaces and set the opt# to that VLAN.  Go in and set that interface IP with no gateway and then go to dhcp server and build dhcp to it.  I make sure to go into firewall and allow all to all for that vlan id.
    sk3 - dumb netgear switch handling all internal home services

    Cisco 3560g

    interface GigabitEthernet0/2  –-- This is plugged into sk2
    switchport trunk encapsulation dot1q
    switchport mode trunk

    some configs I have tried to get working
    I have built out a LACP VDS in vsphere 6 following VMware guide.  Made sure promiscuous mode is on.  The 2 NICS in esxi assigned are below

    interface GigabitEthernet0/3
    switchport trunk encapsulation dot1q
    switchport mode trunk
    channel-group 1 mode active
    !
    interface GigabitEthernet0/4
    switchport trunk encapsulation dot1q
    switchport mode trunk
    channel-group 1 mode active
    !
    and port channel is
    interface Port-channel1
    switchport trunk encapsulation dot1q
    switchport mode trunk

    Group  Port-channel  Protocol    Ports
    ------+-------------+-----------+-----------------------------------------------
    1      Po1(SD)        LACP      Gi0/3(I)    Gi0/4(I)

    liquidsw#sh etherchannel detail
                    Channel-group listing:
                    ----------------------

    Group: 1

    Group state = L2
    Ports: 2  Maxports = 16
    Port-channels: 1 Max Port-channels = 16
    Protocol:  LACP
                    Ports in the group:
                    -------------------
    Port: Gi0/3

    Port state    = Up Sngl-port-Bndl Mstr Not-in-Bndl
    Channel group = 1          Mode = Active          Gcchange = -
    Port-channel  = null        GC  =  -            Pseudo port-channel = Po1
    Port index    = 0          Load = 0x00            Protocol =  LACP

    Flags:  S - Device is sending Slow LACPDUs  F - Device is sending fast LACPDUs.
            A - Device is in active mode.        P - Device is in passive mode.

    Local information:
                                LACP port    Admin    Oper    Port        Port
    Port      Flags  State    Priority      Key      Key    Number      State
    Gi0/3    SA      indep    32768        0x1      0x1    0x3        0x7D

    Age of the port in the current state: 0d:00h:33m:39s

    Port: Gi0/4

    Port state    = Up Sngl-port-Bndl Mstr Not-in-Bndl
    Channel group = 1          Mode = Active          Gcchange = -
    Port-channel  = null        GC  =  -            Pseudo port-channel = Po1
    Port index    = 0          Load = 0x00            Protocol =  LACP

    Flags:  S - Device is sending Slow LACPDUs  F - Device is sending fast LACPDUs.
            A - Device is in active mode.        P - Device is in passive mode.

    Local information:
                                LACP port    Admin    Oper    Port        Port
    Port      Flags  State    Priority      Key      Key    Number      State
    Gi0/4    SA      indep    32768        0x1      0x1    0x4        0x7D

    Age of the port in the current state: 0d:00h:34m:02s

    Port-channels in the group:
                    ---------------------------

    Port-channel: Po1    (Primary Aggregator)


    Age of the Port-channel  = 0d:01h:18m:31s
    Logical slot/port  = 2/1          Number of ports = 0
    HotStandBy port = null
    Port state          = Port-channel Ag-Not-Inuse
    Protocol            =  LACP
    Port security      = Disabled

    I assigned a Win 2012 R2 VM to this port on the VDS with no vlan id, with vlan trunk 1-4094 and also no vlan and I cannot ping the vlan ip assigned to vlan 16 in the pfsense box.  I made sure to turn off the firewall just in case.
    I have also not had the lacp setup on both ends and just had the Win 2012 assigned to a portgroup with vlan tagged 16 and made sure the vmnic# was set to 4095(all vlans) and the specific port on the cisco switch set to trunk, still no pinging.

    I also have 4 other cisco ports plugged into the esxi hosts that are trunked in cisco as above and vmnics set to 4095 (all vlans)

    each dell r610 has 5 ports,

    vmnic 0 on each host is set to the physical management of my home services network 10.66.66.0/24. Home service VMs, media servers and such assigned the this port group (no vlan tagging) on this nic and work fine

    vmnic 4 has a vmkernal assigned to a 10.55.55.0/24 and a port group that all servers run through and is strictly assigned to the storage and I have no issues with these (attached to the synology and centos NFS shares, even mapped all the way down to the nested esxi hosts)

    interface GigabitEthernet0/21
    switchport access vlan 55
    !
    interface GigabitEthernet0/22
    switchport access vlan 55
    !
    interface GigabitEthernet0/23
    switchport access vlan 55
    !
    interface GigabitEthernet0/24
    switchport access vlan 55

    interface Vlan55
    no ip address
    no ip route-cache

    vmnic 1 on each host are plugged into ports 3 and 4 on the cisco switch, intent was for this port to maintain the mgmt network of all nested esxi hosts and servers needed for vmware management services (172.28.16.0/24)

    vmnic 2 was suppose to be for a "tenant space" VDS that NSX will run over

    vmnic 3 was supposed to be for the vmotion network for the nested esxi hosts.

    I do not have a default gateway setup on my cisco 3560, the only ip on it is on vlan 3 which corresponds to vlan 3 on the pfsense just so I can ssh to it.  When I first setup the physical pfsense and built the first vlan and trunked port 2 to sk2 the vlans had been blocked on cisco due to vlan mismatch issues.  I looked up some steps and nothing I tried fixed that till I added vlan 1 to pfsense on sk2

    I cannot ping any IP on pfsense from the 3560 yet from pfsense i can ping 10.3.3.2 (cisco vlan ip) and in the pfsense arp table I can see a mac for vlan 3 on pfsense, vlan 3 on cisco and even the mac for the win 2012 nic assigned to 10.3.3.100.  I can also ping 10.3.3.100 from pfsense but not the opposite way.

    Here are all the routes I have in pfsense, dynamic not static

    Destination Gateway Flags Use Mtu Netif Expire
    default         x.x.x.x UGS 78263 1500 sk0
    10.1.1.0/24 link#11 U 0         1500 sk2_vlan1
    10.1.1.1         link#11 UHS 0 16384 lo0
    10.3.3.0/24 link#10 U 5736         1500 sk2_vlan3
    10.3.3.1         link#10 UHS 198 16384 lo0
    10.8.8.0/24 link#12 U 155844 1500 sk2_vlan8
    10.8.8.1         link#12 UHS 0 16384 lo0
    10.15.15.0/24 link#13 U 0         1500 sk2_vlan15
    10.15.15.1  link#13 UHS 0 16384 lo0
    10.66.66.0/24 link#2 U 3591268 1500 sk1
    10.66.66.1 link#2 UHS 42 16384 lo0
    x.x.x.x/20         link#1 U 182886 1500 sk0
    x.x.x.x         link#1 UHS 0 16384 lo0
    127.0.0.1         link#8 UH 484 16384 lo0
    172.28.16.0/24 link#14 U 0         1500 sk2_vlan16
    172.28.16.1 link#14 UHS 120 16384 lo0

    Sorry for the long post and any errors or missed info in it.  Like I said I have been working on this for a week and banging my head trying to get things to work.  I am open to any suggestions -buying any new gear

    Any help is greatly appreciated.