Home Lab Questions
-
Hello all, I have come to the forums for help as I have been working on a setup for a week now. I will try my best to describe what I have so far and what my intent is. I hope someone has a similar setup or can suggest a better way for me to get to my end goal.
Current equipment and its purpose
2 dell r610 servers (esxi 6.0 U2) hosting internal home network VMs and a nested environment (6 x esxi 6.0 U2)
1 cisco 3560g with multiple vLans and trunk ports to physical esxi hosts
1 netgear 24 port dumb switch for all internal home services
1 physical synology server as a NFS server
1 physical centos server as second NFS server
1 firebox x550e with pfsense 2.3So my intent in all of this is a very extensive home lab to build out NSX, VDI, vRA and vRops. I have this all setup at work fine, but enterprise equipment is not the same as cheap home lab equipment.
sk0 - WAN port = Charter
sk1 - LAN port = 10.66.66.0/24 DHCP and working/routing all internal home services fine
sk2 - vLAN trunk, or what I can gather as much. I have followed the guide to build out multiple VLANs in pfsense. I just add another opt# for sk2, then go and add a VLAN and then go back to interfaces and set the opt# to that VLAN. Go in and set that interface IP with no gateway and then go to dhcp server and build dhcp to it. I make sure to go into firewall and allow all to all for that vlan id.
sk3 - dumb netgear switch handling all internal home servicesCisco 3560g
interface GigabitEthernet0/2 –-- This is plugged into sk2
switchport trunk encapsulation dot1q
switchport mode trunksome configs I have tried to get working
I have built out a LACP VDS in vsphere 6 following VMware guide. Made sure promiscuous mode is on. The 2 NICS in esxi assigned are belowinterface GigabitEthernet0/3
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 1 mode active
!
interface GigabitEthernet0/4
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 1 mode active
!
and port channel is
interface Port-channel1
switchport trunk encapsulation dot1q
switchport mode trunkGroup Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
1 Po1(SD) LACP Gi0/3(I) Gi0/4(I)liquidsw#sh etherchannel detail
Channel-group listing:
----------------------Group: 1
Group state = L2
Ports: 2 Maxports = 16
Port-channels: 1 Max Port-channels = 16
Protocol: LACP
Ports in the group:
-------------------
Port: Gi0/3Port state = Up Sngl-port-Bndl Mstr Not-in-Bndl
Channel group = 1 Mode = Active Gcchange = -
Port-channel = null GC = - Pseudo port-channel = Po1
Port index = 0 Load = 0x00 Protocol = LACPFlags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi0/3 SA indep 32768 0x1 0x1 0x3 0x7DAge of the port in the current state: 0d:00h:33m:39s
Port: Gi0/4
Port state = Up Sngl-port-Bndl Mstr Not-in-Bndl
Channel group = 1 Mode = Active Gcchange = -
Port-channel = null GC = - Pseudo port-channel = Po1
Port index = 0 Load = 0x00 Protocol = LACPFlags: S - Device is sending Slow LACPDUs F - Device is sending fast LACPDUs.
A - Device is in active mode. P - Device is in passive mode.Local information:
LACP port Admin Oper Port Port
Port Flags State Priority Key Key Number State
Gi0/4 SA indep 32768 0x1 0x1 0x4 0x7DAge of the port in the current state: 0d:00h:34m:02s
Port-channels in the group:
---------------------------Port-channel: Po1 (Primary Aggregator)
Age of the Port-channel = 0d:01h:18m:31s
Logical slot/port = 2/1 Number of ports = 0
HotStandBy port = null
Port state = Port-channel Ag-Not-Inuse
Protocol = LACP
Port security = DisabledI assigned a Win 2012 R2 VM to this port on the VDS with no vlan id, with vlan trunk 1-4094 and also no vlan and I cannot ping the vlan ip assigned to vlan 16 in the pfsense box. I made sure to turn off the firewall just in case.
I have also not had the lacp setup on both ends and just had the Win 2012 assigned to a portgroup with vlan tagged 16 and made sure the vmnic# was set to 4095(all vlans) and the specific port on the cisco switch set to trunk, still no pinging.I also have 4 other cisco ports plugged into the esxi hosts that are trunked in cisco as above and vmnics set to 4095 (all vlans)
each dell r610 has 5 ports,
vmnic 0 on each host is set to the physical management of my home services network 10.66.66.0/24. Home service VMs, media servers and such assigned the this port group (no vlan tagging) on this nic and work fine
vmnic 4 has a vmkernal assigned to a 10.55.55.0/24 and a port group that all servers run through and is strictly assigned to the storage and I have no issues with these (attached to the synology and centos NFS shares, even mapped all the way down to the nested esxi hosts)
interface GigabitEthernet0/21
switchport access vlan 55
!
interface GigabitEthernet0/22
switchport access vlan 55
!
interface GigabitEthernet0/23
switchport access vlan 55
!
interface GigabitEthernet0/24
switchport access vlan 55interface Vlan55
no ip address
no ip route-cachevmnic 1 on each host are plugged into ports 3 and 4 on the cisco switch, intent was for this port to maintain the mgmt network of all nested esxi hosts and servers needed for vmware management services (172.28.16.0/24)
vmnic 2 was suppose to be for a "tenant space" VDS that NSX will run over
vmnic 3 was supposed to be for the vmotion network for the nested esxi hosts.
I do not have a default gateway setup on my cisco 3560, the only ip on it is on vlan 3 which corresponds to vlan 3 on the pfsense just so I can ssh to it. When I first setup the physical pfsense and built the first vlan and trunked port 2 to sk2 the vlans had been blocked on cisco due to vlan mismatch issues. I looked up some steps and nothing I tried fixed that till I added vlan 1 to pfsense on sk2
I cannot ping any IP on pfsense from the 3560 yet from pfsense i can ping 10.3.3.2 (cisco vlan ip) and in the pfsense arp table I can see a mac for vlan 3 on pfsense, vlan 3 on cisco and even the mac for the win 2012 nic assigned to 10.3.3.100. I can also ping 10.3.3.100 from pfsense but not the opposite way.
Here are all the routes I have in pfsense, dynamic not static
Destination Gateway Flags Use Mtu Netif Expire
default x.x.x.x UGS 78263 1500 sk0
10.1.1.0/24 link#11 U 0 1500 sk2_vlan1
10.1.1.1 link#11 UHS 0 16384 lo0
10.3.3.0/24 link#10 U 5736 1500 sk2_vlan3
10.3.3.1 link#10 UHS 198 16384 lo0
10.8.8.0/24 link#12 U 155844 1500 sk2_vlan8
10.8.8.1 link#12 UHS 0 16384 lo0
10.15.15.0/24 link#13 U 0 1500 sk2_vlan15
10.15.15.1 link#13 UHS 0 16384 lo0
10.66.66.0/24 link#2 U 3591268 1500 sk1
10.66.66.1 link#2 UHS 42 16384 lo0
x.x.x.x/20 link#1 U 182886 1500 sk0
x.x.x.x link#1 UHS 0 16384 lo0
127.0.0.1 link#8 UH 484 16384 lo0
172.28.16.0/24 link#14 U 0 1500 sk2_vlan16
172.28.16.1 link#14 UHS 120 16384 lo0Sorry for the long post and any errors or missed info in it. Like I said I have been working on this for a week and banging my head trying to get things to work. I am open to any suggestions -buying any new gear
Any help is greatly appreciated.