VLAN internet access plus routing issue - Cisco SG300 & ESX 6.5

SubX

Current Settings:

Internet Modem >
Bell Fiber connected to pfSense via PPPOE passthrough (A PPP interface instead of WAN with VLAN35)

pfSense >

WAN (pppoe) direct connect to internet, WAN shows public IP (dynamic)
LAN 10.0.0.1 (didn't create VLAN 1)
VLAN 8 Interface = 10.0.8.66
FW Rule (allow LAN & VLAB 8 access internet)

SG300 >

Layer3 mode, IP = 10.0.0.6,
VLAN1 = 10.0.0.6 (mgmt), VLAN 8 = 10.0.8.66 (VM Networking), VLAN 18 = 10.0.18.66 (vMotion), VLAN 88 = 10.0.88.66 (iSCSI)
All VLAN can ping to other VLAN's gateway, devices in each VLAN can't ping other VLAN even routing is there
VMs within VLAN8 can communicate with each other. ESXi hosts can communicate with VLAN 88's iscsi storage (FreeNAS)

vSphere 6.5 >

vSwitch 0 (VST tagging) / mgmt vmk = VLAN 1 / VM PG = VLAN 8 / vMotion vmk = VLAN 18
vSwitch 1 (EST tagging, VLAN ID = 0) / iSCSI vmk = VLAN 88

FreeNAS >

iSCSI sharing for 2 x ESX hosts
CIFS for file share

Issues:

VLAN 8 can't access internet. Will pppoe connection prevent this from happening? Notice that you suggest /30 LAN and WAN as Transit Area. If go for pppoe option, can't setup NAT.
Outside traffic can't reach all VLANs behind pfSense
VLANs created and routed by SG300 can't communicate to each other. Would like VLAN 1 & 8 talked to VLAN 88 (there is CIFS share where clients need to access)
Any best practice should be followed on vSphere 6.5 for such setting

Thanks,

Sub

tarakesh

Issue 2: does PFSense have routing entries for each vlan pointing to 10.0.0.6?
Issue 3: What is the default gateway on the devices in each vlan?

imho you should split this up completly

from all vlans make the sg300 the default gateway and seperate the pfsense completly
then build a /30 transfer vlan to the pfsense with only the switch and the pf in it. Set the switch default gateway to pf
vice versa add all vlan networks to the pf routing table pointing to the switch ip in the transfer vlan.
as all is routed you can still firewall it off so only ips from vlan 1 and 8 can access the internet.
But if you do so you will block the return route for other vlans for the access from the internet

SubX

Issue 2
No routing entries for each vlan on pfSense. Fiber internet is facing up and down issue. Changed to pppoe passthrough (no NAT) to see if the issue will be lifted. Two days after chaning it was working fine until tonight. Might try /30 NAT in a couple of days.

Issue 3
default gateway on the devices are 8.66, 18.66, 88.66 per VLAN ip setup in SG300.
I would like to separate switch and pfSense completely. However having some trouble on vlan routing prevent me from doing so. Sorry I am not a network guy here.
pf LAN gw=10.0.0.1, SG300 ip=10.0.0.6, VLAN 8 gw=10.0.8.66, VLAN 18 gw=10.0.18.66, VLAN 88 gw=10.0.88.66 , VLAN 1 gw=10.0.0.6 (same as SG300 ip, any miss-configuration here). As I didn't figure it how vlans can talk to each other, would like to get this clear before moving to /30 transit interface. Any suggest or detail steps/cli here?
Try to have SG300 handle vlan routing rather than routing from pfSense. Does this leads to a dead end here?

Thanks,

johnpoz

What do you want to route your vlans? Your sg300 or pfsense?

From this seems you have SVIs setup on sg300
"SG300 >

Layer3 mode, IP = 10.0.0.6,
VLAN1 = 10.0.0.6 (mgmt), VLAN 8 = 10.0.8.66 (VM Networking), VLAN 18 = 10.0.18.66 (vMotion), VLAN 88 = 10.0.88.66 (iSCSI)"

So it would be doing all the routing?

If you want pfsense to route, then pfsense would have all the vlans setup on it.. You would just use sg300 as layer 2 for these networks..

I your going to route on your sg300.. Then the network between your sg300 and pfsense would be a transit network..

SubX

Johnpoz,

Thanks a lot! You are correct. VLANs are currently setup in SG300. However still have issue to communicate VLAN 8 to VLAN 1 or VLAN 88.

Cause the Ram < 3G on pfSense, try to avoid having to much workload on pfSense, so no plan regarding routing on pfSense. Expect routing on SG300.

I just recently upgrade to Fiber connection (ONT and Bell Hub 3000 fiber modem/router) and found out when pfSense connect to Fiber, the Hub3000 will reboot every hour. Then I moved to direct pppoe connection (a pass-through setting on pfSense to log into pppoe directly instead of a NAT to Hub3000). It worked super fine for two days then the reboot issue start to happen again. Still troubleshooting with Bell. Any experience on that front?

You mentioned that if SG300 takes care of vlan, then SG300 and pfSense will be a transit network. Does it means that since my pfSense LAN = 10.0.0.1, SG300 currently is 10.0.0.6. How to make a transit network between pfSense and SG300. As I am new to networking, a bit of detail is highly appreciated.

Should you need further info regarding my current setting, please let me know.

johnpoz

"Cause the Ram < 3G on pfSense, try to avoid having to much workload on pfSense"

Dude pfsense can run just fine on way less than 3GB.. I have ran it on 256MB as a vm without any issues.. My VM normally just has 512 or at most 1GB..

The network between pfsense and your downstream router (L3 switch) would be a transit network.. No hosts should sit on this network or your going to run into asymmetrical routing conditions if you don't specifically tell that host where to go to get to specific networks. Your transit network could be any network you want that is not used elsewhere in your network. But it rarely needs a mask larger than 30 or /29 if your going to have a few routers on this transit.

Here is a simple drawing a transit setup. (see attached)

You would setup a gateway in pfsense under routing (not default) then you would create a route on pfsense for your networks on pfsense. Be it summary 10.0.0.0/22 or you could even do 10/8 if you wanted, etc. You would then adjust the rules on your transit interface on pfsense (lan quite often) to allow the downstream networks. If your outbound nat is automatic it will auto create the needed outbound nats to allow for the downstream networks to be natted to your wan (internet). If you had messed with outbound nat and placed it in manual for some reason then you would have to add the outbound nats for those downstream networks.

In such a setup your configuration on your switch would just be access to pfsense vs trunking and tagging multiple vlans that you would have setup on your switch for the other vlans, etc. The switch port connected to pfsense would just need to allow the transit network vlan and it wouldn't even need to tagged unless you set the transit up as a vlan on pfsense.

Problem with this is now the only way to control traffic between your downstream networks is via ACLs on the switch - they are way more complicated and limited than the easy setting of firewall rules on pfsense. If you do not need firewall control between these segments. Then it would make more sense to just put them all on the same network vs doing any routing either on pfsense or L3 switch (router).

Use of transit networks and downstream router is more complicated to anyone new to networking, from the vast amount of problems I see here due to asymmetrical routing it seems to be a skill even more experienced networking people lack.

For ease of setup, ease of firewalling, etc. I would really just suggest you let pfsense do all the routing between your segments.. Unless your pfsense is not capable of routing at the speeds you need between your segments there is little need to use downstream L3 switches to route between your vlans. But the Ram on your pfsense should really have little effect on ability to route/firewall between your segments. Keep in mind if you have need of lots of intervlan traffic at high speed that putting all these vlans on top of the same physical interface is going to be a bottleneck due to hairpinning of the traffic between the vlans on the same physical interface. If possible you want your pfsense box to have multiple physical interfaces to spread your networks across so that vlans/networks that need a lot of bandwidth between are not hairpinning on the same interface which cuts your bandwidth between those vlans.

transit.png_thumb

SubX

@johnpoz:

Dude pfsense can run just fine on way less than 3GB.. I have ran it on 256MB as a vm without any issues.. My VM normally just has 512 or at most 1GB..

I found out one mini pc online, spec 1 x Cereon J1900 (4 core 4 threads), 4 x Gigabit NIC, 1G/2G/4G ram, 8G/16G SSD. Low power. Might be a good option for pfSense. My current pfSense box is a borrowed unit. http://www.xcyminipc.com/product/showproduct.php?lang=en&id=51

@johnpoz:

In such a setup your configuration on your switch would just be access to pfsense vs trunking and tagging multiple vlans that you would have setup on your switch for the other vlans, etc. The switch port connected to pfsense would just need to allow the transit network vlan and it wouldn't even need to tagged unless you set the transit up as a vlan on pfsense.

Just to confirm what you mentioned above is L3 switch as downstream. Being a newbie in networking, I would follow you suggestion avoid it and use pfSense as router then.

@johnpoz:

Use of transit networks and downstream router is more complicated to anyone new to networking, from the vast amount of problems I see here due to asymmetrical routing it seems to be a skill even more experienced networking people lack.

My Fiber modem/router is expriencing frequent reboot, could that be a result of asymmetrical routing here?

@johnpoz:

For ease of setup, ease of firewalling, etc. I would really just suggest you let pfsense do all the routing between your segments.. Unless your pfsense is not capable of routing at the speeds you need between your segments there is little need to use downstream L3 switches to route between your vlans. But the Ram on your pfsense should really have little effect on ability to route/firewall between your segments. Keep in mind if you have need of lots of intervlan traffic at high speed that putting all these vlans on top of the same physical interface is going to be a bottleneck due to hairpinning of the traffic between the vlans on the same physical interface. If possible you want your pfsense box to have multiple physical interfaces to spread your networks across so that vlans/networks that need a lot of bandwidth between are not hairpinning on the same interface which cuts your bandwidth between those vlans.

Could you provide more detail (diagram preferred) on pfSense as router with downstream L2 switch.
Thanks a lot!

johnpoz

[qoute]My Fiber modem/router is expriencing frequent reboot, could that be a result of asymmetrical routing here?

I wouldn't think so.. But not going to completely rule it out.. If the device is seeing traffic that its not use to seeing, not out of realm of possibility such odd traffic could cause it problems. I would think it should reboot.. But if its seeing a flood of out of state traffic, guess it could be possible. But I wouldn't think it very likely.

Sure let me dig up one of the previous diagrams, or put together another one. Real work doesn't seem to want to leave me alone today ;) So it might be bit…

SubX

Thanks Johnpoz!

Someone mentioned that "Bell uses 10.x.x.x/12 for their private network side for TV and phone services. HH3000 could be rebooting because of a routing conflict. You might want to try using 192.168.x.x/16."

I didn't sign up for TV or VOIP. Should I try changing vlan to 192.168.x.x subnet to verify.

Thanks,

johnpoz

I doubt that.. So they do carrier grade nat for their tv and phones.. Where did you get the /12 - you mean /8?

BTW found this older drawing did for someone else asking for layer2 switch, etc.

So 1 port could be connected to say vlan 100 your lan lets call.. And vlan 200 and 300 could be connected to another opt port. Where vlan 200 was the native or untagged vlan and 300 was tagged and setup as vlan on pfsense. while your ap managment is on say vlan 100 so it its untagged to your AP and 200 and 300 are tagged.

Does that help?

L2switch.png_thumb

SubX

Thanks Johnpoz! Happy Halloween!

Sorry I was distracted by some family stuff and couldn't spare time on this. Now I am back.

You mentioned about transit network to connect WAN and LAN, in this diagram, I don't see a transit network as vlan 100 might have other hosts and you did mention transit network should be only one ip from pfSense and one ip from switch.

I am also looking at other posts which you helped out with the transit network and hope I will figure out more when trying out myself.

Thanks,
S.

SubX

Johnpoz,

I saw your other post - https://forum.pfsense.org/index.php?topic=123119.15

I followed the idea, create a vlan 99 (192.168.99.0) as transit network, where 192.168.99.1 (SG300 switch) and 99.2 (pfSense). Other VLAN 1 (172.16.0.x) and VLAN 8 (172.16.8.x). Applied default route and static route to all vlan.

Now from the SG300 switch itself, I can ping hosts in internet, 99.2 (pfSense) and other devices in VLAN 1 & 8. However, from those two VLANs, I can ping VLAN 99 gatway (192.168.99.1), but can't ping 192.168.99.2 (pfSense). It is like the last steps. Just wonder if you can give me a hand on how to configure SG300 here to make it works?

If this route is dead end after this last try, I will follow your advice to change back to L2 as downsteam while pfSense take charge of routing.

Thanks,
Stephen

jahonix

@SubX:

I found out one mini pc online, spec 1 x Cereon J1900 (4 core 4 threads), 4 x Gigabit NIC, 1G/2G/4G ram, 8G/16G SSD. Low power. Might be a good option for pfSense.

It surely is not.
Celeron J1900s do not offer AES-NI which will be required from pfSense 2.5 onwards.
Buying such a unit today obsoletes it for pfSense in about a year or so.

Details here:
https://www.netgate.com/blog/pfsense-2-5-and-aes-ni.html
https://www.netgate.com/blog/more-on-aes-ni.html

johnpoz

"Just wonder if you can give me a hand on how to configure SG300 here to make it works?"

Draw up what you did and be happy to..

But lets be clear - downstream router via a transit would not be my suggestion.. As to how you would set it up on a sg300.. It would be just like any other layer 2 vlan. Its just a connection from pfsense to whatever your downstream router is.. If this is the sg300. Then its just a simple access port in whatever vlan you want to setup on on the sg300. Connected to pfsense via native - non vlan interface. Pfsense would not have any clue to what vlan IDs are connected downstream.

It would just need to know what networks are downstream and route to them.. And its firewall rules and outbound nat would have to account for them.

SubX

Johnpoz,

Thanks. Attached please find the diagram as well as the VLAN layout below.
VLAN 1 - 172.16.0.x (172.16.0.6 is router ip / gateway)
VLAN 8 - 172.16.8.x (172.16.8.6 is router ip / gateway)
VLAN 18 - 172.16.18.x (172.16.18.6 is router ip / gateway)
VLAN 88 - 172.16.88.x (172.16.88.6 is router ip / gateway)
VLAN 99 - 192.168.99.x /29 (192.168.99.1 on SG300, 192.168.99.2 on pfSense) - this is transit network

I add static route to allow VLAN 1 and VLAN 8 to route to VLAN 99 next hop = 192.168.99.2 (screenshoot)

Now from 172.16.0.10 can ping VLAN 1, 8, 18, 88 OK. Ping 192.168.99.1 OK.
Ping 192.168.99.2 failed.
From within SG300, I can ping internet host such as google.com, cnn.com etc..

Is it beacuse of VLAN 1 & VLAN 8 route to 192.168.99.2 is inactive? How to activate those two routes?

Thanks,

pfSense_L3R.png_thumb

SG300-inactive-static-route.JPG_thumb

SG300-TransitNetwork-cant-Ping-pfSense.JPG_thumb

SubX

jahonix,

Thanks for the note regarding J9100 limitation.

johnpoz

"VLAN 99 - 192.168.99.x /29 (192.168.99.1 on SG300, 192.168.99.2 on pfSense) - this is transit network"

How is this setup on pfsense? On what interface is this sitting? Your physical lan interface as a vlan or just native network?

What is the sg300 config on this port? It would just be an access port in vlan 99 on the switch.. Pfsense would not have any need to understand this vlan ID. Or any of the other vlan IDs.

What are you routes on pfsense? What does your outbound nat look like… What does the firewall rules look like on the 192.168.99 network?

If your trying to ping pfsense IP on the transit network.. Pfsense has to know how to get back to that downstream network.

SubX

Thanks!!!

pfSense setup only WANpppoe (connect through pppoe directly to internel, gateway becomes dynamic gateway) and LAN (no VLANs) just native.

Port 28 on SG300 as Access, untagged VLAn 99. pfSense doesn't know about any VLAN ID.

Under pfSense routing, there is a WANpppoe gateway (default), follow other post, there is no LAN gateway. Nothing under static route, should I add something here?

Outbound NAT by default, screenshot attached. Firewall rules - WANpppoe allow any to any, LAN allow source 192.168.99.1 > desti 192.168.99.2 , also any to any.

How to config to allow pfSens know how to get back to the downstream network. I setup static route rules in SG300 (screenshot see previous reply), it seems that those two routes are inactive. Or should I set it up in pfSense instead, please show me how to.

Thanks,

PPPoE-n_LAN.JPG_thumb

WANpppoe.JPG_thumb

LAN1.JPG_thumb

LAN2.JPG_thumb

TransitNetwork-SG300-port-AccessPort.JPG_thumb

Routing-GW-WANpppoe1.JPG_thumb

Routing-GW-WANpppoe2.JPG_thumb

Routing-StaticRoute-nothing.JPG_thumb

FireWall-NAT-Auto.JPG_thumb

FireWall-Rules-WANpppoe-default.JPG_thumb

FireWall-Rules-LAN.JPG_thumb

johnpoz

"Nothing under static route, should I add something here?"

How is pfsense suppose to know to get to your networks downstream of your sg300 if you do not have routes?

Once you have created the gateway, not on the interface but in the routing section and create the routes to your downstream network using that gateway it should auto update your outbound nat to include your downstream networks that it can nat outbound.

SubX

Can you be more specific,
Is it like below?
Destination - 172.16.0.0 Gateway - WANpppoe Gateway (or should I create a LAN Gateway 192.168.99.2 ?)
Destination - 172.16.8.0 Gateway - WANpppoe Gateway

Thanks,