VLAN internet access plus routing issue - Cisco SG300 & ESX 6.5

[SubX]

@johnpoz:

Dude pfsense can run just fine on way less than 3GB.. I have ran it on 256MB as a vm without any issues.. My VM normally just has 512 or at most 1GB..

I found out one mini pc online, spec 1 x Cereon J1900 (4 core 4 threads), 4 x Gigabit NIC, 1G/2G/4G ram, 8G/16G SSD. Low power. Might be a good option for pfSense. My current pfSense box is a borrowed unit. http://www.xcyminipc.com/product/showproduct.php?lang=en&id=51

@johnpoz:

In such a setup your configuration on your switch would just be access to pfsense vs trunking and tagging multiple vlans that you would have setup on your switch for the other vlans, etc.  The switch port connected to pfsense would just need to allow the transit network vlan and it wouldn't even need to tagged unless you set the transit up as a vlan on pfsense.

Just to confirm what you mentioned above is L3 switch as downstream. Being a newbie in networking, I would follow you suggestion avoid it and use pfSense as router then.

@johnpoz:

Use of transit networks and downstream router is more complicated to anyone new to networking, from the vast amount of problems I see here due to asymmetrical routing it seems to be a skill even more experienced networking people lack.

My Fiber modem/router is expriencing frequent reboot, could that be a result of asymmetrical routing here?

@johnpoz:

For ease of setup, ease of firewalling, etc.  I would really just suggest you let pfsense do all the routing between your segments.. Unless your pfsense is not capable of routing at the speeds you need between your segments there is little need to use downstream L3 switches to route between your vlans.  But the Ram on your pfsense should really have little effect on ability to route/firewall between your segments.  Keep in mind if you have need of lots of intervlan traffic at high speed that putting all these vlans on top of the same physical interface is going to be a bottleneck due to hairpinning of the traffic between the vlans on the same physical interface.  If possible you want your pfsense box to have multiple physical interfaces to spread your networks across so that vlans/networks that need a lot of bandwidth between are not hairpinning on the same interface which cuts your bandwidth between those vlans.

Could you provide more detail (diagram preferred) on pfSense as router with downstream L2 switch.
Thanks a lot!

[johnpoz]

[qoute]My Fiber modem/router is expriencing frequent reboot, could that be a result of asymmetrical routing here?

I wouldn't think so.. But not going to completely rule it out.. If the device is seeing traffic that its not use to seeing, not out of realm of possibility such odd traffic could cause it problems.  I would think it should reboot.. But if its seeing a flood of out of state traffic, guess it could be possible.  But I wouldn't think it very likely.

Sure let me dig up one of the previous diagrams, or put together another one.  Real work doesn't seem to want to leave me alone today ;)  So it might be bit…

[SubX]

Thanks Johnpoz!

Someone mentioned that "Bell uses 10.x.x.x/12 for their private network side for TV and phone services.  HH3000 could be rebooting because of a routing conflict.  You might want to try using 192.168.x.x/16."

I didn't sign up for TV or VOIP. Should I try changing vlan to 192.168.x.x subnet to verify.

Thanks,

[johnpoz]

I doubt that.. So they do carrier grade nat for their tv and phones.. Where did you get the /12 - you mean /8?

BTW found this older drawing did for someone else asking for layer2 switch, etc.

So 1 port could be connected to say vlan 100 your lan lets call..  And vlan 200 and 300 could be connected to another opt port.  Where vlan 200 was the native or untagged vlan and 300 was tagged and setup as vlan on pfsense.  while your ap managment is on say vlan 100 so it its untagged to your AP and 200 and 300 are tagged.

Does that help?

[SubX]

Thanks Johnpoz! Happy Halloween!

Sorry I was distracted by some family stuff and couldn't spare time on this. Now I am back.

You mentioned about transit network to connect WAN and LAN, in this diagram, I don't see a transit network as vlan 100 might have other hosts and you did mention transit network should be only one ip from pfSense and one ip from switch.

I am also looking at other posts which you helped out with the transit network and hope I will figure out more when trying out myself.

Thanks,
S.

[SubX]

Johnpoz,

I saw your other post - https://forum.pfsense.org/index.php?topic=123119.15

I followed the idea, create a vlan 99 (192.168.99.0) as transit network, where 192.168.99.1 (SG300 switch) and 99.2 (pfSense). Other VLAN 1 (172.16.0.x) and VLAN 8 (172.16.8.x). Applied default route and static route to all vlan.

Now from the SG300 switch itself, I can ping hosts in internet, 99.2 (pfSense) and other devices in VLAN 1 & 8. However, from those two VLANs, I can ping VLAN 99 gatway (192.168.99.1), but can't ping 192.168.99.2 (pfSense). It is like the last steps. Just wonder if you can give me a hand on how to configure SG300 here to make it works?

If this route is dead end after this last try, I will follow your advice to change back to L2 as downsteam while pfSense take charge of routing.

Thanks,
Stephen

[jahonix]

@SubX:

I found out one mini pc online, spec 1 x Cereon J1900 (4 core 4 threads), 4 x Gigabit NIC, 1G/2G/4G ram, 8G/16G SSD. Low power. Might be a good option for pfSense.

It surely is not.
Celeron J1900s do not offer AES-NI which will be required from pfSense 2.5 onwards.
Buying such a unit today obsoletes it for pfSense in about a year or so.

Details here:
https://www.netgate.com/blog/pfsense-2-5-and-aes-ni.html
https://www.netgate.com/blog/more-on-aes-ni.html

[johnpoz]

"Just wonder if you can give me a hand on how to configure SG300 here to make it works?"

Draw up what you did and be happy to..

But lets be clear - downstream router via a transit would not be my suggestion.. As to how you would set it up on a sg300.. It would be just like any other layer 2 vlan.  Its just a connection from pfsense to whatever your downstream router is.. If this is the sg300.  Then its just a simple access port in whatever vlan you want to setup on on the sg300.  Connected to pfsense via native - non vlan interface.  Pfsense would not have any clue to what vlan IDs are connected downstream.

It would just need to know what networks are downstream and route to them.. And its firewall rules and outbound nat would have to account for them.

[SubX]

Johnpoz,

Thanks. Attached please find the diagram as well as the VLAN layout below.
VLAN 1 - 172.16.0.x (172.16.0.6 is router ip / gateway)
VLAN 8 - 172.16.8.x (172.16.8.6 is router ip / gateway)
VLAN 18 - 172.16.18.x (172.16.18.6 is router ip / gateway)
VLAN 88 - 172.16.88.x (172.16.88.6 is router ip / gateway)
VLAN 99 - 192.168.99.x /29 (192.168.99.1 on SG300, 192.168.99.2 on pfSense) - this is transit network

I add static route to allow VLAN 1 and VLAN 8 to route to VLAN 99 next hop = 192.168.99.2 (screenshoot)

Now from 172.16.0.10 can ping VLAN 1, 8, 18, 88 OK. Ping 192.168.99.1 OK.
Ping 192.168.99.2 failed.
From within SG300, I can ping internet host such as google.com, cnn.com etc..

Is it beacuse of VLAN 1 & VLAN 8 route to 192.168.99.2 is inactive? How to activate those two routes?

Thanks,

[SubX]

jahonix,

Thanks for the note regarding J9100 limitation.

[johnpoz]

"VLAN 99 - 192.168.99.x /29 (192.168.99.1 on SG300, 192.168.99.2 on pfSense) - this is transit network"

How is this setup on pfsense?  On what interface is this sitting?  Your physical lan interface as a vlan or just native network?

What is the sg300 config on this port?  It would just be an access port in vlan 99 on the switch..  Pfsense would not have any need to understand this vlan ID.  Or any of the other vlan IDs.

What are you routes on pfsense?  What does your outbound nat look like… What does the firewall rules look like on the 192.168.99 network?

If your trying to ping pfsense IP on the transit network.. Pfsense has to know how to get back to that downstream network.

[SubX]

Thanks!!!

pfSense setup only WANpppoe (connect through pppoe directly to internel, gateway becomes dynamic gateway) and LAN (no VLANs) just native.

Port 28 on SG300 as Access, untagged VLAn 99. pfSense doesn't know about any VLAN ID.

Under pfSense routing, there is a WANpppoe gateway (default), follow other post, there is no LAN gateway. Nothing under static route, should I add something here?

Outbound NAT by default, screenshot attached. Firewall rules - WANpppoe allow any to any, LAN allow source 192.168.99.1 > desti 192.168.99.2 , also any to any.

How to config to allow pfSens know how to get back to the downstream network. I setup static route rules in SG300 (screenshot see previous reply), it seems that those two routes are inactive. Or should I set it up in pfSense instead, please show me how to.

Thanks,

[johnpoz]

"Nothing under static route, should I add something here?"

How is pfsense suppose to know to get to your networks downstream of your sg300 if you do not have routes?

Once you have created the gateway, not on the interface but in the routing section and create the routes to your downstream network using that gateway it should auto update your outbound nat to include your downstream networks that it can nat outbound.

[SubX]

Can you be more specific,
Is it like below?
Destination - 172.16.0.0  Gateway - WANpppoe Gateway (or should I create a LAN Gateway 192.168.99.2 ?)
Destination - 172.16.8.0  Gateway - WANpppoe Gateway

Thanks,

[johnpoz]

Yes you need to create a gateway under gateways.  Then create routes under static routes to use that gateway to get to your downstream networks.

[SubX]

just created LAN gateway and two route one for VLAN 1 and one for VLAN 8.

Now, device from both VLAN 1 and 8 can log in to pfSense (192.168.99.2) , how ever CAN'T ping 192.168.99.2. Check firewall rules, rules allow any port.

What should I check next?

Thanks,

[SubX]

Add two more firewall rules in LAN  - one to allow VLAN 1 to access LAN, one to allow VLAN 8.

Now VLAN 1 & 8 CAN access internet.

So far, everything is fine now. Just wait to see if Bell Hub 3000 will reboot frequently or not. Keep finger cross.

Thanks Johnpoz for all the help !!!! A Big Thank You to you and others who give me a hand !!!

[SubX]

Bad news, this setup works for around 1 hour before Bell Hub 3000 start to reboot. The same symptom returned when I switched to Bell FTTH service. Before the old DSL modem works without any problem.

Any suggestion here, I will go with pfSense +  L2 Switch option where pfSense acts as router to see if the same issue will emerge.

For the pfSense + L2 Switch setup, I will start another post to seek help.