Routing to internal VLANs on switch

[jarush]

I have an existing network with PFSense as the default router for the internal LAN with load balanced internet connections. Primary switch for the LAN is a nexus 3k. I'd like to create some new VLANs on the 3k and have them be able to talk with the existing LAN network defined on the PFSense device. Basic network diagram attached. I created a transit network (10.0.0.0/30) for the 3k and pfsense device to talk, gave the OPT1 interface an IP, defined it as a gateway and created a static route to use it to go to 10.168.20.0/24. Created a new VLAN on the 3k (10.168.20.1/24). From pfsense, i can ping all the way into the new vlan. From an existing host on the network, I can hit the new gateway on pfsense (10.0.0.2), but not the new vlan on the 3k.

I tossed in an allow any/any/any to make sure the firewall wasn't blocking me.

Eventually I want to move the existing 10.168.12.0/24 network onto the 3k, but I need some time to migrate hosts and just wanted to start with a new vlan for some new hosts. It feels like this should work but I'm missing something simple. Can someone point me in the right direction?

[johnpoz]

@jarush said in Routing to internal VLANs on switch:

gave the OPT1 interface an IP, defined it as a gateway

Huh.. Why would your OPT1 be a gateway?

You would create a gateway on pfsense to 10.0.0.1, .2 is not a gateway.. Expect for your switch.

[jarush]

Maybe I'm not stating it correctly. Here is what I did:

• on switch, create vlan 300 as 10.168.20.1/24
• on switch, create vlan 2 as 10.0.0.1/30
• on switch, set native vlan to 2 on e1/49 (connected to pfsense opt 1)
• on pfsense, add gateway on interface opt1 for 10.0.0.1
• on pfsense, set ip address on opt1 as 10.0.0.2/30, ipv4 upstream gateway set to 10.0.0.1
• on pfsense, create static route for 10.168.20.0/24 using the 10.0.0.1 gateway

At this point, from a shell on the pfsense box, i can ping 10.168.20.1. From host1 i can not ping 10.168.20.1. My expectation is that I would be able to - thus I believe I'm missing a setting.

[johnpoz]

@jarush said in Routing to internal VLANs on switch:

on pfsense, set ip address on opt1 as 10.0.0.2/30, ipv4 upstream gateway set to 10.0.0.1

No that is not correct either! When set a gateway on the interface you just created a WAN interface that will do nat..

You create a gateway in the routing section, and then create a route to your downstream networks.

[jarush]

OK, I just removed the upstream gateway from the OPT1 interface. I do have the gateway created and the routes for the downstream networks, however, I still can't get to those downstream networks from hosts using the pfsense as the default gateway.
Attached screenshots of the gw and static routes.

[johnpoz]

What PART do not get about not setting a gateway On the freaking interface!!

[jarush]

Apparently the core concept. In your response, are you telling me the screenshots are the way that I shouldn't be configuring it, or should? That is the way it is currently configured (with my pfsense side being 10.0.0.2 and downstream gateway being 10.0.0.1).

By Interface, I assumed you mean the interface that I see when I go to Interfaces -> OPT1. There is no upstream gateway set there. On the edit gateway screen, if I don't put anything in the gateway field, it says "dynamic". I don't understand how this is supposed to work if I don't tell PFsense what the downstream router IP address is.

My assumption, which is obviously wrong, is that I have to configure an interface on PFsense (interfaces -> OPT1) and then tell PFsense what the downstream router IP address is (System -> Routing -> Gateways), and then create a static route to tell PFsense to use this new gateway when going to certain networks. The routing table on the box seems to indicate that the way I have it configured is correct -
10.168.20.0/24 10.0.0.1 UGS ix0

And ifconfig shows the configured 10.0.0.2 IP address on ix0.

Can you tell me what I'm doing wrong? The current state is that the system is configured as you show above (again, with my pfsense interface being 10.0.0.2 and downstream being 10.0.0.1)

[Derelict]

[johnpoz]

@jarush said in Routing to internal VLANs on switch:

Can you tell me what I'm doing wrong?

Dude I gave you a freaking picture of how to do it - how can you not understand this basic concept??

So I flipped IPs in my picture - sorry... But come on who cares which side .1 or .2 is on... The point is NO gateway on the interface!!! This is the 4th time stating this..

Normally is the upstream router that would have the lower number..

Again if you put the gateway on the INTERFACE!!! Pfsense thinks is a WAN!! And will now NAT to this ip, etc. You do not want that! Its not a wan, its a transit to a downstream router.. That is not public. Since its a "lan" interface there is not gateway set on the interface.

[jarush]

I do not have a gateway specified on the interface.

I do have a gateway set on the routing -> gateways -> edit.

I made my settings look like yours last night prior to posting my response.

[johnpoz]

Ok then do a traceroute from client on your host network to your vlan network 10.168.20.1 interface

Does this hit
12.1, then 0.1 as hops

If you can get your 20.1 interface, but not client on 20.x then you have a problem on the host? (firewall?) Or you host in 20.x is not using 20.1 ?

On your firewall rules on lan and op1 are you forcing a gateway in the rules? If so your overriding pfsense routing tables.

Your L3 should have either default route pointing to 0.0.2 or specific routes to your 10.168.12 network.

[jarush]

Sorry for the delayed response, was traveling.

Ended up being the firewall. Had the rule in but it wasn't working - unclear why as all I did was move it above another pass rule - maybe the rules just needed to be reloaded.

Appreciate the patience.

[Derelict]

Rules are processed top down. First match stops processing. If you examine the rule set you will probably see why the more-specific rule needed to be higher.

[jarush]

But they were all pass rules - I have no deny rules explicitly defined.

[johnpoz]

f you route traffic out a gateway hard to get to your other vlan, etc..

[marvosa]

Derelict already explained this with his pic, but I'll add some specifics.

1. "System -> Routing -> Static Routes" should have a static route for all networks behind the Nexus with a gateway of 10.0.0.1 (Looks like this is done)
2. "System -> Routing -> Gateways" should have an entry on OPT1 with a Gateway of 10.0.0.1 (Looks like this is done)
3. Assuming you've enabled routing on the Nexus, remove the VLAN2 you created, re-configure e1/49 as a routed port then give it an IP of 10.0.0.1/30
4. Configure a default route (not a default-gateway) on your Nexus with the next hop of 10.0.0.2

That's it. I'm running this exact same setup at home. Just to reiterate what's on pic, all hosts behind the Nexus need to be using the IP configured on the SVI of each VLAN as their default gateway in order for the routing to work. You will also need to add helper addresses to each VLAN interface in order to provide DHCP behind the Nexus.