Typical setup and the same issue over and over! L3 switch + Pfsense, Need help!

IrixOS

Hello dear PF users,

I am facing a real dilemma right now. It has been 3 days by now and I am still not able to find how to access Pfsense.
Few posts exist on the subject, but they are rather vague, although, I felt in some way when reading the posts, that I was searching in the right direction.

It has been four days right now, and I am getting more stressed, my hair is getting grey and I feel that it is time for a more straight forward and concrete solution to the GREAT matter

I unfortunately do not have my GNS diagram with me so I will try to sketch it as accurate as possible.

It is a common setup, I forgot the exact networks so the slash notations are fictional:

VLAN2 -> subnet 10.218.1.x /25
VLAN3 -> subnet 10.218.1.x /27 -> L3 switch IP routing port 10.218.1.89/30 –-----> 10.218.1.90/30 LAN Interface PFsense ROUTER
VLAN6 -> subnet 10.218.1.x /28

The problem here is that I can ping from the management IP of the switch or from other switches to 10.218.1.90 but hosts on the vlans cannot ping the default gateway 10.218.1.90 nor access the internet.

The first advise should be:

Configure a default route on the L3 switch and point it to the PFsense GW interface, so i do ip route 0.0.0.0 0.0.0.0 10.218.1.90
Configure a gateway and a static route on the pfsense router under the menu routing, so I do set the LAN interface with a gateway of 10.218.1.89, the IP of the switches router port I assume, then next set the route destination to the specific vlan or create a summary route to the vlans. Can you confirm that?
Change the rules, but I do not want to mess that up unless I know what I am doing or if somebody can provide me specific instructions and the proper steps to do so.

I must admit that I am not a big fan of graphical configured firewalls, why do you think the CLI was invented?

Can you help me out please, I cannot afford no ASA.

Thank you,

Sincerely

Derelict

What are the rules on LAN?

And the subnets matter. You're probably better off posting accurate details if you want accurate assistance.

johnpoz

Quite possible your /30 could be overlapping your vlans.. Since you could not be bother to actually post the exact info..

"so I do set the LAN interface with a gateway of 10.218.1.89"

NO you do not set a gateway on a lan interface.. Did you read what it says
"If this interface is an Internet connection, select an existing Gateway from the list or add a new one using the link above. On local LANs the upstream gateway should be "none"."

Pretty sure they default to lan nat as the source, which your downstream networks would not match.

IrixOS

Well,

As requested:

VLAN2: 10.218.1.0 - 10.218.1.63 /26 GW: 10.218.1.1
VLAN3: 10.218.1.64 - 10.218.1.79 /28 GW: 10.218.1.65
VLAN6: 10.218.1.80 - 10.218.1.87 /29 GW: 10.218.1.81

DEFAULT ROUTE: 10.218.1.88 - 10.218.1.91 /30

PFSENSE Interface IP: 10.218.1.90 /30
L3 SWITCH Interface IP: 10.218.1.89 /30

No Derelict, I have not set any rule,…

What do you have in mind exactly?

JohnPOZ,

I have no idea what you are talking about,...

Thank you for having a first look into the issue,

I am standing by for further instructions,...

Derelict

What does this mean?

DEFAULT ROUTE: 10.218.1.88 - 10.218.1.91 /30

What are the rules on this pfSense interface

PFSENSE Interface IP: 10.218.1.90 /30

IrixOS

Hmm,

I already knew you would react to that… :-[

I do not know, I call it that way, e.g ip route 0.0.0.0 0.0.0.0 10.218.1.90 :-X

It is the subnet between the L3 switch routerport and the PFsense NIC.

....

I include images of the pfsense rules, Nat, Gateway/Routing section.
That might bring up some enlightment,...Other than a few changed fields, I did not touched them and are as is. The route/gateways section is a complete farce when you look at the ip address.
Frankly I cannot see the forest through the trees no more.

Thank you for replying.

NAT.jpg_thumb

Gateways.jpg_thumb

Routes.jpg_thumb

Rules.jpg_thumb

Derelict

set the default route in the switch to 10.218.1.90

create a gateway on pfSense for 10.218.1.89 - call it L3_SWITCH

Create static routes for your /26, /28, and /29 with L3_SWITCH as the gateway.

Make sure the firewall rules on the pfSense interface will pass traffic for the entire /24, not just the /29 (Called LAN net).

IrixOS

Hmm, Derelict,

I performed your steps,

You can see on the image that I tried to ping from a host in VLAN3 to the gateway of another VLAN, VLAN2 and did not succeed, I got a TTL error. I know the routing on the switch works.

Why is the host using the pfsense routers gateway to communicate with another VLAN host? That is very odd.

Also on the image regarding the modem settings, you can see the 10.218.1.88 network is set for outbound, I used that to temporarely access the internet I believe when I configured my new ADSL modem which is in bridged mode.

I do not know about the Rules list either if they are correct after reading your steps,…

Ping.jpg_thumb
![Rules Update.jpg](/public/imported_attachments/1/Rules Update.jpg)
![Rules Update.jpg_thumb](/public/imported_attachments/1/Rules Update.jpg_thumb)

Rules_2.jpg_thumb

Derelict

TTL means you have a routing loop and the traffic is never reaching its destination but is bouncing back and forth usually.

Traffic between VLAN2 and VLAN3 will be handled entirely by your layer 3 switch. pfSense has nothing to do with it. Did you create SVIs for all your VLANs?

IrixOS

Hmm,…

I knew you would mention that. I do not know what happened here but I now have a ping between host on all vlans. I powered the trunked switch, telnetted from that switch to the routing switch to check the vlans and now it works. That is odd!

Derelict, did you see my images?

I successfully pinged the PFsense box from a host on VLAN 3 and I can access the webserver right now.
However I cannot still access the internet from the host, I have set the DNS ip of the host to 10.218.1.90, thus the ip of Pfsense,...

Did you see my images?

Can you see what is obsolete/wrong in the Rules list?
What regarding the screenshot about the bridged modem on the NAT rules outbound?

Derelict

You need to resolve names, route to pfSense, route out WAN and probably NAT in order to get internet from the switch VLANs. Check all that.

IrixOS

Hmm,

I tried everything you said,

Now my hosts can reach the firewall but cannot access the internet.

I tried to logically play with the rules based on the already existing one, the ones created automatically for the 10.218.1.88 /30 subnet, so when I give my PC an ip of 10.218.1.89 it is natting fine.
When I add the vlan subnets to the rules or in NAT, the hosts can not reach the internet. There must be something I overlooked,…but It very difficult to figger out what.

Like I said before, the configuration of pfsense is too complex for this kind of simple and stupid setup.

I added 2 more images,...If you are willing to analyze them,...

Thank you

Update.jpg_thumb

Update_2.jpg_thumb

coxhaus

I have a question in the second picture you have the rules set to manual configuration but on the right side they say auto created? I am wondering because I am running a L3 switch behind pfsense. Mine is still set to auto.

fpsense1.PNG_thumb

IrixOS

Well,

I followed

https://doc.pfsense.org/index.php/Accessing_modem_from_inside_firewall

I have a modem with a router in bridged mode. This is what I had to do for internet access.

coxhaus

Here is my first setup one evening. I don't know if is right but it works. I never got a /30 mask to work for me as I probably did something wrong.

I am running a modem in bridge mode also and I have skipped the IP network on the modem so I can pull up the stats. It is found using the default gateway.

https://forum.pfsense.org/index.php?topic=105825.msg591728#msg591728

PS
I just tried accessing my modem and it does not work. I guess it worked when I was using a router. I just noticed there is a firewall rule to block all private IP addresses. I may have to add one to allow the modem's IP out pfsense. The ISP is going to block it as private IP addresses are not routable. So I don't have a tested answer for accessing the modem.

PSS
Added the rule to allow 192.168.1.100 to pass the WAN interface. The problem I have now is I cannot escalate the rule higher than the general rule blocking all private IP addresses.

Derelict

Like I said before, the configuration of pfsense is too complex for this kind of simple and stupid setup.

A two-router set up is anything but simple for those not used to it.

It is actually quite straightforward.

Derelict

I have a question in the second picture you have the rules set to manual configuration but on the right side they say auto created? I am wondering because I am running a L3 switch behind pfsense. Mine is still set to auto.

I am not sure which version added it but some time ago pfSense got smarter about adding static routes to automatic outbound NAT in addition to LAN interface networks.

Derelict

I added 2 more images,…If you are willing to analyze them,...

That looks good. You are going to have to provide more information than "can not reach the internet." It's impossible to tell from that if you have a routing problem, NAT problem, DNS problem, DHCP problem, Layer 2 problem, etc.

Can the hosts on the L3 switch ping 8.8.8.8? How about 10.218.1.90? How about whatever DHCP is giving them as a default gateway which should be the VIF on the L3 switch for their segment???

coxhaus

@Derelict:

I have a question in the second picture you have the rules set to manual configuration but on the right side they say auto created? I am wondering because I am running a L3 switch behind pfsense. Mine is still set to auto.

I am not sure which version added it but some time ago pfSense got smarter about adding static routes to automatic outbound NAT in addition to LAN interface networks.

Yes my outbound NAT routes were added automatically with pfsense v 2.2.6.

IrixOS

Derelict, >:(

I think we have got winner!!! ;D

I was about to throw the Tyan GS10 through the window along withe the firewall :P

You mentioned to check other factors that might block web access.

I was thinking about a misconfigured NAT or rule, you said it could be anything, well it turned out to be the resolver ::)

I should have known that as I was a former IPCop user.

Derelict I would say BRAVO, I have learned alot about pfsense in a short time and will continue expoiting it to some degree at least.

I will move forward to the access lists, and fine tuning

Many thanks

Sincerely,

IRIXos
I HATE FreeBSD desktops