What am I doing wrong (pfSense behind edge router)
-
Hi there. This is my first time using pfSense so please excuse my ignorance.
TL;DR - pfSense is behind the edge router and in front of my "secure" subnet. Traffic from my desktop seems to be sent out fine, but traffic coming back from the internet seems to be dropped, re-transmitted, or in a loop somewhere. Seems like there may be more NAT or static route setup needed.
This is my home network, but also for a school project. The server at 10.32.0.4 is not attached yet, but everything else is actually set up the way the diagram indicates.
My intent is to create a secure zone for my desktop and a couple of laptops, a DMZ for a FTP server, and then the red zone for every other device. For my home network I don't really need the DMZ, but for this school project I do.
My desktop ip config is
IP - 10.64.0.3
Subnet mask - 255.224.0.0
Default GW - 10.64.0.1DNS Servers - 10.0.0.1; 8.8.8.8
I can ping the edge router (10.0.0.1) without issue, and access its GUI config page, same with pfSense. When I ping somewhere on the internet, like 8.8.8.8, one or two come back and the rest time out. When I go to an actual web page, it loads forever, maybe loads up part of the page, but does not have a clean connection.
From reading here and elsewhere, it sounds like an asynchronous routing issue but I've gone through everything on this page and none of it seems to apply.
Troubleshooting Blocked Log Entries due to Asymmetric Routing
All my wired devices are on the same Cisco switch, so I created VLANs so they would have to go through a router to communicate with each other. The diagram below shows the VLANs and subnets of all connections: Red is the main LAN (10.0.0.0/11 - VLAN 1), Blue is DMZ (10.32.0.0/11 - VLAN 50), Green is secure zone (10.64.0.0/11 - VLAN 100).
It doesn't seem like asynchronous routing should be an issue because there's only one path to get to and from my desktop: 10.64.0.3->10.64.0.1->10.32.0.1->10.0.0.1->Internet and then Internet->10.0.0.1->10.0.0.3->10.32.0.3->10.64.03.
It seems like when I give myself an ip in the DMZ and put my switch port in VLAN 50 everything works fine. So I guess it has something to do with putting another router between my desktop and the firewall instead of the pfSense LAN interface being my default gateway. But I have static routes on pfSense and the router so I'm not sure why they're confused.
Any help is appreciated.
Network diagram:
Switch VLAN configuration:
Router route table:
Firewall Rules:
NAT/Static Route:
-
Well yeah that setup is clearly asymmetrical... If you want to put a router downstream from your edge, then that should be on a transit network (no hosts on this network). Your using 10.0.0/11 as transit (path to other router) - which has hosts on it..
The simple correct fix would be to setup a transit from your edge router to your downstream router.
At a loss to how you came up with a /11 mask? That just makes no sense at all.. You can not possibly have anything close to 2.1 million devices on those segments..
Also a bit confused as to what your doing with default gateway in your different segments of .1, and then calling out .2 as the vlan?
You need to look at the paths between devices, and what that device uses as its gateway to get to other networks, you run into issues with the return traffic and hairpinning... If your going to put hosts on a transit network, then you either need to host route on each device, or you need to nat the downstream, etc. The correct solution is actual transit network to the downstream router (no hosts on it this network)... Be cause talking to or from these devices to your downstream networks is where you run into asymmetrical traffic.
You have 2 transit networks actually both with hosts on them.. your 10.32 is transit to get to your 10.64 through your cisco router..
Yeah that is a mess ;) You could fix it with transit off your edge router, and then your 2 downstreams connected to that transit. I have to run - but can draw it up for you after I get to work.
-
Ok, so you're saying instead of having the edge router LAN side be 10.0.0.1 and the FW WAN be 10.0.0.3 in the same network with all the wireless devices, create like a /29 network that only has routers on it and no hosts? Or a /30 between edge router and FW WAN and another /30 between FW LAN and the cisco router (currently 10.32.0.3)?
-
If it were I, I would need the pfSense at the edge.
-
@NollipfSense Yeah... the thing about that is my wife doesn't necessarily see the value of a dedicated firewall running on a laptop in our living room (where the only cable connection is) at all times. So I'm trying to keep as much stuff in the guest room/office as possible to appease her.
-
Hmm, we don't know what ports those things are connected to but I would assume the three VLANs are completely separated there in the switch?
In which case you do have, effectively, just a very large transit subnet between pfSense and the Cisco router, 10.32.0.0/11. It certainly doesn't need to be that big but shouldn't be a problem either.
I can't really see where you would have an asymmetric route assuming it actually is connected as shown in the diagram and the VLANs are separated entirely in the switch.Can we see the actual blocked traffic logs you're seeing?
Is it possible that desktop has some other path, via wifi maybe?
Steve
-
@duhduhdee My recipe for the perfect device to use in this situation is an Apple Mini...very small form factor, robust hardware, and aesthetically pleasing to look at. That's what I did...picked up an Apple Mini 2011 server on eBay for $200 and maxed out the memory (16GB).
-
@stephenw10 Sure I'll post some logs when I get home.
My desktop is definitely Gi0/20. The others, I'm pretty sure (will confirm when home) are:
Gi0/1 - 10.0.0.1 (Netgear edge router)
Gi0/3 - 10.0.0.3 pfSense Wan int
Gi0/6 - Pretty sure this was my work laptop
Gi0/7 - 10.0.0.4 Asus wireless router in AP mode
Gi0/9 - 10.32.0.1 pfSense LAN int
Gi0/11 - 10.32.0.3 Cisco router
Gi0/19 - 10.64.0.1 Cisco router
Gi0/20 - 10.64.0.3 Desktopre: the desktop having another path, i don't think so, it doesn't have a wireless NIC except for a USB one I plug in sometimes, and it's not connected now. I thought maybe the Asus router in AP mode was confusing things, so I disconnected it but it didn't seem to have much effect.
-
@NollipfSense That's a pretty good idea, she loves Apple stuff. Thanks
-
Mmm, well that looks correct from what I can see.
Gonna have to see the blocked traffic logs I think.
Steve
-
The devices were all connected to the switchports that I thought they were.
Idk what format would be the most helpful, so here's a screenshot.
Here also is part of my states summary page by IP pair:
I seem to be connected enough to make this post from the ip address (10.64.0.3) and VLAN that I want to be on. But frustratingly, I can go to the youtube page below and see all the thumbnails load and get a bunch of the page formatting, but the video itself just loads and loads forever.
As I write this, I seem to also be blocking a bunch of 239.255.255.250:1900 UDP, which is Simple Service Discovery Protocol. But I don't know why I need that to communicate with most sites on the internet.
Thanks for your help.
-
That network is clearly asymmetrical - you can NOT put hosts on a transit network without issues, unless your going to host route or nat.. I have gone over this so many times its becoming sad!!!
I got tied up in meetings all day at work so unable to draw up how it would be done correctly... And now going to watch some TV with the wife, just got home... But will for sure draw up how you would do this correctly with out being asymmetrical..
Again you have to follow the traffic when devices are talking to each other from the different segments and where their default gateway is to see the asymmetrical flow.. If your just talking traffic to and from the outside internet then its not a problem.. The problem is when you have devices talking to each other over such a setup.
-
@johnpoz I appreciate your help.
-
So here is why its asymmetrical
Here is how you fix that - with a transit network.
If you want to put hosts on a transit network, then you need a route on every host saying hey to get to the downstream network go to X vs your default gateway. Or you have to nat at the downstream network.
-
@johnpoz I really appreciate the diagram.
I think I understand you, but I don't think I have the hardware to accomplish it. I have two Cisco routers, and each only has 2 ports. I also have the edge router, a Netgear R7800, which has one WAN port and then 4 switch ports.
The edge router in your diagram has at least 3 ports. I don't think I have any device capable of routing in more than 2 directions.
So I guess I either need to buy another router or figure out how to NAT what I have.
-
do it with vlans then.
-
@duhduhdee said in What am I doing wrong (pfSense behind edge router):
So I guess I either need to buy another router or figure out how to NAT what I have.
Or you could get a cheap managed switch (avoid TP-Link) and use VLANs.
-
Yeah he already has a cisco switch there that clearly can do vlans... So he should be good to go as long as his edge router can do vlans...
-
Unless I'm misreading this the connection seeing issues is from 10.64.0.3 to external IPs, which should not be asymmetric.
Yes you would certainly have asymmetry problems connecting from, say, 10.64.0.3 to 10.2.0.4. Assuming the Cisco is not NATing, which I assume it isn't since there appears to be a static route to 10.64.0.0/11 on pfSense.
None of that blocked traffic you've shown looks like the result of asymmetry either. It would all be TCP flagged if so.
Steve
-
Yeah, I'm reading in the manual about the edge router's VLAN options. It talks about "Setting up a bridge for a VLAN Tag group". Which I guess bridge = trunk, in netgear's terminology?
If that works, that would be ideal. If all else fails I'll take the laptop pfSense is running on, connect it to the modem, make it the edge router, put the netgear router in AP mode, get rid of the other routers and subnets and just let the cisco switch separate everything by VLAN.
Maybe my wife won't mind if I try to arrange things around the laptop in a pleasing fashion.
Thanks everyone.
