What am I doing wrong (pfSense behind edge router)



  • Hi there. This is my first time using pfSense so please excuse my ignorance.

    TL;DR - pfSense is behind the edge router and in front of my "secure" subnet. Traffic from my desktop seems to be sent out fine, but traffic coming back from the internet seems to be dropped, re-transmitted, or in a loop somewhere. Seems like there may be more NAT or static route setup needed.

    This is my home network, but also for a school project. The server at 10.32.0.4 is not attached yet, but everything else is actually set up the way the diagram indicates.

    My intent is to create a secure zone for my desktop and a couple of laptops, a DMZ for a FTP server, and then the red zone for every other device. For my home network I don't really need the DMZ, but for this school project I do.

    My desktop ip config is
    IP - 10.64.0.3
    Subnet mask - 255.224.0.0
    Default GW - 10.64.0.1

    DNS Servers - 10.0.0.1; 8.8.8.8

    I can ping the edge router (10.0.0.1) without issue, and access its GUI config page, same with pfSense. When I ping somewhere on the internet, like 8.8.8.8, one or two come back and the rest time out. When I go to an actual web page, it loads forever, maybe loads up part of the page, but does not have a clean connection.

    From reading here and elsewhere, it sounds like an asynchronous routing issue but I've gone through everything on this page and none of it seems to apply.

    Troubleshooting Blocked Log Entries due to Asymmetric Routing

    All my wired devices are on the same Cisco switch, so I created VLANs so they would have to go through a router to communicate with each other. The diagram below shows the VLANs and subnets of all connections: Red is the main LAN (10.0.0.0/11 - VLAN 1), Blue is DMZ (10.32.0.0/11 - VLAN 50), Green is secure zone (10.64.0.0/11 - VLAN 100).

    It doesn't seem like asynchronous routing should be an issue because there's only one path to get to and from my desktop: 10.64.0.3->10.64.0.1->10.32.0.1->10.0.0.1->Internet and then Internet->10.0.0.1->10.0.0.3->10.32.0.3->10.64.03.

    It seems like when I give myself an ip in the DMZ and put my switch port in VLAN 50 everything works fine. So I guess it has something to do with putting another router between my desktop and the firewall instead of the pfSense LAN interface being my default gateway. But I have static routes on pfSense and the router so I'm not sure why they're confused.

    Any help is appreciated.

    Network diagram:
    Network_Diagram_Troubleshooting.png

    Switch VLAN configuration:
    Switch.PNG

    Router route table:
    Router.PNG

    Firewall Rules:
    Firewall_Rules.PNG

    NAT/Static Route:
    NAT-Static Route.PNG


  • LAYER 8 Global Moderator

    Well yeah that setup is clearly asymmetrical... If you want to put a router downstream from your edge, then that should be on a transit network (no hosts on this network). Your using 10.0.0/11 as transit (path to other router) - which has hosts on it..

    The simple correct fix would be to setup a transit from your edge router to your downstream router.

    At a loss to how you came up with a /11 mask? That just makes no sense at all.. You can not possibly have anything close to 2.1 million devices on those segments..

    Also a bit confused as to what your doing with default gateway in your different segments of .1, and then calling out .2 as the vlan?

    You need to look at the paths between devices, and what that device uses as its gateway to get to other networks, you run into issues with the return traffic and hairpinning... If your going to put hosts on a transit network, then you either need to host route on each device, or you need to nat the downstream, etc. The correct solution is actual transit network to the downstream router (no hosts on it this network)... Be cause talking to or from these devices to your downstream networks is where you run into asymmetrical traffic.

    You have 2 transit networks actually both with hosts on them.. your 10.32 is transit to get to your 10.64 through your cisco router..

    Yeah that is a mess ;) You could fix it with transit off your edge router, and then your 2 downstreams connected to that transit. I have to run - but can draw it up for you after I get to work.



  • Ok, so you're saying instead of having the edge router LAN side be 10.0.0.1 and the FW WAN be 10.0.0.3 in the same network with all the wireless devices, create like a /29 network that only has routers on it and no hosts? Or a /30 between edge router and FW WAN and another /30 between FW LAN and the cisco router (currently 10.32.0.3)?



  • If it were I, I would need the pfSense at the edge.



  • @NollipfSense Yeah... the thing about that is my wife doesn't necessarily see the value of a dedicated firewall running on a laptop in our living room (where the only cable connection is) at all times. So I'm trying to keep as much stuff in the guest room/office as possible to appease her.


  • Netgate Administrator

    Hmm, we don't know what ports those things are connected to but I would assume the three VLANs are completely separated there in the switch?

    In which case you do have, effectively, just a very large transit subnet between pfSense and the Cisco router, 10.32.0.0/11. It certainly doesn't need to be that big but shouldn't be a problem either.
    I can't really see where you would have an asymmetric route assuming it actually is connected as shown in the diagram and the VLANs are separated entirely in the switch.

    Can we see the actual blocked traffic logs you're seeing?

    Is it possible that desktop has some other path, via wifi maybe?

    Steve



  • @duhduhdee My recipe for the perfect device to use in this situation is an Apple Mini...very small form factor, robust hardware, and aesthetically pleasing to look at. That's what I did...picked up an Apple Mini 2011 server on eBay for $200 and maxed out the memory (16GB).



  • @stephenw10 Sure I'll post some logs when I get home.

    My desktop is definitely Gi0/20. The others, I'm pretty sure (will confirm when home) are:

    Gi0/1 - 10.0.0.1 (Netgear edge router)
    Gi0/3 - 10.0.0.3 pfSense Wan int
    Gi0/6 - Pretty sure this was my work laptop
    Gi0/7 - 10.0.0.4 Asus wireless router in AP mode
    Gi0/9 - 10.32.0.1 pfSense LAN int
    Gi0/11 - 10.32.0.3 Cisco router
    Gi0/19 - 10.64.0.1 Cisco router
    Gi0/20 - 10.64.0.3 Desktop

    re: the desktop having another path, i don't think so, it doesn't have a wireless NIC except for a USB one I plug in sometimes, and it's not connected now. I thought maybe the Asus router in AP mode was confusing things, so I disconnected it but it didn't seem to have much effect.



  • @NollipfSense That's a pretty good idea, she loves Apple stuff. Thanks


  • Netgate Administrator

    Mmm, well that looks correct from what I can see.

    Gonna have to see the blocked traffic logs I think.

    Steve



  • @stephenw10

    The devices were all connected to the switchports that I thought they were.

    Idk what format would be the most helpful, so here's a screenshot.

    FW_Log.PNG

    Here also is part of my states summary page by IP pair:

    States_Summary.PNG

    I seem to be connected enough to make this post from the ip address (10.64.0.3) and VLAN that I want to be on. But frustratingly, I can go to the youtube page below and see all the thumbnails load and get a bunch of the page formatting, but the video itself just loads and loads forever.

    Youtube.PNG

    As I write this, I seem to also be blocking a bunch of 239.255.255.250:1900 UDP, which is Simple Service Discovery Protocol. But I don't know why I need that to communicate with most sites on the internet.

    Thanks for your help.


  • LAYER 8 Global Moderator

    That network is clearly asymmetrical - you can NOT put hosts on a transit network without issues, unless your going to host route or nat.. I have gone over this so many times its becoming sad!!!

    I got tied up in meetings all day at work so unable to draw up how it would be done correctly... And now going to watch some TV with the wife, just got home... But will for sure draw up how you would do this correctly with out being asymmetrical..

    Again you have to follow the traffic when devices are talking to each other from the different segments and where their default gateway is to see the asymmetrical flow.. If your just talking traffic to and from the outside internet then its not a problem.. The problem is when you have devices talking to each other over such a setup.



  • @johnpoz I appreciate your help.


  • LAYER 8 Global Moderator

    So here is why its asymmetrical

    asym.jpg

    Here is how you fix that - with a transit network.

    transit.jpg

    If you want to put hosts on a transit network, then you need a route on every host saying hey to get to the downstream network go to X vs your default gateway. Or you have to nat at the downstream network.



  • @johnpoz I really appreciate the diagram.

    I think I understand you, but I don't think I have the hardware to accomplish it. I have two Cisco routers, and each only has 2 ports. I also have the edge router, a Netgear R7800, which has one WAN port and then 4 switch ports.

    The edge router in your diagram has at least 3 ports. I don't think I have any device capable of routing in more than 2 directions.

    So I guess I either need to buy another router or figure out how to NAT what I have.


  • LAYER 8 Global Moderator

    do it with vlans then.



  • @duhduhdee said in What am I doing wrong (pfSense behind edge router):

    So I guess I either need to buy another router or figure out how to NAT what I have.

    Or you could get a cheap managed switch (avoid TP-Link) and use VLANs.


  • LAYER 8 Global Moderator

    Yeah he already has a cisco switch there that clearly can do vlans... So he should be good to go as long as his edge router can do vlans...


  • Netgate Administrator

    Unless I'm misreading this the connection seeing issues is from 10.64.0.3 to external IPs, which should not be asymmetric.

    Yes you would certainly have asymmetry problems connecting from, say, 10.64.0.3 to 10.2.0.4. Assuming the Cisco is not NATing, which I assume it isn't since there appears to be a static route to 10.64.0.0/11 on pfSense.

    None of that blocked traffic you've shown looks like the result of asymmetry either. It would all be TCP flagged if so.

    Steve



  • Yeah, I'm reading in the manual about the edge router's VLAN options. It talks about "Setting up a bridge for a VLAN Tag group". Which I guess bridge = trunk, in netgear's terminology?

    If that works, that would be ideal. If all else fails I'll take the laptop pfSense is running on, connect it to the modem, make it the edge router, put the netgear router in AP mode, get rid of the other routers and subnets and just let the cisco switch separate everything by VLAN.

    Maybe my wife won't mind if I try to arrange things around the laptop in a pleasing fashion.

    Thanks everyone.


  • LAYER 8 Global Moderator

    To external IPs it shouldn't be a problem with asymmetrical true... But the overall design is flawed... Even if he has no need to talk to other devices at this time... The design is not correct to ever allow internal hosts to talk to each other at some future point in time.

    Hosts should never be on a transit. If you have something that forces you do to it, then you have to host routing or natting.. Neither of which are desired things to have to do.


  • Netgate Administrator

    I look forward to pics of your decorative laptop arrangement. 😁


Log in to reply