Routing issue or ?
-
We just replaced our EdgeRouter Pro-8 with a Netgate SG-5100. I’m not sure if we’re having a routing issue, DNS, firewall rule issue, or something else. Everything works how it should when we swap back to the EdgeRouter so it should be an issue with some setting in pfSense.
We just have 1 WAN and 1 LAN setup in use. The pfSense router is at 10.0.5.101, LAN is 10.0.0.0/20. We use Securly DNS servers for our DNS. We have two ApplianSys CACHEBoxes that caches HTTP/HTTPS content for LAN clients. The CACHEBoxes are connected to an ApplianSys LOADBox for load balancing between them. The LOADBox is at 10.0.3.50 and acts as a gateway for the LAN. The CACHEBoxes connect to separate ethernet ports on the LOADBox and are at 192.168.2.2 and 192.168.3.2. I’ve added static routes in pfSense that point 192.168.2.0/24 and 192.168.3.0/24 to 10.0.3.50.
Ping, DNS, NTP tests work on the LoadBox. DNS appears to work on the 2 CacheBoxes but ping, NTP, and HTTP/HTTPS do not. Some websites clients go to load fine, others they get a timeout error page from one of the Cacheboxes that the site was unreachable.
I’ve enabled the option under System > Advanced > Firewall/NAT > bypass rules for traffic on same interface with no luck. I’m fairly sure I have all of the firewall rules we had in EdgeRouter copied over to pfSense correctly.
At a loss as to where to look next to fix this problem so any help is greatly appreciated!
-
@ls112 said in Routing issue or ?:
The CACHEBoxes connect to separate ethernet ports on the LOADBox and are at 192.168.2.2 and 192.168.3.2. I’ve added static routes in pfSense that point 192.168.2.0/24 and 192.168.3.0/24 to 10.0.3.50.
Ping, DNS, NTP tests work on the LoadBox. DNS appears to work on the 2 CacheBoxes but ping, NTP, and HTTP/HTTPS do not.So the CacheBoxes do outbound connections, I guess, and the LoadBox do not natting that traffic. In this case you need an outbound NAT rule for the 192.168.2.0/24 and 192.168.3.0/24 subnets.
To add an outbound NAT rule, switch the outbound NAT to the hybrid mode, save and add a new rule to the WAN for the source 192.168.2.0/23, translation = WAN address.
-
@ls112 said in Routing issue or ?:
The pfSense router is at 10.0.5.101, LAN is 10.0.0.0/20.
How are you doing that? /20 overlaps your wan address
/20 would be 10.0.0.0 - 10.0.15.255
So yeah that is going to be an issue.
Is the 5.101 just pfsense lan IP? That is an odd IP for the gateway ;) Normally its at either end of the range, or smack in the middle ;)
The LOADBox is at 10.0.3.50 and acts as a gateway for the LAN.
Huh?? That is going to be problematic as well - and screams asymmetrical... Could you draw up this network..
-
@johnpoz said in Routing issue or ?:
@ls112 said in Routing issue or ?:
The pfSense router is at 10.0.5.101, LAN is 10.0.0.0/20.
How are you doing that? /20 overlaps your wan address
/20 would be 10.0.0.0 - 10.0.15.255
So yeah that is going to be an issue.
Is the 5.101 just pfsense lan IP? That is an odd IP for the gateway ;) Normally its at either end of the range, or smack in the middle ;)
Yes pfSense LAN IP is 10.0.5.101. We do DHCP on 10.0.12.1 to 10.0.15.254, with another range (10.0.3.1 to 10.0.5.254) set aside for other devices with static IPs. Yeah, odd I know but that's how we've had it set for years and too much of a hassle to change it now, lol.
-
@viragomann said in Routing issue or ?:
@ls112 said in Routing issue or ?:
The CACHEBoxes connect to separate ethernet ports on the LOADBox and are at 192.168.2.2 and 192.168.3.2. I’ve added static routes in pfSense that point 192.168.2.0/24 and 192.168.3.0/24 to 10.0.3.50.
Ping, DNS, NTP tests work on the LoadBox. DNS appears to work on the 2 CacheBoxes but ping, NTP, and HTTP/HTTPS do not.So the CacheBoxes do outbound connections, I guess, and the LoadBox do not natting that traffic. In this case you need an outbound NAT rule for the 192.168.2.0/24 and 192.168.3.0/24 subnets.
To add an outbound NAT rule, switch the outbound NAT to the hybrid mode, save and add a new rule to the WAN for the source 192.168.2.0/23, translation = WAN address.
Thanks, I'll give this a try!
-
Maybe you missed my edit.. Could you draw up this network.. Happy to help but really need to understand what your using as gateway for client, that sit on what IP range, etc. And what is actual gateway to get to internet, etc.
Pointing a client to 10.0.3.x as its gatway, which then uses 10.0.5.x as its gateway is going to be asymmetrical for sure when they all sit on the same /20..
-
@johnpoz said in Routing issue or ?:
Maybe you missed my edit.. Could you draw up this network.. Happy to help but really need to understand what your using as gateway for client, that sit on what IP range, etc. And what is actual gateway to get to internet, etc.
Pointing a client to 10.0.3.x as its gatway, which then uses 10.0.5.x as its gateway is going to be asymmetrical for sure when they all sit on the same /20..
Interfaces:
WAN igb0 is a static IP 74.x.x.8 from ISP that uses the ISP GW 74.x.x.6
LAN igb1 is set to 10.0.5.101 /20, gateway set to NoneDHCP server for LAN set to:
Subnet: 10.0.0.0
Mask: 255.255.240.0
Range set from 10.0.12.1 to 10.0.15.254
Gateway: 10.0.3.50 (load box)
We have a few static IPs mapped from 10.0.3.1 to 10.0.5.252
DNS left blank so should be using 50.18.216.174 and 50.18.216.175 that are set under System > General setupThe Loadbox uses 10.0.5.101 as its gateway and default DNS server. Each CacheBox uses the Loadbox as it's gateway and 10.0.5.101 as DNS. Cachebox 1 at 192.168.2.2 uses 192.168.2.1 as gateway. Cachebox 2 at 192.168.3.2 uses 192.168.3.1 as gateway.
Hope this isn't too confusing!
-
@ls112 said in Routing issue or ?:
DNS left blank so should be using 50.18.216.174 and 50.18.216.175 that are set under System > General setup
Not how it works.. by default if you leave dns blank in dhcpd, then it would point to pfsense IP for dns.. Unless you are not running forwarder or resolver on pfsense.
But you clearly have asymmetrical issue here..
This is never going to be a good thing..
A drawing would make it clear why its asymmetrical..
Where exactly are these 192.168 networks? These cache boxes are connected how to your network.. Where exactly are they - they are networks behind 3.50? 3.50 routes.. or is just some multihomed thing? If its routing and not natting - that is fine and can be setup. But anything in your /20 network should be using 5.101 as its gateway..
When you have a downstream router, that is not natting - it needs to be connected to the upstream router via a transit network.. Or your going to have asymmetrical traffic flow, unless you do host routing on on each device that sits in the transit network.
-
Sorry, a lot of this is starting to get over my head!
Okay, I do have the DNS forwarder turned on.
The cacheboxes are connected to the Loadbox on those networks. The cacheboxes are connected directly to eth2 and eth3 on the Loadbox. LoadBox eth0 just goes to a switch that is fed by the LAN port on pfSense.
Network settings from the Loadbox --
-
-
Hopefully my crude map has enough info about our network setup. I'm not entirely sure I follow why having pfSense at 10.0.5.101 and the Loadbox at 10.0.3.50 is an issue. Does pfSense do routing a little differently than Ubiquiti EdgeRouters do? We have the same IPs when we're running the EdgeRouter - Edge at 10.0.5.101 and Loadbox at 10.0.3.50 - and everything works.
It would sure help if I understood more about how the Loadbox/Cachebox setup actually worked. At our other location where the network only has a single Cachebox the Cachebox acts as a gateway for clients to cache web traffic, e.g.:
Server Running
Deployment Mode Gateway Interception (e.g. PBR)But with this network with the 2 Cacheboxes running behind the Loadbox it looks like the Cacheboxes are basically just a proxy server for clients:
Proxy 192.168.2.2:800
Server Running
Deployment Mode AdvancedProxy 192.168.3.2:800
Server Running
Deployment Mode AdvancedWould I need to open port 800 somewhere in pfSense?
-
Does this loadbox nat? Its a proxy?? 10.0.3.50 is your GW, so if say 10.0.12.56 wanted to go to say 8.8.8.8?
If not then yeah that is asymetrical..
If the loadbox is a downstream router - then it should be connected to your pfsense via a transit network (no hosts on it).. Or you run into the problem I just showed where you send a SYN via red arrows, and your SYN,ACK comes back via green.
-
@johnpoz Thanks. I think I understand what you're getting at now. I've reached out to appliansys support to help me understand what's actually going on with the Loadbox and Caches. I'd be just guessing right now but it doesn't seem like the Loadbox does much but pass traffic on to one or the other Cachebox. I don't see many network options on it, no NAT options.
Looks like the Cacheboxes do some SNATting --
NAT IP Address:192.168.2.2, Source Networks 10.0.0.0/20
NAT IP Address:192.168.3.2, Source Networks 10.0.0.0/20 -
So what happens when user on say this 10.0.3.25 box wants to load say www.google.com?
It pulls data off your cache boxes?
Is you loadbox only used for access local stuff (your applications) What is the whole function of this loadbox and cachebox?
-
@johnpoz Yes, if it's cached the client would get the data for www.google.com from one of the Cacheboxes. We currently just cache HTTP content on the Cacheboxes. The LoadBox is just a load balancer for both of the Cacheboxes. We typically just use a single Cachebox on our networks but a situation come up where we ended up with a secondary spare Cachebox. Rather than sitting around not being used we purchased the LoadBox so we could use the two Cacheboxes together.
Would it be better to run the LoadBox/Cacheboxes off of another port on the pFSense? Like so -
-
@ls112 said in Routing issue or ?:
We currently just cache HTTP content on the Cacheboxes.
Does this even make any sense today? I mean what amount of the net is in the clear?
So all of your clients point to your loadbox as a explicit proxy, which would be better than using it as the gateway..
I would be curious to what is your actual cache hit rate is.. How many clients - clients all do their own caching, etc.
Here is some old data (2019) that points https to being 90 to 95 of all web traffic.. So what exactly are you caching?
https://meterpreter.org/https-encryption-traffic/
I would think its higher now to be honest.. I can not see how such setup makes any sense - just from the electric cost of running the boxes ;) even.
Here is more info
https://transparencyreport.google.com/https/overview?hl=en&time_os_region=chrome-usage:1;series:time;groupby:os&lu=load_os_region&load_os_region=chrome-usage:1;series:page-load;groupby:os -
First be sure you have outbound NAT Rules for the 192.168.2.0/24 and 192.168.3.0/24 networks in place as well as the Firewall Rules in the LAN Tab, which permits the traffic towards the internet. I think your loadbox is acting as a router without any NAT.
Second create a Firewall Rule in Your LAN Tab, pretty much at the top
Source: LAN NET
Destionation: 192.168.2.2/32 and 192.168.3.2/32
tcp destination port 800Advanced Options
State type: None -
@johnpoz I would agree. The Cacheboxes are capable of doing HTTPS caching we just haven't gotten around to enabling that yet. Summer project :)
It still helps some right now:
-
Adding floating rules to allow HTTP, HTTPS, ICMP, and NTP inbound for LAN fixed the issues. No more errors on the Cacheboxes and websites load like they should.
