Routing issue or ?

[ls112]

@johnpoz

Sorry, a lot of this is starting to get over my head!

Okay, I do have the DNS forwarder turned on.

The cacheboxes are connected to the Loadbox on those networks. The cacheboxes are connected directly to eth2 and eth3 on the Loadbox. LoadBox eth0 just goes to a switch that is fed by the LAN port on pfSense.

Network settings from the Loadbox --

[ls112]

@johnpoz

[ls112]

Hopefully my crude map has enough info about our network setup. I'm not entirely sure I follow why having pfSense at 10.0.5.101 and the Loadbox at 10.0.3.50 is an issue. Does pfSense do routing a little differently than Ubiquiti EdgeRouters do? We have the same IPs when we're running the EdgeRouter - Edge at 10.0.5.101 and Loadbox at 10.0.3.50 - and everything works.

It would sure help if I understood more about how the Loadbox/Cachebox setup actually worked. At our other location where the network only has a single Cachebox the Cachebox acts as a gateway for clients to cache web traffic, e.g.:
Server Running
Deployment Mode Gateway Interception (e.g. PBR)

But with this network with the 2 Cacheboxes running behind the Loadbox it looks like the Cacheboxes are basically just a proxy server for clients:
Proxy 192.168.2.2:800
Server Running
Deployment Mode Advanced

Proxy 192.168.3.2:800
Server Running
Deployment Mode Advanced

Would I need to open port 800 somewhere in pfSense?

[johnpoz]

Does this loadbox nat? Its a proxy?? 10.0.3.50 is your GW, so if say 10.0.12.56 wanted to go to say 8.8.8.8?

If not then yeah that is asymetrical..

If the loadbox is a downstream router - then it should be connected to your pfsense via a transit network (no hosts on it).. Or you run into the problem I just showed where you send a SYN via red arrows, and your SYN,ACK comes back via green.

[ls112]

@johnpoz Thanks. I think I understand what you're getting at now. I've reached out to appliansys support to help me understand what's actually going on with the Loadbox and Caches. I'd be just guessing right now but it doesn't seem like the Loadbox does much but pass traffic on to one or the other Cachebox. I don't see many network options on it, no NAT options.

Looks like the Cacheboxes do some SNATting --
NAT IP Address:192.168.2.2, Source Networks 10.0.0.0/20
NAT IP Address:192.168.3.2, Source Networks 10.0.0.0/20

[johnpoz]

So what happens when user on say this 10.0.3.25 box wants to load say www.google.com?

It pulls data off your cache boxes?

Is you loadbox only used for access local stuff (your applications) What is the whole function of this loadbox and cachebox?

[ls112]

@johnpoz Yes, if it's cached the client would get the data for www.google.com from one of the Cacheboxes. We currently just cache HTTP content on the Cacheboxes. The LoadBox is just a load balancer for both of the Cacheboxes. We typically just use a single Cachebox on our networks but a situation come up where we ended up with a secondary spare Cachebox. Rather than sitting around not being used we purchased the LoadBox so we could use the two Cacheboxes together.

Would it be better to run the LoadBox/Cacheboxes off of another port on the pFSense? Like so -

[johnpoz]

@ls112 said in Routing issue or ?:

We currently just cache HTTP content on the Cacheboxes.

Does this even make any sense today? I mean what amount of the net is in the clear?

So all of your clients point to your loadbox as a explicit proxy, which would be better than using it as the gateway..

I would be curious to what is your actual cache hit rate is.. How many clients - clients all do their own caching, etc.

Here is some old data (2019) that points https to being 90 to 95 of all web traffic.. So what exactly are you caching?

https://meterpreter.org/https-encryption-traffic/

I would think its higher now to be honest.. I can not see how such setup makes any sense - just from the electric cost of running the boxes ;) even.

Here is more info
https://transparencyreport.google.com/https/overview?hl=en&time_os_region=chrome-usage:1;series:time;groupby:os&lu=load_os_region&load_os_region=chrome-usage:1;series:page-load;groupby:os

[A Former User]

First be sure you have outbound NAT Rules for the 192.168.2.0/24 and 192.168.3.0/24 networks in place as well as the Firewall Rules in the LAN Tab, which permits the traffic towards the internet. I think your loadbox is acting as a router without any NAT.

Second create a Firewall Rule in Your LAN Tab, pretty much at the top

Source: LAN NET
Destionation: 192.168.2.2/32 and 192.168.3.2/32
tcp destination port 800

Advanced Options
State type: None

[ls112]

@johnpoz I would agree. The Cacheboxes are capable of doing HTTPS caching we just haven't gotten around to enabling that yet. Summer project :)

It still helps some right now:

[ls112]

Adding floating rules to allow HTTP, HTTPS, ICMP, and NTP inbound for LAN fixed the issues. No more errors on the Cacheboxes and websites load like they should.