Double NAT TCP/UDP not returning

redvapor

I have been searching around the forums but haven't found anything yet that fits my case.

I have a Ubuiquiti USG as my perimeter firewall and router, one of it's interfaces gateways is 10.3.3.1 the other is 10.4.4.1.
10.4.4.1 has a single device, a virtual pfsense router with 10.4.4.2 as WAN side IP.
10.3.3.33 is the LAN side pfsense IP.
Devices pointed to a gateway of 10.3.3.33 can resolve dns and ping via IP but when trying to returns any other tcp traffic it just leaves the network and doesn't seem to come back. That's of course if I am reading the packet captures correct :)

I know double NAT's are bad but it's either this or just trash pfsense and use the USG. Was hoping to find a way to use pfsense for fun and learning. Eventually I want to use the pfsense box as a second gateway for vpn traffic but I can't get any traffic to return with just a standard config. I figure something strange is going on with the route tables.

My config in pfsense:
alt text

alt text

I also have my WAN set to the default route.

Derelict

Why would you put the same subnet (10.3.3.X) inside pfSense as you have on the other router?

redvapor

I don't have the gear for vlan's right now so am a little limited in what I can do with subnets. You think that is my problem?

Derelict

Well, yeah, probably. pfSense will think 10.3.3 is a local subnet. If you are putting pfSense LAN on the LAN of the outside router you'll have asymmetric nonsense.

redvapor

But remember, the perimeter router is serving two internal subnets on two interfaces. example; eth0 is 10.3.3.1 and eth1 10.4.4.1. If pfsense WAN is physically on 10.4.4.x wouldn't it not be an issue that pfsense LAN is physically on 10.3.3.x?

I didn't mention because didn't want to complicate things too much but...

pfsense is virtualized (proxmox) i have created a virtual only interface as well on 10.7.7.1 (called dmz in pfsense). In this case I get the same result. That would eliminate the two gateways on one LAN problem (10.3.3.x), but the problem still remained.

Derelict

It makes no sense to me why you would do it that way.

Might need a diagram to see how it is all logically connected.

kiokoman

you are going out from LAN/DMZ -> 10.3.3.33 -> 10.4.4.2 but i bet packet are coming back from 10.3.3.1

redvapor

@kiokoman I think it's this or something like it. But have not been able to pin it down.

@Derelict
alt text

Derelict

Yeah putting a router on the same backside subnet like that will only cause you grief and pain.