LAN > Split DNS > WAN issue

bgachenot

Hello,

I am using PfSense for more than two years and I never could find the solution to a problem with my infrastructure.
Short display of infrastructure:

One dedicated server (Hypervisor: Proxmox).
- Two IPs assigned to the server.
- One IP for Proxmox.
- One for PfSense.
A nginx dockerized reverse proxy, which handles every HTTP/HTTPS request.
Basic use of Pfsense:
- WAN
- LAN
- Port forwarding rules from WAN to LAN for accessing SSH port for Gitlab or other services,...

I am a developer and I migrated from using my own computer to code-server, a self-hosted VS Code accessible through a basic browser. Here comes the problem I'm facing.

I had some issues at the beggining, I couldn't access services in the WAN address. I searched and I found that I had to enable NAT Reflection, I used NAT + proxy which was doing a really good job until now.
When I'm working from my Code-Server, I want to access my Gitlab's SSH port to push some new code or even some Docker images to the Registry.
I am having a lot of errors with this setup, when I push, most of the time I have the same error:

Connection to gitlab.domain.com closed by remote host.
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly

And when pushing big images to the Gitlab Registry, they kept trying to push big layers and ended with EOF.

After some debugging, I found it was related to my front-end reverse proxy (nginx), and not the nginx config of Gitlab itself, I had no push issues with direct access to gitlab or registry.
After some testing and searching with my reverse proxy, I found out that my Pfsense setup was not really great for this situation, because when pushing images for example, it was doing LAN > WAN > LAN and I was loosing the original IP address, so the server couldn't reach the client back.
I changed to Split DNS, so the gitlab.domain.com is now pointing to the nginx IP address. Pushes to the registry are working as expected but I have another issue, I can't access gitlab's SSH port because the hostname is pointing to NGINX, not Gitlab.

I can add a Host in local SSH configuration of my code-server to resolve gitlab.domain.com with the good local IP address but I'm also working from time to time under VPN to access directly my servers, so I will have to do it with each computer and it's quite annoying, I would need to comment every time I'm disconnected from the VPN.

Is there a better solution for my needs?
I think Split Dns is doing very well and is a part of the solution but I didn't find anything for the problem it raise for accessing Gitlab's port.

I tried NAT portforwarding from LAN to the nginx server but it doesn't redirect to Gitlab's SSH port and I also tried Outbound NAT mode but didn't work either

Gitlab IP: 192.168.1.7
Nginx IP: 192.168.1.5

I apologize for speelling mistakes I may have done.
Thanks for your answer!

N.B.: If you need more informations, feel free to ask me!
I didn't find what title it could be, if you have a more accurate one, I'll change it.

viragomann

@bgachenot said in LAN > Split DNS > WAN issue:

I found out that my Pfsense setup was not really great for this situation, because when pushing images for example, it was doing LAN > WAN > LAN

Not really this way. It only does NAT, a combination of port forwarding and masquerading.

and I was loosing the original IP address, so the server couldn't reach the client back.

That's the way NAT reflection + proxy must work. Otherwise you would run into asymmetric routing issue.
But I agree, NAT reflection is not the recommended solution here. If there is no need for natting the traffic, better go with split DNS.

I changed to Split DNS, so the gitlab.domain.com is now pointing to the nginx IP address.

Why do you go through the proxy?

Of course in DNS one host name can only point to one IP. If you need to pass the traffic through the proxy, maybe it's an option to have different host names for both services?

bgachenot

Thank you for the answer!

@viragomann said in LAN > Split DNS > WAN issue:

Why do you go through the proxy?

Of course in DNS one host name can only point to one IP. If you need to pass the traffic through the proxy, maybe it's an option to have different host names for both services?

Well, the front-end nginx is here to provide SSL support for all applications but you mark a good point, I could Split DNS to the gitlab server when I'm on the LAN network. It won't impact the registry which has a different fqdn.
A drawback of doing this would be that, when connected to my network with VPN, I couldn't access the gitlab UI anymore (because HSTS enabled)

Is there really no way to port forward ports on the same subnet?

I will create a subnet and move my code-server to it. It should take the gateway for communication with the LAN subnet and pfsense should be able to perform the port forwarding right?

EDIT: I made the gitlab's SSH server listen at the same port than the external forwarded port, so there is no more needs to port forward or do some port translations on the LAN subnet but I'm still curious to see if it is possible

viragomann

@bgachenot said in LAN > Split DNS > WAN issue:

A drawback of doing this would be that, when connected to my network with VPN, I couldn't access the gitlab UI anymore (because HSTS enabled)

Not clear, why HSTS is an issue. Don't you use HTTPS over the VPN?

You should use an internal DNS system like DNS Resolver on pfSense and configure your host overrides there. So you can provide the DNS to VPN clients, so that the clients resolve to the same internal IPs as when you access it from inside your network.

Is there really no way to port forward ports on the same subnet?

Not without masquerading.
When you forward traffic on the router to another device within the same subnet you get following issue:
routers IP: x.0.1
clients IP: x.0.6
server: x.0.10
client sends a request packet to the router (source: x.0.6, dest: x.0.1)
router forwards it to the server (source: x.0.6, dest: x.0.10)
server responds to the source IP (source: x.0.10, dest: x.0.6)
This packet arrives on the client, but he is awaiting the respond from the routers IP x.0.1, where he sent the request to, not x.0.10. Hence the client will not accept the response packet and the communication will fail.

I will create a subnet and move my code-server to it. It should take the gateway for communication with the LAN subnet and pfsense should be able to perform the port forwarding right?

Might be a solution. So each packet has to pass the router and no masquerading is needed.