LAN cannot access DMZ client

[werkstrom]

Hi,

I'm sure I'm missing something obvious here, so I apologize beforehand.

Somehow I cannot access my DMZ server anymore. And I cannot for the life of me figure out why.

My setup is very simple.

LAN, WAN and DMZ

LAN contains most everything 192.168.1.0
DMZ only has one client 10.11.12.0
WAN is fiber connected

Have had this setup for quite some time, no major issues.

LAN --> WAN = OK
WAN --> HAProxy --> DMZ = OK
LAN --> DMZ = Not Working

The most useful info I have found so far is that I can Ping the DMZclient from PFSense if I ping from the DMZ if but not if I ping from the LAN if. Also From my LAN client (my PC) I can ping the DMZ if (10.11.12.1) but not the client (10.11.12.10)

So:
192.168.1.123 --> 10.11.12.1 = OK
192.168.1.123 --> 10.11.12.10 = Not Working
10.11.12.1 --> 10.11.12.10 = OK
192.168.1.1 --> 10.11.12.10 = Not Working

Also: If I setup HAproxy to forward to services on 10.11.12.10 that works as well. Even if I use my PC on LAN but with the external URL. So WAN --> DMZ appears working.

I thought this must be some issue with rules, but I cannot see any rules triggered in the log... I've also stripped my rules down to a bare minimum, but still no luck. I now only have two rules on the DMZ:
login-to-view

And three rules on LAN:
login-to-view

Any tips on where to look further? Getting desperate and frustrated

Thanks a million for any hints or tips

Ping from LAN PC to DMZ interface works, but not client on that net
login-to-view

Ping in PfSense Diagnostics:
From LAN interface:
login-to-view

From DMZ interface:login-to-view

[viragomann]

@werkstrom
Ping from pfSense itself doesn't need any rule to perform.

So obviously the DMZ device does not respond if request come from outside the DMZ.
(HAproxy uses the DMZ address to connect to the backend device.)

Can you access the internet from the DMZ?

Generally there can be two reasons for this.
If internet access works on the DMZ device check its firewall. Maybe it's blocking outside access.
If internet doesn't work check the network settings on both, pfSense DMZ interface and the device. Maybe there is a wrong mask setting.

[SteveITS]

@werkstrom For your two DMZ rules:
The one with source LAN Net isn't going to match anything because packets from LAN won't arrive on the DMZ interface.

The one with source DMZ Net destination WAN Net isn't going to do much. It only allows DMZ to connect out to the WAN subnet of the pfSense, not the Internet. If you want to allow that, allow from DMZ Net to */any.

I'd also suspect the firewall on the DMZ server.

[werkstrom]

@viragomann
Thanks for super quick answer

"Can you access the internet from the DMZ?"
If I setup HAproxy to forward traffic to services on the server it works just fine from WAN. It is a docker server so it has at least before had no problem accessing the internet pulling images. I have no way of checking quickly at the moment since I have no clients on that net. Will do ASAP though.

"If internet access works on the DMZ device check its firewall. Maybe it's blocking outside access."
Its a Debian docker server so no FW on it (AFAIK)

"check the network settings on both, pfSense DMZ interface and the device."
login-to-view
Haven't changed it, but I would say it looks right in PfSense, no?

@steveits

Thanks for answering:

"For your two DMZ rules:"
You're correct, I've been messing around quite a bit. However they should also not hinder communications right?

"I'd also suspect the firewall on the DMZ server."
The DMZ server is a Debian Docker server and I can access things on it just fine If I setup HAproxy to forward traffic from WAN to it. Also. Has been working for well over half a year. So I do not think that can be an issue. Would make sense though.

[viragomann]

@werkstrom said in LAN cannot access DMZ client:

I have no way of checking quickly at the moment since I have no clients on that net. Will do ASAP though.

To get access to the device you can add an outbound NAT rule to the DMZ interface (masquerading) for the source of LAN network on pfSense for the time being.

[SteveITS]

@werkstrom said in LAN cannot access DMZ client:

"For your two DMZ rules:"
You're correct, I've been messing around quite a bit. However they should also not hinder communications right?

Right. Rules for an interface only affect traffic arriving on that interface. If the packet from LAN was allowed then the response would be allowed because pfSense is a stateful firewall.

You make a top rule on LAN to allow from LAN Net to the DMZ server IP and that would confirm the packets are being allowed (when the counter increases). But that should be allowed now anyway.

The reason we asked about the firewall is because it's a frequent post/answer that the firewall on the server is set to allow inbound from the local subnet and not the Internet. In Windows for instance changing the network from a private network to a public network, or vice versa, changes the firewall ruleset. As viragomann mentioned "HAproxy uses the DMZ address to connect to the backend device" so the traffic is arriving on that server from the DMZ interface IP, ergo packets come from within its own subnet.

Though I see you mention it does not have any sort of firewall on it. Does it have a gateway set in its network config?

[werkstrom]

HAproxy/ping on DMZ:
Aha... That eluded me. I apologize, but now I understand what you meant. Makes perfect sense then, thanks for explaining.

"outbound NAT rule to the DMZ interface"
Don't really understand how to set that up, and I have pulled out a switch and some cables to connect to the DMZ directly. I get your point though, good idea. Thanks.

"Does it have a gateway set in its network config?"
It should have, it worked fine this way this morning. Now, I cannot guarantee it as I have not been able to access it yet... Would make sense but again, nothing I know of has happened with it (not saying it still hasn't ;) )

[viragomann]

@werkstrom said in LAN cannot access DMZ client:

"outbound NAT rule to the DMZ interface"
Don't really understand how to set that up, and I have pulled out a switch and some cables to connect to the DMZ directly. I get your point though, good idea. Thanks.

Firewall > NAT > Outbound

Enable the hybrid mode, save that and add a rule:
inteface; DMZ
source: LAN subnet
destination: any
translation: interface address

[werkstrom]

@viragomann

You guys are awesome. Thank you. :-)

OK, so I can now ping from my LAN ... And SSH to it...

Trying to start one of the Docker images I get an error indicating it cannot access internet (and download).

Checking network config things look allright to me:
login-to-view

login-to-view

cat /etc/resolv.conf
10.11.12.1

DNS is the DMZ interface as excpected (right?)

[viragomann]

@werkstrom
Hmmm. Looks well.

Do you have a firewall rule on pfSense allowing internet access?

Can you ping the DMZ and LAN IP of pfSense. Ensure to have a rule, which permit it.

[werkstrom]

@viragomann
I hooked up a switch and a laptop to the DMZ.
Laptop got IP config nicely using DHCP
Laptop can access Internet...

From Server, Pinging DMZ IP = OK
login-to-view

From server, Pinging the LAN IP = OK
login-to-view

From Server, Pinging 216.58.211.14 (google.com) = OK
login-to-view

From Server, Pinging google.com = OK
login-to-view

[viragomann]

@werkstrom said in LAN cannot access DMZ client:

Trying to start one of the Docker images I get an error indicating it cannot access internet (and download).

From inside the container or when pulling an image.
Docker has a separate subnet and this is natted to the machines IP.

So since you have internet access, the network settings might be ok. Also outbound NAT on pfSense might be working properly.
For enabling access from LAN without masquerading check its firewall settings.

[werkstrom]

@viragomann
I now removed the NAT rule you so kindly helped with.... And... Everything still works...
login-to-view

Now... I - do - not - get - it ....

I did physically change the cat6 cables and added a switch for the DMZ, but I never switched off the FW or the Server...
I also removed two disabled rules on the DMZ (should have zero effect)

There were several services on the server that didn't start. They now start flawlessly

I use Portainer to administer several docker servers. Portainer worked for all except my DMZ until you kindly helped with the NAT setup. Now it works just fine without it...

I tried from three separate computers and my phone on the LAN before. They all had the same issue. They now all can access the services on the DMZ server...

I cannot see what I have done to fix this issue.

[werkstrom]

@viragomann @SteveITS
Very grateful for your kind assistance. Thank you so much.
I'll try to backtrack and see if the issue resurfaces and then be a bit more alert and try to isolate it. Again, thank you so much...

PS. As an anecdote. In 1999 I was leading a development effort to replace an old S/36 system with a SQL server based one. During development we had a huge issue for some weeks where SQL queries didn't return all rows they should. One evening I was staying late and by chance saw a bent network cable for the SQL server. I switched it for a new one and went home for the night. Next morning suddenly we got the full number of rows from our queries... After some debate we switched back to the old network cable and ... The error returned. A number of switches back and forth later we were absolutely sure the cable did in fact cause this strange problem. So what I'm trying to say is that even though everyone is 100% sure something cannot impact something in a certain manner... It still can... Sometimes... Computers work in mysterious ways... And I did in fact change the network cable... DS

[viragomann]

@werkstrom
Strange. The network cable can for sure be a reason for some weird behavior. But this one, access through HAproxy succeed, but from other subnet doesn't...