NAT breaks site browsing

javerleo

Hello pfSense community.

I have pfSense 2.1 release running with this setup:

OpenVPN client –- Internet --- WAN --- pfSense –- OPT --- Private network --- Websphere DataPower XI50 appliance
|
|
|
LAN
|
|
|
Local Network

The problem: There is a secure web interface (Websphere data power XI50) that doesn't load on browser when trying from the local network. From a LAN computer we can reach the host (tracert shows the right route). We can even telnet the host on port 9090 (the port assigned for the application) and there is a response. Other resources on the same remote private network beyond OPT are accesible from local network with no issues. If we connect the same test computer directly to the private branch (no pfsense NAT) and configure the same IP used on pfSense OPT interface (or any other valid on that segment) the application page loads immediately. Also, if a road-warrior user connects to the VPN he can reach the web application without problems.

These facts could be relevant:

Outbound NAT is enabled in OPT interface for traffic comming from LAN subnet and from OpenVPN subnet
There are not firewall restrictions neither on pfsense nor on the remote private network blocking traffic from LAN segment (telnet works and other services on the same remote IP segment are accessible)
There is not application-level firewall blocking traffic on the Websphere DataPower appliance
This is not a DNS issue, since we are using only IP addresses

I'm kinda lost here, so I would appreciate your suggestions.

Thanks in advance.

UPDATE:

I thought it could be a NAT issue, so I tried with static port rules … no luck
From the point of view of the remote appliance, all traffic comes from the OPT IP address (NAT applied), so I don't understand why the problem only affects connections from LAN segment.

So far no help ! Can you contribute?

UPDATE:

Moved from routing to NAT category

javerleo

Bump !

I wonder if there is a way to get the attention of the hero members since this is a hard-to-diagnose issue.

:)

timthetortoise

Sounds like a routing issue to me. Please post your internal IP schema, and results of traceroute to your OPT network from LAN. NAT rules would be very helpful as well.

johnpoz

Why would you be natting between lan and (opt) private?

Lan and OPT are both "local lan network segments" Are they not? Why would you be natting between them?

javerleo

Thanks for the reply timthetortoise.

The IP usage:

LAN –> 172.65.10.1/24 (yes, I know it is a public subnet, but we inherit this mistake and must be kept now)

We added another LAN segment, just to be sure that public segment is not causing the issue --> 192.168.131.1/24

WAN --> 192.168.0.50/24 Goes to Internet through gateway 192.168.0.1 (1:1 NAT with ISP router)

BBVA --> 192.168.5.1/24 Goes to several private corporate networks through gateway 192.168.5.254 (router not controlled by us)

OpenVPN --> 172.31.254.1/24

Trace from LAN (172.65.10.16) to the remote resource. - The browser can not load the Web interface on port 9090 from here.

C:>tracert 172.17.85.91

Traza a 172.17.85.91 sobre caminos de 30 saltos como máximo.

1 <1 ms <1 ms <1 ms 172.65.10.1
2 1 ms <1 ms <1 ms 192.168.5.254
3 5 ms 4 ms 4 ms 10.250.160.6
4 6 ms 164 ms 5 ms 172.16.12.179
5 10 ms 6 ms 5 ms 10.255.252.5
6 * * * Tiempo de espera agotado para esta solicitud.
7 6 ms 5 ms 5 ms 172.17.85.91

Traza completa.

Trace from a OpenVPN client from Internet (172.31.254.6) - The Web interface on port 9090 is loaded without problem from here

c:>tracert 172.17.85.91

Traza a 172.17.85.91 sobre caminos de 30 saltos como máximo.

1 52 ms 79 ms 54 ms 172.31.254.1
2 57 ms 54 ms 72 ms 192.168.5.254
3 59 ms 55 ms 57 ms 10.250.160.6
4 * 79 ms 59 ms 172.16.12.179
5 58 ms 57 ms 59 ms 10.255.252.5
6 * * * Tiempo de espera agotado para esta solicitud.
7 60 ms 80 ms 59 ms 172.17.85.91

Traza completa.

Outbound NAT rules are attached to this post.

outbound_nat.gif_thumb

javerleo

@johnpoz:

Why would you be natting between lan and (opt) private?

Lan and OPT are both "local lan network segments" Are they not? Why would you be natting between them?

Thanks for the answer johnpoz

The OPT interface (we call it BBVA) is the way to connect to several private corporate networks. We need to access resources located beyond OPT. NAT is mandatory since the remote hosts only response to requests made from 192.168.5.0/24 segment.

johnpoz

"LAN –> 172.65.10.1/24 (yes, I know it is a public subnet, but we inherit this mistake and must be kept now)"

Says who?? Fix it - it is bad practice to use public IP space you don't own internally. Bad Network Admin, Bad!!

javerleo

Ok. We removed the public subnet from LAN. Left the valid private subnet 192.168.131.0/24. However, the issue persists.

Any further suggestions?

johnpoz

And your still natting? Why can your systems past opt1 not reply to private IP on your same network, why does it have to nat? Makes no sense. Are they running software firewalls on these systems that private them? Since clearly they can talk back to pfsense - and pfsense is connected to this other segment directly - so how is it they can only reply back to pfsense interface network on opt1?

javerleo

As previously stated, the remote system firewall only pass traffic comming from 192.168.5.0/24 subnet. We can't change that policy.
:(

javerleo

I have not received a useful reply so far. Don't get me wrong: I appreciate your posts, but I would expect to see a good theory about what is causing the issue.

What I see here is weird:

PRIVATE SUBNET X (OPENVPN INTERFACE) –> NAT APPLIED ON OPT --> CONNECTION WORKS

PRIVATE SUBNET Y (LAN INTERFACE) --> NAT APPLIED ON OPT --> CONNECTION FAILS

Some ideas guys ????

johnpoz

I don't have a theory because there is nowhere close to enough info. Where are you NAT rules, since your natting for one.

How is this NOT your problem?

"the remote system firewall only pass traffic comming from 192.168.5.0/24"

How about you actually sniff some traffic vs nonsense like telnet to ports?? How is that troubleshooting?

So lets see a sniff at both the lan interface and then the opt1 interface and maybe we can figure out what is going on.

But understanding your layout kind of requirement - which still seems like a pre clustered type of setup. Can you post your lan and opt1 rules along with your routing table.

timthetortoise

@javerleo:

I have not received a useful reply so far. Don't get me wrong: I appreciate your posts, but I would expect to see a good theory about what is causing the issue.

You really have not provided useful info so far. Your descriptions are inarticulate at best, and there are too many unknowns to really begin to diagnose a cause. To me it sounds like a misconfiguration on your other firewall, or some proxy settings that are getting in the way - but it's really hard to know without being able to actually get on the box and start testing things out. If you want guaranteed support, pony up the cash and buy the support contract. This is a user forum, we're not paid to give you answers.

javerleo

@timthetortoise:

This is a user forum, we're not paid to give you answers.

Who told otherwise ? We all know that !

I published a network scheme, NAT rules, IP addressing and a description of the problem. If you were to seriously contribute to this topic you could ask for specific information, but I'm afraid you even have not read the post.

I just want the advice from users with more knowledge/experience than I, but from your answer I can see you don't belong to that group, so don't waste your valuable time answering to my topics and let others to participate.

johnpoz

Where are your rules for OPT1?

This traceroute
tracert 172.17.85.91

How is that your server on opt1? Thought opt1 network is
BBVA –> 192.168.5.1/24

So you have another network past this gateway router 192.168.5.254 (router not controlled by us)

What are the rules on this router - is it doing NAT? You mention firewall rules on it, what are they?

We don't have a clear understanding of your network - atleast I do not.

So from pfsense OPT1 interface your next hop is that 5.254 address

2 1 ms <1 ms <1 ms 192.168.5.254
3 5 ms 4 ms 4 ms 10.250.160.6
4 6 ms 164 ms 5 ms 172.16.12.179
5 10 ms 6 ms 5 ms 10.255.252.5
6 * * * Tiempo de espera agotado para esta solicitud.
7 6 ms 5 ms 5 ms 172.17.85.91

Why so many hops to get to your 172.17.85.91 -- why are they so slow? If this is a lan? What about hop 6 not answering for your trace - that says time out right?

That seems like a lot of hops for a local lan

You mention
"Other resources on the same remote private network beyond OPT are accesible from local network with no issues."

So that to me says pfsense is fine - and something in either your device your trying to talk to, or something between pfsense and dest is issue. Or rules maybe on opt1 for this specific service or source network? Can not even guess since no rules for your OPT1 interface given.

If works through vpn, that goes through the same nat to your OPT1 network - again points to something past pfsense from how I take it.

If we had sniffs of the traffic on lan and opt1 we might be able to a better guess as to the problem.

timthetortoise

@javerleo:

so don't waste your valuable time answering to my topics and let others to participate.

Done and done. Good luck with it.

javerleo

Thanks for the answer johnpoz.

OPT interface (BBVA) is connected to another router 192.168.5.254. We can not control/check/configure that device (it is under corporate management). The corporate infraestructure could be quite complex and we don' have details about it. Yes, there are several hops since the remote resources include several physical locations around the city.

I will capture SSL traffic on OPT interface to be sure that data packets are allowed through pfsense into next router.

Best regards.

johnpoz

@javerleo:

I will capture SSL traffic on OPT interface to be sure that data packets are allowed through pfsense into next router.

And that there is answer back ;)