Webgui and SSH listening on wrong ip

3zzz · Oct 20, 2016, 1:08 AM

Greetings,

our admin moved our setup to a new location and for some reason the GUI is accessible only on the WAN ip address, from the LAN.
If we try to access on http://192.168.1.1 it times out and I do not see any entries in the pfsense firewall log indicating that it's blocking us. If we access on the public ip address (from inside our network only) then the GUI comes right up.
Similarly SSH works on the public address from our LAN, but not the 192.168.1.1 address and again no entry in firewall log.

I'm not on site and our admin won't be there until Friday, he doesn't want me to reboot it remotely, somewhat understandably.
Any ideas, would a reboot possibly get the GUI back on the correct ip?

thank you in advance,
3

JeGr · Oct 20, 2016, 7:01 AM

Are you positive that someone didn't switch LAN with WAN? It reads like the anti-lockout rule is working on the "WAN" side instead of LAN… I'd check that. If that's the case I doubt a reboot would help. The chances are low that this is the problem, but as we had it quite a short time ago with a customer, I'd not rule it out yet :)

3zzz · Oct 21, 2016, 12:11 AM

@JeGr:

Are you positive that someone didn't switch LAN with WAN? It reads like the anti-lockout rule is working on the "WAN" side instead of LAN… I'd check that. If that's the case I doubt a reboot would help. The chances are low that this is the problem, but as we had it quite a short time ago with a customer, I'd not rule it out yet :)

I'm not positive, but all the other WAN rules are working correctly (ie NAT is working).

My suspicion is that this was triggered when pfsense was brought up with DHCP on the WAN side and received a LAN address (192.168.0.X) due to the upstream modem being in "non-bridge" mode, and then for some reason pfsense incorrectly decided that was where it should serve GUI and SSH. Then we updated to bridge mode and used DHCP to correctly get a public address but pfsense is still listening on the WAN side.
Alternatively, our admin may have done something wrong along the way (switching which interface is WAN vs LAN perhaps?) and I'm not on-site to fiddle with it.

Thanks for your reply!

Derelict · Oct 21, 2016, 7:14 AM

What are the LAN firewall rules?

The WebGUI listens on *:80 (301 redirected to :443) and *:443 by default. Therefore it will answer requests to any address on the firewall coming in any interface that has a rule that passes the traffic.

JeGr · Oct 21, 2016, 8:50 AM

"pfsense incorrectly decided that was where it should serve GUI and SSH"

it doesn't decide that. As Derelict says, it binds on * -> any IP. But normally, the only ports open by default are those by the anti-lockout rule on LAN. If you've setup a rule on WAN, that let's you access 443/22 for mgmt access, that has to come from your setup, not pfSense per se (and I would never recommend that other than it locked down to a source IP that is trusted and verified as some sort of fallback/emergency door in).

Derelict · Oct 21, 2016, 6:16 PM

And the firewall WebGUI will answer queries to the WAN address from the inside without the same being available from outside WAN.

Like I said, the firewall will answer requests to ANY IP address on the firewall coming into any interface with a rule that permits it.

WAN can be completely closed with no inbound rules defined and the firewall will still answer with the WebGUI on https://Wan address:443 from LAN if the LAN rules pass that traffic.

3zzz · Oct 22, 2016, 1:29 AM

and like i said http://192.168.1.1 and https://192.168.1.1 times out, not refused, and no entry appears in pfsense firewall log, as i would expect if we were locked out. The LAN has the anti-lockout rules in place. We can ping (ICMP) 192.168.1.1 from the LAN, no doubt that is pfsense's ip address, but … by all evidence it seems it's not listening on port 80 or port 22 on that address.
Rebooting did not resolve it.

Chrome says:
This webpage is not available
ERR_CONNECTION_TIMED_OUT

First LAN Rule:

- - LAN Address 80/22 * * Anti-Lockout Rule

interface LAN static 192.168.1.1

I ran "packet capture" and I see my requests on LAN interface;
18:09:34.488988 IP 192.168.1.168.52834 > 192.168.1.1.80: tcp 0
18:09:34.489014 IP 192.168.1.168.52835 > 192.168.1.1.80: tcp 0
18:09:34.738972 IP 192.168.1.168.52836 > 192.168.1.1.80: tcp 0
18:09:35.133949 IP 192.168.1.168.52839 > 192.168.1.1.80: tcp 0

I ran packet capture on my public ip address and see something strange when I try to access pfsense public address, packet capture shows responses from an internal address 192.168.1.52 that's mentioned in some aliases and rules but I don't see any rules that would direct public port 80 there - very strange... something is not right, wish i could find it...

Derelict · Oct 22, 2016, 3:47 AM

Just post screen shots of your LAN rules and your System > Advanced webgui settings.

Not going to just sit around and guess. if you provide the information is will likely be obvious.

3zzz · Oct 22, 2016, 6:31 PM

@Derelict:

Just post screen shots of your LAN rules and your System > Advanced webgui settings.

Not going to just sit around and guess. if you provide the information is will likely be obvious.

OK thanks, I hope it's obvious.

Derelict · Oct 22, 2016, 7:23 PM

What is that at the bottom about a port forward on port 80? Is there some sort of port forward in the NAT rules?

You have 443 turned off so it's only on port 80.

3zzz · Oct 22, 2016, 8:00 PM

@Derelict:

What is that at the bottom about a port forward on port 80? Is there some sort of port forward in the NAT rules?

You have 443 turned off so it's only on port 80.

That was me experimenting last night trying to get in, i added a NAT rule to forward port 80 on the WAN side to the LAN interface,
from NAT page, the rule I added last night (it did not grant me access)

UPDATE: I changed those to WAN interface / WAN address and I now can access pfsense from outside on the public ip. Still can't access it from inside on the private ip, times out: http://192.168.1.1/

cheers

Derelict · Oct 22, 2016, 8:06 PM

Diagnostics > Packet Capture

Interface: LAN
Protocol: TCP
Host address: 192.168.1.1 (Whatever LAN address is)
Port: 80
Count: 10000

Press Start

Make a couple connection attempts from LAN to http://192.168.1.1:80/ - Yes, please be that specific in the URL. Try a couple of browsers especially if you have one around that you never use like IE or Safari.

Go back to Diagnostics > Packet Capture and press Stop

Post what is there.

You can change the level of detail there and press View capture to see more detail if you like. You can Download Capture to get the pcap for study in Wireshark, etc, if desired.

3zzz · Oct 22, 2016, 8:14 PM

coincidentally i was basically just doing more or less what you've asked, came back here and found your post.
I connected to a local windows box and captured those packets,

13:11:11.479523 IP (tos 0x0, ttl 128, id 26991, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.168.56280 > 192.168.1.1.80: Flags [s], cksum 0x1234 (correct), seq 1438877430, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
13:11:11.479715 IP (tos 0x0, ttl 128, id 26992, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.168.56281 > 192.168.1.1.80: Flags [s], cksum 0xbc2f (correct), seq 3263197244, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
13:11:11.730817 IP (tos 0x0, ttl 128, id 27004, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.1.168.56282 > 192.168.1.1.80: Flags [s], cksum 0x6c4b (correct), seq 1244346485, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0

no ack from pfsense.  i will try again with the exact specific url, just in case - thank you for your help!
[/s][/s][/s]

3zzz · Oct 22, 2016, 8:19 PM

OK did it again, copied and pasted the exact url with port 80 and the slash. Tried 3 browsers, 2 of which are never used. Here's the result with "normal" detail level:

13:17:39.874241 IP 192.168.1.168.56622 > 192.168.1.1.80: tcp 0
13:17:39.874747 IP 192.168.1.168.56623 > 192.168.1.1.80: tcp 0
13:17:40.124688 IP 192.168.1.168.56625 > 192.168.1.1.80: tcp 0
13:17:42.871083 IP 192.168.1.168.56622 > 192.168.1.1.80: tcp 0
13:17:42.871114 IP 192.168.1.168.56623 > 192.168.1.1.80: tcp 0
13:17:43.120066 IP 192.168.1.168.56625 > 192.168.1.1.80: tcp 0
13:17:43.404719 IP 192.168.1.168.56627 > 192.168.1.1.80: tcp 0
13:17:46.406182 IP 192.168.1.168.56627 > 192.168.1.1.80: tcp 0

Derelict · Oct 22, 2016, 8:27 PM

Diagnostics > Command Prompt

What does this output?

netstat -an | grep -i listen

3zzz · Oct 22, 2016, 8:44 PM

swapped out the hostname and public ip address, this is the output:

[2.2.1-RELEASE][admin@my.hostname.net]/root: netstat -an | grep -i listen
tcp4       0      0 AA.BB.CC.DD.1194      *.*                    LISTEN
tcp6       0      0 *.53                   *.*                    LISTEN
tcp4       0      0 *.53                   *.*                    LISTEN
tcp6       0      0 *.80                   *.*                    LISTEN
tcp4       0      0 *.80                   *.*                    LISTEN
tcp4       0      0 *.222                  *.*                    LISTEN
tcp6       0      0 *.222                  *.*                    LISTEN
[2.2.1-RELEASE][admin@
[/code]

so yeah it's listening, and ipconfig says it has this ip address;
[code]em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether 00:50:c2:24:7a:df
        inet6 fe80::250:c2ff:fe24:7adf%em1 prefixlen 64 scopeid 0x2
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
[/code]

not sure what to think, could some other device on the LAN be interfering somehow?  Or is there some other service config screwing it up?  Still stumped...
thank you.</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,simplex,multicast>

johnpoz · Oct 22, 2016, 8:44 PM

What is the mask on our lan interface?

You have rules that make ZERO sense at all.. If your lan IP is 192.168.1.1 with a /24 bit mask what is the point of any rules allowing access to 192.168.1.anything??

You have 3 rules that say proxied access to printing?? edit machine

Those rules would never come into play.. Do you have some sort of bridge setup?

3zzz · Oct 22, 2016, 8:58 PM

@johnpoz:

What is the mask on our lan interface?

You have rules that make ZERO sense at all.. If your lan IP is 192.168.1.1 with a /24 bit mask what is the point of any rules allowing access to 192.168.1.anything??

You have 3 rules that say proxied access to printing?? edit machine

Those rules would never come into play.. Do you have some sort of bridge setup?

Some of the rules you see may have been from me experimenting trying to get this to work. Other rules may be years old, I just leave well enough alone.
The isp router is in bridge mode if that's what you mean.

I modified my comment above to add more info, netmask 0xffffff00

3zzz · Oct 22, 2016, 8:55 PM

@johnpoz:

You have 3 rules that say proxied access to printing??

fwiw "proxied" is an alias to several LAN addresses. I don't really care about those rules unless they somehow after 6 years are preventing our web access to the LAN address.

Derelict · Oct 22, 2016, 9:03 PM

My money's on some other rule somewhere. Do Diagnostics > Command prompt and cat /tmp/rules.debug and send that to me in a PM.

Nothing shows up in the firewall log for these failures?

Is 192.168.1.165 listed in Diagnostics > Tables sshlockout ??