Webgui and SSH listening on wrong ip



  • Greetings,

    our admin moved our setup to a new location and for some reason the GUI is accessible only on the WAN ip address, from the LAN.
    If we try to access on http://192.168.1.1 it times out and I do not see any entries in the pfsense firewall log indicating that it's blocking us.  If we access on the public ip address (from inside our network only) then the GUI comes right up.
    Similarly SSH works on the public address from our LAN, but not the 192.168.1.1 address and again no entry in firewall log.

    I'm not on site and our admin won't be there until Friday, he doesn't want me to reboot it remotely, somewhat understandably.
    Any ideas, would a reboot possibly get the GUI back on the correct ip?

    thank you in advance,
    3


  • LAYER 8 Moderator

    Are you positive that someone didn't switch LAN with WAN? It reads like the anti-lockout rule is working on the "WAN" side instead of LAN… I'd check that. If that's the case I doubt a reboot would help. The chances are low that this is the problem, but as we had it quite a short time ago with a customer, I'd not rule it out yet :)



  • @JeGr:

    Are you positive that someone didn't switch LAN with WAN? It reads like the anti-lockout rule is working on the "WAN" side instead of LAN… I'd check that. If that's the case I doubt a reboot would help. The chances are low that this is the problem, but as we had it quite a short time ago with a customer, I'd not rule it out yet :)

    I'm not positive, but all the other WAN rules are working correctly (ie NAT is working).

    My suspicion is that this was triggered when pfsense was brought up with DHCP on the WAN side and received a LAN address (192.168.0.X) due to the upstream modem being in "non-bridge" mode, and then for some reason pfsense incorrectly decided that was where it should serve GUI and SSH.  Then we updated to bridge mode and used DHCP to correctly get a public address but pfsense is still listening on the WAN side.
    Alternatively, our admin may have done something wrong along the way (switching which interface is WAN vs LAN perhaps?) and I'm not on-site to fiddle with it.

    Thanks for your reply!


  • LAYER 8 Netgate

    What are the LAN firewall rules?

    The WebGUI listens on *:80 (301 redirected to :443) and *:443 by default. Therefore it will answer requests to any address on the firewall coming in any interface that has a rule that passes the traffic.


  • LAYER 8 Moderator

    "pfsense incorrectly decided that was where it should serve GUI and SSH"

    it doesn't decide that. As Derelict says, it binds on * -> any IP. But normally, the only ports open by default are those by the anti-lockout rule on LAN. If you've setup a rule on WAN, that let's you access 443/22 for mgmt access, that has to come from your setup, not pfSense per se (and I would never recommend that other than it locked down to a source IP that is trusted and verified as some sort of fallback/emergency door in).


  • LAYER 8 Netgate

    And the firewall WebGUI will answer queries to the WAN address from the inside without the same being available from outside WAN.

    Like I said, the firewall will answer requests to ANY IP address on the firewall coming into any interface with a rule that permits it.

    WAN can be completely closed with no inbound rules defined and the firewall will still answer with the WebGUI on https://Wan address:443 from LAN if the LAN rules pass that traffic.



  • and like i said http://192.168.1.1 and https://192.168.1.1 times out, not refused, and no entry appears in pfsense firewall log, as i would expect if we were locked out.  The LAN has the anti-lockout rules in place.  We can ping (ICMP) 192.168.1.1 from the LAN, no doubt that is pfsense's ip address, but … by all evidence it seems it's not listening on port 80 or port 22 on that address.
    Rebooting did not resolve it.

    Chrome says:
    This webpage is not available
    ERR_CONNECTION_TIMED_OUT

    First LAN Rule:

        • LAN Address 80/22 * * Anti-Lockout Rule

    interface LAN static 192.168.1.1

    I ran "packet capture" and I see my requests on LAN interface;
    18:09:34.488988 IP 192.168.1.168.52834 > 192.168.1.1.80: tcp 0
    18:09:34.489014 IP 192.168.1.168.52835 > 192.168.1.1.80: tcp 0
    18:09:34.738972 IP 192.168.1.168.52836 > 192.168.1.1.80: tcp 0
    18:09:35.133949 IP 192.168.1.168.52839 > 192.168.1.1.80: tcp 0

    I ran packet capture on my public ip address and see something strange when I try to access pfsense public address, packet capture shows responses from an internal address 192.168.1.52 that's mentioned in some aliases and rules but I don't see any rules that would direct public port 80 there - very strange... something is not right, wish i could find it...


  • LAYER 8 Netgate

    Just post screen shots of your LAN rules and your System > Advanced webgui settings.

    Not going to just sit around and guess. if you provide the information is will likely be obvious.



  • @Derelict:

    Just post screen shots of your LAN rules and your System > Advanced webgui settings.

    Not going to just sit around and guess. if you provide the information is will likely be obvious.

    OK thanks, I hope it's obvious.






  • LAYER 8 Netgate

    What is that at the bottom about a port forward on port 80? Is there some sort of port forward in the NAT rules?

    You have 443 turned off so it's only on port 80.



  • @Derelict:

    What is that at the bottom about a port forward on port 80? Is there some sort of port forward in the NAT rules?

    You have 443 turned off so it's only on port 80.

    That was me experimenting last night trying to get in, i added a NAT rule to forward port 80 on the WAN side to the LAN interface,
    from NAT page, the rule I added last night (it did not grant me access)

    UPDATE: I changed those to WAN interface / WAN address and I now can access pfsense from outside on the public ip.  Still can't access it from inside on the private ip, times out: http://192.168.1.1/

    cheers


  • LAYER 8 Netgate

    Diagnostics > Packet Capture

    Interface: LAN
    Protocol: TCP
    Host address: 192.168.1.1 (Whatever LAN address is)
    Port: 80
    Count: 10000

    Press Start

    Make a couple connection attempts from LAN to http://192.168.1.1:80/ - Yes, please be that specific in the URL. Try a couple of browsers especially if you have one around that you never use like IE or Safari.

    Go back to Diagnostics > Packet Capture and press Stop

    Post what is there.

    You can change the level of detail there and press View capture to see more detail if you like. You can Download Capture to get the pcap for study in Wireshark, etc, if desired.



  • coincidentally i was basically just doing more or less what you've asked, came back here and found your post.
    I connected to a local windows box and captured those packets,

    13:11:11.479523 IP (tos 0x0, ttl 128, id 26991, offset 0, flags [DF], proto TCP (6), length 52)
        192.168.1.168.56280 > 192.168.1.1.80: Flags [s], cksum 0x1234 (correct), seq 1438877430, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
    13:11:11.479715 IP (tos 0x0, ttl 128, id 26992, offset 0, flags [DF], proto TCP (6), length 52)
        192.168.1.168.56281 > 192.168.1.1.80: Flags [s], cksum 0xbc2f (correct), seq 3263197244, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
    13:11:11.730817 IP (tos 0x0, ttl 128, id 27004, offset 0, flags [DF], proto TCP (6), length 52)
        192.168.1.168.56282 > 192.168.1.1.80: Flags [s], cksum 0x6c4b (correct), seq 1244346485, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
    
    no ack from pfsense.  i will try again with the exact specific url, just in case - thank you for your help!
    [/s][/s][/s]
    


  • OK did it again, copied and pasted the exact url with port 80 and the slash.  Tried 3 browsers, 2 of which are never used.  Here's the result with "normal" detail level:

    13:17:39.874241 IP 192.168.1.168.56622 > 192.168.1.1.80: tcp 0
    13:17:39.874747 IP 192.168.1.168.56623 > 192.168.1.1.80: tcp 0
    13:17:40.124688 IP 192.168.1.168.56625 > 192.168.1.1.80: tcp 0
    13:17:42.871083 IP 192.168.1.168.56622 > 192.168.1.1.80: tcp 0
    13:17:42.871114 IP 192.168.1.168.56623 > 192.168.1.1.80: tcp 0
    13:17:43.120066 IP 192.168.1.168.56625 > 192.168.1.1.80: tcp 0
    13:17:43.404719 IP 192.168.1.168.56627 > 192.168.1.1.80: tcp 0
    13:17:46.406182 IP 192.168.1.168.56627 > 192.168.1.1.80: tcp 0
    

  • LAYER 8 Netgate

    Diagnostics > Command Prompt

    What does this output?

    netstat -an | grep -i listen



  • swapped out the hostname and public ip address, this is the output:

    [2.2.1-RELEASE][admin@my.hostname.net]/root: netstat -an | grep -i listen
    tcp4       0      0 AA.BB.CC.DD.1194      *.*                    LISTEN
    tcp6       0      0 *.53                   *.*                    LISTEN
    tcp4       0      0 *.53                   *.*                    LISTEN
    tcp6       0      0 *.80                   *.*                    LISTEN
    tcp4       0      0 *.80                   *.*                    LISTEN
    tcp4       0      0 *.222                  *.*                    LISTEN
    tcp6       0      0 *.222                  *.*                    LISTEN
    [2.2.1-RELEASE][admin@
    [/code]
    
    so yeah it's listening, and ipconfig says it has this ip address;
    [code]em1: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=4209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso>ether 00:50:c2:24:7a:df
            inet6 fe80::250:c2ff:fe24:7adf%em1 prefixlen 64 scopeid 0x2
            inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
            nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    [/code]
    
    not sure what to think, could some other device on the LAN be interfering somehow?  Or is there some other service config screwing it up?  Still stumped...
    thank you.</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic,vlan_hwtso></up,broadcast,running,simplex,multicast>
    

  • LAYER 8 Global Moderator

    What is the mask on our lan interface?

    You have rules that make ZERO sense at all.. If your lan IP is 192.168.1.1 with a /24 bit mask what is the point of any rules allowing access to 192.168.1.anything??

    You have 3 rules that say proxied access to printing??  edit machine

    Those rules would never come into play..  Do you have some sort of bridge setup?



  • @johnpoz:

    What is the mask on our lan interface?

    You have rules that make ZERO sense at all.. If your lan IP is 192.168.1.1 with a /24 bit mask what is the point of any rules allowing access to 192.168.1.anything??

    You have 3 rules that say proxied access to printing??  edit machine

    Those rules would never come into play..  Do you have some sort of bridge setup?

    Some of the rules you see may have been from me experimenting trying to get this to work.  Other rules may be years old, I just leave well enough alone.
    The isp router is in bridge mode if that's what you mean.

    I modified my comment above to add more info, netmask 0xffffff00



  • @johnpoz:

    You have 3 rules that say proxied access to printing??

    fwiw "proxied" is an alias to several LAN addresses.  I don't really care about those rules unless they somehow after 6 years are preventing our web access to the LAN address.


  • LAYER 8 Netgate

    My money's on some other rule somewhere. Do Diagnostics > Command prompt and cat /tmp/rules.debug and send that to me in a PM.

    Nothing shows up in the firewall log for these failures?

    Is 192.168.1.165 listed in Diagnostics > Tables sshlockout ??



  • Correct, nothing in the firewall log.
    sshlockout table -> No entries exist in this table.

    I'm about to go out for a few hours but I'll be back later and really appreciate your help.


  • LAYER 8 Netgate

    This is your problem:

    LAN = "{ em1 }"

    rdr on em1 proto tcp from any to 192.168.1.1 -> 192.168.1.7

    Find that port forward. Actually that is probably a 1:1 NAT.

    You should probably check that these are necessary and doing what you want too:

    rdr on em1 proto tcp from any to 192.168.1.1 -> 192.168.1.212 port 22
    rdr on em1 proto tcp from any to 192.168.1.1 port 21255 -> 192.168.1.241 port 3389


  • LAYER 8 Global Moderator

    "I don't really care about those rules"

    While your at it - might well clean up the the rest of those rules that are completely pointless.  You don't care that there are rules that don't do anything??  Maybe that is part of your problem lack of understanding how the rules work?

    "just leave well enough alone."

    Another issue if you ask me..  What I would do after you fix this big issue is address all your rules, what do they do - are they need still do they even do anything..

    "experimenting trying to get this to work"

    This points to a BIG problem if you ask me.. Why would you be experimenting?  You experiment on the recipe for the perfect pasta sauce or cupcake.  When it comes to firewall rules there should be no experimenting.. You either understand how they work and what is needed to allow or block what you want or you don't.  If you don't - that is a problem!!

    In wha scenario is that 67.166.x.x address going to be both a source and or a dest address?  In those first 2 rules?  Your lan is 192.168 - how would there be a packet inbound to the lan interface from that IP?  Dest ok maybe.  But seeing that its listed as a source along with those rules to dest IPs that are in the same local lan network to allow printing??



  • @Derelict:

    This is your problem:

    LAN = "{ em1 }"

    rdr on em1 proto tcp from any to 192.168.1.1 -> 192.168.1.7

    Find that port forward. Actually that is probably a 1:1 NAT.

    You should probably check that these are necessary and doing what you want too:

    rdr on em1 proto tcp from any to 192.168.1.1 -> 192.168.1.212 port 22
    rdr on em1 proto tcp from any to 192.168.1.1 port 21255 -> 192.168.1.241 port 3389

    There's only one rule involving that ip address (192.168.1.7) and it's on NAT.
    It should only listen on one specific port, so not sure why it would cause the problem.  Update: I deleted that rule and still can't access on the LAN address.  Further update; I removed another similar rule and now I can access the LAN ip.

    Those other rules are appropriate.  Thank you so much for your help Derelict!



  • To summarize the cause, before the move there were two upstream ISPs connected and after the move only one.  Some rules that applied to the defunct ISP interface somehow got automatically converted to listening on the LAN interface, once I removed those rules the LAN interface started responding normally.

    Thanks again Derelict for your help tracking this down!!!!

    cheers


  • LAYER 8 Netgate

    somehow got automatically converted
    [/sigh]

    Sigh.



  • @Derelict:

    somehow got automatically converted

    Sigh.

    well they weren't changed by me, i'm not on site, if you're sighing then i imagine the guy who moved it must've done something, I'm just trying to figure it out remotely after the fact, in which I have now succeeded thanks to your help.


Log in to reply