Remote Netgate appliance crippled, please help me to avoid the ride over there

Cabledude

Hi all,
I manage two netgate controlled networks, Home and Remote. The remote is giving trouble as of yesterday 1 pm.

Config:
SG-1100, 25.11, IPsec S2S VPN (Netgate example), HAProxy for RP
UniFi stack behind the Netgate: two 8p switches, two AP's, three security cameras
UniFi network is managed by Cloud Key gen 2+
ISP cable modem in bridge mode

Symtoms:

connections stopped working without me doing anything on the remote network
ping successful for all remote subnets
No uptimerobot error messages
remote clients appear to function properly (iPad, VoIP, TV)
webConfigurator (LAN) not reachable
UniFi Cloud Key (LAN) reachable by IP address, not by hostname
clients on other subnets not reachable by IP address or hostname
My local NAS stopped backing up to the remote NAS

Assets at remote location:

my mother in law (83 y/o and totally tech illiterate)

Steps taken to troubleshoot:

checked the S2S VPN tunnel from the local Netgate and it's up. There are three P2 (child) entries, no split connections so clients from any subnet can go anywhere on the other end
Connected the OpenVPN on my MacBook but no dice. Then disconnected. S2S still up.
unplugged the remote Netgate from mains and plugged in again
check unit status LED's: Power solid green, Diamond flashing (update available)
No improvements
grandma wasn't sure about the LED's before the reboot so we'll never know (my bad)
unplugged remote cable modem and plugged in again
No improvements
tried to SSH in, but times out. I'm not sure that I have SSH opened up on the remote unit.

Any cues what to try next?

Thanks!

stephenw10

Hmm, so you don't know if ssh was ever available?

Can you SSH into something at the remote site and try to access from there?

I assume the 1100 console is not connected to anything locally you can access?

luckman212

@Cabledude said in Remote Netgate appliance crippled, please help me to avoid the ride over there:

…my mother in law…

…grandma…

technical issues aside, I'm still trying to wrap my head around this 83 year old person being both your grandma and your mother-in-law at the same time.

Cabledude

@stephenw10 said in Remote Netgate appliance crippled, please help me to avoid the ride over there:

Hmm, so you don't know if ssh was ever available?

Correct. But my hunch is it isn't as I rarely use it.

Can you SSH into something at the remote site and try to access from there?

Actually all I have over there other than the Netgate are one Synology NAS, one iPad, a UniFi Cloud Key, a VoIP Grandstream box and a TV.

I assume the 1100 console is not connected to anything locally you can access?

I'm not an SSH expert by any means, I only use it when hard pressed really, like if I'd need to take a plane to get to the remote location.

Well then I suppose there isn't much I can do now. It's an hour drive to get there so I'll just do that, probably on Saturday. I'll report back on what I found.

Thanks for your input Steve, I appreciate that.

Cabledude

@luckman212 Good question! She is my wife's mom, so she's my mother in law and she is our children's grandma, which is the title used for her most of the time... 8~D

luckman212

@Cabledude ok that's much better than whatever crazy scenario my brain came up with 🫠

As for your SG-1100, what Steve meant is probably "do you have the USB console cable there, and is it connected (or could it be connected) to a local PC so you could get to a command prompt?"

If you don't have that, you might get lucky with a portscan (maybe SSH is running on a different port…) - use something like nmap or SoftPerfect Network Scanner to check for that if you have any remote access to a machine on the far end.

Cabledude

@luckman212 I'm glad we got your brain sorted, then :)

No I'm afraid we don't run any client PC's or the like and even if we would I probably couldn't connect via IP or hostname, at least I cannot get to my Synology that way.
And Granny would have an extremely hard time if she'd have to plug in cables of any sort other than mains plugs. Although I did have her unplug an ethernet cable once, but it took ages.

Mission-Ghost

Is dns down? DNS seems to pack it in from time to time, especially when ports flap. Pfsense seems very intolerant of port status changes and seems to queue up numerous dns stops and starts that seem to interfere with each other until it’s deadlocked.

Can mil unplug and plug things in? Pull the plug, wait 5 seconds and plug back in. What the heck. It’s already down-ish. If config couldn’t have changed a reboot probably won’t make it worse and might fix it, for now. At least until you can get over there at a convenient time.

stephenw10

Mmm, it does sound like DNS but I would have expected a power cycle to fix some transient unbound issue.

Not sure what sort of access Synology gives you. If it has a CLI you could try to connect from there maybe. Or perhaps you can VPN into the Synology? You'd need to have port forwards already in place though.

Cabledude

@Mission-Ghost Thanks for joining the conversation. DNS is a good thing to check, but not sure how.

The weird thing is I didn't interfere with the remote Netgate at all around the time this happened. My Synology backup log shows perfect backups to the remote Synology every hour until 1 PM and it keeps failing since.

Granny says her iPad is functioning well, but I had to explain to her that display on alone (she said "yes I opened the iPad's lid and the display lights up so it works, yay!") doesn't mean a thing until she actually sits down, opens safari and visits a web site. Will try that tomorrow.

She cannot access the Netgate physically.

She has access to the wall plug and we tried a reboot that way but nothing has changed.

Oh well I'll just go there on Saturday, fix the Netgate and help her in the garden a bit. The poor lady is alone after her husband died many years ago so some company will do her good.

stephenw10

You can enable SSH while you're there.

Cabledude

@stephenw10 said in Remote Netgate appliance crippled, please help me to avoid the ride over there:

Mmm, it does sound like DNS but I would have expected a power cycle to fix some transient unbound issue.

Not sure what sort of access Synology gives you. If it has a CLI you could try to connect from there maybe. Or perhaps you can VPN into the Synology? You'd need to have port forwards already in place though.

Good suggestion. I don't think I need ports open as the remote netgate still accepts OpenVPN connections and the Site-2-site has been up all the time even after this trouble started. however I don't have any VPN's set up in the Synology. I might do that just in case I need it some other time.

Basically all I would need is just to access the unit via IP.

Could it be an expired certificate? I have a wildcard certificate in place (created by Acme on the remote Netgate) for HAProxy which I am quite certain is valid as renewal is handled by Acme. I use the same certificate for the HTTPS web dashboard access.

What if I would dump an old laptop at the remote location and have Granny power it up. That would have given me some options I don't have now.

Cabledude

@stephenw10 I'm in. I tethered my laptop to my phone, now on cellular network, connected to the netgate via openvpn and I get the dashboard instantly. So the problem lies in the local network, not the remote network.

luckman212

@Cabledude if you still have remote web access to your Synology there, you could install Container Manager, execute a lightweight container-like netshoot and pivot from there and try to SSH locally into your 1100 if it's still listening on the internal LAN.

I just tested and verified that this works, pasting some screenshots below. It's a long shot, but could possibly save you a trip.

edit: oh I see while I was putting that together you got in! nice.

Cabledude

@luckman212 Still thanks a ton for all the work you put into that post
I'm reading it now...
I checked, SSH was not enabled on the remote Netgate.

For the life of me I can't think of anything I did to my own network that can have caused this.

luckman212

@Cabledude Well now that you've regained access, I'd start by having a dig through /var/log/system.log

Cabledude

@luckman212

For the life of me I can't think of anything I did to my own network that can have caused this.

Got it! On the local Netgate (my trusted SG-2100 here in the closet) I had made an adjustment the day before: I had changed the MTU on the WAN interface from default to 1400 while troubleshooting a different issue. That issue had nothing to do with MTU and it was solved, but I had forgotten to undo the MTU 1400 setting on the WAN interface.

Regardless, everything kept working flawlessly for another day before the trouble started. That is why I didn't associate the MTU adjustment I made on Monday with this new issue on Tuesday.

I just changed the WAN MTU to default and all is working as before.

Now all I would like is to understand what I messed up by setting WAN MTU to 1400...

stephenw10

Your S2S tunnel is IPSec? That could be dropping packets with an MTU issue. But small packets like ping would still pass.