Can't access remote console
-
I have an SG-3100 that was flashing that it had an update. I logged into the GUI and did the update. A few days late I could no longer log into the GUI so I tried to connect via the Remote console, which I've done several times in the past. But for some reason I can't get it to connect. I've triple checked my COM port. And the settings in PUTTY. They are all correct. The port shows in device manager (COM3) but when I try to connect there's no success. I even rebooted the appliance. I can not connect via the GUI, but still not via the Remote Link.
Any suggestions as to why that connection isn't working all the sudden? I do see in the log file "login on ttyu0 as root" many times as I'm attempting.
I'm on version 21.05
-
Log entries like this are expected when bootup completes and are nothing to do with the serial console:
Jun 25 00:31:47 login 49429 login on ttyv0 as root Jun 25 00:31:47 login 50531 login on ttyu0 as root
What are you connecting from?
You should see the output from uboot at the serial console even if it's power cycled even if pfSense doesn't boot.
There's really no reason it would just stop working like that completely (even from uboot).
It's probably an issue in the terminal client where something changed. Windows update changed the permissions for example.
The only other real possibility is that the USB cable or port have been damaged somehow but that's a lot less likely IMO.Steve
-
@stephenw10 Yes that's the type of entry I was seeing in the logs. Since those are normal I'll ignore them.
I was trying to connect via the USB cable directly to the Sg-3100. A couple of months ago it did an update and afterward I couldn't get into the GUI. It just kept retrying over and over saying it wasn't ready yet. That went on all day. I then connected via the USB cable with Putty and did a reboot and everything was normal.
Now a couple months later it does another update and it does the same thing. Only this time I can't get in with Putty. When I click the open button it just makes the computer go "ding" and nothing happens.
I ended up killing the power to the SG-3100 and when it rebooted I was again able to get into the GUI but I still can't connect via the USB. Like you said, it's super unlikely the cable went bad as it hasn't even been touched since the last time. I've checked connections on both ends and tried multiple USB ports on the computer. I've looked in Device Manager and it's still COM3 regardless of which USB I plug it into. The drivers look like they've loaded correctly. It's a Windows 7 Notebook so there haven't been any Microsoft updates. Indeed I'm not sure I've even used this notebook for anything since the last time I used it to access the SG-3100. Maybe I can find another notebook and try it with that. Not sure what else to try.
-
@stephenw10 I used a different computer. Same results. It's showing up as COM3 when I plug it into the computer so the connection is taking place. But Putty isn't able to connect. I've checked the speed/parity/etc on both the computer's com in Device Manager and with Putty to make sure they match and match the setting from the manual (115200, 8 bits, 1 stop, no parity, xon/xoff). Can't make it happen.
I used to run PfSense on an old computer and when it came time to update I moved to one of the appliances as I thought it would simplify things a bit but it's had the opposite effect. This appliance has been trouble from day 1.
-
Try a different USB cable if you have not already.
The client making a noise when you open the terminal seems more like a problem opening the port. I would expect it to show an error though if it's a permissions issue.
-
@stephenw10 I've tried different ports, different cables, different computers. The only thing left is that SG-3100 unit. I've swapped out every other part of the equation.
Is there a setting in PfSense that would disable that connection? I didn't think so, but I don't know what's left.
-
It's almost impossible to disable the serial console on the SG-3100, certainly not something you could so accidentally in the pfSense config. However even if you deliberately disabled it it's not possible to disable the output from uboot when it boots. That will always appear on the console it the device is booting at all.
If the port itself was damaged you would not see the virtual com device at the client when you connect it.
The most common cause for this is the Windows driver. Are you able to try a Linux client where the driver is in the kernel already?
Another alternative is to connect it to another pfSense device which also has a driver included. Then you can ssh into the other firewall and open the 3100 console using the cu command.Steve
-
@stephenw10 I tried to access it with two different computers (Windows). One has never connected to and PfSense before so I had to install the driver. The other one has connected many times without a problem. However this time neither one could establish an SSH connection. It was as if the port wasn't opening at all.
I don't have any linux machines running that I can use for this connection. My critical operations are running on Proxmox but I can't risk having those servers go down so I can't use that machine.
I haven't tried rebooting PfSense while trying to connect via Putty at the same time. How would a person do that? Just continually clicking the Open button on Putty to see if it connects at some point?
-
You should be able to reboot the firewall whilst connected to the console. Or even power cycle it.
The USB-serial IC is powered by the client so it will remain up even if the 3100 is powered down.You should always see the uboot output there when it is powered on.
Steve
-
@stephenw10 I don't understand. If I can't get the console to come up at all, how is it going to stay connected during a reboot when it's never connected in the first place?
It shows up as a port in Windows (COM3) but Putty is never able to connect to the SG-3100 over the port connection.
-
As I understand it the console connects, putty allows you to open the port, you just don't see any output on it?
If you reboot it whilst that console window is open you would see output from uboot even if for some reason pfSense was not using the serial console.Steve
-
@stephenw10 No that's not what's happening. I connect the console, which then shows as COM3 in Windows. I open Putty however it will not connect to the console. When I click on the Open button in putty it does not connect.
I've replaced the cable, the computer, and triple checked the COM and settings but nothing gets the console to open.
-
Ah, OK. Then I would expect Putty to show an error when you try, what does it show?
However that rules out a pfSense config issue. There is nothing you can set to make that happen.
Either it's a hardware issue with the USB-serial device or a problem on the client(s).
A hardware issue is possible but I'm not we've ever seen that on an SG-3100. A driver is far more likely but testing from two different clients makes it a lot less so.
Do you have anything else you can try to connect from?Steve
-
@stephenw10 No, there is no error from Putty. It shows nothing.
I went and got another computer. I tried to connect it as well. I tried with multiple USB cables with it too. So now I've tried three different computers and about six cables. None of them can connect to the SC-3100 console. They all do exactly the same thing.. that is, nothing but bing.
I don't see how three computers can all have failed at the same time and all those cables could have failed at the same time. This HAS to be a problem with the SG-3100 or my serial settings (115200, 8, 1, none, xon/xoff). Otherwise I don't see any possibility other than the SG-3100 console port.
-
Hmm, the 'bing' sounds like Windows complaining trying to open the port. Like a permissions error. But I would expect putty to throw an error if that was the case.
Does Windows log an error?Are you able to test with anything that isn't Windows. Boot a live Linux image?
If it really is the virtual com device that has failed and it's in warranty then open a ticket with us. The only option in that case would be to RMA it.
As I said though, I don't think I've ever seen that sort of failure on the SG-3100.Steve
-
@stephenw10 Everything we have here is Windows except for our Proxmox servers. I get lost quickly in Linux. I hate to mess with those Proxmox machines as they are mission critical to our operation.
I doubt there's a warranty on my unit as it was purchased in October 2019.
-
@cdsjerry When you say there is "no error from Putty. It shows nothing." Do you at least see a blank screen? If so, have you tried hitting return a few times at this blank screen? Ignore me if you've tried this too
-
@theonemcdonald No. I don't even see a blank screen. Normally you hit this open button and get the blank screen and can log in from there etc. In this case it's never getting that far. When you hit the open button the computer bings and that's it. There's no communication being established with the console. No window, not box. Nothing.
-
Well I would expect Windows to log an error of some sort. It does really seem like a driver or permissions error.
Otherwise I would boot a live linux image on whatever hardware you have and test from that.
Steve
-
@stephenw10 As I've said, my Linux is really poor. If I spin up a VM Linux box what commands would I need to issue to open the SSH connection?
This seems like a really long shot given that three different computers have not been able to reach it, including the one that we've always used to connect to it in the past and have never had a problem.
-
I agree it seems unlikely but we have seen Windows driver issues in the past.
If you use a VM you introduce the additional issue or passing through a USB port to it. I would just boot a live Linux USB stick on a laptop and then connect form the live session.
In Linux you can just use screen:
https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/connect-to-console.html#launch-a-terminal-programSteve
-
@stephenw10 I built a Debian Linux machine but I'm having trouble installing the driver due to my lack of Linux experience. I have Putty installed but not the driver so it's not able to connect yet. I don't have a tty/USB0 connection showing. Instead it says /dev/ttyS0 I'm assuming this is because there's no driver.
I have several very busy days ahead of me on other projects so it may be a bit before I can get back to this. I hadn't expected it to take this much effort just to access the console.
-
@stephenw10 My Linux skills (or rather the lack of them) are limiting me. I got a Debian machine built but I can't figure out how to create a com port nor how to install the drivers. When I try to enter the commands I find on the help pages it usually comes back as an unknown command. I'm just lost at this point.
The console is dead to Windows and my Lixux sucks.
-
You should just be able to run something Live from CD or memstick. No need to install.
What version of Debian are you using? Anything recent will have the driver in kernel already.
Try runningdmesg
at the command line, then connecting the console and running it again. You should see some new messages from the USB connection.
For example on mt Linux Mint box here I see:[16678.494072] usb 2-2: New USB device found, idVendor=10c4, idProduct=ea60, bcdDevice= 1.00 [16678.494078] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3 [16678.494082] usb 2-2: Product: CP2104 USB to UART Bridge Controller [16678.494085] usb 2-2: Manufacturer: Silicon Labs [16678.494088] usb 2-2: SerialNumber: 01327307 [16678.495946] cp210x 2-2:1.0: cp210x converter detected [16678.498315] usb 2-2: cp210x converter now attached to ttyUSB0
I can then open ttyUSB0 with
screen /dev/ttyUSB0 115200
.
You may need to use sudo with that depending on your user.
To escape screen pressctl+a
thenk
then answery
.Steve
-
@stephenw10 I already wiped out a Windows machine so I could do this. I installed Debian 4.19.194-2. When I run the dmesg the second time I do get a line just like your last line. I then open Putty and set the port to /dev/ttyUSB0, speed 15200, 8, 1, none, xon/xoff and click open.... and nothing happens. It's exactly the same as what's happened on all the other computers.
-
So it refuses to open (like in Windows) or it opens and shows no output?
No errors shown?
The current version of Debian is 10.10. That looks more like a kernel version but it's still quite old.
The driver has been in there for a long time though so it should still work. -
@stephenw10 It doesn't open at all. Exact same result as when I tried it with Windows. I click the Open button and literally nothing happens. No errors, no window, no blank window... absolutely nothing.
I wonder why my version is so old. I just downloaded the ISO from debian.org yesterday. I don't know how to make up update. I've found that yum doesn't work at all. I've done apt-get update and it insists that I put my DVD back in the drive. It doesn't seem to be getting online updates? Probably a repository error from the install but again... I don't know how to change it. This is my problem, I just don't use Linux enough to know what I'm doing.
-
If you try using screen instead of putty? Any error shown?
What are the messages actually shown in dmesg when you connect the console?
-
@stephenw10 I don't have a way to Copy/paste from that machine but here's a photo of the last lines.
I'm not familiar with screen.
-
@stephenw10 Was that screen post any help?
-
Ok the screenshot looks as expected. Nothing wrong there.
@stephenw10 said in Can't access remote console:
I can then open ttyUSB0 with
screen /dev/ttyUSB0 115200
.
You may need to use sudo with that depending on your user.
To escape screen press ctl+a then k then answer y.Try connecting to it with screen instead of putty as I showed.
Steve
-
Reviewing this whole thread it seems likely Putty is just not configured correctly on either Windows or Linux.
When you click Open and it does nothing or just the Windows error sound it pretty much has to be because it's trying to open something invalid or doesn't have all the settings necessary.If screen doesn't work (I believe it will though) lets see some screenshots of your putty config.
Steve
-
@stephenw10 Tried it with screen. I get an error message "Cannot exec '/dev/ttyUSB0' : No such file or directory
You said Putty probably isn't configured correctly but I've used Putty to access this same appliance in the past with the same computer (as well as the other four computers we've now also tried).
I agree that it's trying to open something invalid... because none of the programs is able to communicate with the 3100. It's being non-responsive which makes it invalid. Attached are images from Putty with both Windows and Linux
-
Ok when running screen you have to run it using this command:
screen /dev/ttyUSB0 115200
It looks like you just tried to run
/dev/ttyUSB0
dircetly.Those Putty screenshots show the serial console settings but not the actual session you're trying to open which will be ssh and nor serial by default.
Make sure you have set 'serial' in the session section at the top.
Steve
-
@stephenw10 said in Can't access remote console:
screen /dev/ttyUSB0 115200
That IS the command I was running.
I have however gotten back into the remote console. I did another reboot and then use Putty on my original Windows machine and I got into the console. I have access again.
-
Ah, good news. Any idea what changed?
At least that shows the console port is good still.
Steve
-
@stephenw10 I have no idea. Something in the reboot must have done the trick.
The computer was still up and running the same instance of Putty from my earlier tests. All I did was plug in the cable (because it had been plugged into the Linux computer), checked to make sure it was still COM3, and hit connect. I got a blank window (which did not happen before) hit enter a couple of times and was presented a # prompt.
-
Hmm, OK. Well glad you were able to connect.
The putty interface can be confusing at times. It's all too easy to think you're opening a serial connection when in fact it's just showing you the serial settings and the session is still set as SSH.
Steve