14

I changed my default SSH port on my home server (in the /etc/ssh/sshd_config file) to port 54747, then restarted the ssh and sshd services (never sure which one so I did both just to be safe). To test my configuration, I logged out and then back in without any problem.

A couple days later, I installed apt updates, and then rebooted my server. When I tried to SSH back in (on port 54747), I got a connection refused error.

For some reason, I tried to SSH on default port, and it worked ! I went back to check on the sshd_config, but it still had the custom port. So I restarted the sshand sshdservices, and it got back to "regular" behaviour (ssh on port 54747). I tried rebooting again, and connection refused again...

Anyone knows what I did wrong ?

Extra details :

  • Ubuntu 16.04.2 LTS
  • Server is also used a HTPC, with an open session (same user as SSH) on my TV
  • I SSH using my laptop's RSA key, and have disabled password auth
  • I used to reboot with sudo reboot -h now, but after searching, I discovered it was discouraged by some people, so I tried sudo reboot, but no differences

EDIT Sequence of events :

  1. Change SSH port from 22 to 54747 in /etc/ssh/sshd_config
  2. Restart ssh and sshd services
  3. End current SSH session
  4. SSH back in successfully on port 54747
  5. Reboot
  6. SSH connection error on port 54747, but successful on port 22
  7. Restart ssh and sshd services
  8. SSH back in successfully on port 54747, connection error on port 22
  9. Reboot and go back to 6

EDIT 1 : netstat output

rgo@ATLAS:~$ sudo netstat -lntp | grep :54747
rgo@ATLAS:~$ sudo netstat -lntp | grep :22
tcp6       0      0 :::22                   :::*                    LISTEN      1/init

EDIT 2 : service sshd status

● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

EDIT 3 : lsof -i | grep ssh

systemd      1     root   46u  IPv6  42724      0t0  TCP ATLAS:ssh->192.168.1.27:49837 (ESTABLISHED)
systemd      1     root   49u  IPv6  14641      0t0  TCP *:ssh (LISTEN)
sshd      4088     root    3u  IPv6  42724      0t0  TCP ATLAS:ssh->192.168.1.27:49837 (ESTABLISHED)
sshd      4088     root    4u  IPv6  42724      0t0  TCP ATLAS:ssh->192.168.1.27:49837 (ESTABLISHED)
sshd      4202      rgo    3u  IPv6  42724      0t0  TCP ATLAS:ssh->192.168.1.27:49837 (ESTABLISHED)
sshd      4202      rgo    4u  IPv6  42724      0t0  TCP ATLAS:ssh->192.168.1.27:49837 (ESTABLISHED)

For reference, ATLAS is the remote server hostname, 192.168.1.27 is my laptop's LAN IP, and command was executed between steps 6 and 7

ufw status

Status: inactive

EDIT 4 : ps -ef |grep sshd

root      4088     1  0 22:40 ?        00:00:00 sshd: rgo [priv]
rgo       4202  4088  0 22:40 ?        00:00:00 sshd: rgo@pts/1 sshd
3rgo
  • 313
  • 4
  • 10
  • I am not disparaging you in any way. But it looks to me like you are not entering the commands on the ssh server as requested. You can't have live ssh connections when the ssh daemon is dead....... on the ssh server, ps -ef |grep sshd should return the /usr/sbin/sshd -D process. There are several folks helping but sending you in all different directions. I'm happy to chat with you on IM if that would be helpful to you. – jones0610 Jun 14 '17 at 21:24
  • Maybe it's because I already have a session with the same user opened and displayed on my TV with Kodi ? – 3rgo Jun 14 '17 at 21:52
  • Hi, @3rgo, did you managed to solve this? – pa4080 Jun 17 '17 at 16:00
  • Hi ! No I'm still experiencing this issue... Luckyly, I don't have to reboot my home server every so often, but it's still a pain, because it breaks some of my automated processes... – 3rgo Jun 18 '17 at 08:19
  • I've got some ideas. (1) You could try to change the port to its default value, then restart the entire system. Then try to change it again to the desired value. (2) Try with different value, for example Port 10285. Google shows couple of results for 54747... (3) Also the SSH server can work with several ports simultaneously. Create two separate directives for each port: Port 22 and Port 54747, then open only the second into the firewall. (4) You can try Match LocalPort directive, placed in the beginning of sshd_c. – pa4080 Jun 18 '17 at 18:50
  • I tried resetting the port to the default value and then back (with restarts in the middle), but still bugged... I don't want to open a secondary port, I mainly want to disable port 22 for security reasons – 3rgo Jun 19 '17 at 21:14
  • Did you look at /var/log file to find out error printed out during reboot? – Lety Jul 24 '19 at 07:57
  • A ha! I think the clue is here tcp6 0 0 :::22 :::* LISTEN 1/init ssh is starting during init, maybe init has the wrong conf file, or more likely can't access the config file when it started. I'll do some more digging and see if I can propose a solution. – Bazz Jul 24 '19 at 10:38

6 Answers6

10

ssh may be "socket activated" by systemd depending on configuration, which means that initially it is systemd that sets up the listening port, and sshd is only started when a client first connects. This is to speed up startup time: service daemons are only started on demand.

However this means that you must also configure systemd to the matching port. You'll find the system configuration in /lib/systemd/system/ssh.socket which lists ListenStream=22. To override this, create a file /etc/systemd/system/ssh.socket.d/port.conf (creating the directory ssh.socket.d if needed) that contains:

[Socket]
ListenStream=
ListenStream=54747

Change the number to the port desired. The first blank entry erases the previous default, and the subsequent entry adds the new one. This overrides the default shipped in /lib/systemd/system/ssh.socket and must be done in addition to changing /etc/ssh/sshd_config.

Then run sudo systemctl daemon-reload to tell systemd about your changes, and sudo systemctl reload ssh if your ssh daemon was previously running.

Robie Basak
  • 15,930
  • This answer looks very promising, but /etc/systemd/system/ssh.socket.d/port.conf is being ignored and rebooting still resets port to 22. Is the file name relevant? Not able to find good documentation on systemd overrides on Ubuntu. – MestreLion Jul 25 '19 at 00:31
  • The file name doesn't matter as long as it ends in .conf. See systemd-system.conf(5) for details on systemd override configuration files. – Robie Basak Jul 25 '19 at 00:55
  • Also you can run systemctl status ssh.socket to see if it is enabled and what it is listening on. – Robie Basak Jul 25 '19 at 00:56
  • 2
    This works!!! Finally this mistery is solved! But afterwards I noticed I was able to access using both ports: the default 22 and the custom one. Adding a ListenStream= line before the custom port prevented this, not sure why. Maybe this "clears" the ListenStream=22 setting in the default /lib/systemd/system/ssh.socket? Weird way to override settings. Maybe worth adding this to the answer? – MestreLion Jul 25 '19 at 01:27
  • @MestreLion ah yes, that's correct. I'll update the answer. Thanks! – Robie Basak Jul 25 '19 at 12:48
0

Possible causes that I can think of

  1. A different sshd binary is started on boot or sshd is started with a different config. Maybe systemd is the culprit here - it has a different way to change port, via file /usr/lib/systemd/system/sshd.socket apparently: https://www.vultr.com/docs/how-to-change-ssh-port-on-coreos
  2. The correct /etc/ or /etc/ssh isn't mounted yet when sshd starts, is it a separate volume on your machine that gets mounted later in the boot process?
  3. sshd is lacking read permissions to the config file at boot time, although I don't know if sshd would even start then at all.
Jay
  • 176
  • 2
    I reckon you're onto it. And if it's a server that's gone through a lot of upgrades, maybe there's a cocktail of startup scripts laying around (sysv-init, upstart, systemd) Maybe a simple search and check all files in /etc/ find /etc/ -iname "*ssh*" to look for more clues. – Bazz Jul 24 '19 at 10:48
0

ssh is the client process that arbitrates and maintains a user session connection to the ssh server. sshd is the daemon that runs on the ssh server to listen for and authenticate ssh connection requests.

The configuration file on the sshd server that is read when starting the sshd service (which requires sudo privileges to edit) is

/etc/ssh/sshd_config

The service should start out of 

/etc/systemd/system/sshd.service

To restart sshd which would involve re-reading the sshd_config file

sudo service sshd restart

To see what port the sshd daemon is listening to, as well as other helpful information, on the ssh server type

sudo service sshd status

Do these steps in the specified order:

Reboot the ssh server

Open a terminal session on the ssh server (not a ssh connection into it)

Type hostname

If hostname does not return the name of the ssh server (Atlas in this case) redo the previous step correctly.

grep Port /etc/ssh/sshd_config - note the port number. Should be the one you specified

sudo service sshd status

If status reports that it is active, running and listening on the custom port you specified, then you are good on that end. If not, the service startup may not be calling the sshd_config file you modified but another config file that contains default info. If the service didn't start (says dead and not active and running, then this is a different problem than what you asked about.

These steps will likely identify the root cause of the problem you are asking about.

For testing purposes and for simplicity: On the client side, from a terminal session you would ssh into the ssh server as follows

ssh -l username -p 54747 hostname

Based on OP feedback, I suspect that sshd isn't starting up on bootup but does start correctly when manually invoked. Successful ssh connections via port 22 may well NOT be connecting to the ssh server but to something else (e.g. localhost). To prove or debunk this, after connecting via ssh type 

hostname

Based on what OP is saying, I'm guessing hostname won't be the ssh server atlas.

To further isolate this, after rebooting the ssh server but before doing anything further, from a terminal session on the ssh server (Atlas) type

ssh localhost

If this fails, as it should, then 

ssh -p 54747 localhost

If this doesn't work either that will confirm the results obtained when running 

sudo service sshd status
jones0610
  • 2,539
  • Hi ! I added a sequence of events so you can understand better. I SSH using the command ssh -p <PORT> <USER>@<IP>, with my private key added to the agent. – 3rgo Jun 14 '17 at 20:38
  • Very good. Make step 6a: on the sshd server, sudo service sshd status. If it reports port 22 then there is a bogus sshd_config file out there being called. – jones0610 Jun 14 '17 at 20:54
  • Says, "inactive (dead)" (see full output in my OP in a second) – 3rgo Jun 14 '17 at 21:05
  • So if it's dead (not active and running) you aren't ssh-ing into the machine you think you are. On the sshd server, type ps -ef |grep sshd. If the sshd daemon on the sshd server is actually dead, no sshd processes will be running and thus you won;t be able to ssh into it regardless of the port used. – jones0610 Jun 14 '17 at 21:10
  • 2 sshd processes found... I've added detailed output – 3rgo Jun 14 '17 at 21:13
  • Based on your last edit, I've tried these 2 commands ,and all is as expected (hostname=ATLAS and ssh localhostconnect without specifying port) – 3rgo Jun 14 '17 at 21:55
  • These connections work even though the sshd service reports that it is dead and ps-ef shows that the sshd daemon isn't running?!?!?! – jones0610 Jun 14 '17 at 21:57
  • Let us continue this discussion in chat. – 3rgo Jun 15 '17 at 16:29
0

Verify your port settings in the /etc/ssh/sshd_config file. Make sure you are editing as sudo or a user in the sudo group. All you have to do to set the port is, on one line type Port 54747. Now, restart the ssh service by running service sshd restart. Then verify that ssh is listening on that port by running sudo netstat -lntp | grep ssh. Reboot and test.

Also check your network settings. If you are on a corporate network, make sure you are in the correct vlan.

G_Style
  • 703
  • I did a backup and edited the file as sudo, and changed the default Port 22 line to Port 54747only. Also, the netstat you gave me, had no output. I added a modified one in my OP – 3rgo Jun 14 '17 at 20:40
  • You are connecting with a key correct? So you should be connecting like: ssh -i key.txt user@ipaddress -p 54747. Also check if anything else is listening on that port. Do sudo lsof -i | grep ssh. You could also check your firewall to make sure its not blocking anything. Do: sudo ufw status. – G_Style Jun 14 '17 at 20:58
  • No port used on 54747 (see my OP, I added it). I'm adding the output of your commands to it too – 3rgo Jun 14 '17 at 21:08
  • After some more thought about your problem, I have a feeling that it is not your setup, but the way you are rebooting, that is causing this issue you are experiencing. When you reboot you should use the command shutdown -r now. Give that a try and let us know the results.

    See this article for reference: https://askubuntu.com/questions/483670/what-causes-ssh-problems-after-rebooting-a-14-04-server

     – G_Style Jun 15 '17 at 20:51
  • I just tried, and got the same result as sudo reboot -h nowor `sudo reboot`` – 3rgo Jun 16 '17 at 18:11
0

Sometimes things just go wrong. If I were on your place, I would try with: 

cp /etc/ssh/sshd_config $HOME
sudo apt-get --reinstall install openssh-server
pa4080
  • 30,711
  • Will it require me to physically access the server ? If so, I can only do it tomorrow evening – 3rgo Jun 14 '17 at 21:51
  • Hi @3rgo, I think you do not need physical access. I just try it onto my VPS. Also onto my home Ubuntu Server, while I was logged in via SSH. Even the connection wasn't interrupted. cp command is just in case, usually reinstallation process doesn't touch configuration files. – pa4080 Jun 14 '17 at 22:04
  • Hi! I tried reinstalling, but nothing changed, I still have the same problem... – 3rgo Jun 15 '17 at 16:29
0

Probably you just answered Y when apt detected differences between your sshd_config and package's one. It asks if you want to install package mantainer's version or keep yours.

Marco
  • 186
  • 1
    I don't remember being asked such a thing, but assuming it's the case, what can I do to fix it ? – 3rgo Jun 23 '17 at 07:07