系统诊断小技巧（10）：ssh/sshd的调试模式，sshsshd


网络应用的逻辑一般都是集中在两端。所以端到端的排查思路是解决问题的重要思路。排查ssh问题也是如此。

那么，怎么让ssh两端输出更多信息呢？

如何启动调试选项？

ssh和sshd都有配置选项和命令行选项来启动调试。两者的配置选项是一样的，都是 LogLevel。

关于配置项的具体用法，请参考ssh_config(5)和sshd_config(5)。我们下面只讨论命令行选项。

ssh的调试选项-v

给ssh命令行加上-v选项就能启动其调试模式。我们对比下有无-v选项的区别。

没有-v选项时，ssh严格遵循了“沉默是金”的训导

root@localhost:~$ ssh -i ~/.ssh/id_rsa root@101.201.155.113 exit
root@localhost:~$

有-v选项时ssh输出如下

root@localhost:~$ ssh -v -i ~/.ssh/id_rsa root@101.201.155.113 exit
OpenSSH_7.6p1 Ubuntu-4, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 101.201.155.113 [101.201.155.113] port 22.
debug1: Connection established.
debug1: identity file /home/demo/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/demo/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000
debug1: Authenticating to 101.201.155.113:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: umac-64@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-64@openssh.com compression: none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<3072<8192) sent
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-rsa SHA256:/CxKtMSkLDdqyBNlGyjmpKtOB5YiiHxQmN8veGnvlgA
debug1: Host '101.201.155.113' is known and matches the RSA host key.
debug1: Found key in /home/demo/.ssh/known_hosts:1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:90VKHu3ur7IZpuMwuS5NoJkyLs5vmmbb2dqNFrM4c64 /home/nerd/.ssh/id_aliyun_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 101.201.155.113 ([101.201.155.113]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: Sending env LC_MEASUREMENT = en_US.UTF-8
debug1: Sending env LC_PAPER = en_US.UTF-8
debug1: Sending env LC_MONETARY = en_US.UTF-8
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_NAME = en_US.UTF-8
debug1: Sending env LC_ADDRESS = en_US.UTF-8
debug1: Sending env LC_NUMERIC = en_US.UTF-8
debug1: Sending env LC_TELEPHONE = en_US.UTF-8
debug1: Sending env LC_IDENTIFICATION = en_US.UTF-8
debug1: Sending env LC_TIME = en_US.UTF-8
debug1: Sending command: exit
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 3640, received 2848 bytes, in 0.0 seconds
Bytes per second: sent 189533.1, received 148294.0
debug1: Exit status 0
root@localhost:~$

可见，使用-v选项时，ssh会给出丰富的交互过程信息。这对排查问题无疑大有帮助。

sshd的调试选项-d

给sshd加上-d选项既能启动其调试模式。对比下

没有-d选项的情形

ssh端的输出

ali-4c3275957b8b:~ $ ssh -i ~/.ssh/id_aliyun_rsa root@39.106.22.39 exit
root@localhost:~ $

sshd端的输出

root@iZ2zeecuf3v6yjfdp82unlZ:~#$(which sshd) -D
Could not load host key: /etc/ssh/ssh_host_ed25519_key
  C-c C-c
root@iZ2zeecuf3v6yjfdp82unlZ:~#

有-d选项的情形

ssh端

root@localhost:~ $ ssh -i ~/.ssh/id_aliyun_rsa root@xx.xx.xx.xx exit
debug1: SELinux support disabled
debug1: PAM: reinitializing credentials
debug1: permanently_set_uid: 0/0
Environment:
  LANG=en_US.UTF-8
  USER=root
  LOGNAME=root
  HOME=/root
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
  MAIL=/var/mail/root
  SHELL=/bin/bash
  SSH_CLIENT=xxx.xxx.xxx.xxx 1513 22
  SSH_CONNECTION=xxx.xxx.xxx.xxx 1513 xxx.xxx.xxx.xxx 22
root@localhost:~ $

sshd端的情形

root@localhost:~ $(which sshd) -D -d
debug1: sshd version OpenSSH_7.2, OpenSSL 1.0.2g  1 Mar 2016
debug1: private host key #0: ssh-rsa SHA256:ljPn4MtPW06wd2I2raiiMjgIc8BkuawPLC/XS90+x2w
debug1: private host key #1: ssh-dss SHA256:2uuXiQWHJf28G8fCSL14aj+eDweQyQzoUgWrm4l6dLE
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:JWEiiREibEzjiT6L9lxnvIHJ+CI7DhRQMHCR/boLF5o
debug1: key_load_private: No such file or directory
debug1: key_load_public: No such file or directory
Could not load host key: /etc/ssh/ssh_host_ed25519_key
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-D'
debug1: rexec_argv[2]='-d'
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug1: inetd sockets after dupping: 3, 3
Connection from 106.11.34.13 port 1513 on 172.17.165.12 port 22
debug1: Client protocol version 2.0; client software version OpenSSH_7.6
debug1: match: OpenSSH_7.6 pat OpenSSH* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: permanently_set_uid: 109/65534 [preauth]
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: curve25519-sha256@libssh.org [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user root service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "106.11.34.13"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user root service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: userauth_pubkey: test whether pkalg/pkblob are acceptable for RSA SHA256:90VKHu3ur7IZpuMwuS5NoJkyLs5vmmbb2dqNFrM4c64 [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /root/.ssh/authorized_keys, line 1 RSA SHA256:90VKHu3ur7IZpuMwuS5NoJkyLs5vmmbb2dqNFrM4c64
debug1: restore_uid: 0/0
Postponed publickey for root from 106.11.34.13 port 1513 ssh2 [preauth]
debug1: userauth-request for user root service ssh-connection method publickey [preauth]
debug1: attempt 2 failures 0 [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /root/.ssh/authorized_keys, line 1 RSA SHA256:90VKHu3ur7IZpuMwuS5NoJkyLs5vmmbb2dqNFrM4c64
debug1: restore_uid: 0/0
debug1: do_pam_account: called
Accepted publickey for root from 106.11.34.13 port 1513 ssh2: RSA SHA256:90VKHu3ur7IZpuMwuS5NoJkyLs5vmmbb2dqNFrM4c64
debug1: monitor_child_preauth: root has been authenticated by privileged process
debug1: monitor_read_log: child log fd closed
debug1: PAM: establishing credentials
debug1: rekey after 134217728 blocks
debug1: rekey after 134217728 blocks
debug1: ssh_packet_set_postauth: called
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 2097152 max 32768
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype no-more-sessions@openssh.com want_reply 0
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request exec reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req exec
Starting session: command for root from 106.11.34.13 port 1513 id 0
debug1: Received SIGCHLD.
debug1: session_by_pid: pid 8729
debug1: session_exit_message: session 0 channel 0 pid 8729
debug1: session_exit_message: release channel 0
debug1: session_by_channel: session 0 channel 0
debug1: session_close_by_channel: channel 0 child 0
Close session: user root from 106.11.34.13 port 1513 id 0
debug1: channel 0: free: server-session, nchannels 1
Received disconnect from 106.11.34.13 port 1513:11: disconnected by user
Disconnected from 106.11.34.13 port 1513
debug1: do_cleanup
debug1: PAM: cleanup
debug1: PAM: closing session
debug1: PAM: deleting credentials
debug1: audit_event: unhandled event 12
root@localhost:~#$

但是，sshd是以守护进程方式运行的，要修改其启动脚本增加选项吗？

如何在前端启动sshd？

的确可以通过修改启动脚本或者有关配置文件来增加-d选项。比如在CentOS上可以通过修改/etc/sysconfig/sshd来实现这个目的。

但是更好的方案是使用命令行参数的方式。这需要把sshd启动到前端。

上面的测试为了突出重点，我们故意忽略了一些准备和善后步骤。现在我们补充齐全

root@localhost:~ netstat -ltpn # 1. 检查sshd是否在运行
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      978/sshd
tcp        0      0 127.0.0.1:32000         0.0.0.0:*               LISTEN      918/java
root@localhost:~ service sshd stop # 2. 暂停sshd的运行
root@localhost:~ netstat -ltpn # 3. 确认sshd已经关闭
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:32000         0.0.0.0:*               LISTEN      918/java
root@localhost:~ $(which sshd) -D -d # 4. 前端启动sshd，并且启动调试模式
... ...

ssh/sshd两端排查的步骤

重要

关闭sshd前要考虑会不会把自己关在系统外。没有办法登录系统可不是个好消息。如果root登录系统只能使用ssh，则请做好准备前，务必不要关闭sshd。
linuxboy的实例可以在官网控制台通过VNC登录系统。使用VNC通道请提前准备好VNC密码和系统的密码登录方式，并且验证VNC通道工作符合预期。

步骤

sshd端的步骤

# 1. 检查sshd是否在运行
netstat -ltpn
# 2. 关闭sshd
service sshd stop
# 3. 检查sshd是否已经关闭
netstat -ltpn
# 4. 启动到前端并且启动调试模式
$(which sshd) -D -d # -D确保sshd在前端运行
# 5. 排查、测试、验证 ... ...
# 6. 重新启动sshd
service sshd start

ssh端的步骤

ssh -vvv login_name@sshd_host

探索愉快。

相关内容