Simplify your life with SSH

 

Background and motivation

I have one Ubuntu Linux server (I will call it as remote server in the following of this post) in my office, which is behind a NAT router and restrictive rewall. Now I want to SSH to the remote server while I'm away from my office. How should I do?

Part 1: Reverse SSH Tunnel

关于 SSH port forwaring 概念的介绍。这个帖子: http://www.linux-france.org/prj/edu/archinet/systeme/ch13s04.html 以及 https://www.ssh.com/ssh/tunneling/example讲的还挺好的。

SSH portforwarding will certainly be an option. However, port forwarding can become tricky if you are dealing with multiple nested NAT environment. Besides, it can be interfered with under various ISP-specific conditions, such as restrictive firewalls which block forwarded ports, or carrier-grade NAT which shares IPv4 addresses among users.

One alternative to SSH port forwarding is reverse SSH tunneling. The concept of reverse SSH tunneling is simple. For this, you will need another host (so-called "relay host") outside your restrictive office network, which you can connect to via SSH from where you are. You could set up a relay host using a VPS instance with a public IP address. What you do then is to set up a persistent SSH tunnel from the remote server in your office network to the public relay host. With that, you can connect "back" to the remote server from the relay host (which is why it's called a "reverse" tunnel). As long as the relay host is reachable to you, you can connect to your remote server wherever you are, or however restrictive your NAT or firewall is in your office network.

1	ssh -fN -R <your_selected_port>:localhost:22 <relayserver_username>@relayserver

Some explanations about the above command: * term <your_selected_port> is any arbitrary unoccupied port number (e.g. 10222) that you can choose on the relay server(for exampler, your VPS). Similarly, term <relayserver_username> should be replaced by your user name on the relay server; * with the option combination fN, SSH will go right into the background once you successfully authenticate with an SSH server (in this case, relay server). This option is useful when you do not want to execute any command on a remote SSH server, and just want to forward ports; * option R, along side with string <your_selected_port>:localhost:22 defines a reverse SSH tunnel. It forwards traffic received by port <your_selected_port> of relay server to port 22 of homeserver.

Now I will log into my VPS with root user and check the opening port on my VPS:

1 2 3

<user_name>@<your_machine_name>:~# netstat -autn | grep 22222 tcp 0 0 127.0.0.1:22222 0.0.0.0:* LISTEN tcp6 0 0 ::1:22222 :::* LISTEN

Now that you can ssh into your remote server with the following command:

1	root@vps639023:~# ssh -p <your_selected_port> <username_at_remote_server>@localhost

, where <your_selected_port> should be replaced by port number such as 22222.

Dynamic port forwarding

可以用来翻墙 但是网速。。。很慢。。。

Special case: outgoing port 22 (i.e. for SSH) is filtered by firewall

During the establishment of reverse SSH tunnel (i.e. from remote server to relay server), it is very possible that the remote server has no outbound SSH access to a machine on the public Internet (in my case: my VPS instance), because port 22 is blokced by restrictive firewall. With port 22 open allows malicious attackers establishing reverse tunnel and bypassing completely bypassing the firewall. Thus, in practice, some enterprise may block the outbound port 22 for the sake of network security. In this case, how we could establish reverse SSH tunnel?

The answer is yes if: * the outbound port 443 (i.e. for HTTPS) is not filtered by the firewall in the network of remote server; * web server (Apache2 or Nginx), SSH server and SSLH1 are installed on the relay server. Please refer to~ for more details.

Compilation and Installation of SSLH

1. Download source code
2. Install pre-requisites: take Debian-based system (e.g. Ubuntu 16.04) as example libwrap0-dev libsystemd-dev libcap-dev libbsd-dev
3. Compilation and Installation

Provided that the operations on the relay server side are done, we now log into the home server can establish a reverse SSH tunnel with relay server with the following command:

1	ssh -fN -R 10022:localhost:22 root@<your>.<own>.<ip>.<address> -p 443

Some explanaition about above command: * with option p, the destination port of SSH is set as 443, instead of 22, which is the default port for SSH. In other words, remote server will establish ssh connection with destination port 443, which won't be blocked by firewall.

Part 3: execute command on remote server over SSH connection

在实际使用Linux的过程中，我们经常会碰到这样的场景：登录到远程服务器上，执行某个命令，比如查看下某个文件夹下的目录，然后再退出。常规的思路是： 1. Run the command "ssh username@host" to log in to the system 2. At the command prompt, run "top" to view process activity on the remote system 3. Exit top and be dropped to the remote command line 4. Type "Exit" to close the command

SSH已经考虑到了用户的这种需求，可以一条命令完成上述四步：ssh -t username@host 'top'。命令中的选项-t, 作用是分配一个为terminal界面给当前命令，允许用户进行一些交互。比如当你需要执行需要sudo的命令时，-t保证了当需要输入管理员密码的时候，可以通过当前的terminal输入密码。

Part 2: Enable SSH connection without password

There are some issues with this password based authentication, mainly, it allows brute-force password guessing. SSH provides better authentication process: SSH Public key access, which allows to password-less log in.

To this end, first you have to generate a public and private key pair in your relay server, namely your VPS

1	root@vps639023:~# ssh-keygen -t rsa -P ''

Then, you have to transfer the newly generated public key to your target server:

1	root@vps639023:~# ssh-copy-id -i .ssh/id_rsa.pub <user_at_remote_server>@localhost -p 22222

Note that command ssh-copy-id relies on scp that uses port 22 as default port. Here we explicitly specify here the used port is 22222. Without option i, ssh-copy-id will copy whatever keys it finds to the remote (might be multiple)!

一些疑问：

1. 产生公钥/私钥对的时候，会被询问是否设置passphrase, 这个和password有何异同(比如会比password更长)? 什么时候需要设置passphrase? 设置了passphrase是否还是真正的password-less登录？
2. SSH agent 又是啥？和SSH 客户端又有何不同呢？SSH agent forwarding 是什么？ SSH agent 第一个用途是用来解决 unlocking private key with a secret passphrase upon each connection 这个问题的。即，采用被passphrase保护的私钥登录remote server时候(ssh -i path/to/private-key remote-server)时, 虽然不再需要输入登录远程服务器所需要的密码，但是我们需要输入passphrase才能获得对本地私钥的使用权！这其实不就是另一种形式的密码么，而且每次登录都需要输入更长的字符！根本不是我们想要的password-less登录。ssh-agent可以解决这个问题。执行ssh-add path/to/your/private-key, 之后在执行ssh -i path/to/private-key remote-server。第一次还需要输入passphrase，之后再登录remoter-server就不再需要输入passphrase了，因为指定的私钥就会被存储在ssh-agent中(私钥应该是存储在内存中了)。

SSH agent forwarding can be used to make deploying to a server simple. It allows you to use your local SSH keys instead of leaving keys (without passphrases!) sitting on your server.

SSH agent forwarding特性有两种方式启用：1）命令行模式下的参数-A, 2）配置文件(如~/.ssh/config)中在相应的Host下，加入directive: ForwardAgent yes。 3.

Part 3: How make a tunnel persistent 

By persistent I mean, that it is made sure the tunnel will always run. For example, once your ssh connection times out (By server-side timeout), your tunnel should be re-established automatically.

Many NAT firewalls time out idle sessions after a certain period of time to keep their trunks clean. Sometimes the interval between session drops is 24 hours, but on many commodity firewalls, connections are killed after as little as 300 seconds.

3.1 SSH command option ServerAliveInterval and ServerAliveCountMax

To avoid having your SSH sessions become unresponsive after e.g. 5 minutes, do the following (On Linux machine):

To enable the keep alive system-wide (root access required), edit /etc/ssh/ssh_config; to set the settings for just your user, edit ~/.ssh/config (create the file if it doesn’t exist).

Insert the following snipet of code:

1 2 3

Host * ServerAliveInterval 300 ServerAliveCountMax 2

Alternatively, you can also make your OpenSSH server keep alive all connections with clients by adding the following to /etc/ssh/sshd_config:

1 2 3

Host * ClientAliveInterval 300 ClientAliveCountMax 2

These settings will make the SSH client or server send a null packet to the other side every 300 seconds (5 minutes), and give up if it doesn’t receive any response after 2 tries, at which point the connection is likely to have been discarded anyway.

修改完openssh-server的配置之后，可以采取两种方式使配置生效： sudo service ssh restart # 重启ssh server sudo service ssh reload # 重载ssh server 相关配置

如果你已经安装了openssh-server的话(没有的话,要安装),因为Ubuntu已经使用Upstart来进行管理/etc/init.d里面的任务. 所以/etc/init.d/sshd start已经无效了. 注意: 应该用sudo service ssh start(注意: 是ssh,不是sshd)启动sshd服务,之后可以通过进程查看sshd进程已在运行.

describes how to use different terminal color schema when establishing SSH connection.

3.2 AutoSSH

Autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic.

AutoSSH basic usage:

1	usage: autossh [-V] [-M monitor_port[:echo_port]] [-f] [SSH_OPTIONS]

The important part to remember is that -f (run in background) is not passed to the ssh command, but handled by autossh itself. Apart from that you can then use it just like you would use ssh to create any forward or reverse tunnels.

Note 1: Before you use autossh, make sure the connection works as expected by trying it with ssh first.

Note 2: Make sure you use public/private key authentification instead of password-based authentification when you use -f. This is required for ssh as well as for autossh, simply because in a background run a passphrase cannot be entered interactively.

With option -M AutoSSH will continuously send data back and forth through the pair of monitoring ports in order to keep track of an established connection. If no data is going through anymore, it will restart the connection. The specified monitoring and the port directly above (+1) must be free. The first one is used to send data and the one above to receive data on. For example, if you specify 11111 as monitoring port, then 11112 will be automatically used for receving data. You shoul ensure that both ports are unoccupied.

Unfortunately, this is not too handy, as it must be made sure both ports (the specified one and the one directly above) are free (not used). So in order to overcome this problem, there is a better solution:

Options ServerAliveInterval and ServerAliveCountMax of ssh command. They cause the SSH client to send traffic through the encrypted link to the server. This will keep the connection alive when there is no other activity and also when it does not receive any alive data, it will tell AutoSSH that the connection is broken and AutoSSH will then restart the connection.

The AutoSSH man page also recommends the second solution: >-M [:echo_port], >… >In many ways this [ServerAliveInterval and ServerAliveCountMax options] may be a better solution than the monitoring port.

You can disable the built-in AutoSSH monitoring port by giving it a value of 0:

1	autossh -M 0

Additionally you will also have to specify values for ServerAliveInterval and ServerAliveCountMax:

1	autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3"

Option	Description
ServerAliveInterval	ServerAliveInterval: number of seconds that the client will wait before sending a null packet to the server (to keep the connection alive). Default: 30
ServerAliveCountMax	Sets the number of server alive messages which may be sent without ssh receiving any messages back from the server. If this threshold is reached while server alive messages are being sent, ssh will disconnect from the server, terminating the session. Default: 3

Part 4: Enable the tunnel at system boot

To make sure that even the remote server is restarted, we still have this reverse SSH tunnel, we configure that this server executes this command at boot time.

In recent years, Linux distributions have increasingly transitioned from other init systems to systemd. The systemd suite of tools provides a fast and flexible init model for managing an entire machine from boot onwards. Refer to 3 for a quick tutorial for systemd.

4.1 Under Ubuntu system

Modify /etc/rc.local

We are going to edit the /etc/rc.local file. This script normally does nothing, but gets executed at boot. If you make any errors in this script, your machine might not boot so make sure to do this correctly.

Add the following instruction with your preferred editor into /etc/rc.local % credit:https://tex.stackexchange.com/questions/116534/lstlisting-line-wrapping

1	autossh -M 10984 -N -f -o "PubkeyAuthentication=yes" -o "PasswordAuthentication=no" -i /path/to/.ssh/id_rsa -R 6666:localhost:22 remy@middleman -p 2222 &

We have three new things in this command:

• -N: Do not execute a command on the middleman machine
• -f: drop in the background
• &: Execute this command but do not wait for output or an exit code. If this is not added, your machine might hang at boot.

Save the file, and as make it executable if necessary:

1	sudo chmod +x /etc/rc.local

Use cron command

TBD

Create systemd service

If you want a permanent SSH tunnel already created during boot time, you will (nowadays) have to create a systemd service and enable it. There is however an important thing to note about systemd and AutoSSH: -f (background usage) already implies AUTOSSH_GATETIME=0, however -f is not supported by systemd.

http://www.freedesktop.org/software/systemd/man/systemd.service.html […] running programs in the background using “&”, and other elements of shell syntax are not supported.

So in the case of systemd we need to make use of AUTOSSH_GATETIME. Let’s look at a very basic service:

1 2 3 4 5 6 7 8 9 10 11 12

# The following command should be run with root priviledge $ vim /etc/systemd/system/revtun2vps.service [Unit] Description=AutoSSH managed reverse SSH tunnel service to bind local port 22 with port 22222 on VPS server After=network.target [Service] Environment="AUTOSSH_GATETIME=0" ExecStart=/usr/bin/autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -N -R 22222:localhost:22 <user_name>@<vps.host.name> -p 443 [Install] WantedBy=multi-user.target

One can refer to 4 for more details about the directives such as after=, WantedBy.

Note 1: 以上systemd service将会以root身份启动autossh进程，如果root身份从未跟VPS建立过SSH connection的话，通过systemctl status revtun2vps.service, 将会看到SSH connection实际上并没有被创建，由于host key verification failed的错误。最后我的解决办法是，sudo su 切换为root，然后ssh-keygen创建root专属的公私钥，最后再ssh-copy-id -i .ssh/id_rsa.pub root@vps.ip.addr -p 443使得root可以免密登录VPS. 之后再启动revtun2vps service，将会顺利创建reverse tunnel了。

虽然我之前一直都以qsong的身份在和VPS建立SSH连接, 但是 VPS 的host key确实被保存在/home/qsong/.ssh/known_hosts文件中的，当我第一次以root身份去发起连接的时候，root用户的工作目录(即/root)下并没有known_hosts这么一个文件！

问题：难道 systemd service 启动的 autossh 难道不需要指明运行在后台？

Tell systemd that we have added some stuff:

1	systemctl daemon-reload

Start the service:

1	systemctl start revtun2vps.service

Enable during boot time:

1 2 3

systemctl enable revtun2vps.service root@vps273731:~# systemctl enable revtun2vps.service Created symlink from /etc/systemd/system/multi-user.target.wants/revtun2vps.service to /etc/systemd/system/revtun2vps.service.

Once the service is correctly started:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

qsong-P650SE# systemctl status revtun2vps.service ● revtun2vps.service - AutoSSH managed reverse SSH tunnel creation service to bind local 22 with port 22222 on remote VPS server Loaded: loaded (/etc/systemd/system/revtun2vps.service; enabled; vendor preset: enabled) Active: active (running) since ven. 2019-05-17 18:23:37 CEST; 11s ago Main PID: 21307 (autossh) Tasks: 2 Memory: 1.3M CPU: 19ms CGroup: /system.slice/revtun2vps.service ├─21307 /usr/lib/autossh/autossh -M 0 -o ServerAliveInterval 30 -o ServerAliveCountMax 3 -N -R 22222:localhost:22 root@xx.xx.xx.xx -p 443 └─21319 /usr/bin/ssh -o ServerAliveInterval 30 -o ServerAliveCountMax 3 -N -R 22222:localhost:22 -p 443 root@xx.xx.xxx.xx mai 17 18:23:37 qsong-P650SE systemd[1]: Started AutoSSH managed reverse SSH tunnel creation service to bind local 22 with port 22222 on remote VPS server. mai 17 18:23:37 qsong-P650SE autossh[21307]: port set to 0, monitoring disabled mai 17 18:23:37 qsong-P650SE autossh[21307]: starting ssh (count 1) mai 17 18:23:37 qsong-P650SE autossh[21307]: ssh child pid is 21319

4.2 Under Mac OSX

On Mac, the alternative to service and systemctl is launchctl. Mac OSX relies on launchctl to manage the services that need to be started at boot time. Unfortunately, Systemd, which systemctl is a part of, is a service manager designed specifically for Linux and uses interfaces and constructs specific to it such as cgroups. There is no port of it to Mac OS.

launchd, a unified service-management framework, starts, stops and manages daemons, applications, processes, and scripts in mac OS. launchd has two main tasks. The first is to boot the system, and the second is to load and maintain services.

Part 5: Simplify your life With an SSH Config File 

Create a SSH config file under path ~/.ssh/config. This is the per-user configuration file.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

# My personal VPS serves as a broker for connection from my Mac to my Ubnuntu server Host my_vps HostName <your>.<own>.<ip>.<address> User root Port 443 # my remote Ubuntu server Host ubuntu_server # example: ProxyCommand ssh -q my_vps nc -qo hostname.or.IP.address.internal.machine 22 # change hostname.or.IP.address.internal.machine and port 22 to machine you like to reach # if you would from the `my_vps` machine ProxyCommand ssh -q my_vps nc -q0 localhost 22222 # my vagrant vm running in my ubuntu_server # the following ssh command options are obtained from `ps aux | grep ssh` on my ubuntu_server. Host sfc103-vm ProxyJump qsong@ubuntu_server HostName 127.0.0.1 User vagrant Port 2222 LogLevel FATAL Compression yes DSAAuthentication yes IdentitiesOnly yes StrictHostKeyChecking no UserKnownHostsFile /dev/null IdentityFile ~/Documents/sfc-patch/sfc-demo/sfc103/.vagrant/machines/odl/virtualbox/private_key

Also note that now you have a single SSH host target name for it, you can use this in other applications as well. E.g.: SCP to copy files.

1	scp somefile user@internalmachine:~/

Copy your public key from A directly to C:

1	ssh-copy-id -i .ssh/id_rsa.pub qsong@ubuntu_server

This allows one to SSH from A to C without password.

On my macbook pro, I have openssh 7.9p1 installed. From OpenSSH 7.3, ProxyJump is supported in ssh_config. I find that even though my remote ubuntu_server (having OpenSSH 7.2) have no support for such an option, this does not bother the using of ProxyJump on my macbook pro.

Part 6: SSH connection with GUI

This post gives a very detailed tutorial about how to install Xfce and VNC server on Ubuntu 16.04.

One thing we need to indicate is: I have to establish one reverse SSH tunnel with my VPS.

On the Ubunutu server, add the following codes into file ~/.ssh/config:

1 2 3 4 5

Host vnc_tunnel_vps HostName <vps.ip.address.or.name> User <your-user-name> Port 443 RemoteForward 59000 127.0.0.1:5901

Port 59000 at VPS server will be on state of listening. Any traffic to this port on VPS will be forwarded to port 5901 on the remote Ubuntu server.

Thus, On the macbook pro, we have to also add some local port forwarding configuration in file ~/.ssh/config:

1 2 3 4 5 6 7 8

# Establish one tunnel between this machine and my vps Host tunnel_vps HostName <vps.ip.address.or.name> LocalForward 33333 127.0.0.1:22222 # Establish one tunnel to bind local 59000 with remote 59000 on my vps LocalForward 59000 127.0.0.1:59000 User <your-user-name> Port 443

Note that in the above command, SSH will create two local port forwarding.

If you want to access to the VNC server running on Ubuntu server, you can either install VNC client or use Screen sharing program integrated into Mac OSX.

Open Finder application, click menu Go and further click Connect to Server.... On the prompted window, type the following command:

1	vnc://127.0.0.1:59000

You will be required to type password, which is the one you set when installing VNC server.

Part 7: Mount remote directory on a remote server with SSHFS

If you use SSHFS on MacOS and the end point stops responding or crashes, you need to do new SSHFS connection. You need to either restart your mac or umount the old connection first. I use sshfs to access data on my vmware ( Ubuntu ) – if that restarts, or crashes – I end up with the following error message:

1	mount_osxfuse: mount point /xx/yy/zz is itself on a OSXFUSE volume

This post gives the cause and solution6. Simply speaking, the remote directory is not correctly unmounted. We need to find its mount point and unmount it another time.

1	sshfs -o defer_permissions root@hostname.or.ip.address:/path/to/your/folder /mnt/path/to/your/mount_point

• -o defer\_permissions: without this option, we will encounter error permission defined if we try to write something into the mounted directory.

Problem description

Now I have three machines: my working laptop (Mac OSX), my remote server (Ubuntu 16.04) and VPS server. I want to mount a directory in my remote Ubuntu server on my working laptop.

FUSE for OS X

The first step to using SSHFS is to install FUSE for OS X (There’s a prior version called MacFUSE, you don’t want that). Once installed, reboot your Mac. By itself, FUSE for OS X doesn’t do much, it provides the layer for userspace filesystems, but no filesystems itself, so download (found on the FfOSX page) and install SSHFS as well.

key idea

All one has to do is bind the one local port (e.g. 33333) of my VPS to a port (e.g. 22222) on remote server. The port of remote server is the one used to establish reverse SSH tunnel.

concret steps

1. Bind one local port with port in relay server

One one terminal and type into the following command:

1	$ ssh -L 33333:localhost:22222 <user_on_relay_server>@<relay_server_addr>

With above command, one has to keep this terminal window open. If you want to keep this binding running in the background:

1	$ ssh -NfL 33333:localhost:22222 <user_on_relay_server>@<relay_server_addr>

Remember to adapt <user_on_relay_server> and <relay_server_addr> for your own case. The -N prevents SSH from executing a command on the remote system. This is useful when just forwarding ports.

If you have a slow network, you may wish to include the -C option in both the ssh and sshfs commands as well to enable compression. Don't bother if you are on a fast network as in the time taken to compress the file and send the smaller version, you can instead just get the uncompressed version quicker.

2.create a local mount point and mount the remote file system

Then, mounting the directory my_remote_server:/home/path/to/folder to a local foler for example named pcB on my_working_laptop. First create a mount point:

1	$ mkdir /mnt/pcB

Where you mount it matters as that’s where the drive icon will appear. In the above example, I create the local mount point in the Desktop folder, which makes the drive appear on the Desktop when mounted. The downside of that, is then when the drive is not mounted, you have an empty folder that appear on the Desktop.

Another popular approach is to mount it in /Volumes, where OS X traditionally mounts drives.

1	$ mkdir /Volumes/server

Then mount the folder with the following command:

1	sshfs -o defer_permissions -o port=33333 -o volname=sfc103demo <user_at_my_remote_server>@localhost:/home/path/to/folder /mnt/pcB

• -o defer_permissions: this option is very important. Without this option, we will encounter error permission defined if we try to write something into the mounted directory.
• -o volname=Server sets the name that the volume will have in the Finder. Otherwise it’s called something like OSXFUSE Volume 0 (sshfs)

By default, the file system is only visible to the person that mounted it. To allow other users to see it you can add -o allow_other to the command line. This can be a little dangerous when combined with -o defer_permissions as everyone will have the permissions of the account that was used for the SSH connection. Use this option with care.

3.unmount the file system

1	umount /mnt/pcB

Remember to replace /mnt/pcB. In Mac OSX, one can also eject the mounted file system in the Finder. (key combination cmd+shift+G to open any folder within Finder window)。

If you encounter the following messages:

1	umount(/mnt/pcB): Resource busy -- try 'diskutil unmount'

or

1	$ fusermount -u /mnt/pcB/

You may use:

1	diskutil unmount /mnt/pcB

You can refer to 7 for more details.

4.Killing the tunnel 8

1 2	$ ps ax | grep "ssh_command" | awk '{print $1}' | \ xargs -i kill {} 2>/dev/null

Firstly, ps is used to find all running processes. We then pipe the output to grep which looks for "ssh_command" (which should be replaced with whichever command you used to establish the tunnel). Then we use awk to return only the PID numbers for the relevant processes and these are finally piped to kill to terminate the process. All errors are redirected to /dev/null. For a simple explanation of output redirection, see this tutorial.

Part 8: Git over SSH 9

Part 9: Rsync through an SSH tunnel

Ref: http://toddharris.net/blog/2005/10/23/rsyncing-through-an-ssh-tunnel/

curl, the HTTP client used by Git, supports SOCKS proxies as provided by ssh -D dynamic tunnel.

1. Set up a dynamic (SOCKS) tunnel with:

   1	ssh <username>@serverA -D 1080 -N

2. Configure Git to use it as a proxy: Globally (or per-repository)

   1	git config [--global] http.proxy socks5://localhost:1080

   For a single command:

   1	git -c http.proxy=socks5://localhost:1080 clone http://serverB/repo.git

Tcpdump over SSH tunnel and read capture with local wireshark

It’s often more useful to capture packets using tcpdump rather than wireshark. For example, you might want to do a remote capture and either don’t have GUI access or don’t have Wireshark installed on the remote machine.

1	tcpdump -i <interface-name> -s 65535 -w <some-file>

Imagine now you want to capture traffic on remote server and use local wirehshark to read captured traffic.

1	ssh <remote_server> "sudo tcpdump -s0 -w - 'port 8080'" | wireshark -k -i -

tcpdump requires priveldged right to be launched. To avoid inputing password, it is better enable password-less sudo

Open the /etc/sudoers file (as root, of course!) by running:

1	sudo visudo

Note that you should never edit /etc/sudoers with a regular text editor, such as Vim or nano, because they do not validate the syntax like the visudo editor!

1	username ALL=(ALL) NOPASSWD:ALL

Replace username with your account username, of course. Save the file and exit. If you have any sort of syntax problem, visudo will warn you and you can abort the change or open the file for editing again.

It is important to add this line at the end of the file, so that the other permissions do not override this directive, since they are processed in order.

Use SSH to create VPN

Refer to this tutorial: https://www.kancloud.cn/kancloud/unix-toolbox/50744

Rsync over ssh

https://gist.github.com/KartikTalwar/4393116

c arcfour: use the weakest but fastest SSH encryption. Must specify "Ciphers arcfour" in sshd_config on destination. 自从 OpenSSH 7.6(?) 之后, OpenSSH 不再支持这么一个Cipher了。https://www.reddit.com/r/archlinux/comments/9wf2nd/ssh_no_arcfour_cipher/ 这个帖子里说，时代在进步，其实有其他的 Cipher 不比 arcfour 慢，但同时还更安全，干嘛不用呢？比如 chacha20-poly1305@openssh.com

Rsync 在大量小文件的情况下，速度要比scp快的多

As a test, you can change into /tmp directory and type as follows:

1	rsync -aHAXxv --numeric-ids --delete --progress -e "ssh -T -c chacha20-poly1305@openssh.com -o Compression=no -x" ubuntu_server:~/Documents/sfc-patch .

关于上述命令的解释:

• a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.
• H: preserves hard-links
• A: preserves ACLs
• X: preserves extended attributes
• x: don't cross file-system boundaries
• --delete: delete extraneous files from destination dirs (differential clean-up during sync)
• --progress: show progress during transfer
• --numeric-ds: don't map uid/gid values by user/group name

离开法国之前，我打算把在里尔的ubuntu_server上所有document下的数据全部拷贝至我的移动硬盘 rsync --exclude-from='rsync_exclude_file.txt' -aHAXxv --numeric-ids --delete --progress -e 'ssh -T -c chacha20-poly1305@openssh.com -o Compression=no -x' ubuntu_server:~/Documents/bake /Volumes/Memory/lille_ubunt_server/bake 注意: 原本bake这个文件夹并不存在，命令执行之后，这个文件夹会被自动创建

最近通过里尔ubuntu_server机器下载了一个新西兰大学网站的一个数据集，该数据集无法通过国内的网址下载获得(不知何故)，现在需要将下载好的文件从ubuntu_server下载到我的Macbook Pro上。但是，通过翻墙服务器建立的SSH链接很不稳定，经常性出现连接中断的问题。导致马上就要下载好的文件，还需要从新下载，很是不方便。经过搜索，发现rsync对于文件夹还是单个文件，都支持断点续传功能。只需要加上--partial参数即可。需要注意的是，对于单个大型文件，用--partial参数的优势非常明显，但是如果在拷贝大量文件（可能有很多小文件不需要进行文件内断点续传）的时候，就要考虑时间上的问题的了。不用-P应该可以节省不少时间。

-c aes128-ctr：选择一个较弱但较快的SSH加密。其他人指定arcfour，这将需要在目标主机上手动修改ssh_config。这并非总是可能的，并且这种加密对我来说效果很好。因为传输的是一个单个大文件，所以我发现采用压缩比不压缩的传输效果好。即compression=yes.

1 2 3

split -a 2 -d -b 10m bigfile.tar.gz bigfile.tar.gz #将压缩包切片 cat bigfile.tar.gz* > bigfile.tar.gz #合并压缩包切片 tar xvf bigfile.tar.gz #解压

对于一个单一的大文件来说, 参考链接:

1 2 3 4 5

for d in `ssh ubuntu_server 'find ~/Documents/comp_aukland -name "dataset.tar.gz[0-9][0-9]"'`; do #d=${d%$'\n'} # Remove a trailing newline. #echo "Start to fetch ${d}...\n" rsync -aHAXxv --numeric-ids --delete --progress --partial -e 'ssh -T -c aes128-ctr -o Compression=yes -x' ubuntu_server:"$d" Downloads/comp_aukland/ & done

ssh -t ubuntu_server 'find ~/Documents/comp_aukland -name "dataset.tar.gz[0-9][0-9]"' | parallel -j 4 rsync -aHAXxv --numeric-ids --delete --progress --partial -e "ssh -T -c aes128-ctr -o Compression=yes -x" ubuntu_server:{='uq'=} ~/Downloads/comp_aukland/

ssh ubuntu_server 'find ~/Documents/comp_aukland -name "dataset.tar.gz[0-9][0-9]"' 一旦加上-t,则文件末尾会莫名其妙地多了#015，导致无法发现真正的文件 /home/qsong/Documents/comp_aukland/dataset.tar.gz10

parallel通过添加选项v，可以帮助debug.

1 2 3

rsync -av --progress --partial -e ssh -T -c aes128-ctr -o Compression=yes -x ubuntu_server:'dataset.tar.gz00 ' Downloads/comp_aukland/ Unexpected remote arg: ubuntu_server:dataset.tar.gz00 rsync error: syntax or usage error (code 1) at main.c(1374) [sender=3.1.3]

不知道为啥文件名dataset.tar.gz00在parallel语句中被额外添加了单引号呢？？？

最终有效的命令是：

1	cat test-rsync.txt | parallel --will-cite -v -j 4 rsync -aHPAXxv --numeric-ids --delete --progress -e \"ssh -T -c aes128-ctr -o Compression=yes -x\" ubuntu_server:{='uq'=} ~/Downloads/comp_aukland/

注意： 1. find命令的name域不采用双引号，即-name dataset.tar.gz[0-9][0-9],则name后面的数据将不会按照正则表达式去解释，无法达到预期效果。 2. parallel里如果需要使用quote,使用转义 3. cat命令输出作为parallel输入的时候，{}作为替换字符串，默认是会加上single quote的，这样会导致rsync出现执行错误，需要使用{='uq'=}避免引入single quote

最终我发现一个很有意思的想象：把一个100M的大文件切割成10多个代销为10M的文件在进行传输，传输的速度是以前的4-10倍！！这一定跟rsync底层实现有关！！！可能是避免了不断来回计算两边的差异么？？？

Part X: Troubleshooting and tips

1. connection closed by remote host after one month without any activity

I haven't connected to my Ubuntu server in my office from my Macbook Pro at home since one month. In the meantime, both my VPS and my Ubuntu server have never been shut down. However, I surprisingly observe that the SSH server on my Ubuntu server does not work. When I log into my VPS and further try to ssh into my laptop server, the following error are prompted (in the terminal of my VPS):

1	ssh_exchange_identification: Connection closed by remote host

At the terminal of my laptop server at school, such an error message prompts up:

1	connect_to localhost port 22: failed.

I found that sshd daemon is not running at my Ubuntu server.

1 2	~ ps aux | grep sshd qsong 31271 0.0 0.0 15972 952 pts/2 S+ 00:40 0:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn sshd

Naturally, I tried to restart sshd service:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21

~ sudo service sshd restart [sudo] password for qsong: Job for ssh.service failed because the control process exited with error code. See "systemctl status ssh.service" and "journalctl -xe" for details. ~ sudo systemctl restart sshd Job for ssh.service failed because the control process exited with error code. See "systemctl status ssh.service" and "journalctl -xe" for details. ~ sudo systemctl status ssh.service ● ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit-hit) since mer. 2019-03-13 00:42:14 CET; 15s ago Process: 31500 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=255) Main PID: 12601 (code=exited, status=0/SUCCESS) mars 13 00:42:14 qsong-P650SE systemd[1]: Failed to start OpenBSD Secure Shell server. mars 13 00:42:14 qsong-P650SE systemd[1]: ssh.service: Unit entered failed state. mars 13 00:42:14 qsong-P650SE systemd[1]: ssh.service: Failed with result 'exit-code'. mars 13 00:42:14 qsong-P650SE systemd[1]: ssh.service: Service hold-off time over, scheduling restart. mars 13 00:42:14 qsong-P650SE systemd[1]: Stopped OpenBSD Secure Shell server. mars 13 00:42:14 qsong-P650SE systemd[1]: ssh.service: Start request repeated too quickly. mars 13 00:42:14 qsong-P650SE systemd[1]: Failed to start OpenBSD Secure Shell server. mars 13 00:42:14 qsong-P650SE systemd[1]: ssh.service: Unit entered failed state. mars 13 00:42:14 qsong-P650SE systemd[1]: ssh.service: Failed with result 'start-limit-hit'.

Finally, I solve this issue by reinstalling openssh-server again.

1 2	sudo apt-get purge openssh-server sudo apt-get install openssh-server

Now the sshd daemon is runing again.

1 2 3 4 5

~ps aux | grep sshd root 1390 0.0 0.0 65512 5404 ? Ss 00:46 0:00 /usr/sbin/sshd -D root 1510 0.0 0.0 96980 6964 ? Ss 00:47 0:00 sshd: qsong [priv] qsong 1576 0.0 0.0 96980 3332 ? S 00:47 0:00 sshd: qsong@pts/15 qsong 2085 0.0 0.0 15940 1020 pts/15 S+ 01:07 0:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn sshd

How to check if a certain port is open or used?

We can use lsof. The syntax is:

1 2 3

$ sudo lsof -i -P -n $ sudo lsof -i -P -n | grep LISTEN $ doas lsof -i -P -n | grep <A port Number>

在国内，SSH连接经常会断掉，某一次断掉之后的DEBUG信息(远程VPS的IP地址已经被隐藏)如下:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1 debug3: send packet: type 100 debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1 debug3: send packet: type 100 debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1 debug3: send packet: type 100 debug3: send packet: type 1 debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1) debug3: fd 1 is not O_NONBLOCK Connection to XX.XX.XX.XX closed by remote host. Connection to XX.XX.XX.XX closed. Transferred: sent 5912, received 8848 bytes, in 2888.1 seconds Bytes per second: sent 2.0, received 3.1 debug1: Exit status -1

Refereces:

---

1. RUTSCHLE YVES, SSLH, http://www.rutschle.net/tech/sslh/README.html

2. SSH TUNNELLING FOR FUN AND PROFIT: AUTOSSH, https://www.everythingcli.org/ssh-tunnelling-for-fun-and-profit-autossh/

3. Systemd Essentials: Working with Services, Units, and the Journal, https://www.digitalocean.com/community/tutorials/systemd-essentials-working-with-services-units-and-the-journal

4. Understanding Systemd Units and Unit Files, https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files

5. Simplify Your Life With an SSH Config File, https://nerderati.com/2011/03/17/simplify-your-life-with-an-ssh-config-file/

6. https://phpsolved.com/mount-point-xyz-osxfuse-volume/

7. FUSE and SSHFS on OS X, https://stuff-things.net/2015/05/20/fuse-and-sshfs-on-os-x/

8. Mounting file systems over two SSH hops, https://www.larkinweb.co.uk/computing/mounting_file_systems_over_two_ssh_hops.html

9. Tunneling git through ssh (to get past a firewall): port specification interperted as relative path?, https://superuser.com/questions/1117698/tunneling-git-through-ssh-to-get-past-a-firewall-port-specification-interpert