HPN-SSH(High Performance SSH/SCP)是高性能SSH/SCP,是在原有openssh上打的一组补丁，可以去除openssh上的性能瓶颈。
http://www.psc.edu/index.php/hpn-ssh
打上对应版本的HPN-SSH补丁后，ssh额外的性能：
1, 实时调整buffer大小的能力，从而将文件传输速率提高。
2, 利用多核进行加密/解密，且允许使用不加密的scp数据copy (none cipher)，性能更优。
在仅仅使用hpn-ssh作为server或者仅仅作为client的情况下，性能一样可以得到提升。但是在LAN中传送数据，提速是徒劳的，因为传统的ssh已经够快了。
首先修改操作系统的TCP参数：
http://www.psc.edu/index.php/networking/641-tcp-tune
每种操作系统修改方法均有不同，这里以Debian6 64bit为例
/proc/sys/net/ipv4/tcp_rmem
查看ssh版本：
输出：
这里的OpenSSH版本是5.5p1
http://www.psc.edu/index.php/hpn-ssh
在这个页面找到对应的补丁包下载地址
OpenSSH_5.5p1对应的补丁包下载地址：
http://www.psc.edu/index.php/component/remository/HPN-SSH/OpenSSH-5.5-patches/OpenSSH-5.5-Kitchen-Sink-patch/
http://www.psc.edu/index.php/component/remository/func-startdown/708/
http://www.psc.edu/index.php/component/remository/func-download/708/chk,fbc8716cf110a440018071fee8f93f9c/no_html,1/
http://www.psc.edu/index.php/component/remository/func-download/708/chk,fbc8716cf110a440018071fee8f93f9c/no_html,1/openssh-5.5p1-hpnssh13v9.diff
apt-get install build-essential devscripts fakeroot
cd /tmp
wget http://www.psc.edu/index.php/component/remository/func-download/708/chk,fbc8716cf110a440018071fee8f93f9c/no_html,1/openssh-5.5p1-hpnssh13v9.diff.gz
zcat ./openssh-5.5p1-hpnssh13v9.diff.gz | patch
--------------------------------------------------------------------------------------------
email: hpn-ssh@psc.edu
Now, to repeat myself from the last few releases, the reason for the delay is that we no longer have funding for this project which means that the team (consisting of one person) has to focus on other projects. Lately we've had a couple significant new projects come up which have really been taking up almost all of my time. I'm sorry about that. However, everyday I hear about more people incorporating HPN-SSH into distributions and operating systems (FreeBSD 9.0 for example!). It's great to hear that people find it useful. Hopefully I'll be able to grab some of that energy and keep moving on this.
I'm still looking into setting up a space on sourceforge or google code with the idea of passing continued development and maintenance on to someone with more time. Once that is in place it will be easier to let people help out. Also, If you work for a funding agency or are able to provide grants for work like this I'd love to talk to you. It's important to note that the AES CTR threaded cipher found in the kitchen sink and threaded patch set still don't handle forking properly. If you use AES CTR ciphers and forking with these patch sets the application will hang. As always, we'd like to get this resolved but A) the person who wrote the threaded cipher no longer works with PSC and B) we no longer have a dedicated funding stream for the HPN-SSH patches. As such, we may never be able to fix them unless someone volunteers to do it (contact us at hpn-ssh@psc.edu) or we can get some funding (again, contact us at hpn-ssh@psc.edu).
We've been lucky enough to have volunteers translate this page into Belorussian, German, and Hindi. You can find these translations here:
The amount of improvement any specific user will see is dependent on a number of issues. Transfer rates cannot exceed the capacity of the network nor the throughput of the I/O subsystem including the disk and memory speed. The improvement will also be highly influenced by the capacity of the processor to perform the encryption and decryption. Less computational expensive ciphers will often provide better throughput than more complex ciphers.
The difference between the red and blue line is, essentially, wasted throughput potential along the path.
Clearly, the HPN patches significantly boost throughput performance. This enhancement is entirely from tuning the SSH buffer sizes.
Solaris Users: Some versions of Solaris use an older version of the patch and diff commands which are incompatible with this patch. Please make sure you are using a recent version of gnu patch.
This cipher mode introduces multi-threading into the OpenSSH application in order to allow it to make full use of CPU resources available on multi-core systems. As the canonical distribution of OpenSSH is unable to make use of more than one core, high performance transfers can be bottlenecked by the cryptographic overhead. HPN12 dealt with this by the introduction of None Cipher Switching. However, this technique is limited to those users who are willing to allow their data to be transferred without encipherment. It also was, by design, limited to bulk data transfers which further restricts its value to some users. The MT-AES-CTR mode will allow users, on multicore platforms, to attain throughput rates comparable or equal to unencrypted data transfers. In both lab and real world tests throughput at full GigE line rates, with full encryption, were commonly seen.
Obviously, the MT AES-CTR mode cipher breaks through the single core bottleneck.
MT-AES-CTR produces a cipherstream that is indistinguishable from the distributed Single Thread AES-CTR (ST-AES-CTR) mode cipher and is, to our knowledge, completely compatible with all other AES-CTR mode implementations. In other words, its completely backward compatible and will function in heterogenous connections with no problem. However, it is important to note MT-AES-CTR does impose additional overhead and may impose a performance penalty on single core machines. Additionally, the MT-AES-CTR mode cipher replaces the default ST-AES-CTR mode cipher. Be aware that if the client process is forked using the -f command line option the process will hang as the parent thread gets 'divorced' from the key generation threads. This issue will be resolved as soon as possible. Additionally, this does rely on posix threads so you need to be sure they are properly supported in the target OS.
The second major difference is in the way in which the HPN patch set will be made available. HPN13 will be provided as both a 'kitchen sink' and 'a la carte' distribution. The kitchen sink distribution will incorporate the dynamic window patches, none cipher switching, MT-AES-CTR mode ciphers, peak throughput display in SCP, and extended server side logging. The a la carte distribution will make each of these available as distinct patches which can be used independently of each other.
Recent problems with buffer_append_space in HPN-SSH. If you are experiencing disconnects due to a failure in buffer_append_space please let us know. We're currently tracking some problems with this and we're trying to gather more information to help resolve it. You may want to try using -oHPNBufferSize=16384 to restrict the growth of the buffer. Let us know if that helps.
This work was made possible in part by grants from Cisco Systems, Inc., The National Science Foundation,
from http://www.psc.edu/index.php/hpn-ssh
http://www.psc.edu/index.php/hpn-ssh
打上对应版本的HPN-SSH补丁后，ssh额外的性能：
1, 实时调整buffer大小的能力，从而将文件传输速率提高。
2, 利用多核进行加密/解密，且允许使用不加密的scp数据copy (none cipher)，性能更优。
在仅仅使用hpn-ssh作为server或者仅仅作为client的情况下，性能一样可以得到提升。但是在LAN中传送数据，提速是徒劳的，因为传统的ssh已经够快了。
首先修改操作系统的TCP参数：
http://www.psc.edu/index.php/networking/641-tcp-tune
每种操作系统修改方法均有不同，这里以Debian6 64bit为例
/proc/sys/net/ipv4/tcp_rmem
查看ssh版本：
ssh -v
OpenSSH_5.5p1 Debian-6+squeeze1, OpenSSL 0.9.8o 01 Jun 2010
usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
[-D [bind_address:]port] [-e escape_char] [-F configfile]
[-I pkcs11] [-i identity_file]
[-L [bind_address:]port:host:hostport]
[-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
[-R [bind_address:]port:host:hostport] [-S ctl_path]
[-W host:port] [-w local_tun[:remote_tun]]
[user@]hostname [command]
http://www.psc.edu/index.php/hpn-ssh
在这个页面找到对应的补丁包下载地址
OpenSSH_5.5p1对应的补丁包下载地址：
http://www.psc.edu/index.php/component/remository/HPN-SSH/OpenSSH-5.5-patches/OpenSSH-5.5-Kitchen-Sink-patch/
http://www.psc.edu/index.php/component/remository/func-startdown/708/
http://www.psc.edu/index.php/component/remository/func-download/708/chk,fbc8716cf110a440018071fee8f93f9c/no_html,1/
http://www.psc.edu/index.php/component/remository/func-download/708/chk,fbc8716cf110a440018071fee8f93f9c/no_html,1/openssh-5.5p1-hpnssh13v9.diff
apt-get install build-essential devscripts fakeroot
cd /tmp
wget http://www.psc.edu/index.php/component/remository/func-download/708/chk,fbc8716cf110a440018071fee8f93f9c/no_html,1/openssh-5.5p1-hpnssh13v9.diff.gz
zcat ./openssh-5.5p1-hpnssh13v9.diff.gz | patch
--------------------------------------------------------------------------------------------
High Performance SSH/SCP : HPN-SSH
(PI) Chris Rapier PSC, Michael Stevens CMU, Benjamin Bennett PSCemail: hpn-ssh@psc.edu
Are you using HPN-SSH?
Please let us know
Please let us know
On this page:
Notes
6.0 and 6.1 patches released. The same excuses for the delay apply for these patches as well. I'm sure you are as tired of hearing that as I am of writing it.Now, to repeat myself from the last few releases, the reason for the delay is that we no longer have funding for this project which means that the team (consisting of one person) has to focus on other projects. Lately we've had a couple significant new projects come up which have really been taking up almost all of my time. I'm sorry about that. However, everyday I hear about more people incorporating HPN-SSH into distributions and operating systems (FreeBSD 9.0 for example!). It's great to hear that people find it useful. Hopefully I'll be able to grab some of that energy and keep moving on this.
I'm still looking into setting up a space on sourceforge or google code with the idea of passing continued development and maintenance on to someone with more time. Once that is in place it will be easier to let people help out. Also, If you work for a funding agency or are able to provide grants for work like this I'd love to talk to you. It's important to note that the AES CTR threaded cipher found in the kitchen sink and threaded patch set still don't handle forking properly. If you use AES CTR ciphers and forking with these patch sets the application will hang. As always, we'd like to get this resolved but A) the person who wrote the threaded cipher no longer works with PSC and B) we no longer have a dedicated funding stream for the HPN-SSH patches. As such, we may never be able to fix them unless someone volunteers to do it (contact us at hpn-ssh@psc.edu) or we can get some funding (again, contact us at hpn-ssh@psc.edu).
We've been lucky enough to have volunteers translate this page into Belorussian, German, and Hindi. You can find these translations here:
- HPN-SSH in Belorussian provided by fatcow
- HPN-SSH in German provided by Andreas Beraz.
- HPN-SSH in Hindi provided by Ashish Jha.
- HPN-SSH in Russian provided by portablecomponentsforall
Abstract
SCP and the underlying SSH2 protocol implementation in OpenSSH is network performance limited by statically defined internal flow control buffers. These buffers often end up acting as a bottleneck for network throughput of SCP, especially on long and high bandwith network links. Modifying the ssh code to allow the buffers to be defined at run time eliminates this bottleneck. We have created a patch that will remove the bottlenecks in OpenSSH and is fully interoperable with other servers and clients. In addition HPN clients will be able to download faster from non HPN servers, and HPN servers will be able to receive uploads faster from non HPN clients. However, the host receiving the data must have a properly tuned TCP/IP stack. Please refer to this tuning page for more information.The amount of improvement any specific user will see is dependent on a number of issues. Transfer rates cannot exceed the capacity of the network nor the throughput of the I/O subsystem including the disk and memory speed. The improvement will also be highly influenced by the capacity of the processor to perform the encryption and decryption. Less computational expensive ciphers will often provide better throughput than more complex ciphers.
Performance Gap
With many high bandwidth connections, there is a performance gap between what SSH is capable of and what the network link has the capacity to do. The difference between these two numbers is the performance gap, or the underutilized portion of your network connection. This gap, in most situations, is the direct cause of undersized receive buffers in the SSH congestion control mechanism. The graph below shows the optimal receive buffer versus the effective SSH channel receive buffer for various round trip times along a 100Mbps pathThe difference between the red and blue line is, essentially, wasted throughput potential along the path.
Normal vs. HPN SCP Performance
The effect of raising the SSH buffer sizes can be seen in the following chart. The standard SSH throughput, represented by the red columns, closely matches the expected throughput for this path if the receive buffer was limited to 64KB. By increasing the size of the SSH channel receive buffers throughput, represented by the blue columns, improved by as much as 1000%. The variation now seen is due to the complexity of the cipher and the limits of the hard drive.Clearly, the HPN patches significantly boost throughput performance. This enhancement is entirely from tuning the SSH buffer sizes.
Patches
All patches should be applied to the OpenSSH source files using the 'patch' utility from the command line. Building SSH from source is actually quite easy and the recommended method. Some binary packages will be made available as a convenience but will not be officially supported.Solaris Users: Some versions of Solaris use an older version of the patch and diff commands which are incompatible with this patch. Please make sure you are using a recent version of gnu patch.
HPN-13
This is the 7th revision of the 13th major version of the HPN patch set. The HPN12 patch set remains available here. There are two fundamental differences between the HPN12 and HPN13 patch set. The most significant of these is the inclusion of the Multi-Threaded AES-CTR (MT-AES-CTR) mode cipher. A paper and presentation about this work are available.This cipher mode introduces multi-threading into the OpenSSH application in order to allow it to make full use of CPU resources available on multi-core systems. As the canonical distribution of OpenSSH is unable to make use of more than one core, high performance transfers can be bottlenecked by the cryptographic overhead. HPN12 dealt with this by the introduction of None Cipher Switching. However, this technique is limited to those users who are willing to allow their data to be transferred without encipherment. It also was, by design, limited to bulk data transfers which further restricts its value to some users. The MT-AES-CTR mode will allow users, on multicore platforms, to attain throughput rates comparable or equal to unencrypted data transfers. In both lab and real world tests throughput at full GigE line rates, with full encryption, were commonly seen.
Obviously, the MT AES-CTR mode cipher breaks through the single core bottleneck.
MT-AES-CTR produces a cipherstream that is indistinguishable from the distributed Single Thread AES-CTR (ST-AES-CTR) mode cipher and is, to our knowledge, completely compatible with all other AES-CTR mode implementations. In other words, its completely backward compatible and will function in heterogenous connections with no problem. However, it is important to note MT-AES-CTR does impose additional overhead and may impose a performance penalty on single core machines. Additionally, the MT-AES-CTR mode cipher replaces the default ST-AES-CTR mode cipher. Be aware that if the client process is forked using the -f command line option the process will hang as the parent thread gets 'divorced' from the key generation threads. This issue will be resolved as soon as possible. Additionally, this does rely on posix threads so you need to be sure they are properly supported in the target OS.
The second major difference is in the way in which the HPN patch set will be made available. HPN13 will be provided as both a 'kitchen sink' and 'a la carte' distribution. The kitchen sink distribution will incorporate the dynamic window patches, none cipher switching, MT-AES-CTR mode ciphers, peak throughput display in SCP, and extended server side logging. The a la carte distribution will make each of these available as distinct patches which can be used independently of each other.
HPN-13 Kitchen Sink
Note: This patch has been gziped. You must gunzip it before applying it.| OpenSSH Version | HPN-SSH Patch |
|---|---|
| OpenSSH 4.7p1 | OpenSSH-4.7p1-hpn13 v3 |
| OpenSSH 5.0p1 | OpenSSH-5.0p1-hpn13 v4 |
| OpenSSH 5.1p1 | OpenSSH-5.1p1-hpn13 v5 |
| OpenSSH 5.2p1 | OpenSSH-5.2p1-hpn13 v6 |
| OpenSSH 5.3p1 | OpenSSH-5.3p1-hpn13 v7 |
| OpenSSH 5.4p1 | OpenSSH-5.4p1-hpn13 v8 |
| OpenSSH 5.5p1 | OpenSSH-5.5p1-hpn13 v9 |
| OpenSSH 5.6p1 | OpenSSH-5.6p1-hpn13 v10 |
| OpenSSH 5.8p1 | OpenSSH-5.8p1-hpn13 v11 |
| OpenSSH 5.9p1 | OpenSSH-5.9p1-hpn13 v12 |
| OpenSSH 6.0p1 | OpenSSH-6.0p1-hpn13 v13 |
| OpenSSH 6.1p1 | OpenSSH-6.1p1-hpn13 v14 |
HPN-13 A la Carte
These are the a la carte patches and some of the version numbers may skew from time to time. For example, if the peak throughput patch doesn't need to be updated for various OpenSSH releases the patch number won't be updated. Not all of the patches are available just yet, as the NONE cipher switching still needs to be broken out from the HPN12 patch set.- Get the OpenSSH source code from OpenSSH.org.
- Untar OpenSSH source.
- cd into the OpenSSH source directory
- If gzipped type 'zcat pathtopatch/patchfile | patch'
Otherwise 'patch < pathtopatch/patchfile' - type 'configure && make'
- type 'make install'
Recent problems with buffer_append_space in HPN-SSH. If you are experiencing disconnects due to a failure in buffer_append_space please let us know. We're currently tracking some problems with this and we're trying to gather more information to help resolve it. You may want to try using -oHPNBufferSize=16384 to restrict the growth of the buffer. Let us know if that helps.
This work was made possible in part by grants from Cisco Systems, Inc., The National Science Foundation,
from http://www.psc.edu/index.php/hpn-ssh
