Total Pageviews

Saturday, 2 June 2018

侦测协议的工具-serve2d

Protocol detecting server.

serve2 GoDoc Build Status Go Report Card

A protocol detecting server library. For examples. look at the docs for the github.com/joushou/serve2/proto package.
serve2 accepts a connection, and runs it through the active Protocols, reading the smallest amount of data necessary to identify the protocol. Protocols do not need to be certain about how much data they need up front - they can ask for more as needed. A default Protocol can be used when the server fails to identify the protocol, or one can rely on the default behaviour that closes the socket.
Protocols can implement transports themselves, by returning a new net.Conn that will be fed through the protocol detection mechanism again, allowing for transparent support of transports. HTTPS (for HTTP/1.1) is implemented in this manner simply by adding the TLS transport independent of HTTP, leaving HTTP completely unaware of TLS. This also allows for things like SSH over TLS, or any other protocol for that matter.
serve2 comes with a set of Protocols ready for consumption, in the form of TLS, HTTP (consuming a http.Handler), ECHO and DISCARD. There is also a very convenient Proxy Protocol, which instead of managing the protocol itself simply checks for a predefined set of bytes, and when matched, dials a configured service, leaving the actual protocol up to someone else. This is useful for things like SSH.
Ensuring that the read bytes are fed back in is done by ProxyConn, a net.Conn-implementing type with a buffered Read.

Installation and documentation

To get:
  go get github.com/joushou/serve2
More info and examples at:

Why?

Well, I always kind of wanted to make something that could understand everything. I get those kinds of ideas occasionally. At one point I remembered that idea, and hving gotten caught by the Go fever, I thought I'd try it out in Go, which proved to be very suitable for the idea.

Uses

Apart from how cool it is to be able to serve everything on any port, it also allows flexibility when firewall rules are present. Ever had to run SSH on port 80 to get through a firewall? You know, annoying corporate networks, or maybe difficulties with annoying ISP's and your home server. Well, now you can still have a nice web server on port 80 and still serve SSH. Or maybe you had a packet inspecting firewall that didn't think that was a good idea? Use openssl's s_client to open a TLS transport to port 443, and SSH to that then!

Limitations

serve2 cannot detect /all/ protocols. It's not really possible to detect DISCARD or ECHO, for example, as the client does not send any recognizable array of bytes before expecting the server to reply. Instead, they require that you send "ECHO" or "DISCARD" as the first message you want echoed or discarded.
In order to be able to detect a protocol, the client will have to send something either immediately on connect, or at the latest before expecting the server to reply/do anything. What it sends must furthermore either be a static magic, or a dynamic message within such boundaries that a pattern can be validated programmatically. Static "magics" can be seen in the form of SSH that starts out by sending "SSH..." (... being a longer version string), and more dynamic ones involve TLS that does not send a magic, but always starts by sending a ClientHello, of which the first byte is 0x16 (to inform that this is a handshake), followed by major/minor version numbers and the handshake type (which for the first message is always ClientHello, 0x01). With this information, one can verify the message/handshake type and major/minor version number ranges, and establish with a decent probability that this is indeed a TLS ClientHello handshake.
While Protocols can ask for as much data as they can dream of, and can incrementally increase how much data they need (in case dynamic patterns also have dynamic lengths, for example), but it is suggested that the detection amount is kept as small as possible while also maintaining good probability of the protocol. SSH can be detected with extremely high probability by reading 3 bytes, which is a nice and small amount, and HTTP can be detected by looking at the HTTP method (and increasing the read amount if longer method names must be tested).

Performance

This depends heavily on both the quantity of registered handlers, and the individual handlers themselves. Every time more data is has to be read, serve2 will read the smallest amount that was requested by the remaining handlers (handlers that need len(requested) >= len(read)), resulting in a few unnecessary calls to handlers. As long as the handlers are simple checks, and it doesn't end up running through this iteration several hundred times, then the overhead should not be too bad. A future optimization to ensure that handlers are only called when len(read) >= len(requested), in order to lower the overhead.

What does the name mean?

Nothing. I called the toy project "serve", and when making a new version I had to use a new folder name, and so "serve2" was born.

from  https://github.com/kennylevinsen/serve2
------

serve2d Go Report Card

A protocol detecting server, based off the https://github.com/joushou/serve2 library. Scroll down for installation and usage info.
You don't like having to have to decide what port to use for a service? Maybe you're annoyed by a firewall that only allows traffic to port 80? Or even a packet inspecting one that only allows real TLS traffic on port 443, but you want to SSH through none the less?
Welcome to serve2, a protocol recognizing and stacking server/dispatcher.
serve2 allows you to serve multiple protocols on a single socket. Example handlers include proxy, HTTP, TLS (through which HTTPS is handled), ECHO and DISCARD. More can easily be added, as long as the protocol sends some data that can be recognized. The proxy handler allows you to redirect the connection to external services, such as OpenSSH or Nginx, in case you don't want or can't use a Go implementation. In most cases, proxy will be sufficient.
So, what does this mean? Well, it means that if you run serve2 for port 22, 80 and 443 (or all the ports, although I would suggest just having your firewall redirect things in that case, rather than having 65535 listening sockets), you could ask for HTTP(S) on port 22, SSH on port 80, and SSH over TLS (Meaning undetectable without a MITM attack) on port 443! You have to admit that it's kind of neat.
All bytes read by serve2 are of course fed into whatever ends up having to handle the protocol. For more details on the protocol matching, look at the serve2 library directly.

Installation

serve2d can either be installed from source (super simple with Go), or by downloading a prepackaged binary release. To download source (requires Go 1.4.2 or above):
go get github.com/joushou/serve2d
It can be run with:
cd $GOPATH/src/github.com/joushou/serve2d
go build
./serve2d example_conf.json
Arch Linux also has serve2d in the AUR: https://aur.archlinux.org/packages/serve2d/
Or, use the unstable version: https://aur.archlinux.org/packages/serve2d-git/

Limitations

serve2, and by extension, serve2d, can only detect protocols initiated by the client. That is, protocols where the client starts out by blindly sending a unique blob that can be used to identify the protocol.

What's up with the name?

I called the first toy version "serve", and needed to call the new directory in my development folder something else, so it became serve2. 'd' was added to this configurable front-end (daemon), to distinguish it from the library.

Usage

Due to potentially large amounts of parameters, serve2d consumes a json configuration file. The only parameter taken by serve2d is the name of this file. The format is as follows:
{
 // Listening address as given directly to net.Listen.
 "address": ":80",

 // Maximum read size for protocol detection before fallback or failure.
 // Defaults to 128.
 "maxRead": 10,

 // Logging to stdout.
 // Defaults to false.
 "logStdout": true,

 // Logging to file. Note that only one logging destination can be
 // enabled at a given time.
 // Defaults to empty string, meaning disabled.
 "logFile": "serve2d.log",

 // The enabled ProtocolHandlers.
 "protocols": [
  {
   // Name of the ProtocolHandler.
   "kind": "proxy",

   // Setting this flag to true means that this ProtocolHandler
   // will not be used in protocol detection, but instead be used
   // as a fallback in case of failed detection.
   // Defaults to false.
   "default": false,

   // Protocol-specific configuration.
   // Defaults to empty.
   "conf": {
    "magic": "SSH",
    "target": "localhost:22"
   }
  }
 ]
}

ProtocolHandlers

proxy

Simply dials another service to handle the protocol. Matches protocol using the user-defined string. If an array of strings is provided, then MultiProxy will be used instead of Proxy internally, which will try to match any of the provided strings, from shortest to longest, progressively requesting more data as necessary.
  • magic (string or []string): The bytes to look for in order to identify the protocol. Example: "SSH" or ["GET", "POST", "HEAD"]
  • target (string): The address as given directly to net.Dial to call the real service. Example: "localhost:22".

tls

Looks for a TLS1.0-1.3 ClientHello handshake, and feeds it into Go's TLS handling. The resulting net.Conn is fed back through the protocol detection, allowing for any other supported protocol to go over TLS. The certificates required can be generated with http://golang.org/src/crypto/tls/generate_cert.go.
  • cert (string): The certificate PEM file path to use for the server. Example: "cert.pem".
  • key (string): The key PEM file path to use for the server. Example: "key.pem".
  • protos ([]string): The protocols the TLS server will advertise support for in the handshake. Example: ["http/1.1", "ssh"]
As tls works as a transport, it can be used for anything, not just HTTP. tls + proxy handler for SSH would make it possible to do the following to grant you stealthy SSH over TLS, which would be indistinguishable from HTTPS traffic:
ssh -oProxyCommand="openssl s_client -connect %h:%p -tls1 -quiet" -p443 serve2dhost
Alternatively, using http://github.com/joushou/tunnel, one can do:
ssh -oProxyCommand="tunnel - tls:%h:%p" -p443 serve2dhost

tlsmatcher

Looks for already established TLS transports, allowing for checks against some of the connection properties, such as SNI.
  • negotiatedProtocols (string): The protocol to look for. Defaults to no check. Example: ["h2", "h2-14"].
  • negotiatedProtocolIsMutual (bool): Check if the protocol was one that was advertised or not.Defaults to no check. Example: true.
  • serverNames (string): The SNI server name to look for. Defaults to no check. Example: ["http2.golang.org"].
  • target (string): The target to dial upon a match. Example: "http2.golang.org:443".
  • dialTLS (bool): Whether or not to use TLS when dialing. This also copies servername and protocol. Example: true.

http

Simple file-server without directory listing (might change in the future). It guards against navigating out of the directory with some simple path magic. It identifies HTTP traffic by checking for possible methods ("GET", "PUT", "HEAD", "POST", "TRACE", "PATCH", "DELETE", "OPTIONS", "CONNECT"). Forwarding to another HTTP server can be done by just putting this list of methods in as magics for a "proxy" handler.
  • path (string): Path to serve from. Example: "/srv/http/"
  • defaultFile (string, optional): File to serve for /. Example: "index.html"
  • notFoundMsg (string, optional): 404 body. Example: "Not Found"
  • notFoundFile (string, optional): 404 file. Example: "404.html"

echo

A test protocol. Requires that the client starts out by sending "ECHO" (which will by echoed by itself, of course). No configuration.

discard

Same as DISCARD, start by sending "DISCARD". No configuration. If you feel silly, try DISCARD over TLS!

More info

For more details about this project, see the underlying library: http://github.com/joushou/serve2

from https://github.com/kennylevinsen/serve2d
Posted by at
Labels:

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)