Total Pageviews

Thursday, 1 April 2021

内网穿透/端口转发/tcp代理工具:tcp_forward

 

为了解决的问题

image

  • 内网主机可以直接访问外网主机(snat、防火墙允许)
  • 外网主机不能直接连接内网主机(防火墙拦截)

一般的解决方法

  • 向网管申请,在防火墙上为内网主机某个端口开通dnat
  • 部署vpn服务,使用vpn接入内网

这些方法的局限

  • 如果需要开的端口较多网管不会允许
  • 如果需要变换端口需要再次申请,费时费力
  • vpn部署需要的工作量较大,且由于内网安全性,不是所有的公司都允许部署

tcp-forward的解决方案

image

  • 在外网和内网分别部署forward_server和forward_client
  • forward_client会主动向forward_server建立一个通道
  • 在forward_server监听一些端口,并配置这些端口到内网主机:端口的映射
  • 外网主机想访问内网主机时,只需要连接外网端口,则forward_server自动向forward_client发送创建连接请求,forward_client会向内网主机建立连接,形成一个通道,传输数据

典型案例

网络拓扑

  • 内网:

    • 主机A:192.168.10.100 部署forward_client
    • 主机B:192.168.10.5 centos server
  • 外网

    • 主机C:221.10.12.34 部署forward_server
  • 其他网络

    • 主机D:任意ip 只要能访问主机C

需求

  • 主机D通过ssh协议连接到主机B

做法

  • 在forward_server配置端口映射 1234=192.168.10.5:22
  • 启动forward_server和forward_client
  • 在主机D执行
ssh usr@221.10.12.23 -p 1234

使用说明

源码安装(server & client)

git clone https://github.com/cao19881125/tcp_forward.git
cd tcp_forward
pip install .

pip源安装

pip install tcp-forward

docker安装

源码编译安装

git clone https://github.com/cao19881125/tcp_forward.git
docker build -t tcp-forward:latest tcp_forward/docker

docker hub安装

docker pull cao19881125/tcp-forward:latest

配置

注意,docker启动由于把/etc/tcp-forward目录mount出来,导致没有配置文件,先执行下面的命令安装配置文件

pip install tcp-forward

server

  • 在/etc/tcp-forward/forward_server.cfg中配置forward_client连入的端口,日志级别,带宽(以实际带宽配置,单位是M)
[DEFAULT]
INNER_PORT=1111
OUTER_PORTS_FILE=/etc/tcp-forward/port_mapper.cfg
LOG_LEVEL=DEBUG
#minimal network BANDWIDTH in Mb
BANDWIDTH=100

api_paste_config=/etc/tcp-forward/api-paste.ini
user_file=/etc/tcp-forward/user_file
  • 在/etc/tcp-forward/port_mapper.cfg中配置外网端口到内网的映射,格式为:外网端口=内网ip:内网端口,以及端口对应的client tag,如下
[MAPPER]
1234=192.168.10.5:80
4444=192.168.105:22

[TAG]
1234=default
4444=default
  • 配置用户
# vim /etc/tcp-forward/user_file
[USER]
test1=123456
test2=654321
  • 格式为用户名=密码
  • 修改完后不需要重启

client

  • 在/etc/tcp-forward/forward_client.cfg中配置forward_server的ip和端口以及日志级别以及client的tag
[DEFAULT]
SERVER_IP=221.10.12.34
SERVER_PORT=1111
LOG_LEVEL=DEBUG
TAG=default

启动

docker启动

server
docker run -t -d --name "forward_server" --network host -v /etc/tcp-forward/:/etc/tcp-forward/ -v /var/log/tcp_forward/:/var/log/tcp_forward/ --restart always tcp-forward:latest tcp-forward --run_type server --config-file /etc/tcp-forward/forward_server.cfg
client
docker run -t -d --name "forward_client" --network host -v /etc/tcp-forward/:/etc/tcp-forward/ -v /var/log/tcp_forward/:/var/log/tcp_forward/ --restart always cao19881125/tcp-forward:latest tcp-forward --run_type client --config-file /etc/tcp-forward/forward_client.cfg

命令启动

server
tcp-forward --run_type server --config-file  /etc/tcp-forward/forward_server.cfg
client
tcp-forward --run_type client --config-file  /etc/tcp-forward/forward_client.cfg

查看日志

  • 日志生成在/var/log/tcp_forward 目录下面

配套的web项目

可以使tcp-forward-web进行设置和监控,项目地址 https://github.com/cao19881125/tcp-forward-web

from https://github.com/cao19881125/tcp_forward

-----------------------------------------

部署

以下两种部署方式任选其一

1.使用dockerhub镜像部署

pull镜像

docker pull cao19881125/tcp-forward-web:latest

run

docker run -it -d --name 'ng1' --network host cao19881125/tcp-forward-web:latest

修改web服务端口和tcp-forward server地址

默认web服务使用的端口是7070,如果需要修改则执行 默认地址是用的http://localhost:8080/,如果不同则需要修改

docker cp ng1:/etc/nginx/nginx.conf ./
# vim docker/nginx.conf
server {
    listen       7070 default_server;
    ...
    location /tcp-forward-web/forward_server/{
                ...
                proxy_pass http://localhost:8080/;
                ...
            }
}
  • 修改listen 后的端口值,即修改了web服务的地址
  • 修改proxy_pass地址为tcp-forward server服务的地址
docker cp ./nginx.conf ng1:/etc/nginx/nginx.conf
docker restart ng1

2.编译部署

修改web服务端口和tcp-forward server地址

默认web服务使用的端口是7070,如果需要修改则执行 默认地址是用的http://localhost:8080/,如果不同则需要修改

docker cp ng1:/etc/nginx/nginx.conf ./
# vim docker/nginx.conf
server {
    listen       7070 default_server;
    ...
    location /tcp-forward-web/forward_server/{
                ...
                proxy_pass http://localhost:8080/;
                ...
            }
}
  • 修改listen 后的端口值,即修改了web服务的地址
  • 修改proxy_pass地址为tcp-forward server服务的地址

build docker image

cd docker
./build.sh
# docker images
REPOSITORY                                                              TAG                 IMAGE ID            CREATED             SIZE
cao19881125/tcp-forward-web                                            latest              709c90d11224        8 minutes ago       145.4 MB

run

docker run -it -d --name 'ng1' --network host cao19881125/tcp-forward-web:latest

访问


from https://github.com/cao19881125/tcp-forward-web

 

XHR vs WebSockets 2019


Comparing the performance characteristics of using WebSockets instead of XHR in 2019 to understand the pros and cons.

How it works

You start a web (Python tornado) server that can accept XHR requests and WebSocket messages. Each request/message is encoded with a count and all the server needs to do is decrement this count.

On the client, it starts a timer and a self-recursive loop that simply asks the server to decrement a number till it reaches 0. When it's done we look at how long that took.

To contribute

For the server, create a virtualenv (wanna contribute a Dockerfile?) and install with:

pip install -r requirements.txt

Start the server in debug mode with:

python app.py --debug

Test that it works:

curl -v http://localhost:8888/

Now start the front-end (which is create-react-app) like this:

yarn
yarn run start

Now open http://localhost:3000 and follow the instructions there.

How I tested this behind a local Nginx (on macOS)

First edit /etc/hosts so it contains this:

▶ cat /etc/hosts | grep sockshootout
127.0.0.1       sockshootout.local
::1             sockshootout.local

Create a self-signed certificate with mkcert (from Homebrew):

▶ mkcert sockshootout.local
Using the local CA at "/Users/peterbe/Library/Application
Support/mkcert" ✨

Created a new certificate valid for the following names 📜
 - "sockshootout.local"

The certificate is at "./sockshootout.local.pem" and the key at
"./sockshootout.local-key.pem" ✅

▶ ls *.pem
sockshootout.local-key.pem sockshootout.local.pem

Next, create an Nginx config that looks something like this:

server {
    server_name sockshootout.local;
    return 301 https://sockshootout.local$request_uri;
}

server {
    root /Users/peterbe/dev/TORNADO/sockshootout2019/build;
    server_name sockshootout.local;

    listen 443 ssl http2;

    # ssl on;
    ssl_certificate /Users/peterbe/dev/TORNADO/sockshootout2019/sockshootout.local.pem;
    ssl_certificate_key /Users/peterbe/dev/TORNADO/sockshootout2019/sockshootout.local-key.pem;
    ssl_session_timeout 1m;
    ssl_session_cache shared:SSL:50m;

    proxy_set_header Host $host;

    charset      utf-8;

    gzip_static on;

    location /ws {
        proxy_pass http://localhost:8888;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
    }

    location / {
        try_files $uri @tornado;
    }
    location @tornado {
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Host "";
        proxy_set_header Host $http_host;
        proxy_pass http://127.0.0.1:8888;
    }

    access_log  /tmp/sockshootout.access.log combined;
    error_log  /tmp/sockshootout.error.log info;
}

Note about running Tornado servers

The fastest thing is to fork out one process per CPU all running under one port. This isn't ideal for production operations as described here and there are other better alternatives. But it is the best way to squeeze out the best possible throughput when you don't have an exernal load server in front of Tornado. You can start the app like this:

python app.py --logging=error --pre-fork

On my macbook that starts 8 processes and run:

hey -n 10000 -c 100 "http://localhost:8888/xhr?count=1000"

...

Requests/sec: 6216.8589

Without pre-fork, still with 100 clients doing 10,000 requests I get:

Requests/sec: 1878.6060
from https://github.com/peterbe/sockshootout2019

shadow-chrome

 Chrome的插件,支持socks5协议和http协议代理的设置,GFW自动更新默认黑名单走代理,可手动设置需要的域名走代理或者直通.

安装shadow-chrome插件

下载插件代码

git clone https://github.com/cao19881125/shadow-chrome

加载插件

  • 打开chrome浏览器,输入chrome://extensions/
  • 点击:加载已解压的扩展程序
  • 选择shadow-chrome目录,确定

3.使用说明

  • 点击chrome浏览器右上角的飞机图标,打开shadow-chrome
  • 默认本地socks代理地址为127.0.0.1:1080,为上面运行shadowsocks client配置的本地代理地址,可以根据上面的配置自己修改
  • 也可以在其他主机上运行shadowsocks client,只要本机能访问到的地方均可,此处配置shadowsocks client的运行地址即可
  • 点击连接按钮即开始代理
  • 默认使用gfw的黑名单确定域名是否走代理,由底部的代理状态可以看出来,当前页面的域名是否走代理
  • 可以手动更改其代理状态
  • 若选中全局代理,则所有域名均走代理
  • shadow-chrome图标有三种颜色,分别为:
    • 黑色:未连接状态
    • 黄色:连接-不走代理状态
    • 绿色:连接-走代理状态

from https://github.com/cao19881125/shadow-chrome