gofs, 一个跨平台的文件同步工具

 A cross-platform file synchronization tool out of the box based on golang.

English | 简体中文

A cross-platform file synchronization tool out of the box based on golang.

Installation

The first need Go installed (version 1.18+ is required), then you can use the below command to install gofs.

go install github.com/no-src/gofs/...@latest

Run In Docker

If you want to run in a docker, you should install or build with the -tags netgo flag or set the environment CGO_ENABLED=0, otherwise you may get an error that the gofs not found, when the docker container is running.

go install -tags netgo github.com/no-src/gofs/...@latest

You can use the build-docker.sh script to build the docker image and you should clone this repository and cd to the root path of the repository first.

$ ./scripts/build-docker.sh

Or pull the docker image directly from DockerHub with the command below.

$ docker pull nosrc/gofs

For more scripts about release and docker, see the scripts directory.

Run In the Background

You can install a program run in the background using the following command on Windows.

go install -ldflags="-H windowsgui" github.com/no-src/gofs/...@latest

Quick Start

Prerequisites

Please ensure the source directory and dest directory exists first, replace the following path with your real path.

$ mkdir source dest

Generate the TLS cert file and key file for testing purposes.

The TLS cert and key files are just used by File Server and Remote Disk Server.

$ go run $GOROOT/src/crypto/tls/generate_cert.go --host 127.0.0.1
2021/12/30 17:21:54 wrote cert.pem
2021/12/30 17:21:54 wrote key.pem

Look up our workspace.

$ ls
cert.pem  key.pem  source  dest

Local Disk

Monitor source directory and sync change files to dest directory.

You can use the logically_delete flag to enable the logically delete and avoid deleting files by mistake.

Set the checkpoint_count flag to use the checkpoint in the file to reduce transfer unmodified file chunks, by default checkpoint_count=10, which means it has 10+2 checkpoints at most. There are two additional checkpoints at the head and tail. The first checkpoint is equal to the chunk_size, it is optional. The last checkpoint is equal to the file size, it is required. The checkpoint offset set by the checkpoint_count is always more than chunk_size, unless the file size is less than or equal to chunk_size, then the checkpoint_count will be zero, so it is optional.

By default, if the file size and file modification time of the source file is equal to the destination file, then ignore the current file transfer. You can use the force_checksum flag to force enable the checksum to compare whether the file is equal or not.

The default checksum hash algorithm is md5, you can use the checksum_algorithm flag to change the default hash algorithm, current supported algorithms: md5, sha1, sha256, sha512, crc32, crc64, adler32, fnv-1-32 , fnv-1a-32, fnv-1-64, fnv-1a-64, fnv-1-128, fnv-1a-128.

If you want to reduce the frequency of synchronization, you can use the sync_delay flag to enable sync delay, start sync when the event count is equal or greater than sync_delay_events, or wait for sync_delay_time interval time since the last sync.

$ gofs -source=./source -dest=./dest

Sync Once

Sync the whole path immediately from source directory to dest directory.

$ gofs -source=./source -dest=./dest -sync_once

Sync Cron

Sync the whole path from source directory to dest directory with cron.

# Per 30 seconds sync the whole path from source directory to dest directory
$ gofs -source=./source -dest=./dest -sync_cron="*/30 * * * * *"

Daemon Mode

Start a daemon to create subprocess to work, and record pid info to pid file.

$ gofs -source=./source -dest=./dest -daemon -daemon_pid

File Server

Start a file server for source directory and dest directory.

The file server is use HTTPS default, set the tls_cert_file and tls_key_file flags to customize the cert file and key file.

You can disable the HTTPS by set the tls flag to false if you don't need it.

If you set the tls to true, the file server default port is 443, otherwise it is 80, and you can customize the default port with the server_addr flag, like -server_addr=":443".

If you enable the tls flag on the server side, you can control whether a client skip verifies the server's certificate chain and host name by the tls_insecure_skip_verify flag, default is true.

You should set the rand_user_count flag to auto generate some random users or set the users flag to customize server users for security reasons.

The server users will output to log if you set the rand_user_count flag greater than zero.

If you need to compress the files, add the server_compress flag to enable gzip compression for response, but it is not fast now, and may reduce transmission efficiency in the LAN.

# Start a file server and create three random users
# Replace the `tls_cert_file` and `tls_key_file` flags with your real cert files in the production environment
$ gofs -source=./source -dest=./dest -server -tls_cert_file=cert.pem -tls_key_file=key.pem -rand_user_count=3

Remote Disk Server

Start a remote disk server as a remote file source.

The source flag detail see Remote Server Source Protocol.

Pay attention to that remote disk server users must have read permission at least, for example, -users="gofs|password|r".

You can use the checkpoint_count and sync_delay flags like the Local Disk.

# Start a remote disk server
# Replace the `tls_cert_file` and `tls_key_file` flags with your real cert files in the production environment
# Replace the `users` flag with complex username and password for security
$ gofs -source="rs://127.0.0.1:8105?mode=server&local_sync_disabled=true&path=./source&fs_server=https://127.0.0.1" -dest=./dest -users="gofs|password|r" -tls_cert_file=cert.pem -tls_key_file=key.pem

Remote Disk Client

Start a remote disk client to sync change files from remote disk server.

The source flag detail see Remote Server Source Protocol.

Use the sync_once flag to sync the whole path immediately from remote disk server to local dest directory, like Sync Once.

Use the sync_cron flag to sync the whole path from remote disk server to local dest directory with cron, like Sync Cron.

Use the force_checksum flag to force enable the checksum to compare whether the file is equal or not, like Local Disk.

You can use the sync_delay flag like the Local Disk.

# Start a remote disk client
# Replace the `users` flag with your real username and password
$ gofs -source="rs://127.0.0.1:8105" -dest=./dest -users="gofs|password"

Remote Push Server

Start a Remote Disk Server as a remote file source, then enable the remote push server with the push_server flag.

Pay attention to that remote push server users must have read and write permission at least, for example, -users="gofs|password|rw".

# Start a remote disk server and enable the remote push server
# Replace the `tls_cert_file` and `tls_key_file` flags with your real cert files in the production environment
# Replace the `users` flag with complex username and password for security
$ gofs -source="rs://127.0.0.1:8105?mode=server&local_sync_disabled=true&path=./source&fs_server=https://127.0.0.1" -dest=./dest -users="gofs|password|rw" -tls_cert_file=cert.pem -tls_key_file=key.pem -push_server

Remote Push Client

Start a remote push client to sync change files to the Remote Push Server.

Use the chunk_size flag to set the chunk size of the big file to upload. The default value of chunk_size is 1048576, which means 1MB.

You can use the checkpoint_count and sync_delay flags like the Local Disk.

More flag usage see Remote Disk Client.

# Start a remote push client and enable local disk sync, sync the file changes from source path to the local dest path and the remote push server
# Replace the `users` flag with your real username and password
$ gofs -source="./source" -dest="rs://127.0.0.1:8105?local_sync_disabled=false&path=./dest" -users="gofs|password"

Relay

If you need to synchronize files between two devices that are unable to establish a direct connection, you can use a reverse proxy as a relay server. In more detail, see also Relay.

Remote Server Source Protocol

The remote server source protocol is based on URI, see RFC 3986.

Scheme

The scheme name is rs.

Host

The remote server source uses 0.0.0.0 or other local ip address as host in Remote Disk Server mode, and use ip address or domain name as host in Remote Disk Client mode.

Port

The remote server source port, default is 8105.

Parameter

Use the following parameters in Remote Disk Server mode only.

• path the Remote Disk Server actual local source directory
• mode running mode, in Remote Disk Server mode is server, default is running in Remote Disk Client mode
• fs_server File Server address, like https://127.0.0.1
• local_sync_disabled disabled Remote Disk Server sync changes to its local dest path, true or false, default is false

Example

For example, in Remote Disk Server mode.

 rs://127.0.0.1:8105?mode=server&local_sync_disabled=true&path=./source&fs_server=https://127.0.0.1
 \_/  \_______/ \__/ \____________________________________________________________________________/
  |       |       |                                      |
scheme   host    port                                parameter

Manage API

Enable manage api base on File Server by using the manage flag.

By default, allow to access manage api by private address and loopback address only.

You can disable it by setting the manage_private flag to false.

$ gofs -source=./source -dest=./dest -server -tls_cert_file=cert.pem -tls_key_file=key.pem -rand_user_count=3 -manage

Profiling API

The pprof url address like this

https://127.0.0.1/manage/pprof/

Config API

Reading the program config, default return the config with json format, and support json and yaml format currently.

https://127.0.0.1/manage/config

Or use the format parameter to specific the config format.

https://127.0.0.1/manage/config?format=yaml

Report API

Use the report flag to enable report api route, and start to collect the report data, need to enable the manage flag first.

The details of the report api see Report API.

https://127.0.0.1/manage/report

Logger

Enable the file logger and console logger by default, and you can disable the file logger by setting the log_file flag to false.

Use the log_level flag to set the log level, default is INFO, (DEBUG=0 INFO=1 WARN=2 ERROR=3).

Use the log_dir flag to set the directory of the log file, default is ./logs/.

Use the log_flush flag to enable auto flush log with interval, default is true.

Use the log_flush_interval flag to set the log flush interval duration, default is 3s.

Use the log_event flag to enable the event log, write to file, default is false.

Use the log_sample_rate flag to set the sample rate for the sample logger, and the value ranges from 0 to 1, default is 1.

Use the log_format flag to set the log output format, current support text and json, default is text.

# set the logger config in "Local Disk" mode
$ gofs -source=./source -dest=./dest -log_file -log_level=0 -log_dir="./logs/" -log_flush -log_flush_interval=3s -log_event

Use Configuration File

If you want, you can use a configuration file to replace all the flags.It supports json and yaml format currently.

All the configuration fields are the same as the flags, you can refer to the Configuration Example or the response of Config API.

$ gofs -conf=./gofs.yaml

Checksum

You can use the checksum flag to calculate the file checksum and print the result.

The chunk_size, checkpoint_count and checksum_algorithm flags are effective here the same as in the Local Disk.

$ gofs -source=./gofs -checksum

For More Information

Help Info

$ gofs -h

Version Info

$ gofs -v

About Info

$ gofs -about

from https://github.com/no-src/gofs 

------- 

基于golang开发的一款开箱即用的跨平台文件同步工具gofs

概述

gofs是基于golang开发的一款开箱即用的跨平台文件同步工具，支持在本地磁盘之间同步、从远程服务器同步变更到本地、将本地文件变更推送到远程服务器三种模式。开源地址如下：https://github.com/no-src/gofs

安装

首先需要确保已经安装了Go (版本必须是1.18+)，然后你就可以使用下面的命令来安装gofs了

如果你不需要自行从源码编译安装，也可以直接下载最新发布的二进制文件

go install github.com/no-src/gofs/...@latest

在Docker中运行

如果你想要在docker中运行，请在安装或者编译的时候使用-tags netgo命令行参数或者设置环境变量CGO_ENABLED=0,否则当你运行docker容器时可能会报找不到gofs的错误

go install -tags netgo github.com/no-src/gofs/...@latest

你可以使用build-docker.sh脚本来构建docker镜像，首先你需要克隆本仓库并且cd到本仓库的根目录

$ ./scripts/build-docker.sh

或者使用以下命令直接从DockerHub中拉取docker镜像

$ docker pull nosrc/gofs

更多关于发布与docker的脚本参见scripts目录

后台运行

在windows系统中，你可以使用下面的命令构建一个在后台运行的不带命令行界面的程序

go install -ldflags="-H windowsgui" github.com/no-src/gofs/...@latest

快速开始

先决条件

请确保文件同步的源目录和目标目录都已经存在，如果目录不存在，则用你实际的目录替换下面的路径进行提前创建

$ mkdir source dest

生成仅用于测试的证书和密钥文件，生产中请替换为正式的证书

TLS证书和密钥文件仅用于与Web文件服务器和远程磁盘服务端进行安全通讯

$ go run $GOROOT/src/crypto/tls/generate_cert.go --host 127.0.0.1
2021/12/30 17:21:54 wrote cert.pem
2021/12/30 17:21:54 wrote key.pem

查看你的工作目录

$ ls
cert.pem  key.pem  source  dest

本地磁盘

监控本地源目录将变更同步到目标目录

你可以使用logically_delete命令行参数来启用逻辑删除，从而避免误删数据

设置checkpoint_count命令行参数来使用文件中的检查点来减少传输未修改的文件块，默认情况下checkpoint_count=10，这意味着它最多有10+2个检查点。在头部和尾部还有两个额外的检查点。第一个检查点等于chunk_size，它是可选的。最后一个检查点等于文件大小，这是必需的。由checkpoint_count设置的检查点偏移量总是大于chunk_size，除非文件大小小于或等于chunk_size，那么checkpoint_count将变为0，所以它是可选的

默认情况下，如果源文件的大小和修改时间与目标文件相同，则忽略当前文件的传输。你可以使用force_checksum命令行参数强制启用校验和来比较文件是否相等

$ gofs -source=./source -dest=./dest

全量同步

执行一次全量同步，直接将整个源目录同步到目标目录

$ gofs -source=./source -dest=./dest -sync_once

定时同步

定时执行全量同步，将整个源目录同步到目标目录

# 每30秒钟将源目录全量同步到目标目录
$ gofs -source=./source -dest=./dest -sync_cron="*/30 * * * * *"

守护进程模式

启动守护进程来创建一个工作进程处理实际的任务，并将相关进程的pid信息记录到pid文件中

$ gofs -source=./source -dest=./dest -daemon -daemon_pid

Web文件服务器

启动一个Web文件服务器用于访问远程的源目录和目标目录

Web文件服务器默认使用HTTPS协议，使用tls_cert_file和tls_key_file命令行参数来指定相关的证书和密钥文件

如果你不需要使用TLS进行安全通讯，可以通过将tls命令行参数指定为false来禁用它

如果将tls设置为true，则服务器默认运行端口为443，反之默认端口为80，你可以使用server_addr命令行参数来自定义服务器运行端口，例如-server_addr=":443"

如果你在服务器端启用tls命令行参数，可以通过tls_insecure_skip_verify命令行参数来控制客户端是否跳过验证服务器的证书链和主机名，默认为true

出于安全考虑，你应该设置rand_user_count命令行参数来随机生成指定数量的用户或者通过users命令行参数自定义用户信息来保证数据的访问安全，禁止用户匿名访问数据

如果rand_user_count命令行参数设置大于0，则随机生成的账户密码将会打印到日志信息中，请注意查看

如果你需要启用gzip压缩响应结果，则添加server_compress命令行参数，但是目前gzip压缩不是很快，在局域网中可能会影响传输效率

# 启动一个Web文件服务器并随机创建3个用户
# 在生产环境中请将`tls_cert_file`和`tls_key_file`命令行参数替换为正式的证书和密钥文件
$ gofs -source=./source -dest=./dest -server -tls_cert_file=cert.pem -tls_key_file=key.pem -rand_user_count=3

远程磁盘服务端

启动一个远程磁盘服务端作为一个远程文件数据源

source命令行参数详见远程磁盘服务端数据源协议

注意远程磁盘服务端的用户至少要拥有读权限，例如：-users="gofs|password|r"

你可以使用checkpoint_count命令行参数就跟本地磁盘一样

# 启动一个远程磁盘服务端
# 在生产环境中请将`tls_cert_file`和`tls_key_file`命令行参数替换为正式的证书和密钥文件
# 为了安全起见，请使用复杂的账户密码来设置`users`命令行参数
$ gofs -source="rs://127.0.0.1:8105?mode=server&local_sync_disabled=true&path=./source&fs_server=https://127.0.0.1" -dest=./dest -users="gofs|password|r" -tls_cert_file=cert.pem -tls_key_file=key.pem

远程磁盘客户端

启动一个远程磁盘客户端将远程磁盘服务端的文件变更同步到本地目标目录

使用sync_once命令行参数，可以直接将远程磁盘服务端的文件整个全量同步到本地目标目录，就跟全量同步一样

使用sync_cron命令行参数，可以定时将远程磁盘服务端的文件整个全量同步到本地目标目录，就跟定时同步一样

使用force_checksum命令行参数强制启用校验和来比较文件是否相等，就跟本地磁盘一样

source命令行参数详见远程磁盘服务端数据源协议

# 启动一个远程磁盘客户端
# 请将`users`命令行参数替换为上面设置的实际账户名密码
$ gofs -source="rs://127.0.0.1:8105" -dest=./dest -users="gofs|password"

远程推送服务端

启动一个远程磁盘服务端作为一个远程文件数据源，并使用push_server命令行参数启用远程推送服务端

注意远程推送服务端的用户至少要拥有读写权限，例如：-users="gofs|password|rw"

# 启动一个远程磁盘服务端并启用远程推送服务端
# 在生产环境中请将`tls_cert_file`和`tls_key_file`命令行参数替换为正式的证书和密钥文件
# 为了安全起见，请使用复杂的账户密码来设置`users`命令行参数
$ gofs -source="rs://127.0.0.1:8105?mode=server&local_sync_disabled=true&path=./source&fs_server=https://127.0.0.1" -dest=./dest -users="gofs|password|rw" -tls_cert_file=cert.pem -tls_key_file=key.pem -push_server

远程推送客户端

启动一个远程推送客户端将本地文件变更同步到远程推送服务端

使用chunk_size命令行参数来设置大文件上传时切分的区块大小，默认值为1048576，即1MB

你可以使用checkpoint_count命令行参数就跟本地磁盘一样

更多命令行参数用法请参见远程磁盘客户端

# 启动一个远程推送客户端并且启用本地磁盘同步，将source目录下的文件变更同步到本地dest目录和远程推送服务器上
# 请将`users`命令行参数替换为上面设置的实际账户名密码
$ gofs -source="./source" -dest="rs://127.0.0.1:8105?local_sync_disabled=false&path=./dest" -users="gofs|password"

远程磁盘服务端数据源协议

远程磁盘服务端数据源协议基于URI基本语法,详见RFC 3986

方案

方案名称为rs

主机名

远程磁盘服务端数据源在远程磁盘服务端模式下使用0.0.0.0或者其他本地网卡IP地址作为主机名，在远程磁盘客户端 模式下使用远程磁盘服务端的IP地址或者域名作为主机名

端口号

远程磁盘服务端数据源端口号，默认为8105

参数

仅在远程磁盘服务端模式下设置以下参数

• path 远程磁盘服务端真实的本地源目录
• mode 指定运行模式，只有在远程磁盘服务端模式下需要手动指定为server，默认为远程磁盘客户端模式
• fs_server Web文件服务器地址，例如https://127.0.0.1
• local_sync_disabled 是否将远程磁盘服务端的文件变更同步到远程本地的目标目录,可选值为true或false，默认值为false

示例

远程磁盘服务端模式下的示例

 rs://127.0.0.1:8105?mode=server&local_sync_disabled=true&path=./source&fs_server=https://127.0.0.1
 \_/  \_______/ \__/ \____________________________________________________________________________/
  |       |       |                                      |
 方案   主机名   端口号                                    参数

管理接口

基于Web文件服务器的应用管理接口

默认情况下，仅允许私有地址和回环地址访问管理接口的相关路由

你可以通过将manage_private命令行参数设置为false来禁用默认行为，允许公网IP访问管理接口的路由

$ gofs -source=./source -dest=./dest -server -tls_cert_file=cert.pem -tls_key_file=key.pem -rand_user_count=3 -manage

性能分析接口

pprof访问地址如下：

https://127.0.0.1/manage/pprof/

配置接口

读取应用程序配置，默认返回json格式，当前支持json和yaml格式

https://127.0.0.1/manage/config

或者使用format参数来指定返回的配置格式

https://127.0.0.1/manage/config?format=yaml

报告接口

使用report命令行参数来启用报告接口的路由并且开始收集报告数据，需要先启用manage命令行参数

报告接口详情参见Report API

https://127.0.0.1/manage/report

日志

默认情况下会启用文件日志与控制台日志，你可以将log_file命令行参数设置为false来禁用文件日志

使用log_level命令行参数设置日志的等级，默认级别是INFO，可选项为：DEBUG=0 INFO=1 WARN=2 ERROR=3

使用log_dir命令行参数来设置日志文件目录，默认为./logs/

使用log_flush命令行参数来设置自动刷新日志到文件中，默认启用

使用log_flush_interval命令行参数设置自动刷新日志到文件中的频率，默认为3s

使用log_event命令行参数启用事件日志，所有事件都会记录到文件中，默认为禁用

使用log_sample_rate命令行参数设置采样日志的采样率，取值范围为0到1，默认值为1

# 在"本地磁盘"模式下设置日志信息
$ gofs -source=./source -dest=./dest -log_file -log_level=0 -log_dir="./logs/" -log_flush -log_flush_interval=3s -log_event

使用配置文件

如果需要的话，你可以使用配置文件来代替所有的命令行参数，当前支持json和yaml格式

所有的配置字段名称跟命令行参数一样，你可以参考配置示例或者配置接口的响应结果

$ gofs -conf=./gofs.yaml

校验和

你可以使用checksum命令行参数来计算并打印文件的校验和

chunk_size和checkpoint_count命令行参数在这里同在本地磁盘中一样有效

$ gofs -source=./gofs -checksum

更多信息

帮助信息

$ gofs -h

版本信息

$ gofs -v

关于信息

$ gofs -about