Pages

▼

Thursday, 30 November 2017

RcloneBrowser-简易的跨平台的 rclone客户端

Simple cross platform GUI for rclone.

RcloneBrowser

Simple cross platform GUI for rclone


Simple cross platfrom GUI for rclone command line tool. Supports Windows, macOS and GNU/Linux.

Features

  • Allows to browse and modify any rclone remote, including encrypted ones
  • Uses same configuration file as rclone, no extra configuration required
  • Supports custom location and encryption for .rclone.conf configuration file
  • Simultaneously navigate multiple repositories in separate tabs
  • Lists files hierarchically with file name, size and modify date
  • All rclone commands are executed asynchronously, no freezing GUI
  • File hierarchy is lazily cached in memory, for faster traversal of folders
  • Allows to upload, download, create new folders, rename or delete files and folders
  • Allows to calculate size of folder, export list of files and copy rclone copmmand to clipboard
  • Can process multiple upload or download jobs in background
  • Drag & drop support for dragging files from local file explorer for uploading
  • Streaming media files for playback in player like mpv or similar
  • Mount and unmount folders on macOS and GNU/Linux
  • Optionally minimizes to tray, with notifications when upload/download finishes
  • Supports portable mode (create .ini file next to executable with same name), rclone and .rclone.conf path now can be relative to executable

Download

Get Windows, macOS and Ubuntu package on releases page.
For Ubuntu you can also install it from Launchpad: Rclone Browser.
ArchLinux users can install latest release from AUR repository: rclone-browser.
Other GNU/Linux users will need to build from source.
from  https://mmozeiko.github.io/RcloneBrowser/
https://github.com/mmozeiko/RcloneBrowser
--------------------------
AutoRclone: rclone copy/move/sync (automatically) with thousands of service accounts  .

AutoRclone: rclone copy/move/sync (automatically) with service accounts (still in the beta stage)

Many thanks for rclone and folderclone.

  • create service accounts using script
  • add massive service accounts into rclone config file
  • add massive service accounts into groups for your organization
  • automatically switch accounts when rclone copy/move/sync
  • Windows system is supported

Step 1. Copy code to your VPS or local machine

Before everything, install python3. Because we use python as our programing language.

For Linux system: Install screen, git and latest rclone. If in Debian/Ubuntu, directly use this command

sudo apt-get install screen git && curl https://rclone.org/install.sh | sudo bash

After all dependency above are successfully installed, run this command

sudo git clone https://github.com/xyou365/AutoRclone && cd AutoRclone && sudo pip3 install -r requirements.txt

For Windows system: Directly download this project then install latest rclone. Then run this command (type in cmd command windows or PowerShell windows) in our project folder

pip3 install -r requirements.txt

Step 2. Generate service accounts What is service account How to use service account in rclone.

Let us create only the service accounts that we need. Warning: abuse of this feature is not the aim of autorclone and we do NOT recommend that you make a lot of projects, just one project and 100 sa allow you plenty of use, its also possible that overabuse might get your projects banned by google.

Enable the Drive API in Python Quickstart and save the file credentials.json into project directory.

If you do not have any project in your account then

  • create 1 projec
  • enable the required services
  • create 100 (1 project, each with 100) Service Accounts
  • and download their credentials into a folder named accounts
Note: 1 service account can copy around 750gb a day, 1 project makes 100 service accounts so thats 75tb a day, for most users this should easily suffice.

The command would look something like python3 gen_sa_accounts.py --quick-setup 1 replace "1" with the number of projects you want

If you have already N projects and want to create service accounts only in newly created projects,

to

  • create additional 1 project (project N+1 to project N+2)
  • enable the required services
  • create 100 (1 project, with 100) Service Accounts
  • and download their credentials into a folder named accounts

run

python3 gen_sa_accounts.py --quick-setup 1 --new-only

If you want to create some service accounts using existing projects (do not create more projects), run python3 gen_sa_accounts.py --quick-setup -1. Note that this will overwrite the existing service accounts.

After it is finished, there will be many json files in one folder named accounts.

Step 3. Add service accounts to Google Groups (Optional but recommended for hassle free long term use)

We use Google Groups to manager our service accounts considering the
Official limits to the members of Team Drive (Limit for individuals and groups directly added as members: 600).

For GSuite Admin

  1. Turn on the Directory API following official steps (save the generated json file to folder credentials).

  2. Create group for your organization in the Admin console. After create a group, you will have an address for examplesa@yourdomain.com.

  3. Run python3 add_to_google_group.py -g sa@yourdomain.com

For meaning of above flags, please run python3 add_to_google_group.py -h

For normal user

Create Google Group then add the service accounts as members by hand. Limit is 10 at a time, 100 a day but if you read our warning and notes above, you would have 1 project and hence easily in your range.

Step 4. Add service accounts or Google Groups into Team Drive

If you do not use Team Drive, just skip. Warning: It is NOT recommended to use service accounts to clone "to" folders that are not in teamdrives, SA work best for teamdrives.

If you have already created Google Groups (Step 2) to manager your service accounts, add the group address sa@yourdomain.com or sa@googlegroups.com to your source Team Drive (tdsrc) and destination Team Drive (tddst).

Otherwise, add service accounts directly into Team Drive.

Enable the Drive API in Python Quickstart and save the credentials.json into project root path if you have not done it in Step 2.

  • Add service accounts into your source Team Drive: python3 add_to_team_drive.py -d SharedTeamDriveSrcID
  • Add service accounts into your destination Team Drive: python3 add_to_team_drive.py -d SharedTeamDriveDstID

Step 5. Start your task

Let us copy hundreds of TB resource using service accounts. Note: Sarcasm, over abuse of this (regardless of what cloning script you use) may get you noticed by google, we recommend you dont be a glutton and clone what is important instead of downloading entire wikipedia.

For server side copy

  • publicly shared folder to Team Drive
  • Team Drive to Team Drive
  • publicly shared folder to publicly shared folder (with write privilege)
  • Team Drive to publicly shared folder
python3 rclone_sa_magic.py -s SourceID -d DestinationID -dp DestinationPathName -b 1 -e 600

  • For meaning of above flags, please run python3 rclone_sa_magic.py -h

  • Add --disable_list_r if rclone cannot read all contents of public shared folder.

  • Please make sure the Rclone can read your source and destination directory. Check it using rclone size:

  1. rclone --config rclone.conf size --disable ListR src001:

  2. rclone --config rclone.conf size --disable ListR dst001:

For local to Google Drive (needs some testing)

  • local to Team Drive
  • local to private folder
  • private folder to any (think service accounts cannot do anything about private folder)
python3 rclone_sa_magic.py -sp YourLocalPath -d DestinationID -dp DestinationPathName -b 1 -e 600
  • Run command tail -f log_rclone.txt to see what happens in details (linux only).

Also let's talk about this project in Telegram Group AutoRclone

[Blog（中文）](Blog (中文) https://gsuitems.com/index.php/archives/13/) | Google Drive Group | Google Drive Channel

from https://github.com/xyou365/AutoRclone

------

Rclone 750G自动换号 – AutoRclone

原理都是一样的：
学习了 folderclone 利用 service accounts；
rclone 的时候通过 rclone rc 来做到对 rclone 任务的监、控；
还可以用 Google Groups 的形式对那么多的 service accounts 进行管理。

功能也一样的，多 sa 账号来接力 750GxN：
从本地到 Team Drive，
从公共分享目录到 Team Drive，
再或者从 Team Drive 到 Team Drive。

好久前写的程序，配的环境，好多细节忘记了，凭记忆写的 Steps （勿喷、欢迎交流）

https://github.com/xyou365/AutoRclone

Q. 看着好复杂？
A. 不复杂的其实，如果你用过 folderclone，或者你用过 rclone，那这种结合方案最适合你啦。以下问题都不是问题啦：
0. 知道能通过多成员，把无限容量的 Team Drive 塞满，但是没成千上万的 Google 账号（一键脚本生成 service account 替代普通 Google 账号并生成所有账号的的授权文件）
1. 配置 rclone 很繁琐 （利用上生成的授权文件一键脚本生成 rclone 配置，再也不用交互方式生成 token 了）
2. rclone copy 即使被榨干750G的日流量限制，还一直 retry 不跳到下一个账号（rclone_sa.py 利用 rclone rc 帮你自动 kill 掉被榨干750G 的 sa 账号，自动切换到下一个 sa 账号）
3. folderclone 不能处理目录结构复杂的任务，导致丢文件，还是需要 rclone 来救急（舍弃 folderclone，用 rclone）
4. folderclone 的功能太单一，压根没法替代 rclone （那为什么还用 folderclone）

关于 Google Groups
Q. 为什么要用 Google Groups
A. 因为 Team Drive 的成员有限制，只能添加 600 个 sa，可是咱们有好几K甚至无限的 sa 呀。另外不删除 Team Drive 想删除 sa，尤其是别人的 sa 也麻烦。

Q. 为什么不能用普通的 Google Groups 来一键管理 service accounts
A. 可以用来管理。但 Google Groups 网页添加成员每次只能添加 10 个，24 小时内只能添加 100 个。调用官方 API 的小程序 add_to_google_group.py 可以无限制地添加 sa 到你的 Google Groups 里面，但是（貌似）只有 Gsuite 可以用 Directory API。

from https://www.goojie.eu/235.html

------

Rclone 750G自动换号– AutoRclone使用教程

 prerequisites:

python2.6 or above.

pip.

a google a/c with google drive enabled.

step1. turn on the drive api

博主经常看到有人在论坛或者QQ/Telegram群组发问:

怎么快速拷贝1000T资源到我的Google Drive？

必须是Rclone/AutoRclone啊！一直不想写关于AutoRclone的（中文）教程，因为其一直处于测试阶段，如果后面代码有大的更新那么前面的教程对于后面的人来说就是一个误导。但是现在应该不会有大的更新了，而且这么长时间的测试发现如果没有一个详细的教程的话，会让很多不会程序的小白错过这么好的工具 :)

AutoRclone是一个用Python写的小程序的集合，其

  • 学习了folderclone利用service accounts来替代普通Google账号进行拷贝/上传；
  • 通过rclone rc来做到对Rclone任务的监、控；
  • 还可以用Google Groups的形式对成百上千的service accounts进行管理。

从而方便快速地

  • 从本地到 Team Drive，
  • 从公共分享目录到 Team Drive，
  • 再或者从 Team Drive 到 Team Drive。

其食用方法极其简单！

步骤 1 下载代码

首先不管你是哪个操作系统，需要安装Python。对于Linux和Windows分别用以下方式安装相关脚本
Linux: 安装screen，git和最新的Rclone。如果是Debian/Ubuntu，直接输入以下命令

sudo apt-get install screen git && curl https://rclone.org/install.sh | sudo bash

然后从Github下载代码并安装相关Python依赖包

sudo git clone https://github.com/xyou365/AutoRclone && cd AutoRclone && sudo pip3 install -r requirements.txt

Windows: 安装最新的Rclone

，直接从Github下载AutoRclone

后在Windows的cmd或PowerShell中切换到AutoRclone的目录，输入以下命令安装相关Python依赖包

pip3 install -r requirements.txt
以下步骤如果输入python3没有反应，请输入python或者py3

以下所有步骤由于需要用到谷歌服务，请确保cmd或者vps能够翻墙

步骤 2 生成service accounts

首先开启Drive API

并将credentials.json保存到你的AutoRclone目录下面

开启Drive API
然后分三种情况

如果你之前没创建过项目, 直接运行

python3 gen_sa_accounts.py --quick-setup 5
  • 创建6个项目（项目0到项目5）
  • 开启相关的服务
  • 创建600个service accounts（6个项目，每个项目100个）
  • 将600个service accounts的授权文件下载到accounts文件夹下面

如果你已经有N个项目，现需要创建新的项目并在新的项目中创建service accounts，直接运行

python3 gen_sa_accounts.py --quick-setup 2 --new-only
  • 额外创建2个项目（项目N+1到项目N+2）
  • 开启相关的服务
  • 创建200个service accounts（2个项目，每个项目100个）
  • 将200个service accounts的授权文件下载到accounts文件夹下面

如果你想用已有的项目来创建service accounts（不创建新的项目），直接运行

python3 gen_sa_accounts.py --quick-setup -1

注意这会覆盖掉已有的service accounts
顺利完成后，AutoRclone文件下面的accounts文件夹下会有很多的json文件。

步骤 3 可选：将service accounts加入Google Groups

为了方便管理service accounts，也是为了让我们的Team Drive可以容纳更多的service accounts，我们这里用到了Google Groups。

Official limits to the members of Team Drive

(Limit for individuals and groups directly added as members is 600).

对于G Suite管理员

  1. 按照官方步骤,开启Directory API，将生成的json文件保存到credentials文件下。
  1. 在控制面版里面创建一个群组，创建好你会获得一个类似域名邮箱的地址sa@yourdomain.com
  2. 利用API将service accounts加入Google Groups
python3 add_to_google_group.py -g sa@yourdomain.com

如果想看参数的具体含义，直接运行python3 add_to_google_group.py -h

对于普通Google账号
直接创建一个Google Group

然后手动地将service accounts对应的邮箱地址（可以在json认证文件中找到）挨个加进去。但每次只能加10个，每24小时只能加100个。

步骤 4 将service accounts或者Google Groups加入到Team Drive

- 如果你没有Team Drive，那可以找朋友帮你开一个或者去买一个，实在不嫌弃也可以留言找博主开一个
- 如果你已经在完成了步骤2中的用Google Groups来管理service accounts，那么直接将Google Groups地址sa@yourdomain.com或者sb@googlegroups.com加入你的源Team Drive（tdsrc）和目标Team Drive（tddst）
- 如果你在步骤2中没有生成并保存credentials.json那么再做一遍

将service accounts加入到源Team Drive

python3 add_to_team_drive.py -d SharedTeamDriveSrcID

将service accounts加入到目标Team Drive

python3 add_to_team_drive.py -d SharedTeamDriveDstID

步骤 5 开始拷贝/上传

准备工作都做好了，以后你只需要将项目文件夹拷贝到任何地方，进行此拷贝/上传步骤就可以啦

拷贝

python3 rclone_sa_magic.py -s SourceID -d DestinationID -dp DestinationPathName -b 1 -e 600
  • 如果想看参数的具体含义，直接运行python3 rclone_sa_magic.py -h
  • 特别地，如果想多开，请用-p参数给不同的复制任务指定不同的端口
  • 如果发现拷贝内容明显少于源Team Drive里面的内容，那么你可能碰到Bug.请给上运行参数再加上--disable_list_r.
  • 如果你一开始就碰到了
    Failed to rc: connection failed: Post http://localhost:5572/core/stats: dial tcp :5572: connectex: No connection could be made because the target machine actively refused it.
    那么可能是权限或者路径导致Rclone任务都没跑起来，请观察日志文件log_rclone.txt，并结合如下简单命令检查出原因rclone --config rclone.conf size --disable ListR src001:，rclone --config rclone.conf size --disable ListR dst001:

上传

python3 rclone_sa_magic.py -sp YourLocalPath -d DestinationID -dp DestinationPathName -b 1 -e 600
拷贝截图

不出意外应该就可以跑起来了。每个service account的总的时间消耗是校验时间+拷贝时间，如果你的拷贝任务比较大的话建议将其（按文件夹）稍微拆分，挨个或者并行完成，这样能极大减小每次切换service accounts后不必要的校验时间。另外关于拷贝速度，API调用限制了每秒不超过10次调用，每次拷贝固定数量的文件数目，所以如果你的任务中的文件比较小的话，拷贝速度可能是每秒几百MB，文件都比较大的话，拷贝速度可能是几十GB/s。

一些 Q & A

Q. 怎么检查是否拷贝完全？发现丢文件了？

A. 拷贝完请做以下检查 
rclone --config rclone.conf size --disable ListR src001:源路径
rclone --config rclone.conf size --disable ListR dst001:目标路径

如果发现目标盘比源盘体积还要大，那么对目标盘进行去重；如果发现目标盘比源盘少文件了，那么再拷贝一次；如果还是少，那么就是源盘里面的有重复文件了，直接对其进行去重复。去重命令：

rclone --config rclone.conf dedupe src001:源路径
rclone --config rclone.conf dedupe dst001:目标路径
from https://gsuitems.com/index.php/archives/13/ 
at

No comments:

Post a Comment