pget - 最快的文件下载工具, 可以取代wget

    

Description

Multi-Connection Download using parallel requests.

• Fast
• Resumable
• Cross-compiled (windows, linux, macOS)

This is an example to download linux kernel. It will be finished between 15s.

Note

• Using a large number of connections to a single URL can lead to DOS attacks.
• The case is increasing that if you use multiple connections to 1 URL does not increase the download speed with the spread of CDNs.
  • I recommend to use multiple mirrors simultaneously for faster downloads (And the number of connections is 1 for each).

Installation

Homebrew

brew update
brew install Code-Hex/tap/pget

Go

$ go install github.com/Code-Hex/pget/cmd/pget@latest （go需为1.17.x版）

（如果你的go 版本为1.16或更低版本，则安装方法为：
 cd $GOPATH

go get -u -v github.com/Code-Hex/pget/cmd/pget/ )

Synopsis

This example will be used 2 connections per URL.

$ pget -p 2 MIRROR1 MIRROR2 MIRROR3

If you have created such as this file

cat list.txt
MIRROR1
MIRROR2
MIRROR3

You can do this

cat list.txt | pget -p 2

Options

  Options:
  -h,  --help                   print usage and exit
  -p,  --procs <num>            the number of connections for a single URL (default 1)
  -o,  --output <filename>      output file to <filename>
  -t,  --timeout <seconds>      timeout of checking request in seconds
  -u,  --user-agent <agent>     identify as <agent>
  -r,  --referer <referer>      identify as <referer>
  --check-update                check if there is update available
  --trace                       display detail error messages

Pget vs Wget

URL: https://mirror.internet.asn.au/pub/ubuntu/releases/21.10/ubuntu-21.10-desktop-amd64.iso

Using

time wget https://mirror.internet.asn.au/pub/ubuntu/releases/21.10/ubuntu-21.10-desktop-amd64.iso
time pget -p 6 https://mirror.internet.asn.au/pub/ubuntu/releases/21.10/ubuntu-21.10-desktop-amd64.iso

Results

wget   3.92s user 23.52s system 3% cpu 13:35.24 total
pget -p 6   10.54s user 34.52s system 25% cpu 2:56.93 total

wget 13:35.24 total, pget -p 6 2:56.93 total (6x faster)

Binary

You can download from here

from https://github.com/Code-Hex/pget

具体例子：

yudeMacBook-Air:~ brite$ pget --procs 3 https://buildmedia.readthedocs.org/media/pdf/python-guide/latest/python-guide.pdf

4.66 MiB / 4.66 MiB [-----------------------------------] 100.00% 477.10 KiB p/s

binding with files...

4.66 MiB / 4.66 MiB [--------------------------------------------] 100.00% ? p/s

yudeMacBook-Air:~ brite$

---------------------------

on mac/linux desktop,

wget https://github.com/hellojukay/golang-multithreaded-download/raw/main/download.go

mv download.go go-downloader.go

go build go-downloader.go

(in the current dir, exec file go-downloader will be generated)

./go-downloader -h

it shows:

Usage of ./go-downloader:

  -o string

    filename 

  -url string

    http[s] link url

example:

./go-downloader -url https://buildmedia.readthedocs.org/media/pdf/python-guide/latest/python-guide.pdf

project url; https://github.com/hellojukay/golang-multithreaded-download