fbox

 

fbox is an easy to use and maintain distributed filesystem written in Go using Reed Solomon Erasure with FUSE and S3 interfaces and a modern Web UI.

https://prologic.github.io/fbox/

  

    

fbox is a distributed file system written in Go that has the the following features:

Current Features:

• Single portable binary
• Simple to setup and maintain
• A Web Interface with Drag 'n Drop
• Data redundancy with erasure coding and sharding

Planned Features:

• POSIX compatible FUSE interface
• S3 Object Storage interface
• Docker Volume Driver

There is also a publicly (freely) available demo instance available at:

• https://files.mills.io/

NOTE: I, James Mills, run this instance on pretty cheap hardware on a limited budget. Please use it fairly so everyone can enjoy using it equally!

Sponsor this project to support the development of new features, improving existings ones and fix bugs!

Table of Contents

• fbox - Files in a Box
• Table of Contents
  • Why?
  • Getting Started
    • Install from Releases
    • Install from Homebrew
    • Install from Source
  • Usage
    • Prepare storage
    • Start a master node
    • Start some data nodes
  • Production Deployments
    • Docker Swarm
  • In the News
  • Sponsor
  • Contributing
  • Contributors
  • Stargazers over time
  • Attributions and Prior Art
  • License

Created by gh-md-toc

Why?

fbox was written primarily and firstly as en education exercise in distributed file systems and a lot of time add effort went into understanding how many distributed file systems in the wild are designed.

Most other distributed file systems are either complex and hard to set up and maintain or come with really expensive license feeds. Therefore fbox is also an attempt at designing and implementing a distributed file system that is free and open source with the reliability, scale and durability of other distributed file systems.

Getting Started

Install from Releases

You can install fbox by simply downloading the latest version from the Release page for your platform and placing the binary in your $PATH.

For convenience you can run one of the following shell pipelines which will download and install the latest release binary into /usr/local/bin (modify to suit):

For Linux x86_64:

curl -s https://api.github.com/repos/prologic/fbox/releases/latest | grep browser_download_url | grep Linux_x86_64 | cut -d '"' -f 4 | wget -q -O - -i - | tar -zxv fbox && mv fbox /usr/local/bin/fbox

For macOS x86_64:

curl -s https://api.github.com/repos/prologic/fbox/releases/latest | grep browser_download_url | grep Darwin_x86_64 | cut -d '"' -f 4 | wget -q -O - -i - | tar -zxv fbox && mv fbox /usr/local/bin/fbox

Install from Homebrew

On macOS you can install fbox using Homebrew:

brew tap prologic/fbox
brew install fbox

Install from Source

To install fbox from source you can run go get directly if you have a Go environment setup:

go get github.com/prologic/fbox

NOTE: Be sure to have $GOBIN (if not empty) or your $GOPATH/bin in your $PATH. See Compile and install packages and dependencies

Or grab the source code and build:

git clone https://github.com/prologic/fbox.git
cd fbox
make build

And optionally run make install to place the binary fbox in your $GOBIN or $GOPATH/bin (again see note above).

Usage

Prepare storage

mkdir data1 data2 data3

Start a master node

fbox -a 127.0.0.1:8000 -b :8000 -d ./data1

Start some data nodes

fbox -a 127.0.0.1:8001 -b :8001 -d ./data2 -m http://127.0.0.1:8000
fbox -a 127.0.0.1:8002 -b :8002 -d ./data3 -m http://127.0.0.1:8000

This will set up a 3-node cluster on your local machine (for testing only).

You can open the Web UI by navigating to:

http://127.0.0.1:8000/ui/

You can also use the command-line client fbox itself; See usage:

$ ./fbox --help
Usage: fbox [options] [command [arguments]]

fbox is a simple distributed file system...

Valid commands:
 - cat <name> -- Downloads the given file given by <name> to stdout
 - put <name> -- Upload the given file given by <name>

Valid options:
  -a, --advertise-addr string   [interface]:port to advertise
  -b, --bind string             [interface]:port to bind to (default "0.0.0.0:8000")
  -d, --data string             path to store data in (default "./data")
  -s, --data-shards int         no. of data shards (default 3)
  -D, --debug                   enable debug logging
  -m, --master string           address:port of master to join (empty if master)
  -p, --parity-shards int       no. of parity shards (default 1)
  -u, --store string            blob store uri (default "disk://data")
  -V, --version                 display version information and exit
pflag: help requested

For example to store a file:

echo "Hello World" > hello.txt
fbox -m http://127.0.0.1:8000 put hello.txt

And to retrieve the file:

fbox -m http://127.0.0.1:8000 cat hello.txt

Production Deployments

Docker Swarm

You can deploy fbox to a Docker Swarm cluster by utilising the provided fbox.yml Docker Stack. This also depends on and uses the Traefik ingress load balancer so you must also have that configured and running in your cluster appropriately.

export DOMAIN=files.yourdomain.tld
docker stack deploy -c fbox.yml fbox

from https://github.com/prologic/fbox