innernet

A private network system that uses WireGuard under the hood. See the announcement blog post for a longer-winded explanation.

innernet is similar in its goals to Slack's nebula or Tailscale, but takes a bit of a different approach. It aims to take advantage of existing networking concepts like CIDRs and the security properties of WireGuard to turn your computer's basic IP networking into more powerful ACL primitives.

innernet is not an official WireGuard project, and WireGuard is a registered trademark of Jason A. Donenfeld.

This has not received an independent security audit, and should be considered experimental software at this early point in its lifetime.

Usage

Server Creation

Every innernet network needs a coordination server to manage peers and provide endpoint information so peers can directly connect to each other. Create a new one with

sudo innernet-server new

The init wizard will ask you questions about your network and give you some reasonable defaults. It's good to familiarize yourself with network CIDRs as a lot of innernet's access control is based upon them. As an example, let's say the root CIDR for this network is 10.60.0.0/16. Server initialization creates a special "infra" CIDR which contains the innernet server itself and is reachable from all CIDRs on the network.

Next we'll also create a humans CIDR where we can start adding some peers.

sudo innernet-server add-cidr <interface>

For the parent CIDR, you can simply choose your network's root CIDR. The name will be humans, and the CIDR will be 10.60.64.0/24 (not a great example unless you only want to support 256 humans, but it works for now...).

By default, peers which exist in this new CIDR will only be able to contact peers in the same CIDR, and the special "infra" CIDR which was created when the server was initialized.

A typical workflow for creating a new network is to create an admin peer from the innernet-server CLI, and then continue using that admin peer via the innernet client CLI to add any further peers or network CIDRs.

sudo innernet-server add-peer <interface>

Select the humans CIDR, and the CLI will automatically suggest the next available IP address. Any name is fine, just answer "yes" when asked if you would like to make the peer an admin. The process of adding a peer results in an invitation file. This file contains just enough information for the new peer to contact the innernet server and redeem its invitation. It should be transferred securely to the new peer, and it can only be used once to initialize the peer.

You can run the server with innernet-server serve <interface>, or if you're on Linux and want to run it via systemctl, run systemctl enable --now innernet-server@<interface>. If you're on a home network, don't forget to configure port forwarding to the Listen Port you specified when creating the innernet server.

Peer Initialization

Let's assume the invitation file generated in the steps above have been transferred to the machine a network admin will be using.

You can initialize the client with

sudo innernet install /path/to/invitation.toml

You can customize the network name if you want to, or leave it at the default. innernet will then connect to the innernet server via WireGuard, generate a new key pair, and register that pair with the server. The private key in the invitation file can no longer be used.

If everything was successful, the new peer is on the network. You can run things like

sudo innernet list

or

sudo innernet list --tree

to view the current network and all CIDRs visible to this peer.

Since we created an admin peer, we can also add new peers and CIDRs from this peer via innernet instead of having to always run commands on the server.

Adding Associations between CIDRs

In order for peers from one CIDR to be able to contact peers in another CIDR, those two CIDRs must be "associated" with each other.

With the admin peer we created above, let's add a new CIDR for some theoretical CI servers we have.

sudo innernet add-cidr <interface>

The name is ci-servers and the CIDR is 10.60.64.0/24, but for this example it can be anything.

For now, we want peers in the humans CIDR to be able to access peers in the ci-servers CIDR.

sudo innernet add-association <interface>

The CLI will ask you to select the two CIDRs you want to associate. That's all it takes to allow peers in two different CIDRs to communicate!

You can verify the association with

sudo innernet list-associations <interface>

and associations can be deleted with

sudo innernet delete-associations <interface>

Enabling/Disabling Peers

For security reasons, IP addresses cannot be re-used by new peers, and therefore peers cannot be deleted. However, they can be disabled. Disabled peers will not show up in the list of peers when fetching the config for an interface.

Disable a peer with

sudo innernet disable-peer <interface>

Or re-enable a peer with

sudo innernet enable-peer <interface>

Specifying a Manual Endpoint

The innernet server will try to use the internet endpoint it sees from a peer so other peers can connect to that peer as well. This doesn't always work and you may want to set an endpoint explicitly. To set an endpoint, use

sudo innernet override-endpoint <interface>

You can go back to automatic endpoint discovery with

sudo innernet override-endpoint -u <interface>

Setting the Local WireGuard Listen Port

If you want to change the port which WireGuard listens on, use

sudo innernet set-listen-port <interface>

or unset the port and use a randomized port with

sudo innernet set-listen-port -u <interface>

Remove Network

To permanently uninstall a created network, use

sudo innernet-server uninstall <interface>

Use with care!

Security recommendations

If you're running a service on innernet, there are some important security considerations.

Enable strict Reverse Path Filtering (RFC 3704)

Strict RPF prevents packets from other interfaces from having internal source IP addresses. This is not the default on Linux, even though it is the right choice for 99.99% of situations. You can enable it by adding the following to a /etc/sysctl.d/60-network-security.conf:

net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1

Bind to the WireGuard device

If possible, to ensure that packets are only ever transmitted over the WireGuard interface, it's recommended that you use SO_BINDTODEVICE on Linux or IP_BOUND_IF on macOS/BSDs. If you have strict reverse path filtering, though, this is less of a concern.

IP addresses alone often aren't enough authentication

Even following all the above precautions, rogue applications on a peer's machines could be able to make requests on their behalf unless you add extra layers of authentication to mitigate this CSRF-type vector.

It's recommended that you carefully consider this possibility before deciding that the source IP is sufficient for your authentication needs on a service.

Installation

innernet has only officially been tested on Linux and MacOS, but we hope to support as many platforms as is feasible!

Runtime Dependencies

It's assumed that WireGuard is installed on your system, either via the kernel module in Linux 5.6 and later, or via the wireguard-go userspace implementation.

WireGuard Installation Instructions

Arch Linux

pacman -S innernet

Ubuntu

Fetch the appropriate .deb packages from https://github.com/tonarino/innernet/releases and install with

sudo apt install ./innernet*.deb

macOS

brew install tonarino/innernet/innernet

Cargo

# to install innernet:
cargo install --git https://github.com/tonarino/innernet --tag v1.5.4 client

# to install innernet-server:
cargo install --git https://github.com/tonarino/innernet --tag v1.5.4 server

Note that you'll be responsible for updating manually.

Development

innernet-server Build dependencies

• rustc / cargo (version 1.50.0 or higher)
• libclang (see more info at https://crates.io/crates/clang-sys)
• libsqlite3

Build:

cargo build --release --bin innernet-server

The resulting binary will be located at ./target/release/innernet-server

innernet Client CLI Build dependencies

• rustc / cargo (version 1.50.0 or higher)
• libclang (see more info at https://crates.io/crates/clang-sys)

Build:

cargo build --release --bin innernet

The resulting binary will be located at ./target/release/innernet

Releases

1. Run cargo release [--dry-run] [minor|major|patch|...] to automatically bump the crates appropriately.
2. Create a new git tag (ex. v0.6.0).
3. Push (with tags) to the repo.

innernet uses GitHub Actions to automatically produce a debian package for the releases page.

from https://github.com/tonarino/innernet

-----

Introducing 'innernet'

Nurture and shape your own private networks with simple, free, open-source infrastructure.

For the past year, all of the infrastructure at tonari — our installations, our laptops, our tools — have run on a single WireGuard network that's organized by an opinionated network manager we've been writing called innernet. Today we're happy to be open sourcing it.

In the beginning, we had a shared manually-edited WireGuard config file and many sighs were heard whenever we needed to add a new peer to the network.

In the middle ages, there were bash scripts and a weird Vault backend with questionable-at-best maintainability that got new machines on the network and coordinated things like IP allocation. Many groans could be heard whenever these flimsy scripts broke for any reason.

In the end, we decided to sit down, sigh one long and hopefully final time, and write innernet.

Think of innernet as an opinionated configuration system on top of WireGuard that comes with some added features to make life easy, and is friendly with various sizes of networks: one for your organization, one for your project, one for your social circle to create an idealistic alternate internet universe — your imagination's the limit.

We had some simple goals:

• Conveniences a typical WireGuard user wants: peer names, auto-updating peer lists, groups based on IP blocks, and automatic NAT holepunching.
• Free, open source, and made to be self-hosted. We think it's especially important for such a vital and low-level piece of our infrastructure to not be dependent on the livelihood of a company one has no control over.
• Straightforward architecture — no Raft consensus here. It's a simple SQLite server/client model.

Below, we'll explain how it works, then build a quick example network. The source is available as of today, so tinker away. We're pretty much a pure Rust house at this point, and this project is no exception!

The basics

innernet establishes three main primitives in defining your network:

• Peers: machines on the network,
• CIDRs: peer groups based on IP blocks, and
• Associations: connections between CIDRs for access control.

Peers: In this land, you are your IP

In a innernet network, peers are added or disabled by the creator of the network, or by other peers with administrative capabilities. Peers also always have only one assigned IP address, and that address is permanently associated with them. That is, a peer's IP address on the innernet network is permanent, unique, and immutable.

Unlike other network interfaces, in the WireGuard world, traffic from peers on your WireGuard interface are cryptographically authenticated and can be used to guarantee that traffic from a specific source IP is indeed from that peer.

However, it's important to note that this property alone does not make an IP alone a solid authentication method — there are a lot of critical security caveats in a world where various applications on your computer can make arbitrary network requests, and arbitrary source IPs can be forged via other network interfaces. We'll be working towards creating a set of recommendations for running services inside an innernet to document these caveats.

CIDRs: In an orchard of ripe IPs, make a CIDR

☝

CIDRs are a simple way of defining IP subnets by prefixing a /N at the end of an IP address, where N represents the number of prefixed "on" bits in the subnet mask. (eg. your home LAN's CIDR is something like 192.168.0.0/24, which is 192.168.0.0 to 192.168.0.255 — 0.0.0.0/0 means all IPv4 addresses)

CIDRs are the "groups" in innernet. A CIDR can either have other CIDRs as children, or peers. Every peer belongs to a CIDR.

CIDR notation is very useful because it makes it much easier to create a tree of these subnet groups that don't have overlapping addresses.

For example, here is the rough CIDR layout of tonari's innernet:

10.80.0.0/15 tonari
    10.80.1.0/24 infrastructure
    10.80.64.0/18 humans
        10.80.64.0/23 engineering
        10.80.66.0/23 non-engineering
		10.80.128.0/20 tooling
    10.81.0.0/16 installations
        10.81.0.0/24 straylight
        10.81.1.0/24 frontier
        10.81.2.0/24 mars

Why is this powerful? Well, what if you want a web service that gives access to everyone in your organization? Allow traffic from 10.80.64.0/18 using your web server or your OS's firewall. Want to give only engineers access to the administrative panel? Check if their IP is in the 10.80.64.0/23 range - if it is, they're an engineer.

Associations: who can talk to who

By default, peers can only see other peers in their same CIDR, as well as the server peer on its "infrastructure" CIDR.

To allow peers in one CIDR to communicate with peers in another, you can create associations between CIDRs. For example, we would create an association between the "engineering" CIDR and the "installations" CIDR in order to give engineers the ability to reach all the tonari installations. From there, of course, we can use tools like iptables on the peers to control the security in a more granular way.

So, with this concept, we have a fairly straightforward method of access control.

Getting started

Let's say we are starting a new company called Kermpany. We sell educational cakes and have raised one billion dollars in venture capital and now it's time to get serious. And to get serious, we'll need an innernet.

Innernet consists of two binaries: innernet-server, the coordinating server, and innernet (also aliased as inn for easy access), the client that peers run to access the network.

First, let's install innernet-server on a server we have sitting around, create a new network, and start it up:

sudo innernet-server new
# Network name: kermpany
# Network CIDR: 10.42.0.0/16
# External endpoint: [Enter]
# Listen port: 51820

sudo systemctl enable --now innernet-server@kermpany

Our network CIDR is 10.42.0.0/16, giving us 64k IP addresses to play with! It can be much bigger than this of course, and it can be IPv6 as well.

Next, let's add a CIDR for all the lovely humans we work with.

sudo innernet-server add-cidr kermpany
# Parent CIDR: kermpany
# Name: humans
# CIDR: 10.42.128.0/17
# Create CIDR "humans"? yes

We could now make further sub-CIDRs within humans like bakers or bosses, but we won't do that here. Here at Kermpany, every human is a baker boss in our eyes.

Instead, we'll get straight to creating our first peer, Ryo.

sudo innernet-server add-peer kermpany
# CIDR: humans (10.42.128.0/17)
# IP: [Enter]
# Name: ryo
# Make ryo an admin? yes
# Create peer ryo? yes

Then we can send the generated invite, ryo.toml, to Ryo using a convenient tool like Magic Wormhole. Invites are redeemed by the first person who uses it, so make sure the invite is only sent to the person you intend to join the network!

Now, Ryo has received their invite and can redeem it on their computer to join the network:

sudo inn install ./ryo.toml
# Interface name: [Enter]

sudo systemctl enable --now innernet@kermpany

Since they're an admin, Ryo can now invite peers with sudo inn add-peer kermpany!

All peers within the same CIDR can see each other, and Ryo could make another CIDR and associate it with "humans" to make them visible to each other with:

sudo inn add-association kermpany

Other peers can connect with Ryo by either using his IP or ryo.kermpany.wg.

And like that, the network was born. Kermpany ended up adding a ton more CIDRs and peers and became a huge success, baking the worlds most educational cakes ever.

Security, though

As a security-minded reader you are, by this point, inner-screaming "well that server now looks like a pretty nice target to compromise." In order to appease you, dear reader, we have decided to try to make the server as unattractive to attackers as possible without losing too much simplicity.

1. The server's only internet-exposed port is its WireGuard listening port. The juicy HTTP API only listens on its internal IP address. Thus, the attacker would need to 1) find a way on to the WireGuard network by compromising an existing peer, or 2) find a vulnerability with WireGuard itself.
2. Peers cache and pin the public key-IP pairs of other peers as they are seen, since (IP, public key) tuples for peers are defined to be unique and immutable on innernet. Thus, if a compromised server swapped out public keys for a peer, existing peers won't take kindly to it.
3. New peers join the network via invitations that contain a temporary WireGuard keypair generated for them by a peer with admin rights. This keypair gives them the ability to communicate with the server API, and invitees are then required to submit a new static keypair's public key to redeem the invite. Thus, the server does not know any of the private keys of peers.
4. The server only shows a given peer the connection information of peers that are in the same CIDR or associated CIDRs. Thus, a compromised peer would not be able to attack the entire network unless they compromised the server or an admin peer first.

To us, that felt like a reasonable amount of thus's to justify our architectural choices, given our specific threat model. We're looking forward to hearing how others intend to use innernet and how we can adapt it to suit a wider range of situations.

Innernet compared to...

Tailscale

Tailscale is much more polished, VC-funded, and requires talking to a closed-source backend that you can't run yourself.

Currently, all connected peers are assigned IPs somewhere in the 100.64.0.0/10 range, and ACL is based off of assignable tags rather than IP ranges, so special awareness of Tailscale features would need to be baked into your applications if you needed more granular awareness for access control or identity.

It uses WireGuard as its underlying VPN protocol (with a packaged userspace implementation, not the kernel module, though), which is great, and we love the usability work they're doing too — we just don't want to be dependent on an external proprietary service for our internal networking.

Nebula

Nebula is Slack's (open-source!) answer to this problem, and it uses a userspace client that implement a custom tunnel protocol based on Noise (which WireGuard is also based on). Implementing the tunnels in userspace adds a speed and latency hit compared to the WireGuard kernel module.

Nebula's ACL system is based off of tags, similar to Tailscale. Because of that, it requires much more firewall/security-type code to exist in their codebase and configuration files, rather than utilizing existing controls for your OS's networking stack.

Hey, thanks

Thanks for reading along! We hope you had a good time here and have a great time on the innernet. Please feel free to say hey or even just hi at hey@tonari.no — we've loved your feedback from previous posts so I don't even feel afraid this time to post this email address again.

Note

This project is not affiliated with the WireGuard project. WireGuard is a registered trademark of Jason A. Donenfeld.

Shout out to cbonsai for the beautiful bonsai cherry blossoms in the cover :).

Errata

1. An earlier revision of this blog post claimed that innernet IP addresses can be used in lieu of other authentication methods in internal services, since IP addresses over WireGuard interfaces are cryptographically authenticated. This is flat-out a bad idea without additional care and authentication, and I'd like to apologize for making that incorrect claim and thank those that brought up the issue up so kindly.
2. It turns out that Tailscale doesn't in fact take advantage of the WireGuard kernel module even if it exists, and always uses a userspace WireGuard implementation.

from https://blog.tonari.no/introducing-innernet

-----

https://news.ycombinator.com/item?id=26628285 , (Innernet: open source Rust based Tailscale's alternative )