Manipulate DNS records on various DNS providers in a standardized/agnostic way.
Introduction
Lexicon provides a way to manipulate DNS records on multiple DNS providers in a standardized way. Lexicon has a CLI but it can also be used as a python library.
Lexicon was designed to be used in automation, specifically letsencrypt.
Providers
Only DNS providers who have an API can be supported by
lexicon
.
The current supported providers are:
- Aliyun.com (docs)
- AuroraDNS (docs)
- AWS Route53 (docs)
- Azure DNS (docs)
- Cloudflare (docs)
- ClouDNS (docs)
- CloudXNS (docs)
- ConoHa (docs)
- Constellix (docs)
- DigitalOcean (docs)
- DNSimple (docs)
- DnsMadeEasy (docs)
- DNSPark (docs)
- DNSPod (docs)
- Dreamhost (docs)
- EasyDNS (docs)
- Easyname (docs)
- ExoScale (docs)
- Gandi (docs: RPC (old) / LiveAPI)
- Gehirn (docs)
- Glesys (docs)
- GoDaddy (docs)
- Google Cloud DNS (docs)
- Hover (docs)
- Hurricane Electric DNS (docs)
- Hetzner (docs)
- Infoblox (docs)
- Internet.bs (docs)
- INWX (docs)
- Linode (docs)
- Linode v4 (docs)
- LuaDNS (docs)
- Memset (docs)
- Namecheap (docs)
- Namesilo (docs)
- Netcup (docs)
- NFSN (NearlyFreeSpeech)
- NS1 (docs)
- OnApp (docs)
- Online
- OVH (docs)
- Plesk (docs)
- PointHQ (docs)
- PowerDNS (docs)
- Rackspace (docs)
- Rage4 (docs)
- Sakura Cloud by SAKURA Internet Inc. (docs)
- SafeDNS by UKFast (docs)
- SoftLayer (docs)
- Subreg (docs)
- Transip (docs)
- Vultr (docs)
- Yandex (docs)
- Zeit (docs)
- Zilore (docs)
- Zonomi (docs)
Potential providers are as follows. If you would like to contribute one, follow the CONTRIBUTING.md and then open a pull request.
- Azure DNS (docs)
- AHNames (docs)
DurableDNS (docs)Can't set TXT records- cyon.ch
- Dyn (docs)
💵 requires paid account - Dynu
- DirectAdmin
- EntryDNS (docs)
💵 requires paid account - FreeDNS (docs)
- Host Virtual DNS (docs)
💵 requires paid account - HostEurope
- Infoblox NIOS
- ironDNS (docs)
💵 requires paid account - ISPConfig
- InternetX autoDNS (docs)
- Knot DNS
- KingHost
- Liquidweb (docs)
💵 requires paid account - Loopia (docs)
💵 requires paid account - Mythic Beasts(docs)
- NFSN (NearlyFreeSpeech) (docs)
💵 requires paid account - RFC2136 (docs)
- Servercow (docs)
- selectel.com
- TELE3 (docs)
- UltraDNS (docs)
💵 requires paid account - UnoEuro API
- VSCALE
- WorldWideDns (docs)
💵 requires paid account - Zerigo (docs)
💵 requires paid account - Zoneedit (docs)
- Any others I missed
Setup
Warning: it is strongly advised with pip to install Lexicon in a Python virtual environment, in order to avoid interference between Python modules preinstalled on your system as OS packages and modules installed by pip (see https://docs.python-guide.org/dev/virtualenvs/).
To use lexicon as a CLI application, do the following:
pip install dns-lexicon
Some providers (like Route53 and TransIP) require additional dependencies. You can install provider specific dependencies separately:
pip install dns-lexicon[route53]
To install lexicon with the additional dependencies of every provider, do the following:
pip install dns-lexicon[full]
You can also install the latest version from the repository directly.
pip install git+https://github.com/AnalogJ/lexicon.git
and with Route 53 provider dependencies:
pip install git+https://github.com/AnalogJ/lexicon.git#egg=dns-lexicon[route53]
As an alternative you can also install Lexicon using the OS packages available for major Linux distributions (see
lexicon
or dns-lexicon
package in https://pkgs.org/download/lexicon).Usage
$ lexicon -h
usage: lexicon [-h] [--version] [--delegated DELEGATED]
{cloudflare,cloudxns,digitalocean,dnsimple,dnsmadeeasy,dnspark,dnspod,easydns,luadns,namesilo,nsone,pointhq,rage4,route53,vultr,yandex,zonomi}
...
Create, Update, Delete, List DNS entries
positional arguments:
{cloudflare,cloudxns,digitalocean,dnsimple,dnsmadeeasy,dnspark,dnspod,easydns,luadns,namesilo,nsone,pointhq,rage4,route53,vultr,yandex,zonomi}
specify the DNS provider to use
cloudflare cloudflare provider
cloudxns cloudxns provider
digitalocean digitalocean provider
...
rage4 rage4 provider
route53 route53 provider
vultr vultr provider
yandex yandex provider
zonomi zonomi provider
optional arguments:
-h, --help show this help message and exit
--version show the current version of lexicon
--delegated DELEGATED
specify the delegated domain
$ lexicon cloudflare -h
usage: lexicon cloudflare [-h] [--name NAME] [--content CONTENT] [--ttl TTL]
[--priority PRIORITY] [--identifier IDENTIFIER]
[--auth-username AUTH_USERNAME]
[--auth-token AUTH_TOKEN]
{create,list,update,delete} domain
{A,AAAA,CNAME,MX,NS,SPF,SOA,TXT,SRV,LOC}
positional arguments:
{create,list,update,delete}
specify the action to take
domain specify the domain, supports subdomains as well
{A,AAAA,CNAME,MX,NS,SPF,SOA,TXT,SRV,LOC}
specify the entry type
optional arguments:
-h, --help show this help message and exit
--name NAME specify the record name
--content CONTENT specify the record content
--ttl TTL specify the record time-to-live
--priority PRIORITY specify the record priority
--identifier IDENTIFIER
specify the record for update or delete actions
--auth-username AUTH_USERNAME
specify email address used to authenticate
--auth-token AUTH_TOKEN
specify token used authenticate
Using the lexicon CLI is pretty simple:
# setup provider environmental variables:
export LEXICON_CLOUDFLARE_USERNAME="myusername@example.com"
export LEXICON_CLOUDFLARE_TOKEN="cloudflare-api-token"
# list all TXT records on cloudflare
lexicon cloudflare list example.com TXT
# create a new TXT record on cloudflare
lexicon cloudflare create www.example.com TXT --name="_acme-challenge.www.example.com." --content="challenge token"
# delete a TXT record on cloudflare
lexicon cloudflare delete www.example.com TXT --name="_acme-challenge.www.example.com." --content="challenge token"
lexicon cloudflare delete www.example.com TXT --identifier="cloudflare record id"
Authentication
Most supported DNS services provide an API token, however each service implements authentication differently. Lexicon attempts to standardize authentication around the following CLI flags:
--auth-username
- For DNS services that require it, this is usually the account id or email address--auth-password
- For DNS services that do not provide an API token, this is usually the account password--auth-token
- This is the most common auth method, the API token provided by the DNS service
You can see all the
--auth-*
flags for a specific service by reading the DNS service specific help: lexicon cloudflare -h
Environmental Variables
Instead of providing Authentication information via the CLI, you can also specify them via Environmental Variables. Every DNS service and auth flag maps to an Environmental Variable as follows:
LEXICON_{DNS Provider Name}_{Auth Type}
So instead of specifying
--auth-username
and --auth-token
flags when calling lexicon cloudflare ...
, you could instead set the LEXICON_CLOUDFLARE_USERNAME
and LEXICON_CLOUDFLARE_TOKEN
environmental variables.Letsencrypt Instructions
Lexicon has an example dehydrated hook file that you can use for any supported provider. All you need to do is set the PROVIDER env variable.
PROVIDER=cloudflare dehydrated --cron --hook dehydrated.default.sh --challenge dns-01
Lexicon can also be used with Certbot and the included Certbot hook file (requires configuration).
TroubleShooting & Useful Tools
There is an included example Dockerfile that can be used to automatically generate certificates for your website.
ToDo list
- Create and Register a lexicon pip package.
- Write documentation on supported environmental variables.
- Wire up automated release packaging on PRs.
- Check for additional dns hosts with apis (from fog, dnsperf, libcloud)
- Get a list of Letsencrypt clients, and create hook files for them (letsencrypt clients)
from https://github.com/AnalogJ/lexicon
----
----
DNS as code - Tools for managing DNS across multiple providers
In the vein of infrastructure as code OctoDNS provides a set of tools & patterns that make it easy to manage your DNS records across multiple providers. The resulting config can live in a repository and be deployed just like the rest of your code, maintaining a clear history and using your existing review & workflow.
The architecture is pluggable and the tooling is flexible to make it applicable to a wide variety of use-cases. Effort has been made to make adding new providers as easy as possible. In the simple case that involves writing of a single
class
and a couple hundred lines of code, most of which is translating between the provider's schema and OctoDNS's. More on some of the ways we use it and how to go about extending it below and in the /docs directory.
It is similar to Netflix/denominator.
Table of Contents
- Getting started
- Supported providers
- Custom Sources and Providers
- Other Uses
- Contributing
- Getting help
- License
- Authors
Getting started
Workspace
Running through the following commands will install the latest release of OctoDNS and set up a place for your config files to live. To determine if provider specific requirements are necessary see the Supported providers table below.
$ mkdir dns
$ cd dns
$ virtualenv env
...
$ source env/bin/activate
$ pip install octodns
$ mkdir config
Config
We start by creating a config file to tell OctoDNS about our providers and the zone(s) we want it to manage. Below we're setting up a
YamlProvider
to source records from our config files and both a Route53Provider
and DynProvider
to serve as the targets for those records. You can have any number of zones set up and any number of sources of data and targets for records for each. You can also have multiple config files, that make use of separate accounts and each manage a distinct set of zones. A good example of this this might be ./config/staging.yaml
& ./config/production.yaml
. We'll focus on a config/production.yaml
.---
providers:
config:
class: octodns.provider.yaml.YamlProvider
directory: ./config
default_ttl: 3600
enforce_order: True
dyn:
class: octodns.provider.dyn.DynProvider
customer: 1234
username: 'username'
password: env/DYN_PASSWORD
route53:
class: octodns.provider.route53.Route53Provider
access_key_id: env/AWS_ACCESS_KEY_ID
secret_access_key: env/AWS_SECRET_ACCESS_KEY
zones:
example.com.:
sources:
- config
targets:
- dyn
- route53
class
is a special key that tells OctoDNS what python class should be loaded. Any other keys will be passed as configuration values to that provider. In general any sensitive or frequently rotated values should come from environmental variables. When OctoDNS sees a value that starts with env/
it will look for that value in the process's environment and pass the result along.
Further information can be found in the
docstring
of each source and provider class.
Now that we have something to tell OctoDNS about our providers & zones we need to tell it about or records. We'll keep it simple for now and just create a single
A
record at the top-level of the domain.config/example.com.yaml
---
'':
ttl: 60
type: A
values:
- 1.2.3.4
- 1.2.3.5
Further information can be found in Records Documentation.
Noop
We're ready to do a dry-run with our new setup to see what changes it would make. Since we're pretending here we'll act like there are no existing records for
example.com.
in our accounts on either provider.$ octodns-sync --config-file=./config/production.yaml
...
********************************************************************************
* example.com.
********************************************************************************
* route53 (Route53Provider)
* Create
* Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
* dyn (DynProvider)
* Create
* Summary: Creates=1, Updates=0, Deletes=0, Existing Records=0
********************************************************************************
...
There will be other logging information presented on the screen, but successful runs of sync will always end with a summary like the above for any providers & zones with changes. If there are no changes a message saying so will be printed instead. Above we're creating a new zone in both providers so they show the same change, but that doesn't always have to be the case. If to start one of them had a different state you would see the changes OctoDNS intends to make to sync them up.
Making changes
WARNING: OctoDNS assumes ownership of any domain you point it to. When you tell it to act it will do whatever is necessary to try and match up states including deleting any unexpected records. Be careful when playing around with OctoDNS. It's best to experiment with a fake zone or one without any data that matters until you're comfortable with the system.
Now it's time to tell OctoDNS to make things happen. We'll invoke it again with the same options and add a
--doit
on the end to tell it this time we actually want it to try and make the specified changes.$ octodns-sync --config-file=./config/production.yaml --doit
...
The output here would be the same as before with a few more log lines at the end as it makes the actual changes. After which the config in Route53 and Dyn should match what's in the yaml file.
Workflow
In the above case we manually ran OctoDNS from the command line. That works and it's better than heading into the provider GUIs and making changes by clicking around, but OctoDNS is designed to be run as part of a deploy process. The implementation details are well beyond the scope of this README, but here is an example of the workflow we use at GitHub. It follows the way GitHub itself is branch deployed.
The first step is to create a PR with your changes.
Assuming the code tests and config validation statuses are green the next step is to do a noop deploy and verify that the changes OctoDNS plans to make are the ones you expect.
After that comes a set of reviews. One from a teammate who should have full context on what you're trying to accomplish and visibility in to the changes you're making to do it. The other is from a member of the team here at GitHub that owns DNS, mostly as a sanity check and to make sure that best practices are being followed. As much of that as possible is baked into
octodns-validate
.
After the reviews it's time to branch deploy the change.
If that goes smoothly, you again see the expected changes, and verify them with
dig
and/or octodns-report
you're good to hit the merge button. If there are problems you can quickly do a .deploy dns/master
to go back to the previous state.Bootstrapping config files
Very few situations will involve starting with a blank slate which is why there's tooling built in to pull existing data out of providers into a matching config file.
$ octodns-dump --config-file=config/production.yaml --output-dir=tmp/ example.com. route53
2017-03-15T13:33:34 INFO Manager __init__: config_file=tmp/production.yaml
2017-03-15T13:33:34 INFO Manager dump: zone=example.com., sources=('route53',)
2017-03-15T13:33:36 INFO Route53Provider[route53] populate: found 64 records
2017-03-15T13:33:36 INFO YamlProvider[dump] plan: desired=example.com.
2017-03-15T13:33:36 INFO YamlProvider[dump] plan: Creates=64, Updates=0, Deletes=0, Existing Records=0
2017-03-15T13:33:36 INFO YamlProvider[dump] apply: making changes
The above command pulled the existing data out of Route53 and placed the results into
tmp/example.com.yaml
. That file can be inspected and moved into config/
to become the new source. If things are working as designed a subsequent noop sync should show zero changes.Supported providers
Provider | Requirements | Record Support | Dynamic/Geo Support | Notes |
---|---|---|---|---|
AzureProvider | azure-mgmt-dns | A, AAAA, CAA, CNAME, MX, NS, PTR, SRV, TXT | No | |
CloudflareProvider | A, AAAA, ALIAS, CAA, CNAME, MX, NS, SPF, SRV, TXT | No | CAA tags restricted | |
DigitalOceanProvider | A, AAAA, CAA, CNAME, MX, NS, TXT, SRV | No | CAA tags restricted | |
DnsMadeEasyProvider | A, AAAA, ALIAS (ANAME), CAA, CNAME, MX, NS, PTR, SPF, SRV, TXT | No | CAA tags restricted | |
DnsimpleProvider | All | No | CAA tags restricted | |
DynProvider | dyn | All | Both | |
EtcHostsProvider | A, AAAA, ALIAS, CNAME | No | ||
GoogleCloudProvider | google-cloud-dns | A, AAAA, CAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, TXT | No | |
Ns1Provider | nsone | All | Partial Geo | No health checking for GeoDNS |
OVH | ovh | A, AAAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, SSHFP, TXT, DKIM | No | |
PowerDnsProvider | All | No | ||
Rackspace | A, AAAA, ALIAS, CNAME, MX, NS, PTR, SPF, TXT | No | ||
Route53 | boto3 | A, AAAA, CAA, CNAME, MX, NAPTR, NS, PTR, SPF, SRV, TXT | Both | CNAME health checks don't support a Host header |
AxfrSource | A, AAAA, CNAME, MX, NS, PTR, SPF, SRV, TXT | No | read-only | |
ZoneFileSource | A, AAAA, CNAME, MX, NS, PTR, SPF, SRV, TXT | No | read-only | |
TinyDnsFileSource | A, CNAME, MX, NS, PTR | No | read-only | |
YamlProvider | All | Yes | config |
Notes
- ALIAS support varies a lot from provider to provider care should be taken to verify that your needs are met in detail.
- Dyn's UI doesn't allow editing or view of TTL, but the API accepts and stores the value provided, this value does not appear to be used when served
- Dnsimple's uses the configured TTL when serving things through the ALIAS, there's also a secondary TXT record created alongside the ALIAS that octoDNS ignores
- octoDNS itself supports non-ASCII character sets, but in testing Cloudflare is the only provider where that is currently functional end-to-end. Others have failures either in the client libraries or API calls
Custom Sources and Providers
You can check out the source and provider directory to see what's currently supported. Sources act as a source of record information. AxfrSource and TinyDnsFileSource are currently the only OSS sources, though we have several others internally that are specific to our environment. These include something to pull host data from gPanel and a similar provider that sources information about our network gear to create both
A
& PTR
records for their interfaces. Things that might make good OSS sources might include an ElbSource
that pulls information about AWS Elastic Load Balancersand dynamically creates CNAME
s for them, or Ec2Source
that pulls instance information so that records can be created for hosts similar to how our GPanelProvider
works.
Most of the things included in OctoDNS are providers, the obvious difference being that they can serve as both sources and targets of data. We'd really like to see this list grow over time so if you use an unsupported provider then PRs are welcome. The existing providers should serve as reasonable examples. Those that have no GeoDNS support are relatively straightforward. Unfortunately most of the APIs involved to do GeoDNS style traffic management are complex and somewhat inconsistent so adding support for that function would be nice, but is optional and best done in a separate pass.
The
class
key in the providers config section can be used to point to arbitrary classes in the python path so internal or 3rd party providers can easily be included with no coordination beyond getting them into PYTHONPATH, most likely installed into the virtualenv with OctoDNS.Other Uses
Syncing between providers
While the primary use-case is to sync a set of yaml config files up to one or more DNS providers, OctoDNS has been built in such a way that you can easily source and target things arbitrarily. As a quick example the config below would sync
githubtest.net.
from Route53 to Dyn.---
providers:
route53:
class: octodns.provider.route53.Route53Provider
access_key_id: env/AWS_ACCESS_KEY_ID
secret_access_key: env/AWS_SECRET_ACCESS_KEY
dyn:
class: octodns.provider.dyn.DynProvider
customer: env/DYN_CUSTOMER
username: env/DYN_USERNAME
password: env/DYN_PASSWORD
zones:
githubtest.net.:
sources:
- route53
targets:
- dyn
Dynamic sources
Internally we use custom sources to create records based on dynamic data that changes frequently without direct human intervention. An example of that might look something like the following. For hosts this mechanism is janitorial, run periodically, making sure the correct records exist as long as the host is alive and ensuring they are removed after the host is destroyed. The host provisioning and destruction processes do the actual work to create and destroy the records.
---
providers:
gpanel-site:
class: github.octodns.source.gpanel.GPanelProvider
host: 'gpanel.site.github.foo'
token: env/GPANEL_SITE_TOKEN
powerdns-site:
class: octodns.provider.powerdns.PowerDnsProvider
host: 'internal-dns.site.github.foo'
api_key: env/POWERDNS_SITE_API_KEY
zones:
hosts.site.github.foo.:
sources:
- gpanel-site
targets:
- powerdns-site
from https://github.com/github/octodns
No comments:
Post a Comment