As I mentioned in my previous post, the open source DNS forwarder Dnsmasq is ideal for the DNS part of DNS unblocking. I’m running Dnsmasq on a $30 Raspberry Pi credit card sized mini computer which is up 24/7 anyway since it also handles all VOIP phone calls at home. I point my Mac, Apple TV and iPad to the RPi as the primary DNS server.
On the server side, I’ve setup a HAProxy instance using just a single IP address as a proof of concept. This poor-man’s approach works beautifully with SNI-capable devices like my Mac and iOS devices. I think newer Android devices are SNI-compatible as well but I haven’t tested it. Windows 7 and up should be OK too. Older devices like the Playstation 3 or Xbox 360 are most likely not SNI-compatible and won’t work with my highly cost-efficient single IP address approach. Unfortunately, even some of the newest multimedia players don’t support SNI.

The HAProxy server is running on a lowend virtual private server in the U.S. As a starting point, feel free to use my proof of concept server as shown in the Dnsmasq configuration below. In the web browser, you should be able to watch Netflix, Hulu/HuluPlus, free episodes/TV shows on MTV, Disney XD, Syfy, NBC, ABC, Vevo, Crackle, PBS and CWTV. Netflix works on iPad and Apple TV too. HuluPlus could work on iOS as well.

Please remember, even though I’m planning to keep the HAProxy server up for some time to come, this is just a proof of concept and not a fully fledged DNS unblocking service.

On Debian-based Linux distributions, add the content below to a file named dnsmasq-catchall.conf in /etc/dnsmasq.d and it will get included by Dnsmasq. If Dnsmasq is running, i.e. on 192.168.178.99, you can test it using:

dig @192.168.178.99 trick77.com

This should bring up a few NS and A records for this site.

dig @192.168.178.99 abc.go.com

The result should be an A record for abc.go.com pointing to 199.204.184.146

/etc/dnsmasq.d/dnsmasq-catchall.conf:

address=/abc.go.com/199.204.184.146

address=/api.watchdisneyxd.go.com/199.204.184.146

address=/api.watchabc.go.com/199.204.184.146

address=/release.theplatform.com/199.204.184.146

address=/www.crackle.com/199.204.184.146

address=/api.crackle.com/199.204.184.146

address=/ios-api.crackle.com/199.204.184.146

address=/appletv.crackle.com/199.204.184.146

address=/ios-api-us.crackle.com/199.204.184.146

address=/ios-api.crackle.com/199.204.184.146

address=/android-api-us.crackle.com/199.204.184.146

address=/xboxone-api-us.crackle.com/199.204.184.146

address=/ps3-api-us.crackle.com/199.204.184.146

address=/roku-api.crackle.com/199.204.184.146

address=/content.uplynk.com/199.204.184.146

address=/content-us-east-1.uplynk.com/199.204.184.146

address=/www.crunchyroll.com/199.204.184.146

address=/api.crunchyroll.com/199.204.184.146

address=/static.discoverymedia.com/199.204.184.146

address=/www.dramafever.com/199.204.184.146

address=/token.dramafever.com/199.204.184.146

address=/link.theplatform.com/199.204.184.146

address=/s.hulu.com/199.204.184.146

address=/play.hulu.com/199.204.184.146

address=/www.iheart.com/199.204.184.146

address=/www.last.fm/199.204.184.146

address=/ws.audioscrobbler.com/199.204.184.146

address=/ext.last.fm/199.204.184.146

address=/www.logotv.com/199.204.184.146

address=/activity.flux.com/199.204.184.146

address=/j.maxmind.com/199.204.184.146

address=/mog.com/199.204.184.146

address=/www.mtv.com/199.204.184.146

address=/c.brightcove.com/199.204.184.146

address=/video.nbcuni.com/199.204.184.146

address=/video.nbc.com/199.204.184.146

address=/video.syfy.com/199.204.184.146

address=/signup.netflix.com/199.204.184.146

address=/www.netflix.com/199.204.184.146

address=/appboot.netflix.com/199.204.184.146

address=/cbp-us.nccp.netflix.com/199.204.184.146

address=/a248.e.akamai.net/199.204.184.146

address=/api-global.netflix.com/199.204.184.146

address=/movies.netflix.com/199.204.184.146

address=/movies1.netflix.com/199.204.184.146

address=/secure.netflix.com/199.204.184.146

address=/moviecontrol.netflix.com/199.204.184.146

address=/api.netflix.com/199.204.184.146

address=/api-us.netflix.com/199.204.184.146

address=/uiboot.netflix.com/199.204.184.146

address=/cbp.nccp.netflix.com/199.204.184.146

address=/ios.nccp.netflix.com/199.204.184.146

address=/xbox.nccp.netflix.com/199.204.184.146

address=/nccp-nrdp-31.cloud.netflix.net/199.204.184.146

address=/nintendo.nccp.netflix.com/199.204.184.146

address=/playstation.nccp.netflix.com/199.204.184.146

address=/nrdp.nccp.netflix.com/199.204.184.146

address=/android.nccp.netflix.com/199.204.184.146

address=/www.pandora.com/199.204.184.146

address=/mediaserver-sv5-rt-1.pandora.com/199.204.184.146

address=/tuner.pandora.com/199.204.184.146

address=/urs.pbs.org/199.204.184.146

address=/video.dl.playstation.net/199.204.184.146

address=/api.wipmania.com/199.204.184.146

address=/www.rdio.com/199.204.184.146

address=/www.smithsonianchannel.com/199.204.184.146

address=/once.unicornmedia.com/199.204.184.146

address=/media.mtvnservices.com/199.204.184.146

address=/www.spike.com/199.204.184.146

address=/udat.mtvnservices.com/199.204.184.146

address=/www.thewb.com/199.204.184.146

address=/www.cwtv.com/199.204.184.146

address=/media.cwtv.com/199.204.184.146

address=/pdl.warnerbros.com/199.204.184.146

address=/cdn.wwtv.warnerbros.com/199.204.184.146

address=/www.vevo.com/199.204.184.146

address=/sb.vevo.com/199.204.184.146

address=/videoplayer.vevo.com/199.204.184.146

address=/www.vh1.com/199.204.184.146

address=/screen.yahoo.com/199.204.184.146

address=/geo.yahoo.com/199.204.184.146

address=/mvid.yql.yahoo.com/199.204.184.146

address=/hls.video.query.yahoo.com/199.204.184.146

server=/beta.abc.go.com/8.8.8.8

server=/cdn.media.abc.go.com/8.8.8.8

server=/site.abc.go.com/8.8.8.8

server=/cdn.abc.go.com/8.8.8.8

server=/cdn.media.abc.go.com/8.8.8.8

server=/a.verdict.abc.go.com/8.8.8.8

server=/theview.abc.go.com/8.8.8.8

server=/adsatt.abc.go.com/8.8.8.8

server=/ll.static.abc.go.com/8.8.8.8

server=/static.east.abc.go.com/8.8.8.8

server=/share.mog.com/8.8.8.8

server=/logger.mog.com/8.8.8.8

server=/search.mog.com/8.8.8.8

server=/support.mog.com/8.8.8.8

server=/www.mog.com/8.8.8.8

server=/api.mog.com/8.8.8.8

server=/api.au.mog.com/8.8.8.8

server=/images0.mog.com/8.8.8.8

server=/images1.mog.com/8.8.8.8

server=/images2.mog.com/8.8.8.8

server=/images3.mog.com/8.8.8.8

server=/cdn2.mog.com/8.8.8.8

Once HAProxy is up and running, it will show a lot of statistics if you’ve enabled the built-in web interface:

Please let me know in the comments below once you have successfully set up your own DNS unblocking solution!

from https://trick77.com/dns-unblocking-using-dnsmasq-haproxy/

－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－

Tunlr-style DNS unblocking for Pandora, Netflix, Hulu

Since Tunlr closed down unexpectedly this week, I decided to publish my ideas and findings on the subject of DNS unblocking. I used Tunlr for some time when I decided to develop my own, private DNS unblocking solution last year.

Why VPNs are no good for streaming

DNS unblocking refers to a technique used to circumvent geo-fenced Internet services without the use of a VPN. When we’re using a VPN to access geo-fenced websites, usually all our Internet traffic gets routed through a remote VPN server. With DNS unblocking, only selected traffic gets routed through a remote proxy server, ideally just the minimum traffic required to trick geo-fenced services like Pandora, Netflix or Hulu into “thinking” our current geolocation is within the United States (or any other country required to pass the geo-fence). One advantage is that DNS unblocking works for all devices that allow custom DNS settings while a VPN only works on a computer or in the router. But the big advantage over a VPN is that DNS unblocking allows the full and intended use of Content Delivery Networks (CDN).

Without going too far into the subject, CDNs usually rely on BGP Anycast or Geocast to find the closest destination server. Here’s a real life example for Anycast: if we ping Google’s DNS server (8.8.8.8), we will usually get a response within, let’s say 30ms or less, no matter where we are in the civilised world. This is because Google operates many DNS servers responding to the same 8.8.8.8 IP address, distributed all over the globe. The announced routes for 8.8.8.8 and the path vector protocol BGP make sure we’re getting the one closest (with the shortest AS path) to us. While using a VPN, we will get the Google DNS server closest to the remote VPN server, which can be far away on a different continent. Far away is bad for bandwidth, and bandwidth is important for high quality video streams!

If a CDN is using Geocast, you will get the destination server closest to the DNS server, not the one closest to your real location. If you’re resolving DNS queries through a (far away) DNS server, you’re essential killing the benefits of Geocast. This will wreak havoc on your download rates and increases latency for every DNS request. The latter applies to every scenario which includes a DNS server which is far away, including DNS unblocking. You should always use the DNS server with the lowest latency – which in almost every case is your provider’s DNS server.

On-demand Internet streaming providers like Netflix rely heavily on CDN technology to transport their video streams to the end user.

And then there is the HTTPS tunnelling problem

There are many ways to tunnel a HTTP connection through a proxy. We could use Nginx for instance, or Squid. Even Apache comes with a HTTP proxy module. However, it gets a bit more difficult once we have to tunnel a HTTPS connection without terminating the SSL certificate in the proxy. As of today, none of the previously mentioned software products are able to tunnel a HTTPS connection without SSL termination.

Another problem are IP addresses. In the old days, every SSL endpoint required a dedicated IP address. Thanks to Server Name Indication (SNI), a client is able to present the desired domain name to a server during the initial SSL handshake. Unfortunately though, SNI only works in more recent browser versions and just a few standalone multimedia devices, iOS devices being among them. If we want to tunnel non-SNI-capable devices through a HTTPS proxy, we will have to use a dedicated IP address for every SSL tunnel.

Let’s go back to the HTTPS tunnelling problem. There are a few solutions available in the open source marketplace but absolutely none of them come even close to HAProxy. HAProxy is the mother of all proxies. Among a myriad of other things, HAProxy is able to tunnel HTTPS connections, SNI-based or not, and it does this… wait for it……. without SSL termination! It will just passthrough any connection we throw at it. HAProxy is incredibly fast, unbelievably lightweight and very reliable. It’s so stable I’m even using snapshot versions from the development branch in production environments (YMMW, that’s just me).

Let’s use HAProxy for DNS unblocking!

Here’s a sample HAProxy configuration which includes support for Pandora, Netflix, Hulu, MTV, ABC and quite a few others. You can’t use it without modification. It’s best to start with the proxies you need and to throw away the parts you don’t need. I’m probably not going to maintain it on a regular basis but feel free to fork it on Github.

# Check the HAProxy documentation for information about the configuration keywords.

# Make sure to use (compile) the latest HAProxy version from the current development branch or some features may not work!

# Please see https://trick77.com/2014/03/01/tunlr-style-dns-unblocking-pandora-netflix-hulu-et-al/ for more information.

# *** THIS CONFIGURATION WILL NOT RUN WITHOUT PROPER MODIFICATION ***

global

  daemon

  maxconn 20000

  user haproxy

  group haproxy

  stats socket /var/run/haproxy.sock mode 0600 level admin

  log /dev/log  local0 debug

  pidfile /var/run/haproxy.pid

  spread-checks 5

defaults

  maxconn 19500

  log global

  mode http

  option httplog

  option abortonclose

  option http-server-close

  option persist

  option accept-invalid-http-response

  timeout connect 20s

  timeout server 120s

  timeout client 120s

  timeout check 10s

  retries 3

listen stats    # Website with useful statistics about our HAProxy frontends and backends

  bind *:6969

  mode http

  stats enable

  stats realm HAProxy

  stats uri /

  stats auth haproxy:secure_password_goes_here

# SNI catchall ------------------------------------------------------------------------

# We're trying to save as many IP addresses as possible that's why we're running as many backends as possible on one IP address.

# Obviously, we're using SNI on the 443 frontend only

frontend f_sni_catchall

  mode http

  bind ip_address_1_here:80

  log global

  option httplog

  option accept-invalid-http-request

  capture request  header Host len 50

  capture request  header User-Agent len 150

  #--- abc

  use_backend b_sni_catchall     if { hdr(host) -i abc.go.com }

  use_backend b_sni_catchall     if { hdr(host) -i api.watchdisneyxd.go.com }

  use_backend b_sni_catchall     if { hdr(host) -i api.watchabc.go.com }

  #--- mylifetime

  use_backend b_sni_catchall     if { hdr(host) -i c.brightcove.com }

  #--- cbs

  use_backend b_sni_catchall     if { hdr(host) -i release.theplatform.com }

  #--- crackle

  use_backend b_sni_catchall     if { hdr(host) -i www.crackle.com }

  use_backend b_sni_catchall     if { hdr(host) -i api.crackle.com }

  use_backend b_sni_catchall     if { hdr(host) -i ios-api.crackle.com }

  use_backend b_sni_catchall     if { hdr(host) -i ios-api-us.crackle.com }

  use_backend b_sni_catchall     if { hdr(host) -i appletv.crackle.com }

  use_backend b_sni_catchall     if { hdr(host) -i android-api-us.crackle.com }

  use_backend b_sni_catchall     if { hdr(host) -i xboxone-api-us.crackle.com }

  use_backend b_sni_catchall     if { hdr(host) -i ps3-api-us.crackle.com }

  use_backend b_sni_catchall     if { hdr(host) -i roku-api.crackle.com }

  use_backend b_sni_catchall     if { hdr(host) -i content.uplynk.com }

  use_backend b_sni_catchall     if { hdr(host) -i content-us-east-1.uplynk.com }

  #--- crunchyroll

  use_backend b_sni_catchall     if { hdr(host) -i www.crunchyroll.com }

  use_backend b_sni_catchall     if { hdr(host) -i api.crunchyroll.com }

  #--- discovery

  use_backend b_sni_catchall     if { hdr(host) -i static.discoverymedia.com }

  #--- dramafever

  use_backend b_sni_catchall     if { hdr(host) -i www.dramafever.com }

  use_backend b_sni_catchall     if { hdr(host) -i token.dramafever.com }

  #--- fox

  use_backend b_sni_catchall     if { hdr(host) -i link.theplatform.com }

  #--- hulu

  use_backend b_sni_catchall     if { hdr(host) -i s.hulu.com }

  #--- iheart

  use_backend b_sni_catchall     if { hdr(host) -i www.iheart.com }

  #--- last.fm

  use_backend b_sni_catchall     if { hdr(host) -i www.last.fm }

  use_backend b_sni_catchall     if { hdr(host) -i ws.audioscrobbler.com }

  use_backend b_sni_catchall     if { hdr(host) -i ext.last.fm }

  #--- logotv

  use_backend b_sni_catchall     if { hdr(host) -i www.logotv.com }

  use_backend b_sni_catchall     if { hdr(host) -i activity.flux.com }

  #--- netflix 

  use_backend b_sni_catchall     if { hdr(host) -i www.netflix.com }

  use_backend b_sni_catchall     if { hdr(host) -i appboot.netflix.com }

  use_backend b_sni_catchall     if { hdr(host) -i cbp-us.nccp.netflix.com }

  use_backend b_sni_catchall     if { hdr(host) -i a248.e.akamai.net }

  ...

Please see the full configuration source on Github.

And where’s the DNS part in DNS unblocking?

I’m using Dnsmasq on my Rasperry Pi to “intercept” the domain names from my HAProxy configuration and forward all other DNS queries to my ISP’s DNS server. You could use BIND as a local caching DNS server as well but you would end up writing many DNS zone files. Dnsmasq is a lot easier to setup. I will publish a sample Dnsmasq configuration for DNS unblocking in a future post.
EDIT: Here it is.

So you want to start a DNS unblocking company?

Good luck, you’re a bit late to the party. You will need DNS servers, lots of IP addresses, redundancy for everything, a way to deal with Akamai’s geo-protected transport streams (=lots of bandwidth!), a frontend for clients (WHMCS comes to mind) and many other things. Please do me and the Internet a favour and make sure your open DNS servers are rate-limited. All professionally operated, open (recursive) DNS server do have some sort of rate-limitation to make them less interesting (and harmful) in DNS amplification DDoS attacks. Hackers, or rather, script kiddies, permanentely scan the Internet for open, recursive DNS servers and they will find your DNS server within hours。

from https://trick77.com/tunlr-style-dns-unblocking-pandora-netflix-hulu-et-al/

人生在世，看得透，又看得远者prevail everywhere. v.gd/YLHADG v.gd/cRqvfD v.gd/sRm3tA

Pages

Wednesday, 28 October 2015

使用dnsmasq和haproxy解决dns污染问题