As I mentioned in my previous post, the open source DNS forwarder Dnsmasq is ideal for the DNS part of DNS unblocking. I’m running Dnsmasq on a $30 Raspberry Pi credit card sized mini computer which is up 24/7 anyway since it also handles all VOIP phone calls at home. I point my Mac, Apple TV and iPad to the RPi as the primary DNS server.
On the server side, I’ve setup a HAProxy instance using just a single IP address as a proof of concept. This poor-man’s approach works beautifully with SNI-capable devices like my Mac and iOS devices. I think newer Android devices are SNI-compatible as well but I haven’t tested it. Windows 7 and up should be OK too. Older devices like the Playstation 3 or Xbox 360 are most likely not SNI-compatible and won’t work with my highly cost-efficient single IP address approach. Unfortunately, even some of the newest multimedia players don’t support SNI.
On the server side, I’ve setup a HAProxy instance using just a single IP address as a proof of concept. This poor-man’s approach works beautifully with SNI-capable devices like my Mac and iOS devices. I think newer Android devices are SNI-compatible as well but I haven’t tested it. Windows 7 and up should be OK too. Older devices like the Playstation 3 or Xbox 360 are most likely not SNI-compatible and won’t work with my highly cost-efficient single IP address approach. Unfortunately, even some of the newest multimedia players don’t support SNI.
The HAProxy server is running on a lowend virtual private server in the U.S. As a starting point, feel free to use my proof of concept server as shown in the Dnsmasq configuration below. In the web browser, you should be able to watch Netflix, Hulu/HuluPlus, free episodes/TV shows on MTV, Disney XD, Syfy, NBC, ABC, Vevo, Crackle, PBS and CWTV. Netflix works on iPad and Apple TV too. HuluPlus could work on iOS as well.
Please remember, even though I’m planning to keep the HAProxy server up for some time to come, this is just a proof of concept and not a fully fledged DNS unblocking service.
On Debian-based Linux distributions, add the content below to a file named dnsmasq-catchall.conf in /etc/dnsmasq.d and it will get included by Dnsmasq. If Dnsmasq is running, i.e. on 192.168.178.99, you can test it using:
dig @192.168.178.99 trick77.com
This should bring up a few NS and A records for this site.
dig @192.168.178.99 abc.go.com
The result should be an A record for abc.go.com pointing to 199.204.184.146
/etc/dnsmasq.d/dnsmasq-catchall.conf:
001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
100
101
102
103
104
105
| address=/abc.go.com/199.204.184.146address=/api.watchdisneyxd.go.com/199.204.184.146address=/api.watchabc.go.com/199.204.184.146address=/release.theplatform.com/199.204.184.146address=/www.crackle.com/199.204.184.146address=/api.crackle.com/199.204.184.146address=/ios-api.crackle.com/199.204.184.146address=/appletv.crackle.com/199.204.184.146address=/ios-api-us.crackle.com/199.204.184.146address=/ios-api.crackle.com/199.204.184.146address=/android-api-us.crackle.com/199.204.184.146address=/xboxone-api-us.crackle.com/199.204.184.146address=/ps3-api-us.crackle.com/199.204.184.146address=/roku-api.crackle.com/199.204.184.146address=/content.uplynk.com/199.204.184.146address=/content-us-east-1.uplynk.com/199.204.184.146address=/www.crunchyroll.com/199.204.184.146address=/api.crunchyroll.com/199.204.184.146address=/static.discoverymedia.com/199.204.184.146address=/www.dramafever.com/199.204.184.146address=/token.dramafever.com/199.204.184.146address=/link.theplatform.com/199.204.184.146address=/s.hulu.com/199.204.184.146address=/play.hulu.com/199.204.184.146address=/www.iheart.com/199.204.184.146address=/www.last.fm/199.204.184.146address=/ws.audioscrobbler.com/199.204.184.146address=/ext.last.fm/199.204.184.146address=/www.logotv.com/199.204.184.146address=/activity.flux.com/199.204.184.146address=/j.maxmind.com/199.204.184.146address=/mog.com/199.204.184.146address=/www.mtv.com/199.204.184.146address=/c.brightcove.com/199.204.184.146address=/video.nbcuni.com/199.204.184.146address=/video.nbc.com/199.204.184.146address=/video.syfy.com/199.204.184.146address=/signup.netflix.com/199.204.184.146address=/www.netflix.com/199.204.184.146address=/appboot.netflix.com/199.204.184.146address=/cbp-us.nccp.netflix.com/199.204.184.146address=/a248.e.akamai.net/199.204.184.146address=/api-global.netflix.com/199.204.184.146address=/movies.netflix.com/199.204.184.146address=/movies1.netflix.com/199.204.184.146address=/secure.netflix.com/199.204.184.146address=/moviecontrol.netflix.com/199.204.184.146address=/api.netflix.com/199.204.184.146address=/api-us.netflix.com/199.204.184.146address=/uiboot.netflix.com/199.204.184.146address=/cbp.nccp.netflix.com/199.204.184.146address=/ios.nccp.netflix.com/199.204.184.146address=/xbox.nccp.netflix.com/199.204.184.146address=/nccp-nrdp-31.cloud.netflix.net/199.204.184.146address=/nintendo.nccp.netflix.com/199.204.184.146address=/playstation.nccp.netflix.com/199.204.184.146address=/nrdp.nccp.netflix.com/199.204.184.146address=/android.nccp.netflix.com/199.204.184.146address=/www.pandora.com/199.204.184.146address=/mediaserver-sv5-rt-1.pandora.com/199.204.184.146address=/tuner.pandora.com/199.204.184.146address=/urs.pbs.org/199.204.184.146address=/video.dl.playstation.net/199.204.184.146address=/api.wipmania.com/199.204.184.146address=/www.rdio.com/199.204.184.146address=/www.smithsonianchannel.com/199.204.184.146address=/once.unicornmedia.com/199.204.184.146address=/media.mtvnservices.com/199.204.184.146address=/www.spike.com/199.204.184.146address=/udat.mtvnservices.com/199.204.184.146address=/www.thewb.com/199.204.184.146address=/www.cwtv.com/199.204.184.146address=/media.cwtv.com/199.204.184.146address=/pdl.warnerbros.com/199.204.184.146address=/cdn.wwtv.warnerbros.com/199.204.184.146address=/www.vevo.com/199.204.184.146address=/sb.vevo.com/199.204.184.146address=/videoplayer.vevo.com/199.204.184.146address=/www.vh1.com/199.204.184.146address=/screen.yahoo.com/199.204.184.146address=/geo.yahoo.com/199.204.184.146address=/mvid.yql.yahoo.com/199.204.184.146address=/hls.video.query.yahoo.com/199.204.184.146server=/beta.abc.go.com/8.8.8.8server=/cdn.media.abc.go.com/8.8.8.8server=/site.abc.go.com/8.8.8.8server=/cdn.abc.go.com/8.8.8.8server=/cdn.media.abc.go.com/8.8.8.8server=/a.verdict.abc.go.com/8.8.8.8server=/theview.abc.go.com/8.8.8.8server=/adsatt.abc.go.com/8.8.8.8server=/ll.static.abc.go.com/8.8.8.8server=/static.east.abc.go.com/8.8.8.8server=/share.mog.com/8.8.8.8server=/logger.mog.com/8.8.8.8server=/search.mog.com/8.8.8.8server=/support.mog.com/8.8.8.8server=/www.mog.com/8.8.8.8server=/api.mog.com/8.8.8.8server=/api.au.mog.com/8.8.8.8server=/images0.mog.com/8.8.8.8server=/images1.mog.com/8.8.8.8server=/images2.mog.com/8.8.8.8server=/images3.mog.com/8.8.8.8server=/cdn2.mog.com/8.8.8.8 |
Once HAProxy is up and running, it will show a lot of statistics if you’ve enabled the built-in web interface:
Please let me know in the comments below once you have successfully set up your own DNS unblocking solution!
from https://trick77.com/dns-unblocking-using-dnsmasq-haproxy/
－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－－
Tunlr-style DNS unblocking for Pandora, Netflix, Hulu
Since Tunlr closed down unexpectedly this week, I decided to publish my ideas and findings on the subject of DNS unblocking. I used Tunlr for some time when I decided to develop my own, private DNS unblocking solution last year.
Why VPNs are no good for streaming
DNS unblocking refers to a technique used to circumvent geo-fenced Internet services without the use of a VPN. When we’re using a VPN to access geo-fenced websites, usually all our Internet traffic gets routed through a remote VPN server. With DNS unblocking, only selected traffic gets routed through a remote proxy server, ideally just the minimum traffic required to trick geo-fenced services like Pandora, Netflix or Hulu into “thinking” our current geolocation is within the United States (or any other country required to pass the geo-fence). One advantage is that DNS unblocking works for all devices that allow custom DNS settings while a VPN only works on a computer or in the router. But the big advantage over a VPN is that DNS unblocking allows the full and intended use of Content Delivery Networks (CDN).
Without going too far into the subject, CDNs usually rely on BGP Anycast or Geocast to find the closest destination server. Here’s a real life example for Anycast: if we ping Google’s DNS server (8.8.8.8), we will usually get a response within, let’s say 30ms or less, no matter where we are in the civilised world. This is because Google operates many DNS servers responding to the same 8.8.8.8 IP address, distributed all over the globe. The announced routes for 8.8.8.8 and the path vector protocol BGP make sure we’re getting the one closest (with the shortest AS path) to us. While using a VPN, we will get the Google DNS server closest to the remote VPN server, which can be far away on a different continent. Far away is bad for bandwidth, and bandwidth is important for high quality video streams!
If a CDN is using Geocast, you will get the destination server closest to the DNS server, not the one closest to your real location. If you’re resolving DNS queries through a (far away) DNS server, you’re essential killing the benefits of Geocast. This will wreak havoc on your download rates and increases latency for every DNS request. The latter applies to every scenario which includes a DNS server which is far away, including DNS unblocking. You should always use the DNS server with the lowest latency – which in almost every case is your provider’s DNS server.
On-demand Internet streaming providers like Netflix rely heavily on CDN technology to transport their video streams to the end user.
And then there is the HTTPS tunnelling problem
There are many ways to tunnel a HTTP connection through a proxy. We could use Nginx for instance, or Squid. Even Apache comes with a HTTP proxy module. However, it gets a bit more difficult once we have to tunnel a HTTPS connection without terminating the SSL certificate in the proxy. As of today, none of the previously mentioned software products are able to tunnel a HTTPS connection without SSL termination.
Another problem are IP addresses. In the old days, every SSL endpoint required a dedicated IP address. Thanks to Server Name Indication (SNI), a client is able to present the desired domain name to a server during the initial SSL handshake. Unfortunately though, SNI only works in more recent browser versions and just a few standalone multimedia devices, iOS devices being among them. If we want to tunnel non-SNI-capable devices through a HTTPS proxy, we will have to use a dedicated IP address for every SSL tunnel.
Let’s go back to the HTTPS tunnelling problem. There are a few solutions available in the open source marketplace but absolutely none of them come even close to HAProxy. HAProxy is the mother of all proxies. Among a myriad of other things, HAProxy is able to tunnel HTTPS connections, SNI-based or not, and it does this… wait for it……. without SSL termination! It will just passthrough any connection we throw at it. HAProxy is incredibly fast, unbelievably lightweight and very reliable. It’s so stable I’m even using snapshot versions from the development branch in production environments (YMMW, that’s just me).
Let’s use HAProxy for DNS unblocking!
Here’s a sample HAProxy configuration which includes support for Pandora, Netflix, Hulu, MTV, ABC and quite a few others. You can’t use it without modification. It’s best to start with the proxies you need and to throw away the parts you don’t need. I’m probably not going to maintain it on a regular basis but feel free to fork it on Github.
001
002
003
004
005
006
007
008
009
010
011
012
013
014
015
016
017
018
019
020
021
022
023
024
025
026
027
028
029
030
031
032
033
034
035
036
037
038
039
040
041
042
043
044
045
046
047
048
049
050
051
052
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
072
073
074
075
076
077
078
079
080
081
082
083
084
085
086
087
088
089
090
091
092
093
094
095
096
097
098
099
100
101
102
103
104
105
106
107
108
109
110
111
112
| # Check the HAProxy documentation for information about the configuration keywords.# Make sure to use (compile) the latest HAProxy version from the current development branch or some features may not work!# Please see https://trick77.com/2014/03/01/tunlr-style-dns-unblocking-pandora-netflix-hulu-et-al/ for more information.# *** THIS CONFIGURATION WILL NOT RUN WITHOUT PROPER MODIFICATION ***global daemon maxconn 20000 user haproxy group haproxy stats socket /var/run/haproxy.sock mode 0600 level admin log /dev/log local0 debug pidfile /var/run/haproxy.pid spread-checks 5defaults maxconn 19500 log global mode http option httplog option abortonclose option http-server-close option persist option accept-invalid-http-response timeout connect 20s timeout server 120s timeout client 120s timeout check 10s retries 3listen stats # Website with useful statistics about our HAProxy frontends and backends bind *:6969 mode http stats enable stats realm HAProxy stats uri / stats auth haproxy:secure_password_goes_here# SNI catchall ------------------------------------------------------------------------# We're trying to save as many IP addresses as possible that's why we're running as many backends as possible on one IP address.# Obviously, we're using SNI on the 443 frontend onlyfrontend f_sni_catchall mode http bind ip_address_1_here:80 log global option httplog option accept-invalid-http-request capture request header Host len 50 capture request header User-Agent len 150 #--- abc use_backend b_sni_catchall if { hdr(host) -i abc.go.com } use_backend b_sni_catchall if { hdr(host) -i api.watchdisneyxd.go.com } use_backend b_sni_catchall if { hdr(host) -i api.watchabc.go.com } #--- mylifetime use_backend b_sni_catchall if { hdr(host) -i c.brightcove.com } #--- cbs use_backend b_sni_catchall if { hdr(host) -i release.theplatform.com } #--- crackle use_backend b_sni_catchall if { hdr(host) -i www.crackle.com } use_backend b_sni_catchall if { hdr(host) -i api.crackle.com } use_backend b_sni_catchall if { hdr(host) -i ios-api.crackle.com } use_backend b_sni_catchall if { hdr(host) -i ios-api-us.crackle.com } use_backend b_sni_catchall if { hdr(host) -i appletv.crackle.com } use_backend b_sni_catchall if { hdr(host) -i android-api-us.crackle.com } use_backend b_sni_catchall if { hdr(host) -i xboxone-api-us.crackle.com } use_backend b_sni_catchall if { hdr(host) -i ps3-api-us.crackle.com } use_backend b_sni_catchall if { hdr(host) -i roku-api.crackle.com } use_backend b_sni_catchall if { hdr(host) -i content.uplynk.com } use_backend b_sni_catchall if { hdr(host) -i content-us-east-1.uplynk.com } #--- crunchyroll use_backend b_sni_catchall if { hdr(host) -i www.crunchyroll.com } use_backend b_sni_catchall if { hdr(host) -i api.crunchyroll.com } #--- discovery use_backend b_sni_catchall if { hdr(host) -i static.discoverymedia.com } #--- dramafever use_backend b_sni_catchall if { hdr(host) -i www.dramafever.com } use_backend b_sni_catchall if { hdr(host) -i token.dramafever.com } #--- fox use_backend b_sni_catchall if { hdr(host) -i link.theplatform.com } #--- hulu use_backend b_sni_catchall if { hdr(host) -i s.hulu.com } #--- iheart use_backend b_sni_catchall if { hdr(host) -i www.iheart.com } #--- last.fm use_backend b_sni_catchall if { hdr(host) -i www.last.fm } use_backend b_sni_catchall if { hdr(host) -i ws.audioscrobbler.com } use_backend b_sni_catchall if { hdr(host) -i ext.last.fm } #--- logotv use_backend b_sni_catchall if { hdr(host) -i www.logotv.com } use_backend b_sni_catchall if { hdr(host) -i activity.flux.com } #--- netflix use_backend b_sni_catchall if { hdr(host) -i www.netflix.com } use_backend b_sni_catchall if { hdr(host) -i appboot.netflix.com } use_backend b_sni_catchall if { hdr(host) -i cbp-us.nccp.netflix.com } use_backend b_sni_catchall if { hdr(host) -i a248.e.akamai.net } ... |
Please see the full configuration source on Github.
And where’s the DNS part in DNS unblocking?
I’m using Dnsmasq on my Rasperry Pi to “intercept” the domain names from my HAProxy configuration and forward all other DNS queries to my ISP’s DNS server. You could use BIND as a local caching DNS server as well but you would end up writing many DNS zone files. Dnsmasq is a lot easier to setup. I will publish a sample Dnsmasq configuration for DNS unblocking in a future post.
EDIT: Here it is.
EDIT: Here it is.
So you want to start a DNS unblocking company?
Good luck, you’re a bit late to the party. You will need DNS servers, lots of IP addresses, redundancy for everything, a way to deal with Akamai’s geo-protected transport streams (=lots of bandwidth!), a frontend for clients (WHMCS comes to mind) and many other things. Please do me and the Internet a favour and make sure your open DNS servers are rate-limited. All professionally operated, open (recursive) DNS server do have some sort of rate-limitation to make them less interesting (and harmful) in DNS amplification DDoS attacks. Hackers, or rather, script kiddies, permanentely scan the Internet for open, recursive DNS servers and they will find your DNS server within hours。
from https://trick77.com/tunlr-style-dns-unblocking-pandora-netflix-hulu-et-al/
