Doublehops

In November last year I again updated and hosted the website for the NAB Australian Football League Draft Tracker and flew up to the Gold Coast to sit in on the event to ensure it all went well. The website (http://drafttracker.afl.com.au/) was built as a live tracker so the public can watch the picks as they happen.

It was designed to be lightweight to both server and browser so that any client requests only pulled in all site assets on the initial page load and then tiny JSON files every ten seconds looking for updates [1].  Adding drafted players as they happened by the admin updated records in the database via PHP which would create static files (JSON) for clients to pull down to update the page. NGINX was used as the webserver. All this allowed the server to run with minimal effort during the busy night.

However the trade period weeks earlier showed that the online interest in the AFL had lifted significantly and that I should prepare further to ensure that load was not a problem. As I host on Rackspace with their OpenStack cloud hosting I was able to take advantage of their versatile system to easily build a solution to meet potential demand. I created an image of the server and made four new server instances from it which were to become slaves. I then modified the site so that updates on the master would copy (via rsync) any changed JSON files to the slaves. Then I created a load balancer within the same interface with a few clicks and added the four slaves behind it before finally pointing the domain name to the load balancer’s IP address.  Another benefit of this design was that the site administrator could make updates to the site from an instance that was experiencing no load and therefore unhindered by too much traffic.

The draft went for about 1.5 hours and saw 100,000 unique visitors, each of which would poll the site every ten seconds. Monitoring the servers showed that my solution was complete overkill and probably only the one server was enough. But it’s better to plan for the worst and it was a great experience putting the solution together.

COSTING

Each of the four slaves ran with 512MB memory which costs $0.03 per hour, $0.15 total (including master). The load balancer costs a base of $0.015 per hour but scaled up per number of connections. Therefore for the 1.5 hours the expense of the set-up would have cost just a few dollars. Of course I had this set-up running for quite a few days beforehand though but overall the costing is negligible.

[1] The site was designed before the days of NodeJS and websockets not solution for older browsers.

For some years now I’ve been inspired to travel to the United States to attend the Open Source Convention OSCON in Portland. I hoped to learn what new open source tools and resources developers from around the world are using to get their work done.

This year I made the journey and it was well worth it. About 3000 people attended over the five days and they are all so passionate about open source software. Most are developers but all are working with open source software in one way or another. Everyone is very willing to share their skills and experience.

A main focus of the conference was Open Stack (http://www.openstack.org/) which is an open source alternative to Amazon’s cloud services and the primary thing I hoped to learn about when leaving Melbourne. Open Stack is being embraced by many businesses and the founders from NASA have moved on to build their own businesses that use Open Stack technologies. As some speakers discussed there is still a lot of work to do before Open Stack has all the features required to be a complete cloud services platform but it’s looking very promising.

I also got a lot out of talks about PHP, Vim, Twitter’s Bootcamp and system performance tuning.

I also met lots of interesting people. Sitting down to lunch I found myself sitting next to Sebastion Bergman who created PHPUnit and on another day with an Open Stack founder Josh McKenty. I also met some Ubuntu community members and some people behind MySQL (and MariaSQL), Linode, Rackspace and many more.

Everyone is pushing the open source movement in the same direction. Forward. It was a fantastic event and I hope to attend next year. However tomorrow in day one of Defcon which I’m very excited about.

Recently I have been working on sites that don’t allow ssh access and I always find installing phpMyAdmin overkill and unnecessary for my needs. Therefore I wrote a single file script that you upload to your webserver, login to the database and immediately start writing queries.

It allows you to write complex queries with joins or just simple inserts, updates or deletes and returns the result set quickly with the number of rows found or affected on a page that’s clear and easy to read.

In the future I would like to add support for PostgreSQL and the ability to add or modify the data through a form.
The code is available to download from github. Please try it and let me know what you think.

For the last three months I have been working on different websites that I inherited that rely heavily on Redis. Redis is another no-sql database that uses key/value pairs to store data but does not give you the flexibility to write queries like you’re used to with relational databases like MySQL and Postgres.

Interestingly, these sites also use MySQL. I never had a chance to use or learn about No-SQL databases beforehand but the idea of using two types of databases for one application sounded like a convoluted and unnecessary solution. However, the more I use Redis (especially in these applications) the more I love it.

Redis is used in these applications for caching. When a request it made, rather than PHP sending queries to MySQL, it requests the data directly from Redis, which is pulled straight from the RAM. This results is a much faster response time and requires less resources from the server.

As there are already some good tutorials on what Redis is and how to use it, I will show you some great tricks I found through my travels that I did not see in the general documentation. Start the Redis client (redis-cli) and try the following two tricks:

List All Keys In The Database:

keys *

Show Variable Type:

type <variable-name>

Monitor Queries:
Monitor the queries being sent to Redis by using telnet to login to redis on the port number it’s running (default 6379) and type monitor. This is very helpful if the queries are being sent from an application and you need to debug exactly what’s the query is.

Running Multiple Instances Of Redis:
As I am running multiple applications that require Redis, I needed to learn how to run multiple versions of Redis. This is because you don’t define separate databases like you would with MySQL for example. There is no logins and no way to clearly separate data between applications. An excellent description on how to run multiple version of Redis can be found at chrislaskey.com.

Start Learning:
To get a better understanding of Redis I recommend using the online practical guide found at try.redis-db.com. This guide explains what different types of variables are available and how to access them.

Conclusion:
Redis has shown me what’s available in no-sql databases and that relational databases may not always be the answer. I can see that as I use Redis more in my own projects I will find that it’s useful for other purposes.  One possibility is the storing of variables that I may have previously put into sessions.

I have been maintaining Debian based Linux servers now for some years and at times I find it frustrating that the latest versions of my favourite packages are not available yet because of the delay in getting the newest version into the selected repository. This leaves you needing to build the program from source.

I recently discovered a project designed to get around this problem. Dotdeb is a repository for Debian systems that have the latest versions of PHP, MySQL, Redis, Apache, Nginx and other common web type packages ready to install or upgrade. With doing little more than adding the Dotdeb repository URL to your sources.list file and updating, the newest version of each package are immediately available.

I love now that I can easily have the most up to date packages with minimal fuss and leaving me with time to get back to development.

To find out more visit the project’s website.