Doublehops

I have been using Dotdeb, the custom Debian package repository for the last 15 months to keep all web packages up to the latest version. It’s incredibly easy to install and beats waiting for the Debian team to update their versions. However there was an issue when upgrading PHP from 5.3.x to 5.4.x for systems using the PHP5-fpm package under Nginx.

After returning to the problem after a couple of months I found that when upgrading to 5.4 a major config option was being changed. The listening parameter changed in /etc/php5/fpm/pool.d/www.conf from:

listen = 127.0.0.1:9000;

to:

listen = /var/run/php5-fpm.sock;

This was causing an error about an invalid gateway. Once I discovered this change, I found that correcting it is a simple change in the virtual host file, from:

fastcgi_pass   127.0.0.1:9000;

to:

fastcgi_pass unix:/var/run/php5-fpm.sock;

Finally restarting Nginx resolved the issue leaving you with latest version of PHP 5.4 running on your server.

My most recent appointment required me to build a CMS and front-end for the Australian Football League for the trade period. The CMS was built to allow editors to add news items, trades and free agency movements between the 18 clubs.  The front-end was to display the inserted items, but allow the end-user to filter them to given rules. Again, I chose Yii to build this as it’s a great framework for rapid development but also robust and a pleasure to work with.

After designing the database I started building the models, views and controllers before modifying the forms to match the experience required for an easy to use and intuitive CMS. For the main news feed section, the front-end results could be filtered with different filters such as club, date and result type, eg. Trade only or general comment. These filters work together for fine control over the results shown. As each filter is used, the results are returned and populated by AJAX requests with filters being cleared by selecting Live Feed. The challenging part here was deciding on how to have the filters work together in the browser. I ended up building the URL that would be passed in the AJAX request. Session could have worked also but was an issue in load balancing and caching as I’ll point out later.

The second view was a breakdown of trades in and trades out by club. The result for the view were pulled from the same data as in the main feed to save on repetition will adding content. Also with filters that load with AJAX this came together quickly. I’m impressed the way that Yii allows you to reload content for partial views with just a few extra lines of code writing the jQuery for you.

The third view shows the players that fans most want traded. This data is pulled from another website trademachine.afl.com.au which the results are user generated. I could build this view quickly also by implementing a second database that is easy to do in Yii.

The site went live on October 1 and the demand was a lot greater than I was expecting. This resulting in the server becoming overwhelmed and some slow or failed page loads. Being a little unprepared I quickly made new instances of the server and put them all under a load balancer to meet demand. Cloning servers and putting them under a load balancer couldn’t be easier than what is available with Rackspace. This was quick and saved me a lot of pain early on. I then spent some time adding and fine tuning the built-in caching that Yii provides. I had not used caching in Yii before but I was surprised at how easy and effective this is. Although the content should only be cached for 60 seconds on the live feed, the resources being used on the server were dramatically reduced.

This is an example of adding caching to a given part of the site with Yii:

if($this->beginCache('main-content', array(
            'duration'=>60,
            'varyByParam'=>array('filter','club','dateRange'),
            ))) {
                $this->renderPartial('_entryList', array(
                    'dataProvider'=>$dataProvider,
                ));
    $this->endCache(); }

This would cache the view to 60 seconds and the varyByParam parameter tells the cache to use GET variables filter, club and dataRange as values to take into account when caching to ensure that each unique request is cached and returned as expected. This is essential as the view has a single URL but the content will change depending on what GET variables are also supplied. If I was to use sessions to keep track of what filters the browser had selected, it would fail through the cache and load balancers so sessions here was not an option.

Overall this was a fun project that required me to provide a solution for an event that I have a lot of interest in. The result is an easy to use CMS with a great user experience in the front-end also.

Last month I was one of 15,000 people that attended the Defcon computer security convention in Las Vegas. It was a fantastic four day event with presenters talking about their findings and projects in regards to all things security.

Upon paying the $200 entry fee we were given our badge required for entry. This year’s badge was electronic and a puzzle in a way. Through onboard lights and light sensor the badges would communicate with each other as they past by. Also via a USB port we were encouraged to program some hacks so that they behaved differently.

Defcon 2012 Badge

One of the most interesting events in Capture The Flag where teams are set against each other to hack into their opponents servers and capture so called flags. Each team would harden their own servers before beginning to attack others. From what I could gather they do this non-stop throughout the event and the team who has gathered the most flags is deemed the winner.

My highlights were sitting in on talks by Kevin Mitnick on social engineering and Kevin Poulsen discussing the exploits he used to get up to in his past. Having read books by both presenters I was keen to see what they had to say.

I would love to attend again next year. Anyone feel like sponsoring my trip?

For some years now I’ve been inspired to travel to the United States to attend the Open Source Convention OSCON in Portland. I hoped to learn what new open source tools and resources developers from around the world are using to get their work done.

This year I made the journey and it was well worth it. About 3000 people attended over the five days and they are all so passionate about open source software. Most are developers but all are working with open source software in one way or another. Everyone is very willing to share their skills and experience.

A main focus of the conference was Open Stack (http://www.openstack.org/) which is an open source alternative to Amazon’s cloud services and the primary thing I hoped to learn about when leaving Melbourne. Open Stack is being embraced by many businesses and the founders from NASA have moved on to build their own businesses that use Open Stack technologies. As some speakers discussed there is still a lot of work to do before Open Stack has all the features required to be a complete cloud services platform but it’s looking very promising.

I also got a lot out of talks about PHP, Vim, Twitter’s Bootcamp and system performance tuning.

I also met lots of interesting people. Sitting down to lunch I found myself sitting next to Sebastion Bergman who created PHPUnit and on another day with an Open Stack founder Josh McKenty. I also met some Ubuntu community members and some people behind MySQL (and MariaSQL), Linode, Rackspace and many more.

Everyone is pushing the open source movement in the same direction. Forward. It was a fantastic event and I hope to attend next year. However tomorrow in day one of Defcon which I’m very excited about.

Update: I and others have found hfsplus support has been unreliable on Linux so I have converted the drive to EXT4. Read this post’s comments for more.

I recently purchased a 2TB external drive for my Linux media centre but could not work out why I couldn’t write to the drive regardless of the permissions I had set.

When mounting the drive I would get the following error:

mount: warning: /media/drive seems to be mounted read-only.

This is the solution I found thanks to a collection of findings on the web by firstly installing hfsprogs:

sudo apt-get install hfsprogs // For Debian based distro
sudo fsck.hfsplus /dev/sdb2 // depending on your device and partition

Then try mounting the drive again as a normal user and hopefully it will work.

I purchased a Macbook Pro many months ago and tried using it as a development machine for my PHP/MySQL/Nginx projects but I found I hated the lack of a good package manager like Aptitude (apt-get) and my favourite window manager Fluxbox so I ultimately removed Macosx for my preferred platform of Ubuntu/Kubuntu/Debian.

This was great but I now had new frustrations of having no wireless as Apple had sourced a little known Broadcom wireless device (B4331) for their products that had no native drivers for Linux. I had been perseveering for over five months using LAN cables and tethering from my phone whilst waiting for an elegant solution to become available to get the device to work. Up until now, at best there seemed to just be poor and unreliable methods to get the device to work that I couldn’t risk my dev machine on in case it affected my machine too much, like the constant lockups some people experience.

But yesterday I found an article from Raimer Sandner which walked you through installing the driver and avoiding the pitfalls found with the other methods. It took me about 10 minutes to follow the steps and now I’m finally free of wires.

I bought my first Mac several months ago because I travel around the city a lot I wanted something smaller than my older 15 inch laptop. I used Macosx on it for sometime but found development too difficult. I guess I’ve been using Linux for so long now that I am so intune with the way it works, I was lost without it. My two main gripes was no good (from limited effort trying homebrew) package managers and less than ideal window manager. I realised more than ever how much I need Aptitude and Fluxbox to be productive.

So I installed Kubuntu 11.04 and felt at home again. However there were some things that didn’t work they way I needed them to. The touchpad is great and I could never go back to a system that doesn’t have one but with Linux it was way too sensitive. As someone else on the web pointed out, you could move the mouse without actually touching the device. This made typing emails and most typing tasks a real pain because the cursor would jump all over the place. However thanks to Texstar’s post I was able to correct this. I went with solution 4. I found that setting FingerHigh to a value of 50 was better than the recommended 30. Also on another forum I found an option to detect if your palm has brushed the touchpad and ignore the action.

My config now looks like this: /etc/X11/xorg.conf.d/50-synaptics.conf

Section "InputClass"
    Identifier "touchpad catchall"
        Driver "synaptics"
        MatchIsTouchpad "on"
        MatchDevicePath "/dev/input/event*"
        #Option  "Device"        "/dev/input/mouse0"
        Option  "Protocol"      "auto-dev"
        #Option  "LeftEdge"      "1700"
        #Option  "RightEdge"     "5300"
        #Option  "TopEdge"       "1700"
        #Option  "BottomEdge"    "4200"
        Option  "FingerLow"     "25"
        Option  "FingerHigh"    "50"
        Option  "MaxTapTime"    "180"
        Option  "MaxTapMove"    "220"
        Option  "VertScrollDelta" "30"
        Option  "MinSpeed"      "1.48"
        Option  "MaxSpeed"      "2.00"
        Option  "AccelFactor" "0.3830"
        Option  "SHMConfig"     "1"
        Option  "VertTwoFingerScroll"   "1"
        Option  "HorizTwoFingerScroll"  "1"
        Option  "VertEdgeScroll"        "1"
        Option  "HorizEdgeScroll"       "1"
        Option  "TapButton1"            "1"
        Option  "TapButton2"            "3"
        Option  "TapButton3"            "2"
        Option  "PalmDetect"            "1"
EndSection

These settings can also be changed on the fly using synclient.

synclient FingerHigh=40

I found that this set-up helped a lot getting the mouse gestures in Firefox to respond better. Now I’m just waiting on the development of a native driver for the wireless card.

Update: The option that makes the biggest improvement is the PalmDetect. Setting this to 1 stops the touchpad from reacting when you’re plan slides across. The other settings should then be tweaked to suit your needs.

I have been maintaining Debian based Linux servers now for some years and at times I find it frustrating that the latest versions of my favourite packages are not available yet because of the delay in getting the newest version into the selected repository. This leaves you needing to build the program from source.

I recently discovered a project designed to get around this problem. Dotdeb is a repository for Debian systems that have the latest versions of PHP, MySQL, Redis, Apache, Nginx and other common web type packages ready to install or upgrade. With doing little more than adding the Dotdeb repository URL to your sources.list file and updating, the newest version of each package are immediately available.

I love now that I can easily have the most up to date packages with minimal fuss and leaving me with time to get back to development.

To find out more visit the project’s website.

I discovered something completely new over the weekend; running multiple window managers simultaneously in your Linux environment. This came about because of a conflict I encountered between Fluxbox and an application I was trying to run, Bitcoin. It seems that two are somewhat incompatible.

After finding that Bitcoin ran fine in WindowMaker, it was pointed out to me that I could run WindowMaker inside my Fluxbox session by running a second X server. This saves me from having to logout of one window manager to use another, just for the sake of one application. I installed X window server Xephyr which then allowed me to run a second window manager which then allowed WindowMaker within.

Thanks to qole and his forum thread I was able to put the whole thing together with a basic script with a few minor modifications to suit my environment. My modified script follows:

#!/bin/sh
##################################
#Change these two variables to the WM of your choice.
export WMNAME="windowmakerWM"
export WMEXEC="wmaker"
##################################
export DISPLAY=:0
export TSLIB_TSDEVICE=/dev/input/event3
export TSLIB_CALIBFILE=/etc/pointercal
export GTK_MODULES=libgtkstylus.so
export PARWIN=`wmctrl -l | grep -i $WMNAME | awk '{print $1}'`
wmctrl -i -r $PARWIN -b toggle,fullscreen
xinit /bin/su chippy -c 'export DISPLAY=:1; $WMEXEC' -- /usr/bin/Xephyr :1 -br -screen 1000x700x16 -dpi 96 -parent $PARWIN +kb -ac
export DISPLAY=:0

Running this script loads the X server and then runs a WindowMaker session within it. It’s quite like running a virtual machine, however you are not running another instance of an operating system, just another window within your current operating system with all your installed applications and environment as you would expect.

Sometimes I am reminded the power available to Linux and all things open source. This was another example.

WindowMaker running with Fluxbox

This week I attended the Open Source Developers Conference in Melbourne. OSDC is a three day event that is aimed at sharing skills and thoughts on software development using open source technologies and is attended by many developers passionate about open source software. This year I believe there were about 150 attendees from all around Australia and around 60 talks in total.

The talks covered everything from best practices in version control, Perl 6, PHP frameworks and CMSs, mobile application development and much more.

It was great to see what others are doing with and getting out of open source software. I was most amazed by Ben Martin’s talk about his project called Libferris. Libferris is a virtual file system which can mount vastly different things that can then be accessed from the shell. The types of things that can be mounted are LDAP, Evolution, images, Flicker, Youtube, PostgreSQL, RDP documents, the DOM in Firefox and MP3 files. This allows the extraction of data available to the shell or custom applications.

Another highlight was the closing keynote by Damian Conroy. This was a talk that lasted an hour and was exhausting to listen to as it was complex and fast, yet extremely compelling. Damian effectively used the general theory of relativity, quantum mechanics and perl to find prime numbers. The idea and implementation is completely amazing and I suggest everyone watches the video to see for yourself (shown at the bottom of this post.

All in all everyone was very impressed with the event and took a lot away from it. Sharing a beer at the end of each day was a great way to meet other like minded people and discuss what we learnt throughout the day. I am already looking forward to next years event which will be held in Canberra.

Damien Conway’s closing keynote speech – Temporally Quaquaversal Virtual Nanomachine Programming In Multiple Topologically…