RSS

Setting Up Development Environments With Vagrant and Ansible

0 Comments | This entry was posted on Feb 19 2014

One of the reasons I love running Linux on my main laptop/workstations is that I have an ideal environment to develop web projects. However there’s been many developments in software that moves away from this model which I have grown to love, and that is running your dev environments in virtual machines.

Instead of running Apache (or Nginx), MySQL and PHP natively on my dev machine, I have found it’s now easier to setup and run dev environments in virtual machines that are configured specifically for a given project, which can be automated through server management scripts. Initially this sounds like additional work, and it is but it has several advantages:

  • Custom environments for each project
  • Easily deployable for other developers in your team
  • No knowledge required for other team members.
  • Scripts can be reused for staging and development environments.

What are Vagrant and Ansible:

Vagrant is software that allows you to easily build reproducible development environments for various operating systems. It runs on top of other virtual machine platforms such as Virtualbox but, among other things, creates a sync drive that is accessible to your local file system, allowing you to use you IDE as you would normally without the need to transfer files to the machine.

Ansible, like Puppet or Chef is a server management scripting language. However the learning curve is a lot simpler and doesn’t require any software running on the remote servers. It configures the hosts over ssh.

By combining Vagrant with Ansible, it’s very easy to create development environments for developers who are running any common operating system within minutes without having to manually configure their dev environments to suit their operating system.

I have created Vagrant/Ansible setup script which can be found on Github. This will configure a development virtual machine that will have installed the latest versions of Nginx, MariaDB and PHP on Debian 7.

I think it’s worthwhile for any development teams to investigate using virtual machines like this, especially where complex environments are required.

New AFL Websites Built with Yii

1 Comment | This entry was posted on May 14 2013

I was tasked with building the new AFL websites Mark of the Year and Goal of the Year for the 2013 season for the Australian Football League. After gathering the requirements it was excited to find that Yii will be perfect to build these sites.

I love the sink my teeth into a project by starting with database design. The requirements were clear and not too exhaustive so it didn’t take too long to design. From there I was quickly able to build the models, views and controllers with Yii’s great scaffolding tool, Gii.

From this point I could start with the basic page layout and add the business rules to the the models and controllers and tweak the views to meet the functionality. Once the site was fully functional I could then add all the design elements to meet the page layout and styles created by the client.

The tricky part was including the video code into the site with Telstra’s proprietary video embedding code.

It was a great experience and really highlighted how easy and fun it is to build functional websites with Yii.

Websites:

www.markoftheyear.afl.com.au
www.goaloftheyear.afl.com.au

 

NGINX config for CakePHP 1.3 (& PHP 5.4)

1 Comment | This entry was posted on May 05 2013

This afternoon I setup a virtual host in NGINX for a CakePHP 1.3.x project in readiness for starting work with a new client tomorrow. However once I had what looked correct, CakePHP would complain that friendly URLs where not setup correctly. I am running PHP 5.4.14 on my laptop and CakePHP 1.3 for the site, as this is what the current project is running.

There seems to be no examples on the web of how to get these two versions to run together. So here is my example that I got to work for anyone who’s also stuck:

server {
    listen 80;
    server_name cakephp;
    root /var/www/cakephp/app/webroot/;

    access_log /var/log/nginx/cakephp/access.log;
    error_log /var/log/nginx/cakephp/error.log;

    location / {
        index index.php index.html index.htm;

        if (-f $request_filename) {
            break;
        }

        if (-d $request_filename) {
            break;
        }

        rewrite ^(.+)$ /index.php?url=$1 last;
    }

    location ~ .*\.php[345]?$ {
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME
        /var/www/cakephp/app/webroot$fastcgi_script_name;

        include fastcgi_params;
    }
}

Upgrade to PHP 5.4 with Dotdeb

0 Comments | This entry was posted on Nov 08 2012

I have been using Dotdeb, the custom Debian package repository for the last 15 months to keep all web packages up to the latest version. It’s incredibly easy to install and beats waiting for the Debian team to update their versions. However there was an issue when upgrading PHP from 5.3.x to 5.4.x for systems using the PHP5-fpm package under Nginx.

After returning to the problem after a couple of months I found that when upgrading to 5.4 a major config option was being changed. The listening parameter changed in /etc/php5/fpm/pool.d/www.conf from:

listen = 127.0.0.1:9000;

to:

listen = /var/run/php5-fpm.sock;

 

This was causing an error about an invalid gateway. Once I discovered this change, I found that correcting it is a simple change in the virtual host file, from:

fastcgi_pass   127.0.0.1:9000;

to:

fastcgi_pass unix:/var/run/php5-fpm.sock;

 

Finally restarting Nginx resolved the issue leaving you with latest version of PHP 5.4 running on your server.

Gillette AFL Trade Tracker

0 Comments | This entry was posted on Oct 16 2012

My most recent appointment required me to build a CMS and front-end for the Australian Football League for the trade period. The CMS was built to allow editors to add news items, trades and free agency movements between the 18 clubs.  The front-end was to display the inserted items, but allow the end-user to filter them to given rules. Again, I chose Yii to build this as it’s a great framework for rapid development but also robust and a pleasure to work with.

After designing the database I started building the models, views and controllers before modifying the forms to match the experience required for an easy to use and intuitive CMS. For the main news feed section, the front-end results could be filtered with different filters such as club, date and result type, eg. Trade only or general comment. These filters work together for fine control over the results shown. As each filter is used, the results are returned and populated by AJAX requests with filters being cleared by selecting Live Feed. The challenging part here was deciding on how to have the filters work together in the browser. I ended up building the URL that would be passed in the AJAX request. Session could have worked also but was an issue in load balancing and caching as I’ll point out later.

The second view was a breakdown of trades in and trades out by club. The result for the view were pulled from the same data as in the main feed to save on repetition will adding content. Also with filters that load with AJAX this came together quickly. I’m impressed the way that Yii allows you to reload content for partial views with just a few extra lines of code writing the jQuery for you.

The third view shows the players that fans most want traded. This data is pulled from another website trademachine.afl.com.au which the results are user generated. I could build this view quickly also by implementing a second database that is easy to do in Yii.

The site went live on October 1 and the demand was a lot greater than I was expecting. This resulting in the server becoming overwhelmed and some slow or failed page loads. Being a little unprepared I quickly made new instances of the server and put them all under a load balancer to meet demand. Cloning servers and putting them under a load balancer couldn’t be easier than what is available with Rackspace. This was quick and saved me a lot of pain early on. I then spent some time adding and fine tuning the built-in caching that Yii provides. I had not used caching in Yii before but I was surprised at how easy and effective this is. Although the content should only be cached for 60 seconds on the live feed, the resources being used on the server were dramatically reduced.

This is an example of adding caching to a given part of the site with Yii:

if($this->beginCache('main-content', array(
            'duration'=>60,
            'varyByParam'=>array('filter','club','dateRange'),
            ))) {
                $this->renderPartial('_entryList', array(
                    'dataProvider'=>$dataProvider,
                ));
    $this->endCache(); }

 

This would cache the view to 60 seconds and the varyByParam parameter tells the cache to use GET variables filter, club and dataRange as values to take into account when caching to ensure that each unique request is cached and returned as expected. This is essential as the view has a single URL but the content will change depending on what GET variables are also supplied. If I was to use sessions to keep track of what filters the browser had selected, it would fail through the cache and load balancers so sessions here was not an option.

Overall this was a fun project that required me to provide a solution for an event that I have a lot of interest in. The result is an easy to use CMS with a great user experience in the front-end also.

 

 

runQuery: A New Tool To Query MySQL Databases

0 Comments | This entry was posted on May 07 2012

Recently I have been working on sites that don’t allow ssh access and I always find installing phpMyAdmin overkill and unnecessary for my needs. Therefore I wrote a single file script that you upload to your webserver, login to the database and immediately start writing queries.

It allows you to write complex queries with joins or just simple inserts, updates or deletes and returns the result set quickly with the number of rows found or affected on a page that’s clear and easy to read.

 

 

In the future I would like to add support for PostgreSQL and the ability to add or modify the data through a form.
The code is available to download from github. Please try it and let me know what you think.

Why Use A No-SQL Database Like Redis

0 Comments | This entry was posted on Dec 01 2011

For the last three months I have been working on different websites that I inherited that rely heavily on Redis. Redis is another no-sql database that uses key/value pairs to store data but does not give you the flexibility to write queries like you’re used to with relational databases like MySQL and Postgres.

Interestingly, these sites also use MySQL. I never had a chance to use or learn about No-SQL databases beforehand but the idea of using two types of databases for one application sounded like a convoluted and unnecessary solution. However, the more I use Redis (especially in these applications) the more I love it.

Redis is used in these applications for caching. When a request it made, rather than PHP sending queries to MySQL, it requests the data directly from Redis, which is pulled straight from the RAM. This results is a much faster response time and requires less resources from the server.

As there are already some good tutorials on what Redis is and how to use it, I will show you some great tricks I found through my travels that I did not see in the general documentation. Start the Redis client (redis-cli) and try the following two tricks:

List All Keys In The Database:

keys *

Show Variable Type:

type <variable-name>

Monitor Queries:
Monitor the queries being sent to Redis by using telnet to login to redis on the port number it’s running (default 6379) and type monitor. This is very helpful if the queries are being sent from an application and you need to debug exactly what’s the query is.

Running Multiple Instances Of Redis:
As I am running multiple applications that require Redis, I needed to learn how to run multiple versions of Redis. This is because you don’t define separate databases like you would with MySQL for example. There is no logins and no way to clearly separate data between applications. An excellent description on how to run multiple version of Redis can be found at chrislaskey.com.

Start Learning:
To get a better understanding of Redis I recommend using the online practical guide found at try.redis-db.com. This guide explains what different types of variables are available and how to access them.

Conclusion:
Redis has shown me what’s available in no-sql databases and that relational databases may not always be the answer. I can see that as I use Redis more in my own projects I will find that it’s useful for other purposes.  One possibility is the storing of variables that I may have previously put into sessions.

Yii Framework Issue With Nginx And SSL

0 Comments | This entry was posted on Oct 09 2011

During the process of moving my websites from one host to another and from Apache to Nginx I came across an issue that had me frustrated yet intrigued. I have a Yii application that I use for my business that I ran over SSL. This was working fine under Apache and for the most part under Nginx. The problem would only come about in Nginx when a form was posted, the browser would be redirected from HTTPS to HTTP mode. Why was this?

After some Google searching, Yii core framework investigation and some trial and error I found that Yii relies on what I determined is a non standard HTTP header ‘HTTPS’ with a value of ‘on’. Nginx does not send this header and on a form post Yii would make the assumption that the browser was in non HTTPS mode.

The core Yii method that determined the protocol is:

public function getIsSecureConnection()
{
    return isset($_SERVER['HTTPS']) && !strcasecmp($_SERVER['HTTPS'],'on');
}

This can be overcome by adding a parameter into the Nginx virtual host record but I see this as a workaround and not a real solution. I thought a simple change from the non standard ‘HTTPS’ header to ‘SERVER_PORT’ (checking that it’s set to 443), which is sent by both Apache and Nginx would be a better solution. I have sent a bug report off to the Yii project with my suggestion so maybe we’ll see it in a future release.

Update: 18 October 2011
My suggested fix was added to the Yii code base but was reverted back out because a site running on port 443 does not necessarily mean it’s running over SSL. There is no standard header that browsers send when running over SSL so therefore a non-ideal solution must be used. I believe the best solution is to add the HTTPS header to the Nginx config as suggested.

Keeping Your LAMP Server Up To Date With Dotdeb

0 Comments | This entry was posted on Sep 13 2011

I have been maintaining Debian based Linux servers now for some years and at times I find it frustrating that the latest versions of my favourite packages are not available yet because of the delay in getting the newest version into the selected repository. This leaves you needing to build the program from source.

I recently discovered a project designed to get around this problem. Dotdeb is a repository for Debian systems that have the latest versions of PHP, MySQL, Redis, Apache, Nginx and other common web type packages ready to install or upgrade. With doing little more than adding the Dotdeb repository URL to your sources.list file and updating, the newest version of each package are immediately available.

I love now that I can easily have the most up to date packages with minimal fuss and leaving me with time to get back to development.

To find out more visit the project’s website.

Integrating Campaign Monitor API With WordPress

1 Comment | This entry was posted on May 04 2011

One of the most enjoyable things in software development is combining different tools or datasets together into one application. Sometimes this is done to build interesting mashups or more usually to automate an otherwise manual task. Is the case of using the Campaign Monitor API is to prevent having to do a repetitive task manually.

The project I have been working on for the last two weeks is a WordPress blog that allows site visitors to subscribe to a section (or category) of the site to be notified when a new post has been added. As with this project it is expected to have many hundreds or thousands of subscribers, it is not feasible to send mail directly from the web server because of the overhead and the potential of having the server black listed by spam prevention services. Therefore Campaign Monitor was chosen to send any new post notifications out to the subscribers via the API.

The Campaign Monitor API is very easy to use and does nearly everything that is possible through the normal web based interface.  For this solution there were three API calls required to achieve the necessary task, 1. add new subscribers to a list; 2. create a new campaign for each new post; and 3. send the campaign to the subscribers in the list.

A WordPress plugin was created to display a form to populate required Campaign Monitor values such as client and list Ids and to define the functions that will execute the tasks. The tasks are broken down as follows:

  1. A new client list was first added manually to the Campaign Manager client. In the site, new subscribers were already being saved to the WordPress wp_users table. A new function was created and called from this block of code to add subscribers to the specified list.
  2. Each time a new post is created, a hook is used to call a function that gathers that post information and creates a new campaign which is assigned to the existing list. However, before this is done a new html campaign (edm) is created from a template but populated with the post’s title and URL so that the link can be clicked in the receivers email client. This URL is supplied in the API call when creating the new campaign.
  3. After the campaign has been created, another function is been called to send it of to all those in the subscriber list.

And it’s done. The API is simple to use and requires little effort to get started. It can be downloaded from Github in 5 popular web development languages and is supplied with sample code. Each call to update or read from the API is achieved with just a handful of lines and can be sent and received in either XML or JSON. The documentation is clear and intuitive.

For more information, visit the website.