RSS

Migrating From Rackspace to Amazon AWS.

0 Comments | This entry was posted on Jan 05 2017

Over the Christmas break I took on the task of migrating my websites from Rackspace Opencloud to Amazon’s AWS. There were several reasons for doing so but the main ones were because of the ever increasing amount of services I’m using there through work that I want to include in my own projects. I feel I’ve been missing out. Also, the more I that my head is in the AWS ecosystem, the more I’ll learn and be able to pass on to my clients.

As my projects (roughly 7, including this blog) are all rather small, I host them all on the one server instance. This could be a nightmare to migrate but fortunately I had scripted everything with Ansible, making the process fast and straight forward. I first had to tweak my scripts to use PHP7 as I had no yet upgraded my Rackspace instance. By starting out I created a EC2 instance running Debian Jessie, updated .ssh/config with the right credentials and ensure that I could ssh in to the new server. Once verified that all was OK, I ran the Ansible script over the new server which automatically installed:

  • Required services such as Nginx, MariaDB, PHP7 and miscellanous tools such as htop, git, vim, etc…
  • All the Nginx hosts records
  • Any Basic Auth protection I had created for some hosts and paths
  • Each database and database users for each project (I’m not using RDS for these small projects)
  • Cronjobs and associated scripts that the projects require – which include the onsite backups
  • Created a second user that only has privileges to retrieve the backups to store offsite

From there, it was a simple matter of using mysqldump to export all databases from the Rackspace server, SCP them to the new server and import. I then zipped up the web root directory and SCPed them across to the server also. Lastly came the SSL certificates that I needed to move across. Before long I had a fully functioning server created from scratch that included all sites, their data and full backups. I updated the DNS records to match the new IP address and I was done.

Offensive Security Training

0 Comments | This entry was posted on Sep 11 2016

I’ve always been interested in computer security and although it’s something I consciously think about when building web applications, it’s not something I’ve ever giving solid time to solely focus on. However, over the last three months I spent all my spare time in the evenings and on weekends working through the Offensive Security certificate, a certificate that is taught by the developers of the Kali Linux distro.

The certificate is a very hands on approach to learning how to compromise computer systems. Along with a guide that goes into detail many of the ways in which vulnerabilities can be found and exploited, you are given access to a VPN with about two dozen vulnerable machines where you can explore and hone your skills. Starting out by scanning networks and profiling each server by learning its operating system, open ports and applications (and their versions) running on each. I found that it can be a tedious exercise but very interesting at the same time. SQL injection was fun but maybe because I’ve played around with that before and already had a great understanding of how it works. The buffer overflow exploits, although tough, was made much easier than I would have guessed because of the tools available today that make attempts quite transparent.

You quickly learn to write your own scripts to automate things that you find yourself repeating. As a result it improved my skills in both Python and Bash. This mostly to do things like scan a network for webservers or servers had MySQL ports open. I thought that sqlmap was a useful tool as it takes the tedious guess work out of finding applications that are not properly escaping user data before running them through an SQL query. The certificate introduces the student to many useful tools ranging from discovery and exploit execution.

It was an exhausting exercise to take outside of my day job but very rewarding. I learned a lot about a topic that has fascinated me for over 20 years and I can use these new skills to build and test that the applications and environments that I build are as secure as possible. In 2017, I will start working with IoT devices and build the APIs that they will communicate with. These devices will need to be secure and not become part of the growing botnets that we read about. Keeping on top of security issues is an ongoing task that I’m glad to be a part of.

Presentation on Building an API with Yii2 at PHP Melbourne

0 Comments | This entry was posted on Mar 22 2016

Last week I made a presentation at the Melbourne PHP usergroup phpMelb. The presentation was a live demonstration on how someone would go about creating an API with Yii2 from a clean install. I went through the steps of creating a migration and building the model with Gii. Then, following the Yii2 guide on how to turn a controller into a RESTful API controller. The steps are very straight forward and very quick.

I discussed using Chrome extension Postman to create and submit the payload and to view results. I showed how to add behaviours to the models and controllers but could only touch on authentication because of time constraints. I also demonstrated that with the advanced template you can also have a backend which I built again with Gii to show how easy it is to create a web based admin part of the API/site.

You can view the presentation here: https://doublehops.com/presentations/yii2-api-presentation.

Setting up development environments for Yii2 with Vagrant and Ansible

0 Comments | This entry was posted on Feb 18 2015

In an effort to streamline all work and learning that I do I have put together an Ansible script that will build a web server with Yii2 advanced application template framework ready to start development with the newly released Yii2 PHP framework.

After Vagrant is used to create the virtual machine with Debian 7, Ansible is called to provision the server with PHP5.5, MariaDB and Nginx. It will then setup to virtual host for Yii2 framework with both frontend and backend configuation.

The scripts are available on Github for anyone who wants to have a try with Yii2.

Spreading the Word on Vagrant and Ansible

0 Comments | This entry was posted on May 31 2014

Over the last two months I have presented the advantages of using Vagrant and Ansible to the PHP melbourne and Melbourne Linux user groups and on both occasions it was well received. I demonstrated how development environments can be automated for teams to ensure that everyone is running the same software and at the same versions.

Getting development environments up and running for your current project can very time consuming on some occasions hard to debug when things are behaving strangely. If you have several developers running environments of Windows, Mac and Linux, getting each developer’s rig set to start work can be unnecessarily difficult. With Vagrant and Ansible, one person can easily script the configuration, allowing others to just run it to get the environment setup.

When talking with the Linux group I focused the talk about Ansible more on deploying to production servers (web, mail, etc..) which has no real need for Vagrant. However Vagrant is helpful here also because it allows you to test your Ansible scripts locally before deploying to production systems, saving time and money.

Just a 20 minute presentation is enough to give some examples and a live demonstration to show how easy it is to implement and why they should look at using these technologies in their own work.

My slides are available here and the working script is available on Github.

Setting Up Development Environments With Vagrant and Ansible

0 Comments | This entry was posted on Feb 19 2014

One of the reasons I love running Linux on my main laptop/workstations is that I have an ideal environment to develop web projects. However there’s been many developments in software that moves away from this model which I have grown to love, and that is running your dev environments in virtual machines.

Instead of running Apache (or Nginx), MySQL and PHP natively on my dev machine, I have found it’s now easier to setup and run dev environments in virtual machines that are configured specifically for a given project, which can be automated through server management scripts. Initially this sounds like additional work, and it is but it has several advantages:

  • Custom environments for each project
  • Easily deployable for other developers in your team
  • No knowledge required for other team members.
  • Scripts can be reused for staging and development environments.

What are Vagrant and Ansible:

Vagrant is software that allows you to easily build reproducible development environments for various operating systems. It runs on top of other virtual machine platforms such as Virtualbox but, among other things, creates a sync drive that is accessible to your local file system, allowing you to use you IDE as you would normally without the need to transfer files to the machine.

Ansible, like Puppet or Chef is a server management scripting language. However the learning curve is a lot simpler and doesn’t require any software running on the remote servers. It configures the hosts over ssh.

By combining Vagrant with Ansible, it’s very easy to create development environments for developers who are running any common operating system within minutes without having to manually configure their dev environments to suit their operating system.

I have created Vagrant/Ansible setup script which can be found on Github. This will configure a development virtual machine that will have installed the latest versions of Nginx, MariaDB and PHP on Debian 7.

I think it’s worthwhile for any development teams to investigate using virtual machines like this, especially where complex environments are required.

New AFL Websites Built with Yii

1 Comment | This entry was posted on May 14 2013

I was tasked with building the new AFL websites Mark of the Year and Goal of the Year for the 2013 season for the Australian Football League. After gathering the requirements it was excited to find that Yii will be perfect to build these sites.

I love the sink my teeth into a project by starting with database design. The requirements were clear and not too exhaustive so it didn’t take too long to design. From there I was quickly able to build the models, views and controllers with Yii’s great scaffolding tool, Gii.

From this point I could start with the basic page layout and add the business rules to the the models and controllers and tweak the views to meet the functionality. Once the site was fully functional I could then add all the design elements to meet the page layout and styles created by the client.

The tricky part was including the video code into the site with Telstra’s proprietary video embedding code.

It was a great experience and really highlighted how easy and fun it is to build functional websites with Yii.

Websites:

www.markoftheyear.afl.com.au
www.goaloftheyear.afl.com.au

 

NGINX config for CakePHP 1.3 (& PHP 5.4)

1 Comment | This entry was posted on May 05 2013

This afternoon I setup a virtual host in NGINX for a CakePHP 1.3.x project in readiness for starting work with a new client tomorrow. However once I had what looked correct, CakePHP would complain that friendly URLs where not setup correctly. I am running PHP 5.4.14 on my laptop and CakePHP 1.3 for the site, as this is what the current project is running.

There seems to be no examples on the web of how to get these two versions to run together. So here is my example that I got to work for anyone who’s also stuck:

server {
    listen 80;
    server_name cakephp;
    root /var/www/cakephp/app/webroot/;

    access_log /var/log/nginx/cakephp/access.log;
    error_log /var/log/nginx/cakephp/error.log;

    location / {
        index index.php index.html index.htm;

        if (-f $request_filename) {
            break;
        }

        if (-d $request_filename) {
            break;
        }

        rewrite ^(.+)$ /index.php?url=$1 last;
    }

    location ~ .*\.php[345]?$ {
        fastcgi_pass unix:/var/run/php5-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME
        /var/www/cakephp/app/webroot$fastcgi_script_name;

        include fastcgi_params;
    }
}

Upgrade to PHP 5.4 with Dotdeb

0 Comments | This entry was posted on Nov 08 2012

I have been using Dotdeb, the custom Debian package repository for the last 15 months to keep all web packages up to the latest version. It’s incredibly easy to install and beats waiting for the Debian team to update their versions. However there was an issue when upgrading PHP from 5.3.x to 5.4.x for systems using the PHP5-fpm package under Nginx.

After returning to the problem after a couple of months I found that when upgrading to 5.4 a major config option was being changed. The listening parameter changed in /etc/php5/fpm/pool.d/www.conf from:

listen = 127.0.0.1:9000;

to:

listen = /var/run/php5-fpm.sock;

 

This was causing an error about an invalid gateway. Once I discovered this change, I found that correcting it is a simple change in the virtual host file, from:

fastcgi_pass   127.0.0.1:9000;

to:

fastcgi_pass unix:/var/run/php5-fpm.sock;

 

Finally restarting Nginx resolved the issue leaving you with latest version of PHP 5.4 running on your server.

Gillette AFL Trade Tracker

0 Comments | This entry was posted on Oct 16 2012

My most recent appointment required me to build a CMS and front-end for the Australian Football League for the trade period. The CMS was built to allow editors to add news items, trades and free agency movements between the 18 clubs.  The front-end was to display the inserted items, but allow the end-user to filter them to given rules. Again, I chose Yii to build this as it’s a great framework for rapid development but also robust and a pleasure to work with.

After designing the database I started building the models, views and controllers before modifying the forms to match the experience required for an easy to use and intuitive CMS. For the main news feed section, the front-end results could be filtered with different filters such as club, date and result type, eg. Trade only or general comment. These filters work together for fine control over the results shown. As each filter is used, the results are returned and populated by AJAX requests with filters being cleared by selecting Live Feed. The challenging part here was deciding on how to have the filters work together in the browser. I ended up building the URL that would be passed in the AJAX request. Session could have worked also but was an issue in load balancing and caching as I’ll point out later.

The second view was a breakdown of trades in and trades out by club. The result for the view were pulled from the same data as in the main feed to save on repetition will adding content. Also with filters that load with AJAX this came together quickly. I’m impressed the way that Yii allows you to reload content for partial views with just a few extra lines of code writing the jQuery for you.

The third view shows the players that fans most want traded. This data is pulled from another website trademachine.afl.com.au which the results are user generated. I could build this view quickly also by implementing a second database that is easy to do in Yii.

The site went live on October 1 and the demand was a lot greater than I was expecting. This resulting in the server becoming overwhelmed and some slow or failed page loads. Being a little unprepared I quickly made new instances of the server and put them all under a load balancer to meet demand. Cloning servers and putting them under a load balancer couldn’t be easier than what is available with Rackspace. This was quick and saved me a lot of pain early on. I then spent some time adding and fine tuning the built-in caching that Yii provides. I had not used caching in Yii before but I was surprised at how easy and effective this is. Although the content should only be cached for 60 seconds on the live feed, the resources being used on the server were dramatically reduced.

This is an example of adding caching to a given part of the site with Yii:

if($this->beginCache('main-content', array(
            'duration'=>60,
            'varyByParam'=>array('filter','club','dateRange'),
            ))) {
                $this->renderPartial('_entryList', array(
                    'dataProvider'=>$dataProvider,
                ));
    $this->endCache(); }

 

This would cache the view to 60 seconds and the varyByParam parameter tells the cache to use GET variables filter, club and dataRange as values to take into account when caching to ensure that each unique request is cached and returned as expected. This is essential as the view has a single URL but the content will change depending on what GET variables are also supplied. If I was to use sessions to keep track of what filters the browser had selected, it would fail through the cache and load balancers so sessions here was not an option.

Overall this was a fun project that required me to provide a solution for an event that I have a lot of interest in. The result is an easy to use CMS with a great user experience in the front-end also.