RSS

AWS CloudFormation

0 Comments | This entry was posted on Apr 17 2019

As I’m building more complex applications to be hosted on AWS, it’s becoming tedious to manually configure the same thing over and over again. This is especially the case when a project requires multiple environments such as production, staging and development.

For this Amazon provide an automation tool named CloudFormation which scripts the architecture to make orchestration easier and less repetitive. The script almost anything that can be manually done through the web console, but commonly includes things such as:

  • EC2 image settings such as type, size and startup scripts
  • Load balancer configuration
  • Security groups
  • Databases (along with users and passwords)
  • S3 buckets

Like any infrastructure as code language or configuration management tool, CloudFormation scripts are great to ensure things are built exactly as intended without forgetting anything, but can also be reused for other projects.

The learning curve is a little slow but there are a lot of examples online to learn from and knowing how to build things using the console makes the transition to CloudFormation straight forward.

I think that if you’re building systems on AWS regularly, it’s worthwhile in getting your head around CloudFormation as it will streamline your work and ensure integrity in your systems.

Value in Gaining AWS Certification

0 Comments | This entry was posted on Feb 20 2019

I’ve been building systems with Linux for 20 years now but things have really changed since the cloud has evolved. Most of my experience with creating hosting solutions was to host web applications that I was working on, either personal or commercial.

Over this time I have moved away from requesting that a data centre provision a new server for me, and then waiting days, to opening up a web console to create new servers on the fly just by clicking a mouse – which creates them immediately. Technology has advanced so much that now Amazon, Google and Microsoft all provide cutting-edge cloud environments to host software applications.

I have been using Amazon’s AWS for over 4 years but had failed to learn a lot of what it offered. This is due to web applications only requiring a small subset of services but also because Amazon keeps adding new services. In order to stay current and to show my skill at developing environments on AWS I thought it was a good idea to get certified. By studying for the exam, what I already knew was reinforced but I also learnt about the things I hadn’t had the time or need to learn.

It was a worthwhile experience and can give potential employers and clients the comfort that I know what I’m doing.

Complex Document Management System Build

0 Comments | This entry was posted on Jun 21 2018

A new client whose job it is to audit water management systems approached me to build a complex document management system to automate an existing system that required a lot of manual labour. The audits of varying intervals result in written reports that needed to be kept for future reference, and notifications need to be sent when the system determines that a report is late. In point form, the major requirements of the new system were:

  • Allow a new site (such as a building) to be added/updated with the following attributes:
    • List or auditing companies for given site
    • Report types and their required intervals
    • Email credentials
  • Include an interface to create a report scanner – which will be used to determine the report type by scan a incoming report (pdf, xls, doc file) and scrape metadata such as report date and report type. This has to be done for each report type for each auditing company.
  • Automatically download from the mail server any audits emailed from any number of reporters for the given site (eg. building)
  • Scan the report using the report scanner, give it a filename that includes relevant metarandata and move to a CDN.
  • Have a reporting page for each site to show the latest reports and show whether they we received on-time, late or overdue.
  • Have another service hosting a Nextcloud service that would make available the reports for a user using their preferred Nextcloud client (web browser or mobile app). Therefore any facility manager could easily see and search for any reports for a location the were responsible for.
  • Include a cronjob to look for any outstanding reports and send notifications to the facility manager for the given location.

Taking in all requirements, I determined AWS was the best platform to host the application as I could easily host the application and Nextcloud, use S3 for the CDN, use SNS for sending notifications and load balancers to place the servers behind.

Ansible was used to provision the local development environments as well as the EC2 instances on AWS for the application servers and Nextcloud servers which were all run on Debian.

I chose Yii2 framework to build the application for several reasons but mainly because I could easily scaffold each form and add validation and the business logic required. A custom component was written to retrieve new documents from the email server and scan them, and running the metadata through the scanner app. The documents were named according to their metadata and then moved across to an S3 bucket. If a document type was not found, then it was copied to a folder waiting to be fixed manually. A cronjob runs daily to scan the S3 folder for report types and check against the location’s report interval setting to determine if any reports were overdue and sent notifications if any were found.

The client is really happy with the application as it has taken away manual work as well as notify when reports are late which wasn’t available before.

Easily Generating New Ansible Playbooks with a Python Script.

0 Comments | This entry was posted on Nov 16 2017


Due to the team continually starting new projects that have different stack requirements, we decided we decided to build a Python script that would read a configuration file that contained which OS (CentOS, Debian, Ubuntu), webserver (Apache or Nginx), database (MySQL, MariaDB) and PHP version (set to latest 7.x). The file also contains project hostname (local dev name only), IP address and local output path.

When the configuration file is run, it will create a full Ansible script in the output path will the correct playbooks for the chosen stack. The same scripts can be used to provision remote servers such as AWS EC2 instances. It really helps in getting the project started quickly.

The project can be found here: https://github.com/doublehops/ansible-builder

Migrating From Rackspace to Amazon AWS.

0 Comments | This entry was posted on Jan 05 2017

Over the Christmas break I took on the task of migrating my websites from Rackspace Opencloud to Amazon’s AWS. There were several reasons for doing so but the main ones were because of the ever increasing amount of services I’m using there through work that I want to include in my own projects. I feel I’ve been missing out. Also, the more I that my head is in the AWS ecosystem, the more I’ll learn and be able to pass on to my clients.

As my projects (roughly 7, including this blog) are all rather small, I host them all on the one server instance. This could be a nightmare to migrate but fortunately I had scripted everything with Ansible, making the process fast and straight forward. I first had to tweak my scripts to use PHP7 as I had no yet upgraded my Rackspace instance. By starting out I created a EC2 instance running Debian Jessie, updated .ssh/config with the right credentials and ensure that I could ssh in to the new server. Once verified that all was OK, I ran the Ansible script over the new server which automatically installed:

  • Required services such as Nginx, MariaDB, PHP7 and miscellanous tools such as htop, git, vim, etc…
  • All the Nginx hosts records
  • Any Basic Auth protection I had created for some hosts and paths
  • Each database and database users for each project (I’m not using RDS for these small projects)
  • Cronjobs and associated scripts that the projects require – which include the onsite backups
  • Created a second user that only has privileges to retrieve the backups to store offsite

From there, it was a simple matter of using mysqldump to export all databases from the Rackspace server, SCP them to the new server and import. I then zipped up the web root directory and SCP’d them across to the server also. Lastly came the SSL certificates that I needed to move across. Before long I had a fully functioning server created from scratch that included all sites, their data and full backups. I updated the DNS records to match the new IP address and I was done.

Offensive Security Training

0 Comments | This entry was posted on Sep 11 2016

I’ve always been interested in computer security and although it’s something I consciously think about when building web applications, it’s not something I’ve ever giving solid time to solely focus on. However, over the last three months I spent all my spare time in the evenings and on weekends working through the Offensive Security certificate, a certificate that is taught by the developers of the Kali Linux distro.

The certificate is a very hands on approach to learning how to compromise computer systems. Along with a guide that goes into detail many of the ways in which vulnerabilities can be found and exploited, you are given access to a VPN with about two dozen vulnerable machines where you can explore and hone your skills. Starting out by scanning networks and profiling each server by learning its operating system, open ports and applications (and their versions) running on each. I found that it can be a tedious exercise but very interesting at the same time. SQL injection was fun but maybe because I’ve played around with that before and already had a great understanding of how it works. The buffer overflow exploits, although tough, was made much easier than I would have guessed because of the tools available today that make attempts quite transparent.

You quickly learn to write your own scripts to automate things that you find yourself repeating. As a result it improved my skills in both Python and Bash. This mostly to do things like scan a network for webservers or servers had MySQL ports open. I thought that sqlmap was a useful tool as it takes the tedious guess work out of finding applications that are not properly escaping user data before running them through an SQL query. The certificate introduces the student to many useful tools ranging from discovery and exploit execution.

It was an exhausting exercise to take outside of my day job but very rewarding. I learned a lot about a topic that has fascinated me for over 20 years and I can use these new skills to build and test that the applications and environments that I build are as secure as possible. In 2017, I will start working with IoT devices and build the APIs that they will communicate with. These devices will need to be secure and not become part of the growing botnets that we read about. Keeping on top of security issues is an ongoing task that I’m glad to be a part of.

Presentation on Building an API with Yii2 at PHP Melbourne

0 Comments | This entry was posted on Mar 22 2016

Last week I made a presentation at the Melbourne PHP usergroup phpMelb. The presentation was a live demonstration on how someone would go about creating an API with Yii2 from a clean install. I went through the steps of creating a migration and building the model with Gii. Then, following the Yii2 guide on how to turn a controller into a RESTful API controller. The steps are very straight forward and very quick.

I discussed using Chrome extension Postman to create and submit the payload and to view results. I showed how to add behaviours to the models and controllers but could only touch on authentication because of time constraints. I also demonstrated that with the advanced template you can also have a backend which I built again with Gii to show how easy it is to create a web based admin part of the API/site.

You can view the presentation here: https://doublehops.com/presentations/yii2-api-presentation.

Building RESTful APIs with Yii2

0 Comments | This entry was posted on Sep 21 2015

I have been building an API with Yii2 for the last four months and and I’m really enjoying it. The developers of Yii have done a tremendous job putting together a framework that makes building applications both fun and fast. The API is to be used by iOS and Android apps as well as a web app (Angular JS) to facilitate phone calls and messages, among other things.

Many of Yii’s strengths come from the clever and well thought-out design that makes common tasks incredibly easy to implement and more complex tasks easy to build whilst keeping the code and structure of your application clean. Some of my favourite attributes to see are:

RESTful services

Building a RESTful service is incredibly quick to build by calling the inbuilt scaffolding tool. It will read the database table to build the model, and also views if you’re building a traditional website. Then with just a few changes in the controller you can start creating, reading, updating and deleting records through your browser.

Behaviours

Attaching a behaviour to a component or controller is similar to extending a class but gives you more control and can act on events. Yii2 has a collection of behaviours that are built in but allows you to create your own. Two built-in behaviors that I’m using are Authenticator and CORS.

Authenticator behaviour: This behaviour allows you to easily implement common authentication models such as HTTP Basic Auth and OAuth. But I was able to add a custom authentication module to the auth behaviour to meet our unique rrequirements.

CORS behaviour: CORS (Cross Origin Resource Sharing) is a standard that allows browsers to make Javascript requests to servers that the Javascript files themselves weren’t server from. This is generally not possible due to security issues. CORS can be quite complicated and can cause major headaches but Yii2 allows you to simply include the behaviour to a controller and the API will respond with the required headers to each CORS request from the browser.

Custom authentication

Yii2 has a great solution for Authentication and has the common ones, such as OAuth and Basic Http Auth built in. But it also quite simple to add your own authentication if need be. The requirements for us was to allow each user to connect to the API with multiple devices. Therefore each device is required to send its individual token for each request made to the API. Yii2 makes this quite trivial.

Testing

Yii makes it easy to use use test databases and environment variables which makes testing easy. My solution was to setup an alias domain (env.myapi.test) for each environment. When Yii receives a request on a domain ending with .test it will use test variables and database to run my tests and not pollute the dev or prod data with test data. Every API endpoint is tested and it’s very reassuring watching the tests that you have not broken any endpoints when new features are added.

Growing pains building RESTful APIs with Yii2

0 Comments | This entry was posted on Feb 26 2015

Update:

The issues discussed in this post were easily resolved and surfaced due to my unfamiliarity with the new version of Yii. Issue one fixed with one line of code. When JSON output is required you can set this by adding a new line into the controller action:

\Yii::$app->response->format = \yii\web\Response::FORMAT_JSON;

Issue two is also easily resolved by creating an actions method in your controller and unsetting the default action that you need.

As most things in software development, the things you’re looking for are a lot easier to find when you have a better idea of what you’re looking for.

Original post:

I have begun playing with Yii2 to build RESTful APIs with the intention of creating backends for single-page websites and mobile apps. Although I am excited at how well Yii2 applications are built I am seeing issues and bugs when following documentation and examples. Unfortunately for me it doesn’t seem that many people are building RESTful apps with Yii2 at this time to ask questions from. Here are some issues that I’m experiencing.

Issue 1

When extending a controller from the RESTful ActiveController the behavior method is no longer available. It triggers an exception with the error:

Response content must not be an array

Is this due to a bug that’s been introduced?

Issue 2

When extending a controller from RESTful ActiveController, the normal action methods are no longer used or can be modified. There doesn’t seem to be a way to add additional functionality because of this such as pagination rules. How would I go about hooking into these actions?

Answers

I feel that the documentation I’m reading is from earlier iterations of Yii2 and not valid for the latest version. I would love to hear from anyone who can shed some light on these issues.

Example I’m referring to

http://www.yiiframework.com/doc-2.0/guide-rest-quick-start.html

Setting up development environments for Yii2 with Vagrant and Ansible

0 Comments | This entry was posted on Feb 18 2015

In an effort to streamline all work and learning that I do I have put together an Ansible script that will build a web server with Yii2 advanced application template framework ready to start development with the newly released Yii2 PHP framework.

After Vagrant is used to create the virtual machine with Debian 7, Ansible is called to provision the server with PHP5.5, MariaDB and Nginx. It will then setup to virtual host for Yii2 framework with both frontend and backend configuation.

The scripts are available on Github for anyone who wants to have a try with Yii2.