RSS

Complex Document Management System Build

0 Comments | This entry was posted on Jun 21 2018

A new client whose job it is to audit water management systems approached me to build a complex document management system to automate an existing system that required a lot of manual labour. The audits of varying intervals result in written reports that needed to be kept for future reference, and notifications need to be sent when the system determines that a report is late. In point form, the major requirements of the new system were:

  • Allow a new site (such as a building) to be added/updated with the following attributes:
    • List or auditing companies for given site
    • Report types and their required intervals
    • Email credentials
  • Include an interface to create a report scanner – which will be used to determine the report type by scan a incoming report (pdf, xls, doc file) and scrape metadata such as report date and report type. This has to be done for each report type for each auditing company.
  • Automatically download from the mail server any audits emailed from any number of reporters for the given site (eg. building)
  • Scan the report using the report scanner, give it a filename that includes relevant metarandata and move to a CDN.
  • Have a reporting page for each site to show the latest reports and show whether they we received on-time, late or overdue.
  • Have another service hosting a Nextcloud service that would make available the reports for a user using their preferred Nextcloud client (web browser or mobile app). Therefore any facility manager could easily see and search for any reports for a location the were responsible for.
  • Include a cronjob to look for any outstanding reports and send notifications to the facility manager for the given location.

Taking in all requirements, I determined AWS was the best platform to host the application as I could easily host the application and Nextcloud, use S3 for the CDN, use SNS for sending notifications and load balancers to place the servers behind.

Ansible was used to provision the local development environments as well as the EC2 instances on AWS for the application servers and Nextcloud servers which were all run on Debian.

I chose Yii2 framework to build the application for several reasons but mainly because I could easily scaffold each form and add validation and the business logic required. A custom component was written to retrieve new documents from the email server and scan them, and running the metadata through the scanner app. The documents were named according to their metadata and then moved across to an S3 bucket. If a document type was not found, then it was copied to a folder waiting to be fixed manually. A cronjob runs daily to scan the S3 folder for report types and check against the location’s report interval setting to determine if any reports were overdue and sent notifications if any were found.

The client is really happy with the application as it has taken away manual work as well as notify when reports are late which wasn’t available before.

Easily Generating New Ansible Playbooks with Python Script.

0 Comments | This entry was posted on Nov 16 2017


Due to the team continually starting new projects that have different stack requirements, we decided we decided to build a Python script that would read a configuration file that contained which OS (CentOS, Debian, Ubuntu), webserver (Apache or Nginx), database (MySQL, MariaDB) and PHP version (set to latest 7.x). The file also contains project hostname (local dev name only), IP address and local output path.

When the configuration file is run, it will create a full Ansible script in the output path will the correct playbooks for the chosen stack. The same scripts can be used to provision remote servers such as AWS EC2 instances. It really helps in getting the project started quickly.

The project can be found here: https://github.com/doublehops/ansible-builder

Migrating From Rackspace to Amazon AWS.

0 Comments | This entry was posted on Jan 05 2017

Over the Christmas break I took on the task of migrating my websites from Rackspace Opencloud to Amazon’s AWS. There were several reasons for doing so but the main ones were because of the ever increasing amount of services I’m using there through work that I want to include in my own projects. I feel I’ve been missing out. Also, the more I that my head is in the AWS ecosystem, the more I’ll learn and be able to pass on to my clients.

As my projects (roughly 7, including this blog) are all rather small, I host them all on the one server instance. This could be a nightmare to migrate but fortunately I had scripted everything with Ansible, making the process fast and straight forward. I first had to tweak my scripts to use PHP7 as I had no yet upgraded my Rackspace instance. By starting out I created a EC2 instance running Debian Jessie, updated .ssh/config with the right credentials and ensure that I could ssh in to the new server. Once verified that all was OK, I ran the Ansible script over the new server which automatically installed:

  • Required services such as Nginx, MariaDB, PHP7 and miscellanous tools such as htop, git, vim, etc…
  • All the Nginx hosts records
  • Any Basic Auth protection I had created for some hosts and paths
  • Each database and database users for each project (I’m not using RDS for these small projects)
  • Cronjobs and associated scripts that the projects require – which include the onsite backups
  • Created a second user that only has privileges to retrieve the backups to store offsite

From there, it was a simple matter of using mysqldump to export all databases from the Rackspace server, SCP them to the new server and import. I then zipped up the web root directory and SCP’d them across to the server also. Lastly came the SSL certificates that I needed to move across. Before long I had a fully functioning server created from scratch that included all sites, their data and full backups. I updated the DNS records to match the new IP address and I was done.

Offensive Security Training

0 Comments | This entry was posted on Sep 11 2016

I’ve always been interested in computer security and although it’s something I consciously think about when building web applications, it’s not something I’ve ever giving solid time to solely focus on. However, over the last three months I spent all my spare time in the evenings and on weekends working through the Offensive Security certificate, a certificate that is taught by the developers of the Kali Linux distro.

The certificate is a very hands on approach to learning how to compromise computer systems. Along with a guide that goes into detail many of the ways in which vulnerabilities can be found and exploited, you are given access to a VPN with about two dozen vulnerable machines where you can explore and hone your skills. Starting out by scanning networks and profiling each server by learning its operating system, open ports and applications (and their versions) running on each. I found that it can be a tedious exercise but very interesting at the same time. SQL injection was fun but maybe because I’ve played around with that before and already had a great understanding of how it works. The buffer overflow exploits, although tough, was made much easier than I would have guessed because of the tools available today that make attempts quite transparent.

You quickly learn to write your own scripts to automate things that you find yourself repeating. As a result it improved my skills in both Python and Bash. This mostly to do things like scan a network for webservers or servers had MySQL ports open. I thought that sqlmap was a useful tool as it takes the tedious guess work out of finding applications that are not properly escaping user data before running them through an SQL query. The certificate introduces the student to many useful tools ranging from discovery and exploit execution.

It was an exhausting exercise to take outside of my day job but very rewarding. I learned a lot about a topic that has fascinated me for over 20 years and I can use these new skills to build and test that the applications and environments that I build are as secure as possible. In 2017, I will start working with IoT devices and build the APIs that they will communicate with. These devices will need to be secure and not become part of the growing botnets that we read about. Keeping on top of security issues is an ongoing task that I’m glad to be a part of.

Presentation on Building an API with Yii2 at PHP Melbourne

0 Comments | This entry was posted on Mar 22 2016

Last week I made a presentation at the Melbourne PHP usergroup phpMelb. The presentation was a live demonstration on how someone would go about creating an API with Yii2 from a clean install. I went through the steps of creating a migration and building the model with Gii. Then, following the Yii2 guide on how to turn a controller into a RESTful API controller. The steps are very straight forward and very quick.

I discussed using Chrome extension Postman to create and submit the payload and to view results. I showed how to add behaviours to the models and controllers but could only touch on authentication because of time constraints. I also demonstrated that with the advanced template you can also have a backend which I built again with Gii to show how easy it is to create a web based admin part of the API/site.

You can view the presentation here: https://doublehops.com/presentations/yii2-api-presentation.

Building RESTful APIs with Yii2

0 Comments | This entry was posted on Sep 21 2015

I have been building an API with Yii2 for the last four months and and I’m really enjoying it. The developers of Yii have done a tremendous job putting together a framework that makes building applications both fun and fast. The API is to be used by iOS and Android apps as well as a web app (Angular JS) to facilitate phone calls and messages, among other things.

Many of Yii’s strengths come from the clever and well thought-out design that makes common tasks incredibly easy to implement and more complex tasks easy to build whilst keeping the code and structure of your application clean. Some of my favourite attributes to see are:

RESTful services

Building a RESTful service is incredibly quick to build by calling the inbuilt scaffolding tool. It will read the database table to build the model, and also views if you’re building a traditional website. Then with just a few changes in the controller you can start creating, reading, updating and deleting records through your browser.

Behaviours

Attaching a behaviour to a component or controller is similar to extending a class but gives you more control and can act on events. Yii2 has a collection of behaviours that are built in but allows you to create your own. Two built-in behaviors that I’m using are Authenticator and CORS.

Authenticator behaviour: This behaviour allows you to easily implement common authentication models such as HTTP Basic Auth and OAuth. But I was able to add a custom authentication module to the auth behaviour to meet our unique rrequirements.

CORS behaviour: CORS (Cross Origin Resource Sharing) is a standard that allows browsers to make Javascript requests to servers that the Javascript files themselves weren’t server from. This is generally not possible due to security issues. CORS can be quite complicated and can cause major headaches but Yii2 allows you to simply include the behaviour to a controller and the API will respond with the required headers to each CORS request from the browser.

Custom authentication

Yii2 has a great solution for Authentication and has the common ones, such as OAuth and Basic Http Auth built in. But it also quite simple to add your own authentication if need be. The requirements for us was to allow each user to connect to the API with multiple devices. Therefore each device is required to send its individual token for each request made to the API. Yii2 makes this quite trivial.

Testing

Yii makes it easy to use use test databases and environment variables which makes testing easy. My solution was to setup an alias domain (env.myapi.test) for each environment. When Yii receives a request on a domain ending with .test it will use test variables and database to run my tests and not pollute the dev or prod data with test data. Every API endpoint is tested and it’s very reassuring watching the tests that you have not broken any endpoints when new features are added.

Growing pains building RESTful APIs with Yii2

0 Comments | This entry was posted on Feb 26 2015

Update:

The issues discussed in this post were easily resolved and surfaced due to my unfamiliarity with the new version of Yii. Issue one fixed with one line of code. When JSON output is required you can set this by adding a new line into the controller action:

\Yii::$app->response->format = \yii\web\Response::FORMAT_JSON;

Issue two is also easily resolved by creating an actions method in your controller and unsetting the default action that you need.

As most things in software development, the things you’re looking for are a lot easier to find when you have a better idea of what you’re looking for.

Original post:

I have begun playing with Yii2 to build RESTful APIs with the intention of creating backends for single-page websites and mobile apps. Although I am excited at how well Yii2 applications are built I am seeing issues and bugs when following documentation and examples. Unfortunately for me it doesn’t seem that many people are building RESTful apps with Yii2 at this time to ask questions from. Here are some issues that I’m experiencing.

Issue 1

When extending a controller from the RESTful ActiveController the behavior method is no longer available. It triggers an exception with the error:

Response content must not be an array

Is this due to a bug that’s been introduced?

Issue 2

When extending a controller from RESTful ActiveController, the normal action methods are no longer used or can be modified. There doesn’t seem to be a way to add additional functionality because of this such as pagination rules. How would I go about hooking into these actions?

Answers

I feel that the documentation I’m reading is from earlier iterations of Yii2 and not valid for the latest version. I would love to hear from anyone who can shed some light on these issues.

Example I’m referring to

http://www.yiiframework.com/doc-2.0/guide-rest-quick-start.html

Setting up development environments for Yii2 with Vagrant and Ansible

0 Comments | This entry was posted on Feb 18 2015

In an effort to streamline all work and learning that I do I have put together an Ansible script that will build a web server with Yii2 advanced application template framework ready to start development with the newly released Yii2 PHP framework.

After Vagrant is used to create the virtual machine with Debian 7, Ansible is called to provision the server with PHP5.5, MariaDB and Nginx. It will then setup to virtual host for Yii2 framework with both frontend and backend configuation.

The scripts are available on Github for anyone who wants to have a try with Yii2.

Crypto-Currency Market Watcher Built with Yii2

0 Comments | This entry was posted on Feb 02 2015

Some time ago I wanted to sink my teeth into Yii2. The completely rewritten framework built with PHP. To do so I decided to build a crypto-currency market watcher. My idea was to build a site that would download all market data at regular intervals from different exchanges and record the values, which could then be graphed and sorted in an effort to watch near real-time market trends.

It proved to be both fun and challenging. What I enjoyed a lot was using Composer to install Yii and third-party packages required to meet some tasks. Quite a lot changed with the new version of Yii. One of those is the way that you create and use components. But once you’ve got it, it makes sense with the new format.

At the time I built components for markets at Cryptsy and Mintpal but it is rather easy to add new crypto exchanges. If you use a marketplace that offer an API then you can most likely add it to Coinhawk.

It really is a pleasure to work with Yii2. If anyone would like to see the code and even run the site themselves, I have made the repo public on Github: https://github.com/doublehops/coinhawk

The repo includes a vagrant machine provisioned with Ansible so it’s incredibly easy for a developer to get started. I would love to hear some feedback

Defcon 22 – Security Conference

0 Comments | This entry was posted on Aug 13 2014

I love Defcon. I attended two years ago and was disappointed in myself for not going in 2013. Defcon is about security and fun. Although there is a lot to learn and share, there are also parties and other events that fill up the days and nights. Despite being to Las Vegas twice now I haven’t really been a tourist there. I spend all of my time at Defcon.

The presentations at Defcon are the most entertaining talks I see at conferences. They are fascinating and educational. Presenters talk about exploits they’ve made or discovered and the impact the vulnerabilities can have to vendors and users. Presenters bring ideas forward that make you question technologies and think about them in new and different ways. One presentation I saw was about a project where a quadcopter was used to fly autonomously over regions, picking up wireless access points and devices and map their locations. In regards to devices it would record their current location, but using the devices broadcast request would record all the locations that device (and therefore device owner’s) that it had ever been. This is scary from a privacy point of view but it’s amazing to think that hardware has become so accessible and cheap now that anyone can start a similar project to this that can achieve so much.

I also sat on a presentation by Ladar Levison from Lavabit. The owner of the company that hosted Edward Snowden’s email before he become a whistle blower. He discussed his new project that would prevent authorities or anyone getting access to your emails. This is a project that you could host yourself if you desired.

Apart from presentations there are many competitions. These range from lock picking to capture-the-packet and capture-the-flag. Each with their own technical challenges that test your understanding of the underlying technologies.

It was also good to spend time with friends and share stories of our own security challenges and concerns. We walk away from the event with a list of tasks that we force upon ourselves to achieve before the next Defcon. I really hope to attend again next year.