Doublehops

A new client whose job it is to audit water management systems approached me to build a complex document management system to automate an existing system that required a lot of manual labour. The audits of varying intervals result in written reports that needed to be kept for future reference, and notifications need to be sent when the system determines that a report is late. In point form, the major requirements of the new system were:

• Allow a new site (such as a building) to be added/updated with the following attributes:
  • List or auditing companies for given site
  • Report types and their required intervals
  • Email credentials
• Include an interface to create a report scanner – which will be used to determine the report type by scan a incoming report (pdf, xls, doc file) and scrape metadata such as report date and report type. This has to be done for each report type for each auditing company.
• Automatically download from the mail server any audits emailed from any number of reporters for the given site (eg. building)
• Scan the report using the report scanner, give it a filename that includes relevant metarandata and move to a CDN.
• Have a reporting page for each site to show the latest reports and show whether they we received on-time, late or overdue.
• Have another service hosting a Nextcloud service that would make available the reports for a user using their preferred Nextcloud client (web browser or mobile app). Therefore any facility manager could easily see and search for any reports for a location the were responsible for.
• Include a cronjob to look for any outstanding reports and send notifications to the facility manager for the given location.

Taking in all requirements, I determined AWS was the best platform to host the application as I could easily host the application and Nextcloud, use S3 for the CDN, use SNS for sending notifications and load balancers to place the servers behind.

Ansible was used to provision the local development environments as well as the EC2 instances on AWS for the application servers and Nextcloud servers which were all run on Debian.

I chose Yii2 framework to build the application for several reasons but mainly because I could easily scaffold each form and add validation and the business logic required. A custom component was written to retrieve new documents from the email server and scan them, and running the metadata through the scanner app. The documents were named according to their metadata and then moved across to an S3 bucket. If a document type was not found, then it was copied to a folder waiting to be fixed manually. A cronjob runs daily to scan the S3 folder for report types and check against the location’s report interval setting to determine if any reports were overdue and sent notifications if any were found.

The client is really happy with the application as it has taken away manual work as well as notify when reports are late which wasn’t available before.

Over the Christmas break I took on the task of migrating my websites from Rackspace Opencloud to Amazon’s AWS. There were several reasons for doing so but the main ones were because of the ever increasing amount of services I’m using there through work that I want to include in my own projects. I feel I’ve been missing out. Also, the more I that my head is in the AWS ecosystem, the more I’ll learn and be able to pass on to my clients.

As my projects (roughly 7, including this blog) are all rather small, I host them all on the one server instance. This could be a nightmare to migrate but fortunately I had scripted everything with Ansible, making the process fast and straight forward. I first had to tweak my scripts to use PHP7 as I had no yet upgraded my Rackspace instance. By starting out I created a EC2 instance running Debian Jessie, updated .ssh/config with the right credentials and ensure that I could ssh in to the new server. Once verified that all was OK, I ran the Ansible script over the new server which automatically installed:

• Required services such as Nginx, MariaDB, PHP7 and miscellanous tools such as htop, git, vim, etc…
• All the Nginx hosts records
• Any Basic Auth protection I had created for some hosts and paths
• Each database and database users for each project (I’m not using RDS for these small projects)
• Cronjobs and associated scripts that the projects require – which include the onsite backups
• Created a second user that only has privileges to retrieve the backups to store offsite

From there, it was a simple matter of using mysqldump to export all databases from the Rackspace server, SCP them to the new server and import. I then zipped up the web root directory and SCP’d them across to the server also. Lastly came the SSL certificates that I needed to move across. Before long I had a fully functioning server created from scratch that included all sites, their data and full backups. I updated the DNS records to match the new IP address and I was done.

I’ve always been interested in computer security and although it’s something I consciously think about when building web applications, it’s not something I’ve ever giving solid time to solely focus on. However, over the last three months I spent all my spare time in the evenings and on weekends working through the Offensive Security certificate, a certificate that is taught by the developers of the Kali Linux distro.

The certificate is a very hands on approach to learning how to compromise computer systems. Along with a guide that goes into detail many of the ways in which vulnerabilities can be found and exploited, you are given access to a VPN with about two dozen vulnerable machines where you can explore and hone your skills. Starting out by scanning networks and profiling each server by learning its operating system, open ports and applications (and their versions) running on each. I found that it can be a tedious exercise but very interesting at the same time. SQL injection was fun but maybe because I’ve played around with that before and already had a great understanding of how it works. The buffer overflow exploits, although tough, was made much easier than I would have guessed because of the tools available today that make attempts quite transparent.

You quickly learn to write your own scripts to automate things that you find yourself repeating. As a result it improved my skills in both Python and Bash. This mostly to do things like scan a network for webservers or servers had MySQL ports open. I thought that sqlmap was a useful tool as it takes the tedious guess work out of finding applications that are not properly escaping user data before running them through an SQL query. The certificate introduces the student to many useful tools ranging from discovery and exploit execution.

It was an exhausting exercise to take outside of my day job but very rewarding. I learned a lot about a topic that has fascinated me for over 20 years and I can use these new skills to build and test that the applications and environments that I build are as secure as possible. In 2017, I will start working with IoT devices and build the APIs that they will communicate with. These devices will need to be secure and not become part of the growing botnets that we read about. Keeping on top of security issues is an ongoing task that I’m glad to be a part of.

Last week I made a presentation at the Melbourne PHP usergroup phpMelb. The presentation was a live demonstration on how someone would go about creating an API with Yii2 from a clean install. I went through the steps of creating a migration and building the model with Gii. Then, following the Yii2 guide on how to turn a controller into a RESTful API controller. The steps are very straight forward and very quick.

I discussed using Chrome extension Postman to create and submit the payload and to view results. I showed how to add behaviours to the models and controllers but could only touch on authentication because of time constraints. I also demonstrated that with the advanced template you can also have a backend which I built again with Gii to show how easy it is to create a web based admin part of the API/site.

You can view the presentation here: https://doublehops.com/presentations/yii2-api-presentation.

I have been building an API with Yii2 for the last four months and and I’m really enjoying it. The developers of Yii have done a tremendous job putting together a framework that makes building applications both fun and fast. The API is to be used by iOS and Android apps as well as a web app (Angular JS) to facilitate phone calls and messages, among other things.

Many of Yii’s strengths come from the clever and well thought-out design that makes common tasks incredibly easy to implement and more complex tasks easy to build whilst keeping the code and structure of your application clean. Some of my favourite attributes to see are:

RESTful services

Building a RESTful service is incredibly quick to build by calling the inbuilt scaffolding tool. It will read the database table to build the model, and also views if you’re building a traditional website. Then with just a few changes in the controller you can start creating, reading, updating and deleting records through your browser.

Behaviours

Attaching a behaviour to a component or controller is similar to extending a class but gives you more control and can act on events. Yii2 has a collection of behaviours that are built in but allows you to create your own. Two built-in behaviors that I’m using are Authenticator and CORS.

Authenticator behaviour: This behaviour allows you to easily implement common authentication models such as HTTP Basic Auth and OAuth. But I was able to add a custom authentication module to the auth behaviour to meet our unique rrequirements.

CORS behaviour: CORS (Cross Origin Resource Sharing) is a standard that allows browsers to make Javascript requests to servers that the Javascript files themselves weren’t server from. This is generally not possible due to security issues. CORS can be quite complicated and can cause major headaches but Yii2 allows you to simply include the behaviour to a controller and the API will respond with the required headers to each CORS request from the browser.

Custom authentication

Yii2 has a great solution for Authentication and has the common ones, such as OAuth and Basic Http Auth built in. But it also quite simple to add your own authentication if need be. The requirements for us was to allow each user to connect to the API with multiple devices. Therefore each device is required to send its individual token for each request made to the API. Yii2 makes this quite trivial.

Testing

Yii makes it easy to use use test databases and environment variables which makes testing easy. My solution was to setup an alias domain (env.myapi.test) for each environment. When Yii receives a request on a domain ending with .test it will use test variables and database to run my tests and not pollute the dev or prod data with test data. Every API endpoint is tested and it’s very reassuring watching the tests that you have not broken any endpoints when new features are added.

Update:

The issues discussed in this post were easily resolved and surfaced due to my unfamiliarity with the new version of Yii. Issue one fixed with one line of code. When JSON output is required you can set this by adding a new line into the controller action:

\Yii::$app->response->format = \yii\web\Response::FORMAT_JSON;

Issue two is also easily resolved by creating an actions method in your controller and unsetting the default action that you need.

As most things in software development, the things you’re looking for are a lot easier to find when you have a better idea of what you’re looking for.

Original post:

I have begun playing with Yii2 to build RESTful APIs with the intention of creating backends for single-page websites and mobile apps. Although I am excited at how well Yii2 applications are built I am seeing issues and bugs when following documentation and examples. Unfortunately for me it doesn’t seem that many people are building RESTful apps with Yii2 at this time to ask questions from. Here are some issues that I’m experiencing.

Issue 1

When extending a controller from the RESTful ActiveController the behavior method is no longer available. It triggers an exception with the error:

Response content must not be an array

Is this due to a bug that’s been introduced?

Issue 2

When extending a controller from RESTful ActiveController, the normal action methods are no longer used or can be modified. There doesn’t seem to be a way to add additional functionality because of this such as pagination rules. How would I go about hooking into these actions?

Answers

I feel that the documentation I’m reading is from earlier iterations of Yii2 and not valid for the latest version. I would love to hear from anyone who can shed some light on these issues.

Example I’m referring to

http://www.yiiframework.com/doc-2.0/guide-rest-quick-start.html

In an effort to streamline all work and learning that I do I have put together an Ansible script that will build a web server with Yii2 advanced application template framework ready to start development with the newly released Yii2 PHP framework.

After Vagrant is used to create the virtual machine with Debian 7, Ansible is called to provision the server with PHP5.5, MariaDB and Nginx. It will then setup to virtual host for Yii2 framework with both frontend and backend configuation.

The scripts are available on Github for anyone who wants to have a try with Yii2.

Some time ago I wanted to sink my teeth into Yii2. The completely rewritten framework built with PHP. To do so I decided to build a crypto-currency market watcher. My idea was to build a site that would download all market data at regular intervals from different exchanges and record the values, which could then be graphed and sorted in an effort to watch near real-time market trends.

It proved to be both fun and challenging. What I enjoyed a lot was using Composer to install Yii and third-party packages required to meet some tasks. Quite a lot changed with the new version of Yii. One of those is the way that you create and use components. But once you’ve got it, it makes sense with the new format.

At the time I built components for markets at Cryptsy and Mintpal but it is rather easy to add new crypto exchanges. If you use a marketplace that offer an API then you can most likely add it to Coinhawk.

It really is a pleasure to work with Yii2. If anyone would like to see the code and even run the site themselves, I have made the repo public on Github: https://github.com/doublehops/coinhawk

The repo includes a vagrant machine provisioned with Ansible so it’s incredibly easy for a developer to get started. I would love to hear some feedback

It was great to be back in Portland to attend OSCON again and again was inspired by a community of enthusiastic people that truly love open source software. People travel from all over the US and overseas for the event. All with different careers but often using the same tools and always a love for sharing software and experiences.

Topics covered over the five days ranged from web front and backend technologies as well as compiled languages to philosophical talks on what is open source and open source communities and hardware such as RaspberryPi, Arduino wireless scanning devices.

I focused my time on technologies that I already use or intend to use for my work as a web developer. These included AngularJS, PHP and Golang. As all technologies are evolving so quickly it can be hard to keep up so it helps to be at a convention where you are immersed in it all. It was great to hear that Golang and AngularJS really are being embraced in the commercial world as leading technologies.

I was reminded how I want to adapt some hardware into the work I do. Either Arduino, Raspberry Pi and quadcopters. There is so much happening in to tech world and I just want to get immersed in it all.

See you next time, Portland.

I found enough time recently to finally look into Yii2. I decided that a good test project would be to build a crypto currency exchange tracker. It would download latest prices of all markets from both Cryptsy and Mintpal and then display the data in charts so I could quickly scan trends of all currencies.

Yii2 and it’s dependencies can be installed and managed through Composer which I enjoy. It prevents you from needing to keep any third party packages in version control and makes installs, upgrades and deployments easier. The Yii2 documentation is again great and the community is already solid. Any questions I had that were Yii2 specific were answered on the forum in good time. Some things are different and migrating projects from Yii 1.x to 2.x will take a lot of work. Yii2 uses namespaces and this means namespaces need to be declared at the top of views and other files that wasn’t previously necessary. Getting instances of records is slightly different and was changed several times during Yii2’s evolution. However this is stable now.

Many things are still the same. Migrations, scaffolding, commands and nearly everything else is the same. In my opinion, Yii2 is still the best PHP framework and I can’t wait to start a production project with it. Yii2 is still beta but the code base has mostly settled with only bug fixes remaining. My next task is to incorporate AngularJS into my Yii projects.