Over the Christmas break I took on the task of migrating my websites from Rackspace Opencloud to Amazon’s AWS. There were several reasons for doing so but the main ones were because of the ever increasing amount of services I’m using there through work that I want to include in my own projects. I feel I’ve been missing out. Also, the more I that my head is in the AWS ecosystem, the more I’ll learn and be able to pass on to my clients.
As my projects (roughly 7, including this blog) are all rather small, I host them all on the one server instance. This could be a nightmare to migrate but fortunately I had scripted everything with Ansible, making the process fast and straight forward. I first had to tweak my scripts to use PHP7 as I had no yet upgraded my Rackspace instance. By starting out I created a EC2 instance running Debian Jessie, updated .ssh/config with the right credentials and ensure that I could ssh in to the new server. Once verified that all was OK, I ran the Ansible script over the new server which automatically installed:
- Required services such as Nginx, MariaDB, PHP7 and miscellanous tools such as htop, git, vim, etc…
- All the Nginx hosts records
- Any Basic Auth protection I had created for some hosts and paths
- Each database and database users for each project (I’m not using RDS for these small projects)
- Cronjobs and associated scripts that the projects require – which include the onsite backups
- Created a second user that only has privileges to retrieve the backups to store offsite
From there, it was a simple matter of using mysqldump to export all databases from the Rackspace server, SCP them to the new server and import. I then zipped up the web root directory and SCPed them across to the server also. Lastly came the SSL certificates that I needed to move across. Before long I had a fully functioning server created from scratch that included all sites, their data and full backups. I updated the DNS records to match the new IP address and I was done.
I’ve always been interested in computer security and although it’s something I consciously think about when building web applications, it’s not something I’ve ever giving solid time to solely focus on. However, over the last three months I spent all my spare time in the evenings and on weekends working through the Offensive Security certificate, a certificate that is taught by the developers of the Kali Linux distro.
The certificate is a very hands on approach to learning how to compromise computer systems. Along with a guide that goes into detail many of the ways in which vulnerabilities can be found and exploited, you are given access to a VPN with about two dozen vulnerable machines where you can explore and hone your skills. Starting out by scanning networks and profiling each server by learning its operating system, open ports and applications (and their versions) running on each. I found that it can be a tedious exercise but very interesting at the same time. SQL injection was fun but maybe because I’ve played around with that before and already had a great understanding of how it works. The buffer overflow exploits, although tough, was made much easier than I would have guessed because of the tools available today that make attempts quite transparent.
You quickly learn to write your own scripts to automate things that you find yourself repeating. As a result it improved my skills in both Python and Bash. This mostly to do things like scan a network for webservers or servers had MySQL ports open. I thought that sqlmap was a useful tool as it takes the tedious guess work out of finding applications that are not properly escaping user data before running them through an SQL query. The certificate introduces the student to many useful tools ranging from discovery and exploit execution.
It was an exhausting exercise to take outside of my day job but very rewarding. I learned a lot about a topic that has fascinated me for over 20 years and I can use these new skills to build and test that the applications and environments that I build are as secure as possible. In 2017, I will start working with IoT devices and build the APIs that they will communicate with. These devices will need to be secure and not become part of the growing botnets that we read about. Keeping on top of security issues is an ongoing task that I’m glad to be a part of.
Last week I made a presentation at the Melbourne PHP usergroup phpMelb. The presentation was a live demonstration on how someone would go about creating an API with Yii2 from a clean install. I went through the steps of creating a migration and building the model with Gii. Then, following the Yii2 guide on how to turn a controller into a RESTful API controller. The steps are very straight forward and very quick.
I discussed using Chrome extension Postman to create and submit the payload and to view results. I showed how to add behaviours to the models and controllers but could only touch on authentication because of time constraints. I also demonstrated that with the advanced template you can also have a backend which I built again with Gii to show how easy it is to create a web based admin part of the API/site.
You can view the presentation here: https://doublehops.com/presentations/yii2-api-presentation.
I have been building an API with Yii2 for the last four months and and I’m really enjoying it. The developers of Yii have done a tremendous job putting together a framework that makes building applications both fun and fast. The API is to be used by iOS and Android apps as well as a web app (Angular JS) to facilitate phone calls and messages, among other things.
Many of Yii’s strengths come from the clever and well thought-out design that makes common tasks incredibly easy to implement and more complex tasks easy to build whilst keeping the code and structure of your application clean. Some of my favourite attributes to see are:
Building a RESTful service is incredibly quick to build by calling the inbuilt scaffolding tool. It will read the database table to build the model, and also views if you’re building a traditional website. Then with just a few changes in the controller you can start creating, reading, updating and deleting records through your browser.
Attaching a behaviour to a component or controller is similar to extending a class but gives you more control and can act on events. Yii2 has a collection of behaviours that are built in but allows you to create your own. Two built-in behaviors that I’m using are Authenticator and CORS.
Authenticator behaviour: This behaviour allows you to easily implement common authentication models such as HTTP Basic Auth and OAuth. But I was able to add a custom authentication module to the auth behaviour to meet our unique rrequirements.
Yii2 has a great solution for Authentication and has the common ones, such as OAuth and Basic Http Auth built in. But it also quite simple to add your own authentication if need be. The requirements for us was to allow each user to connect to the API with multiple devices. Therefore each device is required to send its individual token for each request made to the API. Yii2 makes this quite trivial.
Yii makes it easy to use use test databases and environment variables which makes testing easy. My solution was to setup an alias domain (env.myapi.test) for each environment. When Yii receives a request on a domain ending with .test it will use test variables and database to run my tests and not pollute the dev or prod data with test data. Every API endpoint is tested and it’s very reassuring watching the tests that you have not broken any endpoints when new features are added.
The issues discussed in this post were easily resolved and surfaced due to my unfamiliarity with the new version of Yii. Issue one fixed with one line of code. When JSON output is required you can set this by adding a new line into the controller action:
\Yii::$app->response->format = \yii\web\Response::FORMAT_JSON;
Issue two is also easily resolved by creating an actions method in your controller and unsetting the default action that you need.
As most things in software development, the things you’re looking for are a lot easier to find when you have a better idea of what you’re looking for.
I have begun playing with Yii2 to build RESTful APIs with the intention of creating backends for single-page websites and mobile apps. Although I am excited at how well Yii2 applications are built I am seeing issues and bugs when following documentation and examples. Unfortunately for me it doesn’t seem that many people are building RESTful apps with Yii2 at this time to ask questions from. Here are some issues that I’m experiencing.
When extending a controller from the RESTful ActiveController the behavior method is no longer available. It triggers an exception with the error:
Response content must not be an array
Is this due to a bug that’s been introduced?
When extending a controller from RESTful ActiveController, the normal action methods are no longer used or can be modified. There doesn’t seem to be a way to add additional functionality because of this such as pagination rules. How would I go about hooking into these actions?
I feel that the documentation I’m reading is from earlier iterations of Yii2 and not valid for the latest version. I would love to hear from anyone who can shed some light on these issues.
Example I’m referring to
In an effort to streamline all work and learning that I do I have put together an Ansible script that will build a web server with Yii2 advanced application template framework ready to start development with the newly released Yii2 PHP framework.
After Vagrant is used to create the virtual machine with Debian 7, Ansible is called to provision the server with PHP5.5, MariaDB and Nginx. It will then setup to virtual host for Yii2 framework with both frontend and backend configuation.
The scripts are available on Github for anyone who wants to have a try with Yii2.
Some time ago I wanted to sink my teeth into Yii2. The completely rewritten framework built with PHP. To do so I decided to build a crypto-currency market watcher. My idea was to build a site that would download all market data at regular intervals from different exchanges and record the values, which could then be graphed and sorted in an effort to watch near real-time market trends.
It proved to be both fun and challenging. What I enjoyed a lot was using Composer to install Yii and third-party packages required to meet some tasks. Quite a lot changed with the new version of Yii. One of those is the way that you create and use components. But once you’ve got it, it makes sense with the new format.
At the time I built components for markets at Cryptsy and Mintpal but it is rather easy to add new crypto exchanges. If you use a marketplace that offer an API then you can most likely add it to Coinhawk.
It really is a pleasure to work with Yii2. If anyone would like to see the code and even run the site themselves, I have made the repo public on Github: https://github.com/doublehops/coinhawk
The repo includes a vagrant machine provisioned with Ansible so it’s incredibly easy for a developer to get started. I would love to hear some feedback
I love Defcon. I attended two years ago and was disappointed in myself for not going in 2013. Defcon is about security and fun. Although there is a lot to learn and share, there are also parties and other events that fill up the days and nights. Despite being to Las Vegas twice now I haven’t really been a tourist there. I spend all of my time at Defcon.
The presentations at Defcon are the most entertaining talks I see at conferences. They are fascinating and educational. Presenters talk about exploits they’ve made or discovered and the impact the vulnerabilities can have to vendors and users. Presenters bring ideas forward that make you question technologies and think about them in new and different ways. One presentation I saw was about a project where a quadcopter was used to fly autonomously over regions, picking up wireless access points and devices and map their locations. In regards to devices it would record their current location, but using the devices broadcast request would record all the locations that device (and therefore device owner’s) that it had ever been. This is scary from a privacy point of view but it’s amazing to think that hardware has become so accessible and cheap now that anyone can start a similar project to this that can achieve so much.
I also sat on a presentation by Ladar Levison from Lavabit. The owner of the company that hosted Edward Snowden’s email before he become a whistle blower. He discussed his new project that would prevent authorities or anyone getting access to your emails. This is a project that you could host yourself if you desired.
Apart from presentations there are many competitions. These range from lock picking to capture-the-packet and capture-the-flag. Each with their own technical challenges that test your understanding of the underlying technologies.
It was also good to spend time with friends and share stories of our own security challenges and concerns. We walk away from the event with a list of tasks that we force upon ourselves to achieve before the next Defcon. I really hope to attend again next year.
It was great to be back in Portland to attend OSCON again and again was inspired by a community of enthusiastic people that truly love open source software. People travel from all over the US and overseas for the event. All with different careers but often using the same tools and always a love for sharing software and experiences.
Topics covered over the five days ranged from web front and backend technologies as well as compiled languages to philosophical talks on what is open source and open source communities and hardware such as RaspberryPi, Arduino wireless scanning devices.
I focused my time on technologies that I already use or intend to use for my work as a web developer. These included AngularJS, PHP and Golang. As all technologies are evolving so quickly it can be hard to keep up so it helps to be at a convention where you are immersed in it all. It was great to hear that Golang and AngularJS really are being embraced in the commercial world as leading technologies.
I was reminded how I want to adapt some hardware into the work I do. Either Arduino, Raspberry Pi and quadcopters. There is so much happening in to tech world and I just want to get immersed in it all.
See you next time, Portland.
I found enough time recently to finally look into Yii2. I decided that a good test project would be to build a crypto currency exchange tracker. It would download latest prices of all markets from both Cryptsy and Mintpal and then display the data in charts so I could quickly scan trends of all currencies.
Yii2 and it’s dependencies can be installed and managed through Composer which I enjoy. It prevents you from needing to keep any third party packages in version control and makes installs, upgrades and deployments easier. The Yii2 documentation is again great and the community is already solid. Any questions I had that were Yii2 specific were answered on the forum in good time. Some things are different and migrating projects from Yii 1.x to 2.x will take a lot of work. Yii2 uses namespaces and this means namespaces need to be declared at the top of views and other files that wasn’t previously necessary. Getting instances of records is slightly different and was changed several times during Yii2’s evolution. However this is stable now.
Many things are still the same. Migrations, scaffolding, commands and nearly everything else is the same. In my opinion, Yii2 is still the best PHP framework and I can’t wait to start a production project with it. Yii2 is still beta but the code base has mostly settled with only bug fixes remaining. My next task is to incorporate AngularJS into my Yii projects.