How-To Tutorials

article-image-ensuring-five-star-rating-marketplace

05 May 2015

43 min read

Ensuring Five-star Rating in the MarketPlace

05 May 2015

0
0
1847

Packt

05 May 2015

8 min read

Installation and Upgrade

Packt

05 May 2015

8 min read

0
0
1851

article-image-symmetric-messages-and-asynchronous-messages-part-1

Packt

05 May 2015

31 min read

Symmetric Messages and Asynchronous Messages (Part 1)

Packt

05 May 2015

31 min read

0
0
9638

Packt

04 May 2015

7 min read

Git Teaches – Great Tools Don't Make Great Craftsmen

Packt

04 May 2015

7 min read

This article is written by Ferdinando Santacroce, author of the book, Git Essentials. (For more resources related to this topic, see here.) Git is a powerful tool. In case you need to retain multiple versions of files—even if you may not be a software developer—Git can perform this task easily. As a Git user, in my humble career, I have never found a dead-end street—a circumstance where I had to give up because of a lack of solutions. Git always offers a wide range of alternatives even when you make a mistake; you can use either git revert to revert your change, or git reset if there is no need to preserve the previous commit. Another key strength of Git is its ability to let your project grow and take different ways when needed. Git branching is a killer feature of this tool. Every versioning system is able to manage branches. However in Git, using this feature is a pleasure; it is super-fast (it does all the work locally), and it does not require a great amount of space. For those who are used to work with other versioning systems such as Subversion, this probably makes a difference. In my career as a developer, I have assisted in situations where developers wouldn't create new branches for new features, because branching was a time-consuming process. Their versioning system, on large repositories, required 5-6 minutes to create a new branch. Git usually doesn't concede alibis. The git branch command and the consecutive git merge operations are fast and very reliable. Even when you commit, git commit doesn't allow you to store a new commit without a message to protect you from our laziness and grow a talking repository with a clear history and not a mute one. However, Git can't perform miracles. So, to get the most out of it, we need a little discipline. This discipline distinguishes an apprentice from a good craftsman. One of the most difficult things in software development is with regard to the sharing of a common code base. Often, programmers are solitary people who love instructions that are typed in their preferred editor, which helps them make working software without any hassles. However, in professional software development, you usually deal with big projects that require more than a single developer at a time; everyone contributes their own code. At this point, if you don't have an effective tool to share code like Git and a little bit of discipline, you can easily screw up. When I talk about discipline, I talk about two main concepts—writing good commits and using the right workflow. Let's start with the first point. What is a good commit? What makes a commit either good or bad? Well, you will come across highly opinionated answers to this question. So here, I will provide mine. First of all, good commits are those commits that do not mix apples and oranges; when you commit something, you have to focus on resolving one problem at a time (fix a bug and implement a new feature or make a clear step forward towards the final target) without modifying anything that is not strictly related to the task you are working on. While writing some code, especially when you have to refactor or modify the existing code, you may too often fall into the temptation to fix here and there some other things. This is just your nature, I know. Developers hate ugly code, even though they often are the ones who wrote it some time ago; they can't leave it there even for a minute. It's a compulsive reaction. So, in a matter of a few minutes, you end up with a ton of modified files with dozens of cross-modifications that are quite difficult to comment in a commit message. They are also hard to merge and quite impossible to cherry-pick, if necessary. So, one of the first things that you have to learn is to make consistent commits. It has to became a habit, and we all know that habits are hard to grow and hard to break. There are some simple tricks that have helped me become a better committer day by day (yes, I'm still far from becoming a good committer). One of the most effective tricks that you can use to make consistent commits is to have a pencil and a paper with you; when you find something wrong with your code that is not related to what you are working on at the moment, pick up the pencil and write down a note on a piece of paper. Don't work on it immediately. However, remind yourself that there is something that you have to fix in the next commit. In the same manner, when you feel that the feature you're going to implement either is too wide for a single commit, or requires more than a bunch of hours to terminate (I tend to avoid long coding sessions), make an effort and try to split the work in two or three parts, writing down these steps on the paper. Thus, you are unconsciously wrapping up your next commits. Another way to avoid a loss of focus is to write the commit message before you start coding. This may sound a little weird, but having the target of your actual work in front of your eyes helps a lot. If you practice Test Driver Development (TDD), or even better, Behavior Drive Development (BDD), you probably already know that they have a huge side-effect despite their main testing purpose—they force you to look at the final results, maintaining the focus on what the code has to do, and not the implementation details. Writing preemptive commit messages is the same thing. When the target of your commit is clear and you can keep an eye on it every time you look at your paper notebook, then you know that you can code peacefully because you will not go off the rails. Now that we have a clear vision of what makes a good commit, let's move your attention to good workflows. Generally speaking, the sharing of a common way to work is the most taken-for-granted advice that you can give. However, it often represents exactly the biggest problem when you look at underperforming firms. Having a versioning workflow that is decided with the help of common agreement is the most important thing about a development team (even for a team of one) because it lets you feel comfortable even in case of emergency. When I talk about emergencies, I talk about common hitches for a software developer—urgently fixing a bug on a specific software version, developing different features in parallel, and building beta or testing versions to let testers and users give you feedback. There are plenty of good Git workflows out there. You can take inspiration from them. You can use a workflow as it is, or you can take some inspiration and adapt it to fit your project peculiarities. However, the important thing is that you have to keep on using it not only to be consistent (don't cheat!), but also to adapt it when premises change. Don't blindly follow a workflow if you don't feel comfortable with it, and don't even try to use the same workflow every time. There are good workflows for web projects where there's usually no need to keep multiple versions of the same software and the ones that fit desktop applications better, where multiple versions are the order of the day. Every kind of project needs its perfectly tailored workflow. The last thing I wish to suggest to the developers interested in Git is to share common sense. Good developers share coding standards, and a good team has to share the same committing policy and the same workflow. Lone cowboys and outlaws represent a problem even in software development, and not just in Spaghetti Western movies. Resources for Article: Further resources on this subject: Configuration [article] Maintaining Your GitLab Instance [article] Searching and Resolving Conflicts [article]

0
0
1335

Packt

04 May 2015

8 min read

Using Firebase: Learn how and why to use Firebase

Packt

04 May 2015

8 min read

In this article by Manoj Waikar, author of the book Data-oriented Development with AngularJS, we will learn a brief description about various types of persistence mechanisms, local versus hosted databases, what Firebase is, why to use it, and different use cases where Firebase can be useful. (For more resources related to this topic, see here.) We can write web applications by using the frameworks of our choice—be it server-side MVC frameworks, client-side MVC frameworks, or some combination of these. We can also use a persistence store (a database) of our choice—be it an RDBMS or a more modern NoSQL store. However, making our applications real time (meaning, if you are viewing a page and data related to that page gets updated, then the page should be updated or at least you should get a notification to refresh the page) is not a trivial task and we have to start thinking about push notifications and what not. This does not happen with Firebase. Persistence One of the very early decisions a developer or a team has to make when building any production-quality application is the choice of a persistent storage mechanism. Until a few years ago, this choice, more often than not, boiled down to a relational database such as Oracle, SQL Server, or PostgreSQL. However, the rise of NoSQL solutions such as MongoDB (http://www.mongodb.org/) and CouchDB (http://couchdb.apache.org/)—document-oriented databases or Redis (http://redis.io/), Riak (http://basho.com/riak/), keyvalue stores, Neo4j (http://www.neo4j.org/), and a graph database—has widened the choice for us. Please check the Wikipedia page on NoSQL (http://en.wikipedia.org/wiki/NoSQL) solutions for a detailed list of various NoSQL solutions including their classification and performance characteristics. There is one more buzzword that everyone must have already heard of, Cloud, the short form for cloud computing. Cloud computing briefly means that shared resources (or software) are provided to consumers on a paid/free basis over a network (typically, the Internet). So, we now have the luxury of choosing our preferred RDBMS or NoSQL database as a hosted solution. Consequently, we have one more choice to make—whether to install the database locally (on our own machine or inside the corporate network) or use a hosted solution (in the cloud). As with everything else, there are pros and cons to each of the approaches. The pros of a local database are fast access and one-time buying cost (if it's not an open source database), and the cons include the initial setup time. If you have to evaluate some another database, then you'll have to install the other database as well. The pros of a hosted solution are ease of use and minimal initial setup time, and the cons are the need for a reliable Internet connection, cost (again, if it's not a free option), and so on. Considering the preceding pros and cons, it's a safe bet to use a hosted solution when you are still evaluating different databases and only decide later between a local or a hosted solution, when you've finally zeroed in on your database of choice. What is Firebase? So, where does Firebase fit into all of this? Firebase is a NoSQL database that stores data as simple JSON documents. We can, therefore, compare it to other document-oriented databases such as CouchDB (which also stores data as JSON) or MongoDB (which stores data in the BSON, which stands for binary JSON, format). Although Firebase is a database with a RESTful API, it's also a real-time database, which means that the data is synchronized between different clients and with the backend server almost instantaneously. This implies that if the underlying data is changed by one of the clients, it gets streamed in real time to every connected client; hence, all the other clients automatically get updates with the newest set of data (without anyone having to refresh these clients manually). So, to summarize, Firebase is an API and a cloud service that gives us a real-time and scalable (NoSQL) backend. It has libraries for most server-side languages/frameworks such as Node.js, Java, Python, PHP, Ruby, and Clojure. It has official libraries for Node.js and Java and unofficial third-party libraries for Python, Ruby, and PHP. It also has libraries for most of the leading client-side frameworks such as AngularJS, Backbone, Ember, React, and mobile platforms such as iOS and Android. Firebase – Benefits and why to use? Firebase offers us the following benefits: It is a cloud service (a hosted solution), so there isn't any setup involved. Data is stored as native JSON, so what you store is what you see (on the frontend, fetched through a REST API)—WYSIWYS. Data is safe because Firebase requires 2048-bit SSL encryption for all data transfers. Data is replicated and backed-up to multiple secure locations, so there are minimal chances of data loss. When data changes, apps update instantly across devices. Our apps can work offline—as soon as we get connectivity, the data is synchronized instantly. Firebase gives us lightning fast data synchronization. So, combined with AngularJS, it gives us three-way data binding between HTML, JavaScript, and our backend (data). With two-way data binding, whenever our (JavaScript) model changes, the view (HTML) updates itself and vice versa. But, with three-way data binding, even when the data in our database changes, our JavaScript model gets updated, and consequently, the view gets updated as well. Last but not the least, it has libraries for the most popular server-side languages/frameworks (such as Node.js, Ruby, Java, and Python) as well as the popular client-side frameworks (such as Backbone, Ember, and React), including AngularJS. The Firebase binding for AngularJS is called AngularFire (https://www.firebase.com/docs/web/libraries/angular/). Firebase use cases Now that you've read how Firebase makes it easy to write applications that update in real time, you might still be wondering what kinds of applications are most suited for use with Firebase. Because, as often happens in the enterprise world, either you are not at liberty to choose all the components of your stack or you might have an existing application and you just have to add some new features to it. So, let's study the three main scenarios where Firebase can be a good fit for your needs. Apps with Firebase as the only backend This scenario is feasible if: You are writing a brand-new application or rewriting an existing one from scratch You don't have to integrate with legacy systems or other third-party services Your app doesn't need to do heavy data processing or it doesn't have complex user authentication requirements In such scenarios, Firebase is the only backend store you'll need and all dynamic content and user data can be stored and retrieved from it. Existing apps with some features powered by Firebase This scenario is feasible if you already have a site and want to add some real-time capabilities to it without touching other parts of the system. For example, you have a working website and just want to add chat capabilities, or maybe, you want to add a comment feed that updates in real time or you have to show some real-time notifications to your users. In this case, the clients can connect to your existing server (for existing features) and they can connect to Firebase for the newly added real-time capabilities. So, you can use Firebase together with the existing server. Both client and server code powered by Firebase In some use cases, there might be computationally intensive code that can't be run on the client. In situations like these, Firebase can act as an intermediary between the server and your clients. So, the server talks to the clients by manipulating data in Firebase. The server can connect to Firebase using either the Node.js library (for Node.js-based server-side applications) or through the REST API (for other server-side languages). Similarly, the server can listen to the data changes made by the clients and can respond appropriately. For example, the client can place tasks in a queue that the server will process later. One or more servers can then pick these tasks from the queue and do the required processing (as per their availability) and then place the results back in Firebase so that the clients can read them. Firebase is the API for your product You might not have realized by now (but you will once you see some examples) that as soon as we start saving data in Firebase, the REST API keeps building side-by-side for free because of the way data is stored as a JSON tree and is associated on different URLs. Think for a moment if you had a relational database as your persistence store; you would then need to specially write REST APIs (which are obviously preferable to old RPC-style web services) by using the framework available for your programming language to let external teams or customers get access to data. Then, if you wanted to support different platforms, you would need to provide libraries for all those platforms whereas Firebase already provides real-time SDKs for JavaScript, Objective-C, and Java. So, Firebase is not just a real-time persistence store, but it doubles up as an API layer too. Summary In this article, we learned a brief description about Firebase is, why to use it, and different use cases where Firebase can be useful. Resources for Article: Further resources on this subject: AngularJS Performance [article] An introduction to testing AngularJS directives [article] Our App and Tool Stack [article]

0
0
5775

article-image-getting-started-codeception

Packt

04 May 2015

17 min read

Getting started with Codeception

Packt

04 May 2015

17 min read

In this article by Matteo Pescarin, the author of Learning Yii Testing, we will get introduced to Codeception. Not everyone has been exposed to testing. The ones who actually have are aware of the quirks and limitations of the testing tools they've used. Some might be more efficient than others, and in either case, you had to rely on the situation that was presented to you: legacy code, hard to test architectures, no automation, no support whatsoever on the tools, and other setup problems, just to name a few. Only certain companies, because they have either the right skillsets or the budget, invest in testing, but most of them don't have the capacity to see beyond the point that quality assurance is important. Getting the testing infrastructure and tools in place is the immediate step following getting developers to be responsible for their own code and to test it. (For more resources related to this topic, see here.) Even if testing is something not particularly new in the programming world, PHP always had a weak point regarding it. Its history is not the one of a pure-bred programming language done with all the nice little details, and only just recently has PHP found itself in a better position and started to become more appreciated. Because of this, the only and most important tool that came out has been PHPUnit, which was released just 10 years ago, in 2004, thanks to the efforts of Sebastian Bergmann. PHPUnit was and sometimes is still difficult to master and understand. It requires time and dedication, particularly if you are coming from a non-testing experience. PHPUnit simply provided a low-level framework to implement unit tests and, up to a certain point, integration tests, with the ability to create mocks and fakes when needed. Although it still is the quickest way to discover bugs, it didn't cover everything and using it to create large integration tests will end up being an almost impossible task. On top of this, PHPUnit since version 3.7, when it switched to a different autoloading mechanism and moved away from PEAR, caused several headaches rendering most of the installations unusable. Other tools developed since mostly come from other environments and requirements, programming languages, and frameworks. Some of these tools were incredibly strong and well-built, but they came with their own way of declaring tests and interacting with the application, set of rules, and configuration specifics. A modular framework rather than just another tool Clearly, mastering all these tools required a bit of understanding, and the learning curve wasn't promised to be the same among all of them. So, if this is the current panorama, why create another tool if you will end up in the same situation we were in before? Well, one of the most important things to be understood about Codeception is that it's not just a tool, rather a full stack, as noted on the Codeception site, a suite of frameworks, or if you want to go meta, a framework for frameworks. Codeception provides a uniform way to design different types of test by using as much as possible the same semantic and logic, a way to make the whole testing infrastructure more coherent and approachable. Outlining concepts behind Codeception Codeception has been created with the following basic concepts in mind: Easy to read: By using a declarative syntax close to the natural language, tests can be read and interpreted quite easily, making them an ideal candidate to be used as documentation for the application. Any stakeholder and engineer close to the project can ensure that tests are written correctly and cover the required scenarios without knowing any special lingo. It can also generate BDD-style test scenarios from code test cases. Easy to write: As we already underlined, every testing framework uses its own syntax or language to write tests, resulting in some degree of difficulty when switching from one suite to the other, without taking into account the learning curve each one has. Codeception tries to bridge this gap of knowledge by using a common declarative language. Further, abstractions provide a comfortable environment that makes maintenance simple. Easy to debug: Codeception is born with the ability to see what's behind the scenes without messing around with the configuration files or doing random print_r around your code. On top of this all, Codeception has also been written with modularity and extensibility in mind, so that organizing your code is simple while also promoting code reuse throughout your tests. But let's see what's provided by Codeception in more detail. Types of tests As we've seen, Codeception provides three basic types of test: Unit tests Functional tests Acceptance tests Each one of them is self-contained in its own folder where you can find anything needed, from the configuration and the actual tests to any additional piece of information that is valuable, such as the fixtures, database snapshots, or specific data to be fed to your tests. In order to start writing tests, you need to initialize all the required classes that will allow you to run your tests, and you can do this by invoking codecept with the build argument: $ cd tests $ ../vendor/bin/codecept build Building Actor classes for suites: functional, acceptance, unit FunctionalTester includes modules: Filesystem, Yii2 FunctionalTester.php generated successfully. 61 methods added AcceptanceTester includes modules: PhpBrowser AcceptanceTester.php generated successfully. 47 methods added UnitTester includes modules: UnitTester.php generated successfully. 0 methods added $ The codecept build command needs to be run every time you modify any configuration file owned by Codeception when adding or removing any module, in other words, whenever you modify any of the .suite.yml files available in the /tests folder. What you have probably already noticed in the preceding output is the presence of a very peculiar naming system for the test classes. Codeception introduces the Guys that have been renamed in Yii terminology as Testers, and are as follows: AcceptanceTester: This is used for acceptance tests FunctionalTester: This is used for functional tests UnitTester: This is used for unit tests These will become your main interaction points with (most of) the tests and we will see why. By using such nomenclature, Codeception shifts the point of attention from the code itself to the person that is meant to be acting the tests you will be writing. This way we will become more fluent in thinking in a more BDD-like mindset rather than trying to figure out all the possible solutions that could be covered, while losing the focus of what we're trying to achieve. Once again, BDD is an improvement over TDD, because it declares in a more detailed way what needs to be tested and what doesn't. AcceptanceTester AcceptanceTester can be seen as a person who does not have any knowledge of the technologies used and tries to verify the acceptance criteria that have been defined at the beginning. If we want to re-write our previously defined acceptance tests in a more standardized BDD way, we need to remember the structure of a so-called user story. The story should have a clear title, a short introduction that specifies the role that is involved in obtaining a certain result or effect, and the value that this will reflect. Following this, we will then need to specify the various scenarios or acceptance criteria, which are defined by outlining the initial scenario, the trigger event, and the expected outcome in one or more clauses. Let's discuss login using a modal window, which is one of the two features we are going to implement in our application. Story title – successful user login I, as an acceptance tester, want to log in into the application from any page. Scenario 1: Log in from the homepage I am on the homepage. I click on the login link. I enter my username. I enter my password. I press submit. The login link now reads "logout (<username>)" and I'm still on the homepage. Scenario 2: Log in from a secondary page I am on a secondary page. I click on the login link. I enter my username. I enter my password. I press Submit. The login link now reads "logout (<username>)" and I'm still on the secondary page. As you might have noticed I am limiting the preceding example to successful cases. The preceding story can be immediately translated into something along the lines of the following code: // SuccessfulLoginAcceptanceTest.php $I = new AcceptanceTester($scenario); $I->wantTo("login into the application from any page"); // scenario 1 $I->amOnPage("/"); $I->click("login"); $I->fillField("username", $username); $I->fillField("password", $password); $I->click("submit"); $I->canSee("logout (".$username.")"); $I->seeInCurrentUrl("/"); // scenario 2 $I->amOnPage("/"); $I->click("about"); $I->seeLink("login"); $I->click("login"); $I->fillField("username", $username); $I->fillField("password", $password); $I->click("submit"); $I->canSee("logout (".$username.")"); $I->amOnPage("about"); As you can see this is totally straightforward and easy to read, to the point that anyone in the business should be able to write any case scenario (this is an overstatement, but you get the idea). Clearly, the only thing that is needed to understand is what the AcceptanceTester is able to do: The class generated by the codecept build command can be found in tests/codeception/acceptance/AcceptanceTester.php, which contains all the available methods. You might want to skim through it if you need to understand how to assert a particular condition or perform an action on the page. The online documentation available at http://codeception.com/docs/04-AcceptanceTests will also give you a more readable way to get this information. Don't forget that at the end AcceptanceTester is just a name of a class, which is defined in the YAML file for the specific test type: $ grep class tests/codeception/acceptance.suite.yml class_name: AcceptanceTester Acceptance tests are the topmost level of tests, as some sort of high-level user-oriented integration tests. Because of this, acceptance tests end up using an almost real environment, where no mocks or fakes are required. Clearly, we would need some sort of initial state that we can revert to, particularly if we're causing actions that modify the state of the database. As per Codeception documentation, we could have used a snapshot of the database to be loaded at the beginning of each test. Unfortunately, I didn't have much luck in finding this feature working. So later on, we'll be forced to use the fixtures. Everything will then make more sense. When we will write our acceptance tests, we will also explore the various modules that you can also use with it, such as PHPBrowser and Selenium WebDriver and their related configuration options. FunctionalTester As we said earlier, FunctionalTester represents our character when dealing with functional tests. You might think of functional tests as a way to leverage on the correctness of the implementation from a higher standpoint. The way to implement functional tests bears the same structure as that of acceptance tests, to the point that most of the time the code we've written for an acceptance test in Codeception can be easily swapped with that for a functional test, so you might ask yourself: "where are the differences?" It must be noted that the concept of functional tests is something specific to Codeception and can be considered almost the same as that of integration tests for the mid-layer of your application. The most important thing is that functional tests do not require a web server to run, and they're called headless: For this reason, they are not only quicker than acceptance tests, but also less "real" with all the implications of running on a specific environment. And it's not the case that the acceptance tests provided by default by the basic application are, almost, the same as the functional tests. Because of this, we will end up having more functional tests that will cover more use cases for specific parts of our application. FunctionalTester is somehow setting the $_GET, $_POST and $_REQUEST variables and running the application from within a test. For this reason, Codeception ships with modules that let it interact with the underlying framework, be it Symfony2, Laravel4, Zend, or, in our case, Yii 2. In the configuration file, you will notice the module for Yii 2 already enabled: # tests/functional.suite.yml class_name: FunctionalTester modules: enabled: - Filesystem - Yii2 # ... FunctionalTester has got a better understanding of the technologies used although he might not have the faintest idea of how the various features he's going to test have been implemented in detail; he just knows the specifications. This makes a perfect case for the functional tests to be owned or written by the developers or anyone that is close to the knowledge of how the various features have been exposed for general consumption. The base functionality of the REST application, exposed through the API, will also be heavily tested, and in this case, we will have the following scenarios: I can use POST to send correct authentication data and will receive a JSON containing the successful authentication I can use POST to send bad authentication data and will receive a JSON containing the unsuccessful authentication After a correct authentication, I can use GET to retrieve the user data After a correct authentication, I will receive an error when doing a GET for a user stating that it's me I can use POST to send my updated hashed password Without a correct authentication, I cannot perform any of the preceding actions The most important thing to remember is that at the end of each test, it's your responsibility to keep the memory clean: The PHP application will not terminate after processing a request. All requests happening in the same memory container are not isolated. If you see your tests failing for some unknown reason when they shouldn't, try to execute a single test separately. UnitTester I've left UnitTester for the end as it's a very special guy. For all we know, until now, Codeception must have used some other framework to cover unit tests, and we're pretty much sure that PHPUnit is the only candidate to achieve this. If any of you have already worked with PHPUnit, you will remember the learning curve together with the initial problem of understanding its syntax and performing even the simplest of tasks. I found that most developers have a love-and-hate relationship with PHPUnit: either you learn its syntax or you spend half of the time looking at the manual to get to a single point. And I won't blame you. We will see that Codeception will come to our aid once again if we're struggling with tests: remember that these unit tests are the simplest and most atomic part of the work we're going to test. Together with them come the integration tests that cover the interaction of different components, most likely with the use of fake data and fixtures. If you're used to working with PHPUnit, you won't find any particular problems writing tests; otherwise, you can make use of UnitTester and implement the same tests by using the Verify and Specify syntax. UnitTester assumes a deep understanding of the signature and how the infrastructure and framework work, so these tests can be considered the cornerstone of testing. They are super fast to run, compared to any other type of test, and they should also be relatively easy to write. You can start with adequately simple assertions and move to data providers before needing to deal with fixtures. Other features provided by Codeception On top of the types of tests, Codeception provides some more aids to help you organize, modularize, and extend your test code. As we've seen, functional and acceptance tests have a very plain and declarative structure, and all the code and the scenarios related to specific acceptance criteria are kept in the same file at the same level and these are executed linearly. In most of the situations, as it is in our case, this is good enough, but when your code starts growing and the number of components and features become more and more complex, the list of scenarios and steps to perform an acceptance or functional test can be quite lengthy. Further, some tests might end up depending on others, so you might want to start considering writing more compact scenarios and promote code reuse throughout your tests or split your test into two or more tests. If you feel your code needs a better organization and structure, you might want to start generating CEST classes instead of normal tests, which are called CEPT instead. A CEST class groups the scenarios all together as methods as highlighted in the following snippet: <?php // SuccessfulLoginCest.php class SuccessfulLoginCest { public function _before(CodeceptionEventTestEvent $event) {} CodeceptionEventTestEvent $event public function _fail(CodeceptionEventTestEvent $event) {} // tests public function loginIntoTheApplicationTest(AcceptanceTester $I) { $I->wantTo("login into the application from any page"); $I->amOnPage("/"); $I->click("login"); $I->fillField("username", $username); $I->fillField("password", $password); $I->click("submit"); $I->canSee("logout (".$username.")"); $I->seeInCurrentUrl("/"); // ... } } ?> Any method that is not preceded by the underscore is considered a test, and the reserved methods _before and _after are executed at the beginning and at the end of the list of tests contained in the test class, while the _fail method is used as a cleanup method in case of failure. This alone might not be enough, and you can use document annotations to create reusable code to be run before and after the tests with the use of @before <methodName> and @after <methodName>. You can also be stricter and require a specific test to pass before any other by using the document annotation @depends <methodName>. We're going to use some of these document annotations, but before we start installing Codeception, I'd like to highlight two more features: PageObjects and StepObjects. The PageObject is a common pattern amongst test automation engineers. It represents a web page as a class, where its DOM elements are properties of the class, and methods instead provide some basic interactions with the page. The main reason for using PageObjects is to avoid hardcoding CSS and XPATH locators in your tests. Yii provides some example implementation of the PageObjects used in /tests/codeception/_pages. StepObject is another way to promote code reuse in your tests: It will define some common actions that can be used in several tests. Together with PageObjects, StepObjects can become quite powerful. StepObject extends the Tester class and can be used to interact with the PageObject. This way your tests will become less dependent on a specific implementation and will save you the cost of refactoring when the markup and the way to interact with each component in the page changes. For future reference, you can find all of these in the Codeception documentation in the section regarding the advanced use at http://codeception.com/docs/07-AdvancedUsage together with other features, like grouping and an interactive console that you can use to test your scenarios at runtime. Summary In this article, we got hands-on with Codeception and looked at the different types of tests available. Resources for Article: Further resources on this subject: Building a Content Management System [article] Creating an Extension in Yii 2 [article] Database, Active Record, and Model Tricks [article]

0
0
3286

article-image-hadoop-monitoring-and-its-aspects

Packt

04 May 2015

8 min read

Hadoop Monitoring and its aspects

Packt

04 May 2015

8 min read

In this article by Gurmukh Singh, the author of the book Monitoring Hadoop, tells us the importance of monitoring Hadoop and its importance. It also explains various other concepts of Hadoop, such as its architecture, Ganglia (a tool used to monitor Hadoop), and so on. (For more resources related to this topic, see here.) In any enterprise, how big or small it could be, it is very important to monitor the health of all its components like servers, network devices, databases, and many more and make sure things are working as intended. Monitoring is a critical part for any business dependent upon infrastructure, by giving signals to enable necessary actions incase of any failures. Monitoring can be very complex with many components and configurations in a real production environment. There might be different security zones; different ways in which servers are setup or a same database might be used in many different ways with servers listening on various service ports. Before diving into setting up Monitoring and logging for Hadoop, it is very important to understand the basics of monitoring, how it works and some commonly used tools in the market. In Hadoop, we can do monitoring of the resources, services and also do metrics collection of various Hadoop counters. There are many tools available in the market and one of them is Nagios, which is widely used. Nagios is a powerful monitoring system that provides you with instant awareness of your organization's mission-critical IT infrastructure. By using Nagios, you can: Plan release cycle and rollouts, before things get outdated Early detection, before it causes an outage Have automation and a better response across the organization Nagios Architecture It is based on a simple server client architecture, in which the server has the capability to execute checks remotely using NRPE agents on the Linux clients. The results of execution are captured by the server and accordingly alerted by the system. The checks could be for memory, disk, CPU utilization, network, database connection and many more. It provides the flexibility to use either active or passive checks. Ganglia Ganglia, it is a beautiful tool for aggregating the stats and plotting them nicely. Nagios, give the events and alerts, Ganglia aggregates and presents it in a meaningful way. What if you want to look for total CPU, memory per cluster of 2000 nodes or total free disk space on 1000 nodes. Some of the key feature of Ganglia. View historical and real time metrics of a single node or for the entire cluster Use the data to make decision on cluster sizing and performance Ganglia Components Ganglia Monitoring Daemon (gmond): This runs on the nodes that need to be monitored, captures state change and sends updates using XDR to a central daemon. Ganglia Meta Daemon (gmetad): This collects data from gmond and other gmetad daemons. The data is indexed and stored to disk in round robin fashion. There is also a Ganglia front-end for meaningful display of information collected. All these tools can be integrated with Hadoop, to monitor it and capture its metrics. Integration with Hadoop There are many important components in Hadoop that needs to be monitored, like NameNode uptime, disk space, memory utilization, and heap size. Similarly, on DataNode we need to monitor disk usage, memory utilization or job execution flow status across the MapReduce components. To know what to monitor, we must understand how Hadoop daemons communicate with each other. There are lots of ports used in Hadoop, some are for internal communication like scheduling jobs, and replication, while others are for user interactions. They may be exposed using TCP or HTTP. Hadoop daemons provide information over HTTP about logs, stacks, metrics that could be used for troubleshooting. NameNode can expose information about the file system, live or dead nodes or block reports by the DataNode or JobTracker for tracking the running jobs. Hadoop uses TCP, HTTP, IPC or socket for communication among the nodes or daemons. YARN Framework The YARN (Yet Another resource Negotiator) is the new MapReduce framework. It is designed to scale for large clusters and performs much better as compared to the old framework. There are new sets of daemons in the new framework and it is good to understand how to communicate with each other. The diagram that follows, explains the daemons and ports on which they talk. Logging in Hadoop In Hadoop, each daemon writes its own logs and the severity of logging is configurable. The logs in Hadoop can be related to the daemons or the jobs submitted. Useful to troubleshoot slowness, issue with map reduce tasks, connectivity issues and platforms bugs. The logs generated can be user level like task tracker logs on each node or can be related to master daemons like NameNode and JobTracker. In the newer YARN platform, there is a feature to move the logs to HDFS after the initial logging. In Hadoop 1.x the user log management is done using UserLogManager, which cleans and truncates logs according to retention and size parameters like mapred.userlog.retain.hours and mapreduce.cluster.map.userlog.retain-size respectively. The tasks standard out and error are piped to Unix tail program, so it retains the require size only. The following are some of the challenges of log management in Hadoop: Excessive logging: The truncation of logs is not done till the tasks finish, this for many jobs could cause disk space issues as the amount of data written is quite large. Truncation: We cannot always say what to log and how much is good enough. For some users 500KB of logs might be good but for some 10MB might not suffice. Retention: How long to retain logs, 1 or 6 months?. There is no rule, but there are best practices or governance issues. In many countries there is regulation in place to keep data for 1 year. Best practice for any organization is to keep it for at least 6 months. Analysis: What if we want to look at historical data, how to aggregate logs onto a central system and do analyses. In Hadoop logs are served over HTTP for a single node by default. Some of the above stated issues have been addressed in the YARN framework. Rather then truncating logs and that to on individual nodes, the logs can be moved to HDFS and processed using other tools. The logs are written at the per application level into directories per application. The user can access these logs through command line or web UI. For example, $HADOOP_YARN_HOME/bin/yarn logs. Hadoop metrics In Hadoop there are many daemons running like DataNode, NameNode, JobTracker, and so on, each of these daemons captures a lot of information about the components they work on. Similarly, in YARN framework we have ResourceManager, NodeManager, and Application Manager, each of which exposes metrics, explained in the following sections under Metrics2. For example, DataNode collects metrics like number of blocks it has for advertising to the NameNode, the number of replicated blocks, metrics about the various read or writes from clients. In addition to this there could be metrics related to events, and so on. Hence, it is very important to gather it for the working of the Hadoop cluster and also helps in debugging, if something goes wrong. For this, Hadoop has a metrics system, for collecting all this information. There are two versions of the metrics system, Metrics and Metrics2 for Hadoop 1.x and Hadoop 2.x respectively. The file hadoop-metrics.properties and hadoop-metrics2.properties for each Hadoop version can be configured respectively. Configuring Metrics2 For Hadoop version 2, which uses YARN framework, the metrics can be configured using hadoop-metrics2.properties, under the $HADOOP_HOME directory. *.sink.file.class=org.apache.hadoop.metrics2.sink.FileSink *.period=10 namenode.sink.file.filename=namenode-metrics.out datanode.sink.file.filename=datanode-metrics.out jobtracker.sink.file.filename=jobtracker-metrics.out tasktracker.sink.file.filename=tasktracker-metrics.out maptask.sink.file.filename=maptask-metrics.out reducetask.sink.file.filename=reducetask-metrics.out Hadoop metrics Configuration for Ganglia Firstly, we need to define a sink class, as per Ganglia. *.sink.ganglia.class=org.apache.hadoop.metrics2.sink.ganglia.GangliaSink31 Secondly, we need to define the frequency of how often the source showed be polled for data. We are polling every 30 seconds: *.sink.ganglia.period=30 Define retention for the metrics: *.sink.ganglia.dmax=jvm.metrics.threadsBlocked=70,jvm.metrics.memHeapUsedM=40 Summary In this article, we learned about Hadoop monitoring and its importance, and also the various concepts of Hadoop. Resources for Article: Further resources on this subject: Hadoop and MapReduce [article] YARN and Hadoop [article] Hive in Hadoop [article]

0
0
2832

article-image-using-your-smart-watch-control-networked-leds

Andrew Fisher

04 May 2015

15 min read

Using your smart watch to control networked LEDs

Andrew Fisher

04 May 2015

15 min read

Introduction In the middle of this year, the hacker’s watchmaker, Pebble, will launch their next products. These watches, and the non-color ones before them, are remarkably capable devices that can easily talk with other services - whether it’s to get notified of your latest mention on Twitter or get the latest weather report. Pebble have also made building applications easy by allowing you to build in JavaScript. In this post I’ll use JavaScript to show you how easy it is to build a watch application that can interact with an external service. I could build a simple web application that gets your latest train times or Yelp reviews, but where’s the fun in that? Instead I’m going to pair a Pebble with one of the other current darlings of the hacker community, the ESP8266 WiFi module. The ESP8266 wireless module made its entrance into the hardware community in late 2014 and things haven’t been quite the same at hacker spaces around the world. This is a device that has more memory than an Arduino, natively uses WiFi, can be programmed in C or Lua (with a custom firmware), has low power consumption and costs less than $5 per module. It is still early days in the ESP8266 community so the edges are a bit rough, but things are stable enough to get playing and what better combination of things to play with than making some LEDs controllable wirelessly using your smart watch as an interface. Design approach To make the wireless LED device I’m going to use an ESP8266 with the NodeMCU firmware on it with some custom code that exposes a web server. The web server will accept a post request that has values for red, green and blue (between 0 - off, and 255 - full on, for each channel). These values will be interpreted and used to set the color of a strip of WS2812 controllable LEDs (aka NeoPixels). Once the web service is exposed, the Pebble watch application simply needs to make an HTTP request to the ESP8266 web server, POSTing the data to it of the colour to turn the pixels. Bill of materials You’ll need the following to build this project Qty Item Price Notes 1 Pebble Watch $99 Any Pebble watch will work 1 USB to Serial programmer $15 A 3.3 and 5v switchable USB-Serial converter is really useful. http://www.dfrobot.com/index.php?route=product/product&product_id=147 This project requires a 3.3V one 1 ESP8266 $5 The ESP-01 is the most readily available board (even available on Amazon) http://www.amazon.com/ESP8266-ESP-01-Serial-Wireless-Transceiver/dp/B00NF0FCW8 Lots Jumper wires M-M, M-F $1 Get a mix 1 2xAA battery holder $1 http://www.jameco.com/1/1/1260-bh-321-4a-r-2xaa-battery-holder-wires-mounting-tabs.html 1 NeoPixel Strip $10 http://www.adafruit.com/neopixel(Strips work well of any size eg: http://www.adafruit.com/products/1426 or else the circular rings http://www.adafruit.com/products/1643) In addition you’ll probably also want the usual hacker tools of hot glue, solder and band aids. Prerequisites There are a few things you’ll need to set up before you get started which will be specific to your system. This may take a little while (30-60 minutes) but the documentation linked below is exhaustive for Linux, Mac and Windows. Sign up for CloudPebble and get your developer account (it’s free) Download the Pebble SDK and install it following these directions - you need this to be able to install your JS application on your watch Install ESP Tool (for flashing your ESP8266 module) - this uses python so you may need a python environment if you’re using Windows. Install ESPlora (for uploading your application to the ESP8266 module) - this uses Java If you want some additional background on the ESP8266 and the development process, this excellent presentation / documentation by Andy Gelme is well worth a read. Once your environment is ready to go, grab the files that go along with this post with the following command: mkdir ~/watch-led git clone https://gist.github.com/ee6fadcd837a0f46be8d.git ~/watch-led && cd ~/watch-led Configuring the ESP8266 In order to configure the wireless module you will need to do the following things: Wire the module up Flash a binary file with the NodeMCU firmware onto the module Configure the application and upload that to the NodeMCU environment Test that the code is working Wire the module The ESP-01 module is relatively simple inasmuch as it only has 8 pins, however the use of double pin header means you can’t just plug it into a breadboard. You have a few options here, many people make a converter that uses 2x4 female header and converts this to a 1x8 strip of male header that can be plugged into a breadboard. Others (like me) make custom cables for different applications - most people just use jumper wires to join to a breadboard or other modules. You choose whatever works best for you but the wiring diagram is below. ESP-01 USB Serial adapter 1 RXD <-- TXD 2 VCC 3.3V (make sure in range 3-3.5V) 3 GPIO 0 Connect to ground when flashing firmware 4 RESET 3.3V resets (Don’t connect) 5 GPIO 2 Signal line on WS2812 strip 6 CH_PD 3.3V (enables the module) 7 GND Ground 8 TXD --> RXD With the LEDs, connect VCC to 3.3V and ground to ground as well. These LEDs are happy to run off as little as 3V. Install NodeMCU firmware Before you install the firmware you need to put the module into “flash mode”. You do this by joining GPIO 0 (zero) to ground as you can see in the photo below illustrated by the pink wire. If you don’t do this, you can’t put new firmware on your module and esptool will tell you it can’t connect. Next, issue the following commands from a terminal (assuming esptool.py is in your path): cd ~/watch-led esptool.py -p <<PORT>> write_flash 0x00000 nodemcu_dev_0x00000.bin 0x10000 nodemcu_dev_0x10000.bin Note that you need to change <<PORT>> to the path to your serial port (eg /dev/ttyUSB0) - esptool will now go through the process of erasing and then flashing the NodeMCU firmware onto the module. ESPtool is very simple and is fully automated Assuming you have installed esptool and wired your module correctly you should not expect to see any errors in this process. If you do, check your wiring and your setup and then head to the GitHub repo for more information. Once the firmware upload is complete, power down your module, remove the jumper on GPIO0 to ground and then power the module back up again. The module is now using 9600 baud rate - if you connect using screen or minicom etc you will be dropped into a lua interpreter. Here you can issue commands and get responses. screen /dev/ttyUSB0 9600 Change /dev/ttyUSB0 for your port The firmware has a full lua interpreter you can play with. Once you’ve had your fill playing with a lua interpreter, disconnect your serial connection. Configure application Now the module has NodeMCU on it, it’s possible to use the ESPlora IDE to talk to it, upload files to the module and write applications using Lua. Open the ESPlora IDE and connect to your module. Select your connection from the marked area and then hit “open”. From here, you can upload files to the module. Open up the five Lua files in the code folder you downloaded earlier. You should have setup.lua, server.lua, init.lua, config.lua and application.lua - these files can be edited directly from the code screen and then uploaded onto the ESP8266. Two files that are of interest are config and server. Set your wireless details Switch to the config.lua tab and update the line: module.SSID["SSID"] = "PASSWORD" And change SSID and PASSWORD to the details for your network. You’ll notice when you hit save (CTRL+S) that it will save and then upload the file to the module as well and confirm that is complete. In addition you can hit the Save to ESP button which will upload it as well. Go ahead now and upload config.lua, setup.lua, init.lua and application.lua. Configure your LEDs Open up the server.lua file and look for the line: local NO_PIXELS = 17 Change this to however many LEDs you have on your strip (the maximum limit is probably 50 or so before you have memory issues). There are other config options at the top of this file as well which should be fairly self explanatory. Save this file to the module as well. Test application is working As a check before we test if the module is working properly, hit the “File List” button and you will get the files that are uploaded. Check that all five are there and if they are hit the “Reset ESP” button. This will cause the module to restart. You’ll see a bootup sequence and after 10-15 seconds you’ll get a message showing the application has started, and the IP address of the module and the server. You can now issue commands from curl: curl --data "red=255&green=0&blue=255" http://<IP> Where <<IP>> is the IP address of the module. You should at this point have some LEDs turning a nice shade of magenta. If not, backtrack a few steps and make sure each component is working - the debug log will tell you if there’s any serious issues and you can use print() statements in server.lua to print out messages. Also try things like ping IP from a command line to make sure you can see the module on your network from your computer. Once you have your LED web service running it’s time to make it accessible from your Pebble watch. Building the Pebble watch app Building a Pebble watch app using a combination of the cloud pebble IDE and JavaScript makes things super easy. If you’ve done any sort of web development with JavaScript before you’ll find building apps this way really fast. Whilst the amount of access to the hardware is a bit reduced compared to C, you can make calls to external services, create interfaces and have access to the sensors so there’s plenty to play with. If you haven’t already, link your Pebble account to a developer account and login to CloudPebble.net - create a new project and select Pebble JS from the project type. Give your project a descriptive name. Once created you’ll be taken to a blank workspace. Click on the app.js file under source files. The app.js file is the main file you build your application in - you can add more but ours is really simple so we only need the one. Open up the app.js file in the repository folder on your computer and copy and paste the contents of it into the IDE. Before you do anything else, change the HOST IP address to the IP of the ESP8266 at the top of the file. The application code is broken down with further explanations below. var UI = require('ui'); var ajax = require('ajax'); Sets up the required UI library to draw things to the screen and gets an ajax library to make requests to web services. function colour_request(r, g, b) { var req = { url: HOST, method: 'post', data: {red:r, green:g, blue:b} }; ajax(req, function(data, status, request) { console.log(data); }, function(data, status, request) { console.log('The ajax request failed: ' + data + status + JSON.stringify(request)); } ); } This function makes the AJAX request to our ESP8266 service posting the data. This is set up so it can be called from any interaction point within the application as we need it, taking a value for red, green and blue. var colours = [ { title: "OFF", r: 0, g: 0, b: 0, }, { title: "RED", r: 255, g: 0, b: 0, }, { title: "GREEN", r: 0, g: 255, b: 0, }, { title: "BLUE", r: 0, g: 0, b: 255, }, { title: "YELLOW", r: 255, g: 255, b: 0, }, { title: "MAGENTA", r: 255, g: 0, b: 255, }, { title: "CYAN", r: 0, g: 255, b: 255, }, { title: "WHITE", r: 255, g: 255, b: 255, }, ]; Next we define an array of objects that represent the menu items in our application. The title’s will be used for the text on the menu items and then the r,g,b values will be used in order to provide the colour values for the colour_request function. var menu = new UI.Menu({ sections: [{ title: 'Choose LED colour', items: colours }] }); menu.on('select', function(e) { colour_request(e.item.r, e.item.g, e.item.b); }); menu.show(); The last part of the code creates the UI menu items and then defines an event handler for each menu item to call the colour_request function when it is selected and then finally menu.show() puts everything in the screen. Compile and deploy the application When it comes time to test or deploy your application you can either download a PBW file which can then be installed on the Pebble using command line tools or else you can use the emulator that is in the IDE. To use the emulator simple save and then hit the “Play” button to run your app within a watch emulator in the IDE. The great thing about the emulator is that you can test things like UI interactions and ensure configuration happens properly in a nice tight development loop. Unfortunately, because the emulator runs in Pebble’s network, it more than likely won’t have access to your ESP8266 which is sitting on your LAN. One way to fix this is simply create a route through your firewall and map it to your ESP8266. I won’t get into how to do this for your particular router. Simply change the HOST IP address in the app.js file to your public Internet Address and then configure a route to your internal LAN address. If you do this you can test directly from the emulator quite happily. The other option, and one you’ll need to do at some point anyway, is to install the application on your watch directly. Ensure your HOST IP address is updated in app.js and your phone is on the same network as the ESP8266. Now, from the CloudPebble IDE, click “Compilation” and then run a build. This will sit as pending for a little while and then if everything works it will go into the build log with the status “succeeded”. Download the PBW file by clicking on the “PBW” button and make note of the name of the file as it saves it to your downloads. Open up your phone and ensure developer mode is selected as instructed in the build tools set up at the top of this post. Get the IP address from the developer connection screen, as shown below. Open up a terminal and make sure you’ve activated the Pebble development environment. Export your phone’s IP address as shown from the developer connection tool in the Pebble app. export PEBBLE_PHONE=<<IP ADDRESS>> Now you should be able to do things like this: pebble ping And you’ll receive a notification on your watch. If that’s the case then simply install the application with: pebble install ~/Downloads/Watch_LED.pbw The Pebble will vibrate when it is working. Make it wireless Once you have everything working, the last step is to get onto battery power. This is easy - simply remove the USB-Serial adapter then plug in the Ground and VCC of your battery pack into the ground and VCC of your setup. At that point the module will power up - wait approximately 15 seconds and you should be able to ping it (and then control the LEDs from your watch). Going further The Pebble Watch and the ESP8266 are both extremely interesting devices. With a small amount of JS, the watch can be connected to any service that talks standard web protocols and with only $5 and a little bit of Lua it’s possible to make a wireless hardware service that the watch can interact with. Using this basic model you can make all sorts of internet connected devices. Here are some other things you could try now you’ve got this working: Extend the watch app to be able to talk to multiple ESP8266s scattered around your workspace. Change the event model so that the LEDs update on different notifications from your watch such as blue for a new tweet, red for a new email etc. Flip the service around - instead of controlling hardware, attach a sensor to the ESP8266 and then pass the data back to the watch periodically where it can display what’s happening. About the Author Andrew Fisher is a creator and destroyer of things that combine mobile web, ubicomp and lots of data. He is a sometime programmer, interaction researcher and CTO at JBA, a data consultancy in Melbourne, Australia. He can be found on Twitter @ajfisher.

0
0
10099

How-To Tutorials

Packt

04 May 2015

51 min read

Saying Hello to Unity and Android

Packt

04 May 2015

51 min read

Welcome to the wonderful world of mobile game development. Whether you are still looking for the right development kit or have already chosen one, this article will be very important. In this article by Tom Finnegan, the author of Learning Unity Android Game Development, we explore the various features that come with choosing Unity as your development environment and Android as the target platform. Through comparison with major competitors, it is discovered why Unity and Android stand at the top of the pile. Following this, we will examine how Unity and Android work together. Finally, the development environment will be set up and we will create a simple Hello World application to test whether everything is set up correctly. (For more resources related to this topic, see here.) In this article, we will cover the following topics: Major Unity features Major Android features Unity licensing options Installing the JDK Installing the Android software development kit (SDK) Installing Unity 3D Installing Unity Remote Understanding what makes Unity great Perhaps the greatest feature of Unity is how open-ended it is. Nearly all game engines currently on the market are limited in what one can build with them. It makes perfect sense but it can limit the capabilities of a team. The average game engine has been highly optimized for creating a specific game type. This is great if all you plan on making is the same game again and again. It can be quite frustrating when one is struck with inspiration for the next great hit, only to find that the game engine can't handle it and everyone has to retrain in a new engine or double the development time to make the game engine capable. Unity does not suffer from this problem. The developers of Unity have worked very hard to optimize every aspect of the engine, without limiting what types of games can be made using it. Everything ranging from simple 2D platformers to massive online role-playing games are possible in Unity. A development team that just finished an ultrarealistic first-person shooter can turn right around and make 2D fighting games without having to learn an entirely new system. Being so open-ended does, however, bring a drawback. There are no default tools that are optimized for building the perfect game. To combat this, Unity grants the ability to create any tool one can imagine, using the same scripting that creates the game. On top of that, there is a strong community of users that have supplied a wide selection of tools and pieces, both free and paid, that can be quickly plugged in and used. This results in a large selection of available content that is ready to jump-start you on your way to the next great game. When many prospective users look at Unity, they think that, because it is so cheap, it is not as good as an expensive AAA game engine. This is simply not true. Throwing more money at the game engine is not going to make a game any better. Unity supports all of the fancy shaders, normal maps, and particle effects that you could want. The best part is that nearly all of the fancy features that you could want are included in the free version of Unity, and 90 percent of the time beyond that, you do not even need to use the Pro-only features. One of the greatest concerns when selecting a game engine, especially for the mobile market, is how much girth it will add to the final build size. Most game engines are quite hefty. With Unity's code stripping, the final build size of the project becomes quite small. Code stripping is the process by which Unity removes every extra little bit of code from the compiled libraries. A blank project compiled for Android that utilizes full code stripping ends up being around 7 megabytes. Perhaps one of the coolest features of Unity is its multi-platform compatibility. With a single project, one can build for several different platforms. This includes the ability to simultaneously target mobiles, PCs, and consoles. This allows you to focus on real issues, such as handling inputs, resolution, and performance. In the past, if a company desired to deploy their product on more than one platform, they had to nearly double the development costs in order to essentially reprogram the game. Every platform did, and still does, run by its own logic and language. Thanks to Unity, game development has never been simpler. We can develop games using simple and fast scripting, letting Unity handle the complex translation to each platform. Unity – the best among the rest There are of course several other options for game engines. Two major ones that come to mind are cocos2d and Unreal Engine. While both are excellent choices, you will find them to be a little lacking in certain respects. The engine of Angry Birds, cocos2d, could be a great choice for your next mobile hit. However, as the name suggests, it is pretty much limited to 2D games. A game can look great in it, but if you ever want that third dimension, it can be tricky to add it to cocos2d; you may need to select a new game engine. A second major problem with cocos2d is how bare bones it is. Any tool for building or importing assets needs to be created from scratch, or it needs to be found. Unless you have the time and experience, this can seriously slow down development. Then there is the staple of major game development, Unreal Engine. This game engine has been used successfully by developers for many years, bringing great games to the world Unreal Tournament and Gears of War not the least among them. These are both, however, console and computer games, which is the fundamental problem with the engine. Unreal is a very large and powerful engine. Only so much optimization can be done on it for mobile platforms. It has always had the same problem; it adds a lot of girth to a project and its final build. The other major issue with Unreal is its rigidity in being a first-person shooter engine. While it is technically possible to create other types of games in it, such tasks are long and complex. A strong working knowledge of the underlying system is a must before achieving such a feat. All in all, Unity definitely stands strong amidst game engines. But these are still great reasons for choosing Unity for game development. Unity projects can look just as great as AAA titles. The overhead and girth in the final build are small and this is very important when working on mobile platforms. The system's potential is open enough to allow you to create any type of game that you might want, where other engines tend to be limited to a single type of game. In addition, should your needs change at any point in the project's life cycle, it is very easy to add, remove, or change your choice of target platforms. Understanding what makes Android great With over 30 million devices in the hands of users, why would you not choose the Android platform for your next mobile hit? Apple may have been the first one out of the gate with their iPhone sensation, but Android is definitely a step ahead when it comes to smartphone technology. One of its best features is its blatant ability to be opened up so that you can take a look at how the phone works, both physically and technically. One can swap out the battery and upgrade the micro SD card on nearly all Android devices, should the need arise. Plugging the phone into a computer does not have to be a huge ordeal; it can simply function as a removable storage media. From the point of view of the cost of development as well, the Android market is superior. Other mobile app stores require an annual registration fee of about 100 dollars. Some also have a limit on the number of devices that can be registered for development at one time. The Google Play market has a one-time registration fee of 25 dollars, and there is no concern about how many Android devices or what type of Android devices you are using for development. One of the drawbacks of some of the other mobile development kits is that you have to pay an annual registration fee before you have access to the SDK. With some, registration and payment are required before you can view their documentation. Android is much more open and accessible. Anybody can download the Android SDK for free. The documentation and forums are completely viewable without having to pay any fee. This means development for Android can start earlier, with device testing being a part of it from the very beginning. Understanding how Unity and Android work together As Unity handles projects and assets in a generic way, there is no need to create multiple projects for multiple target platforms. This means that you could easily start development with the free version of Unity and target personal computers. Then, at a later date, you can switch targets to the Android platform with the click of a button. Perhaps, shortly after your game is launched, it takes the market by storm and there is a great call to bring it to other mobile platforms. With just another click of the button, you can easily target iOS without changing anything in your project. Most systems require a long and complex set of steps to get your project running on a device. However, once your device is set up and recognized by the Android SDK, a single button click will allow Unity to build your application, push it to a device, and start running it. There is nothing that has caused more headaches for some developers than trying to get an application on a device. Unity makes this simple. With the addition of a free Android application, Unity Remote, it is simple and easy to test mobile inputs without going through the whole build process. While developing, there is nothing more annoying than waiting for 5 minutes for a build every time you need to test a minor tweak, especially in the controls and interface. After the first dozen little tweaks, the build time starts to add up. Unity Remote makes it simple and easy to test everything without ever having to hit the Build button. These are the big three reasons why Unity works well with Android: Generic projects A one-click build process Unity Remote We could, of course, come up with several more great ways in which Unity and Android can work together. However, these three are the major time and money savers. You could have the greatest game in the world, but if it takes 10 times longer to build and test, what is the point? Differences between the Pro and Basic versions of Unity Unity comes with two licensing options, Pro and Basic, which can be found at https://store.unity3d.com. If you are not quite ready to spend the 3,000 dollars that is required to purchase a full Unity Pro license with the Android add-on, there are other options. Unity Basic is free and comes with a 30-day free trial of Unity Pro. This trial is full and complete, as if you have purchased Unity Pro, the only downside being a watermark in the bottom-right corner of your game stating Demo Use Only. It is also possible to upgrade your license at a later date. Where Unity Basic comes with mobile options for free, Unity Pro requires the purchase of Pro add-ons for each of the mobile platforms. An overview of license comparison License comparisons can be found at http://unity3d.com/unity/licenses. This section will cover the specific differences between Unity Android Pro and Unity Android Basic. We will explore what the features are and how useful each one is in the following points: NavMeshes, pathfinding, and crowd simulation This feature is Unity's built-in pathfinding system. It allows characters to find their way from a point to another around your game. Just bake your navigation data in the editor and let Unity take over at runtime. Until recently, this was a Unity Pro only feature. Now the only part of it that is limited in Unity Basic is the use of off-mesh links. The only time you are going to need them is when you want your AI characters to be able to jump across and otherwise navigate around gaps. LOD support LOD (short for level of detail) lets you control how complex a mesh is, based on its distance from the camera. When the camera is close to an object, you can render a complex mesh with a bunch of detail in it. When the camera is far from that object, you can render a simple mesh because all that detail is not going to be seen anyway. Unity Pro provides a built-in system to manage this. However, this is another system that could be created in Unity Basic. Whether or not you are using the Pro version, this is an important feature for game efficiency. By rendering less complex meshes at a distance, everything can be rendered faster, leaving more room for awesome gameplay. The audio filter Audio filters allow you to add effects to audio clips at runtime. Perhaps you created gravel footstep sounds for your character. Your character is running and we can hear the footsteps just fine, when suddenly they enter a tunnel and a solar flare hits, causing a time warp and slowing everything down. Audio filters would allow us to warp the gravel footstep sounds to sound as if they were coming from within a tunnel and were slowed by a time warp. Of course, you could also just have the audio guy create a new set of tunnel gravel footsteps in the time warp sounds, although this might double the amount of audio in your game and limit how dynamic we can be with it at runtime. We either are or are not playing the time warp footsteps. Audio filters would allow us to control how much time warp is affecting our sounds. Video playback and streaming When dealing with complex or high-definition cut scenes, being able to play videos becomes very important. Including them in a build, especially with a mobile target, can require a lot of space. This is where the streaming part of this feature comes in. This feature not only lets us play videos but also lets us stream a video from the Internet. There is, however, a drawback to this feature. On mobile platforms, the video has to go through the device's built-in video-playing system. This means that the video can only be played in fullscreen and cannot be used as a texture for effects such as moving pictures on a TV model. Theoretically, you could break your video into individual pictures for each frame and flip through them at runtime, but this is not recommended for build size and video quality reasons. Fully-fledged streaming with asset bundles Asset bundles are a great feature provided by Unity Pro. They allow you to create extra content and stream it to users without ever requiring an update to the game. You could add new characters, levels, or just about any other content you can think of. Their only drawback is that you cannot add more code. The functionality cannot change, but the content can. This is one of the best features of Unity Pro. The 100,000 dollar turnover This one isn't so much a feature as it is a guideline. According to Unity's End User License Agreement, the basic version of Unity cannot be licensed by any group or individual who made $100,000 in the previous fiscal year. This basically means that if you make a bunch of money, you have to buy Unity Pro. Of course, if you are making that much money, you can probably afford it without an issue. This is the view of Unity at least and the reason why there is a 100,000 dollar turnover. Mecanim – IK Rigs Unity's new animation system, Mecanim, supports many exciting new features, one of which is IK (short form for Inverse Kinematics). If you are unfamiliar with the term, IK allows one to define the target point of an animation and let the system figure out how to get there. Imagine you have a cup sitting on a table and a character that wants to pick it up. You could animate the character to bend over and pick it up; but, what if the character is slightly to the side? Or any number of other slight offsets that a player could cause, completely throwing off your animation? It is simply impractical to animate for every possibility. With IK, it hardly matters that the character is slightly off. We just define the goal point for the hand and leave the animation of the arm to the IK system. It calculates how the arm needs to move in order to get the hand to the cup. Another fun use is making characters look at interesting things as they walk around a room: a guard could track the nearest person, the player's character could look at things that they can interact with, or a tentacle monster could lash out at the player without all the complex animation. This will be an exciting one to play with. Mecanim – sync layers and additional curves Sync layers, inside Mecanim, allow us to keep multiple sets of animation states in time with each other. Say you have a soldier that you want to animate differently based on how much health he has. When he is at full health, he walks around briskly. After a little damage to his health, the walk becomes more of a trudge. If his health is below half, a limp is introduced into his walk, and when he is almost dead he crawls along the ground. With sync layers, we can create one animation state machine and duplicate it to multiple layers. By changing the animations and syncing the layers, we can easily transition between the different animations while maintaining the state machine. The additional curves feature is simply the ability to add curves to your animation. This means we can control various values with the animation. For example, in the game world, when a character picks up its feet for a jump, gravity will pull them down almost immediately. By adding an extra curve to that animation, in Unity, we can control how much gravity is affecting the character, allowing them to actually be in the air when jumping. This is a useful feature for controlling such values alongside the animations, but you could just as easily create a script that holds and controls the curves. The custom splash screen Though pretty self-explanatory, it is perhaps not immediately evident why this feature is specified, unless you have worked with Unity before. When an application that is built in Unity initializes on any platform, it displays a splash screen. In Unity Basic, this will always be the Unity logo. By purchasing Unity Pro, you can substitute for the Unity logo with any image you want. Real-time spot/point and soft shadows Lights and shadows add a lot to the mood of a scene. This feature allows us to go beyond blob shadows and use realistic-looking shadows. This is all well and good if you have the processing space for it. However, most mobile devices do not. This feature should also never be used for static scenery; instead, use static lightmaps, which is what they are for. However, if you can find a good balance between simple needs and quality, this could be the feature that creates the difference between an alright and an awesome game. If you absolutely must have real-time shadows, the directional light supports them and is the fastest of the lights to calculate. It is also the only type of light available to Unity Basic that supports real-time shadows. HDR and tone mapping HDR (short for high dynamic range) and tone mapping allow us to create more realistic lighting effects. Standard rendering uses values from zero to one to represent how much of each color in a pixel is on. This does not allow for a full spectrum of lighting options to be explored. HDR lets the system use values beyond this range and processes them using tone mapping to create better effects, such as a bright morning room or the bloom from a car window reflecting the sun. The downside of this feature is in the processor. The device can still only handle values between zero and one, so converting them takes time. Additionally, the more complex the effect, the more time it takes to render it. It would be surprising to see this used well on handheld devices, even in a simple game. Maybe the modern tablets could handle it. Light probes Light probes are an interesting little feature. When placed in the world, light probes figure out how an object should be lit. Then, as a character walks around, they tell it how to be shaded. The character is, of course, lit by the lights in the scene, but there are limits on how many lights can shade an object at once. Light probes do all the complex calculations beforehand, allowing for better shading at runtime. Again, however, there are concerns about processing power. Too little power and you won't get a good effect; too much and there will be no processing power left for playing the game. Lightmapping with global illumination and area lights All versions of Unity support lightmaps, allowing for the baking of complex static shadows and lighting effects. With the addition of global illumination and area lights, you can add another touch of realism to your scenes. However, every version of Unity also lets you import your own lightmaps. This means that you could use some other program to render the lightmaps and import them separately. Static batching This feature speeds up the rendering process. Instead of spending time grouping objects for faster rendering on each frame , this allows the system to save the groups generated beforehand. Reducing the number of draw calls is a powerful step towards making a game run faster. That is exactly what this feature does. Render-to-texture effects This is a fun feature, but of limited use. It allows you to use the output from a camera in your game as a texture. This texture could then, in its most simple form, be put onto a mesh and act as a surveillance camera. You could also do some custom post processing, such as removing the color from the world as the player loses their health. However, this option could become very processor-intensive. Fullscreen post-processing effects This is another processor-intensive feature that probably will not make it into your mobile game. However, you can add some very cool effects to your scene, such as adding motion blur when the player is moving really fast or a vortex effect to warp the scene as the ship passes through a warped section of space. One of the best effects is using the bloom effect to give things a neon-like glow. Occlusion culling This is another great optimization feature. The standard camera system renders everything that is within the camera's view frustum, the view space. Occlusion culling lets us set up volumes in the space our camera can enter. These volumes are used to calculate what the camera can actually see from those locations. If there is a wall in the way, what is the point of rendering everything behind it? Occlusion culling calculates this and stops the camera from rendering anything behind that wall. Deferred rendering If you desire the best looking game possible, with highly detailed lighting and shadows, this is a feature of interest for you. Deferred rendering is a multi-pass process for calculating your game's light and shadow detail. This is, however, an expensive process and requires a decent graphics card to fully maximize its use. Unfortunately, this makes it a little outside of our use for mobile games. Stencil buffer access Custom shaders can use the stencil buffer to create special effects by selectively rendering over specific pixels. It is similar to how one might use an alpha channel to selectively render parts of a texture. GPU skinning This is a processing and rendering method by which the calculations for how a character or object appears, when using a skeleton rig, is given to the graphics card rather than getting it done by the central processor. It is significantly faster to render objects in this way. However, this is only supported on DirectX 11 and OpenGL ES 3.0, leaving it a bit out of reach for our mobile games. Navmesh – dynamic obstacles and priority This feature works in conjunction with the pathfinding system. In scripts, we can dynamically set obstacles, and characters will find their way around them. Being able to set priorities means that different types of characters can take different types of objects into consideration when finding their way around. For example, a soldier must go around the barricades to reach his target. The tank, however, could just crash through, should the player desire. Native code plugins' support If you have a custom set of code in the form of a Dynamic Link Library (DLL), this is the Unity Pro feature you need access to. Otherwise, the native plugins cannot be accessed by Unity for use with your game. Profiler and GPU profiling This is a very useful feature. The profiler provides tons of information about how much load your game puts on the processor. With this information, we can get right down into the nitty-gritties and determine exactly how long a script takes to process. Script access to the asset pipeline This is an alright feature. With full access to the pipeline, there is a lot of custom processing that can be done on assets and builds. The full range of possibilities is beyond the scope of this article. However, you can think of it as something that can make tint all of the imported textures slightly blue. Dark skin This is entirely a cosmetic feature. Its point and purpose are questionable. However, if a smooth, dark-skinned look is what you desire, this is the feature that you want. There is an option in the editor to change it to the color scheme used in Unity Basic. For this feature, whatever floats your boat goes. Setting up the development environment Before we can create the next great game for Android, we need to install a few programs. In order to make the Android SDK work, we will first install the Java Development Kit (JDK). Then we will install the Android SDK. After that, we will install Unity. We then have to install an optional code editor. To make sure everything is set up correctly, we will connect to our devices and take a look at some special strategies if the device is a tricky one. Finally, we will install Unity Remote, a program that will become invaluable in your mobile development. Installing the JDK Android's development language of choice is Java; so, to develop for it, we need a copy of the Java SE Development Kit on our computer. The process of installing the JDK is given in the following steps: The latest version of the JDK can be downloaded from http://www.oracle.com/technetwork/java/javase/downloads/index.html. So open the site in a web browser, and you will be able to see the screen showed in the following screenshot: Select Java Platform (JDK) from the available versions and you will be brought to a page that contains the license agreement and allows you to select the type of file you wish to download. Accept the license agreement and select your appropriate Windows version from the list at the bottom. If you are unsure about which version to choose, then Windows x86 is usually a safe choice. Once the download is completed, run the new installer. After a system scan, click on Next two times, the JDK will initialize, and then click on the Next button one more time to install the JDK to the default location. It is as good there as anywhere else, so once it is installed, hit the Close button. We have just finished installing the JDK. We need this so that our Android development kit will work. Luckily, the installation process for this keystone is short and sweet. Installing the Android SDK In order to actually develop and connect to our devices, we need to have installed the Android SDK. Having the SDK installed fulfills two primary requirements. First, it makes sure that we have the bulk of the latest drivers for recognizing devices. Second, we are able to use the Android Debug Bridge (ADB). ADB is the system used for actually connecting to and interacting with a device. The process of installing the Android SDK is given in the following steps: The latest version of the Android SDK can be found at http://developer.android.com/sdk/index.html, so open a web browser and go to the given site. Once there, scroll to the bottom and find the SDK Tools Only section. This is where we can get just the SDK, which we need to make Android games with Unity, without dealing with the fancy fluff of the Android Studio. We need to select the .exe package with (Recommended) underneath it (as shown in the following screenshot): You will then be sent to a Terms and Conditions page. Read it if you prefer, but agree to it to continue. Then hit the Download button to start downloading the installer. Once it has finished downloading, start it up. Hit the first Next button and the installer will try to find an appropriate version of the JDK. You will come to a page that will notify you about not finding the JDK if you do not have it installed. If you skipped ahead and do not have the JDK installed, hit the Visit java.oracle.com button in the middle of the page and go back to the previous section for guidance on installing it. If you do have it, continue with the process. Hitting Next again will bring you to a page that will ask you about the person for whom you are installing the SDK . Select Install for anyone using this computer because the default install location is easier to get to for later purposes. Hit Next twice, followed by Install to install the SDK to the default location. Once this is done, hit Next and Finish to complete the installation of the Android SDK Manager. If Android SDK Manager does not start right away, start it up. Either way, give it a moment to initialize. The SDK Manager makes sure that we have the latest drivers, systems, and tools for developing with the Android platform. However, we have to actually install them first (which can be done from the following screen): By default, the SDK manager should select a number of options to install. If not, select the latest Android API (Android L (API 20) as of the time of writing this article), Android Support Library and Google USB Driver found in Extras. Be absolutely sure that Android SDK Platform-tools is selected. This will be very important later. It actually includes the tools that we need to connect to our device. Once everything is selected, hit Install packages at the bottom-right corner. The next screen is another set of license agreements. Every time a component is installed or updated through the SDK manager, you have to agree to the license terms before it gets installed. Accept all of the licenses and hit Install to start the process. You can now sit back and relax. It takes a while for the components to be downloaded and installed. Once this is all done, you can close it out. We have completed the process, but you should occasionally come back to it. Periodically checking the SDK manager for updates will make sure that you are using the latest tools and APIs. The installation of the Android SDK is now finished. Without it, we would be completely unable to do anything on the Android platform. Aside from the long wait to download and install components, this was a pretty easy installation. Installing Unity 3D Perform the following steps to install Unity: The latest version of Unity can be found at http://www.unity3d.com/unity/download. As of the time of writing this article, the current version is 5.0. Once it is downloaded, launch the installer and click on Next until you reach the Choose Components page, as shown in the following screenshot: Here, we are able to select the features of Unity installation: Example Project: This is the current project built by Unity to show off some of its latest features. If you want to jump in early and take a look at what a complete Unity game can look like, leave this checked. Unity Development Web Player: This is required if you plan on developing browser applications with Unity. As this article is focused on Android development, it is entirely optional. It is, however, a good one to check. You never know when you may need a web demo and since it is entirely free to develop for the web using Unity, there is no harm in having it. MonoDevelop: It is a wise choice to leave this option unchecked. There is more detail in the next section, but it will suffice for now to say that it just adds an extra program for script editing that is not nearly as useful as it should be. Once you have selected or deselected your desired options, hit Next. If you wish to follow this article, note that we will uncheck MonoDevelop and leave the rest checked. Next is the location of installation. The default location works well, so hit Install and wait. This will take a couple of minutes, so sit back, relax, and enjoy your favorite beverage. Once the installation is complete, the option to run Unity will be displayed. Leave it checked and hit Finish. If you have never installed Unity before, you will be presented with a license activation page (as shown in the following screenshot): While Unity does provide a feature-rich, free version, in order to follow the entirety of this article, one is required to make use of some of the Unity Pro features. At https://store.unity3d.com, you have the ability to buy a variety of licenses. Once they are purchased, you will receive an e-mail containing your new license key. Enter that in the provided text field. If you are not ready to make a purchase, you have two alternatives. We will go over how to reset your license in the Building a simple application section later in the article. The alternatives are as follows: The first alternative is that you can check the Activate the free version of Unity checkbox. This will allow you to use the free version of Unity. As discussed earlier, there are many reasons to choose this option. The most notable at the moment is cost. Alternatively, you can select the Activate a free 30-day trial of Unity Pro option. Unity offers a fully functional, one-time installation and a free 30-day trial of Unity Pro. This trial also includes the Android Pro add-on. Anything produced during the 30 days is completely yours, as if you had purchased a full Unity Pro license. They want you to see how great it is, so you will come back and make a purchase. The downside is that the Trial Version watermark will be constantly displayed at the corner of the game. After the 30 days, Unity will revert to the free version. This is a great option, should you choose to wait before making a purchase. Whatever your choice is, hit OK once you have made it. The next page simply asks you to log in with your Unity account. This will be the same account that you used to make your purchase. Just fill out the fields and hit OK. If you have not yet made a purchase, you can hit Create Account and have it ready for when you do make a purchase. The next page is a short survey on your development interests. Fill it out and hit OK or scroll straight to the bottom and hit Not right now. Finally, there is a thank you page. Hit Start using Unity. After a short initialization, the project wizard will open and we can start creating the next great game. However, there is still a bunch of work to do to connect the development device. So for now, hit the X button in the top-right corner to close the project wizard. We will cover how to create a new project in the Building a simple application section later on. We just completed installing Unity 3D. We also had to make a choice about licenses. The alternatives, though, will have a few shortcomings. You will either not have full access to all of the features or be limited to the length of the trial period while making due with a watermark in your games. The optional code editor Now a choice has to be made about code editors. Unity comes with a system called MonoDevelop. It is similar in many respects to Visual Studio. And like Visual Studio, it adds many extra files and much girth to a project, all of which it needs to operate. All this extra girth makes it take an annoying amount of time to start up, before one can actually get to the code. Technically, you can get away with a plain text editor, as Unity doesn't really care. This article recommends using Notepad++, which is found at http://notepad-plus-plus.org/download. It is free to use and it is essentially Notepad with code highlighting. There are several fancy widgets and add-ons for Notepad++ that add even greater functionality to it, but they are not necessary for following this article. If you choose this alternative, installing Notepad++ to the default location will work just fine. Connecting to a device Perhaps the most annoying step in working with Android devices is setting up the connection to your computer. Since there are so many different kinds of devices, it can get a little tricky at times just to have the device recognized by your computer. A simple device connection The simple device connection method involves changing a few settings and a little work in the command prompt. It may seem a little scary, but if all goes well you will be connected to your device shortly: The first thing you need to do is turn on the phone's Developer options. In the latest version of Android, these have been hidden. Go to your phone's settings page and find the About phone page. Next, you need to find the Build number information slot and tap it several times. At first, it will appear to do nothing, but it will shortly display that you need to press the button a few more times to activate the Developer options. The Android team did this so that the average user does not accidentally make changes. Now go back to your settings page and there should be a new Developer options page; select it now. This page controls all of the settings you might need to change while developing your applications. The only checkbox we are really concerned with checking right now is USB debugging. This allows us to actually detect our device from the development environment. If you are using Kindle, be sure to go into Security and turn on Enable ADB as well. There are several warning pop-ups that are associated with turning on these various options. They essentially amount to the same malicious software warnings associated with your computer. Applications with immoral intentions can mess with your system and get to your private information. All these settings need to be turned on if your device is only going to be used for development. However, as the warnings suggest, if malicious applications are a concern, turn them off when you are not developing. Next, open a command prompt on your computer. This can be done most easily by hitting your Windows key, typing cmd.exe, and then hitting Enter. We now need to navigate to the ADB commands. If you did not install the SDK to the default location, replace the path in the following commands with the path where you installed it. If you are running a 32-bit version of Windows and installed the SDK to the default location, type the following in the command prompt: cd c:\program files\android\android-sdk\platform-tools If you are running a 64-bit version, type the following in the command prompt: cd c:\program files (x86)\android\android-sdk\platform-tools Now, connect your device to your computer, preferably using the USB cable that came with it. Wait for your computer to finish recognizing the device. There should be a Device drivers installed type of message pop-up when it is done. The following command lets us see which devices are currently connected and recognized by the ADB system. Emulated devices will show up as well. Type the following in the command prompt: adb devices After a short pause for processing, the command prompt will display a list of attached devices along with the unique IDs of all the attached devices. If this list now contains your device, congratulations! You have a developer-friendly device. If it is not completely developer-friendly, there is one more thing that you can try before things get tricky. Go to the top of your device and open your system notifications. There should be one that looks like the USB symbol. Selecting it will open the connection settings. There are a few options here and by default Android selects to connect the Android device as a Media Device. We need to connect our device as a Camera. The reason is the connection method used. Usually, this will allow your computer to connect. We have completed our first attempt at connecting to our Android devices. For most, this should be all that you need to connect to your device. For some, this process is not quite enough. The next little section covers solutions to resolve the issue for connecting trickier devices. For trickier devices, there are a few general things that we can try; if these steps fail to connect your device, you may need to do some special research. Start by typing the following commands. These will restart the connection system and display the list of devices again: adb kill-server adb start-server adb devices If you are still not having any luck, try the following commands. These commands force an update and restart the connection system: cd ../tools android update adb cd ../platform-tools adb kill-server adb start-server adb devices If your device is still not showing up, you have one of the most annoying and tricky devices. Check the manufacturer's website for data syncing and management programs. If you have had your device for quite some time, you have probably been prompted to install this more than once. If you have not already done so, install the latest version even if you never plan on using it. The point is to obtain the latest drivers for your device, and this is the easiest way. Restart the connection system again using the first set of commands and cross your fingers! If you are still unable to connect, the best, professional recommendation that can be made is to google for the solution to your problem. Conducting a search for your device's brand with adb at the end should turn up a step-by-step tutorial that is specific to your device in the first couple of results. Another excellent resource for finding out all about the nitty-gritties of Android devices can be found at http://www.xda-developers.com/. Some of the devices that you will encounter while developing will not connect easily. We just covered some quick steps and managed to connect these devices. If we could have covered the processes for every device, we would have. However, the variety of devices is just too large and the manufacturers keep making more. Unity Remote Unity Remote is a great application created by the Unity team. It allows developers to connect their Android-powered devices to the Unity Editor and provide mobile inputs for testing. This is a definite must for any aspiring Unity and Android developer. If you are using a non-Amazon device, acquiring Unity Remote is quite easy. At the time of writing this article, it could be found on Google Play at https://play.google.com/store/apps/details?id=com.unity3d.genericremote. It is free and does nothing but connects your Android device to the Unity Editor, so the app permissions are negligible. In fact, there are currently two versions of Unity Remote. To connect to Unity 4.5 and later versions, we must use Unity Remote 4. If, however, you like the ever-growing Amazon market or seek to target Amazon's line of Android devices, adding Unity Remote will become a little trickier. First, you need to download a special Unity Package from the Unity Asset Store. It can be found at https://www.assetstore.unity3d.com/en/#!/content/18106. You will need to import the package into a fresh project and build it from there. Import the package by going to the top of Unity, navigate to Assets | Import Package | Custom Package, and then navigate to where you saved it. In the next section, we will build a simple application and put it on our device. After you have imported the package, follow along from the step where we open the Build Settings window, replacing the simple application with the created APK. Building a simple application We are now going to create a simple Hello World application. This will familiarize you with the Unity interface and how to actually put an application on your device. Hello World To make sure everything is set up properly, we need a simple application to test with and what better to do that with than a Hello World application? To build the application, perform the following steps: The first step is pretty straightforward and simple: start Unity. If you have been following along so far, once this is done you should see a screen resembling the next screenshot. As the tab might suggest, this is the screen through which we open our various projects. Right now, though, we are interested in creating one; so, select New Project from the top-right corner and we will do just that: Use the Project name* field to give your project a name; Ch1_HelloWorld fits well for a project name. Then use the three dots to the right of the Location* field to choose a place on your computer to put the new project. Unity will create a new folder in this location, based on the project name, to store your project and all of its related files: For now, we can ignore the 3D and 2D buttons. These let us determine the defaults that Unity will use when creating a new scene and importing new assets. We can also ignore the Asset packages button. This lets you select from the bits of assets and functionality that is provided by Unity. They are free for you to use in your projects. Hit the Create Project button, and Unity will create a brand-new project for us. The following screenshot shows the windows of the Unity Editor: The default layout of Unity contains a decent spread of windows that are needed to create a game: Starting from the left-hand side, Hierarchy contains a list of all the objects that currently exist in our scene. They are organized alphabetically and are grouped under parent objects. Next to this is the Scene view. This window allows us to edit and arrange objects in the 3D space. In the top left-hand side, there are two groups of buttons. These affect how you can interact with the Scene view. The button on the far left that looks like a hand lets you pan around when you click and drag with the mouse. The next button, the crossed arrows, lets you move objects around. Its behavior and the gizmo it provides will be familiar if you have made use of any modeling programs. The third button changes the gizmo to rotation. It allows you to rotate objects. The fourth button is for scale. It changes the gizmo as well. The fifth button lets you adjust the position and the scale based on the bounding box of the object and its orientation relative to how you are viewing it. The second to last button toggles between Pivot and Center. This will change the position of the gizmo used by the last three buttons to be either at the pivot point of the selected object, or at the average position point of all the selected objects. The last button toggles between Local and Global. This changes whether the gizmo is orientated parallel with the world origin or rotated with the selected object. Underneath the Scene view is the Game view. This is what is currently being rendered by any cameras in the scene. This is what the player will see when playing the game and is used for testing your game. There are three buttons that control the playback of the Game view in the upper-middle section of the window. The first is the Play button. It toggles the running of the game. If you want to test your game, press this button. The second is the Pause button. While playing, pressing this button will pause the whole game, allowing you to take a look at the game's current state. The third is the Step button. When paused, this button will let you progress through your game one frame at a time. On the right-hand side is the Inspector window. This displays information about any object that is currently selected. In the bottom left-hand side is the Project window. This displays all of the assets that are currently stored in the project. Behind this is Console. It will display debug messages, compile errors, warnings, and runtime errors. At the top, underneath Help, is an option called Manage License.... By selecting this, we are given options to control the license. The button descriptions cover what they do pretty well, so we will not cover them in more detail at this point. The next thing we need to do is connect our optional code editor. At the top, go to Edit and then click on Preferences..., which will open the following window: By selecting External Tools on the left-hand side, we can select other software to manage asset editing. If you do not want to use MonoDevelop, select the drop-down list to the right of External Script Editor and navigate to the executable of Notepad++, or any other code editor of your choice. Your Image application option can also be changed here to Adobe Photoshop or any other image-editing program that you prefer, in the same way as the script editor. If you installed the Android SDK to the default location, do not worry about it. Otherwise, click on Browse... and find the android-sdk folder. Now, for the actual creation of this application, right-click inside your Project window. From the new window that pops up, select Create and C# Script from the menu. Type in a name for the new script (HelloWorld will work well) and hit Enter twice: once to confirm the name and once to open it. In this article, this will be a simple Hello World application. Unity supports C#, JavaScript, and Boo as scripting languages. For consistency, this article will be using C#. If you, instead, wish to use JavaScript for your scripts, copies of all of the projects can be found with the other resources for this article, under a _JS suffix for JavaScript. Every script that is going to attach to an object extends the functionality of the MonoBehaviour class. JavaScript does this automatically, but C# scripts must define it explicitly. However, as you can see from the default code in the script, we do not have to worry about setting this up initially; it is done automatically. Extending the MonoBehaviour class lets our scripts access various values of the game object, such as the position, and lets the system automatically call certain functions during specific events in the game, such as the Update cycle and the GUI rendering. For now, we will delete the Start and Update functions that Unity insists on including in every new script. Replace them with a bit of code that simply renders the words Hello World in the top-left corner of the screen; you can now close the script and return to Unity: public void OnGUI() { GUILayout.Label("Hello World"); } Drag the HelloWorld script from the Project window and drop it on the Main Camera object in the Hierarchy window. Congratulations! You have just added your first bit of functionality to an object in Unity. If you select Main Camera in Hierarchy, then Inspector will display all of the components attached to it. At the bottom of the list is your brand-new HelloWorld script. Before we can test it, we need to save the scene. To do this, go to File at the top and select Save Scene. Give it the name HelloWorld and hit Save. A new icon will appear in your Project window, indicating that you have saved the scene. You are now free to hit the Play button in the upper-middle section of the editor and witness the magic of Hello World. We now get to build the application. At the top, select File and then click on Build Settings.... By default, the target platform is PC. Under Platform, select Android and hit Switch Platform in the bottom-left corner of the Build Settings window. Underneath the Scenes In Build box, there is a button labeled Add Current. Click on it to add our currently opened scene to the build. Only scenes that are in this list and checked will be added to the final build of your game. The scene with the number zero next to it will be the first scene that is loaded when the game starts. There is one last group of things to change before we can hit the Build button. Select Player Settings... at the bottom of the Build Settings window. The Inspector window will open Player Settings (shown in the following screenshot) for the application. From here, we can change the splash screen, icon, screen orientation, and a handful of other technical options: At the moment, there are only a few options that we care about. At the top, Company Name is the name that will appear under the information about the application. Product Name is the name that will appear underneath the icon on your Android device. You can largely set these to anything you want, but they do need to be set immediately. The important setting is Bundle Identifier, underneath Other Settings and Identification. This is the unique identifier that singles out your application from all other applications on the device. The format is com.CompanyName.ProductName, and it is a good practice to use the same company name across all of your products. For this article, we will be using com.TomPacktAndBegin.Ch1.HelloWorld for Bundle Identifier and opt to use an extra dot (period) for the organization. Go to File and then click on Save again. Now you can hit the Build button in the Build Settings window. Pick a location to save the file, and a file name ( Ch1_HelloWorld.apk works well). Be sure to remember where it is and hit Save. If during the build process Unity complains about where the Android SDK is, select the android-sdk folder inside the location where it was installed. The default would be C:\Program Files\Android\android-sdk for a 32-bit Windows system and C:\Program Files (x86)\Android\android-sdk for a 64-bit Windows system. Once loading is done, which should not be very long, your APK will have been made and we are ready to continue. We are through with Unity for this article. You can close it down and open a command prompt. Just as we did when we were connecting our devices, we need to navigate to the platform-tools folder in order to connect to our device. If you installed the SDK to the default location, use: For a 32-bit Windows system: cd c:\program files\android\android-sdk\platform-tools For a 64-bit Windows system: cd c:\program files (x86)\android\android-sdk\platform-tools Double-check to make sure that the device is connected and recognized by using the following command: adb devices Now we will install the application. This command tells the system to install an application on the connected device. The -r indicates that it should override if an application is found with the same Bundle Identifier as the application we are trying to install. This way you can just update your game as you develop, rather than uninstalling before installing the new version each time you need to make an update. The path to the .apk file that you wish to install is shown in quotes as follows: adb install -r "c:\users\tom\desktop\packt\book\ch1_helloworld.apk" Replace it with the path to your APK file; capital letters do not matter, but be sure to have all the correct spacing and punctuations. If all goes well, the console will display an upload speed when it has finished pushing your application to the device and a success message when it has finished the installation. The most common causes for errors at this stage are not being in the platform-tools folder when issuing commands and not having the correct path to the .apk file, surrounded by quotes. Once you have received your success message, find the application on your phone and start it up. Now, gaze in wonder at your ability to create Android applications with the power of Unity. We have created our very first Unity and Android application. Admittedly, it was just a simple Hello World application, but that is how it always starts. This served very well for double-checking the device connection and for learning about the build process without all the clutter from a game. If you are looking for a further challenge, try changing the icon for the application. It is a fairly simple procedure that you will undoubtedly want to perform as your game develops. How to do this was mentioned earlier in this section, but, as a reminder, take a look at Player Settings. Also, you will need to import an image. Take a look under Assets, in the menu bar, to know how to do this. Summary There were a lot of technical things in this article. First, we discussed the benefits and possibilities when using Unity and Android. That was followed by a whole lot of installation; the JDK, the Android SDK, Unity 3D, and Unity Remote. We then figured out how to connect to our devices through the command prompt. Our first application was quick and simple to make. We built it and put it on a device. Resources for Article: Further resources on this subject: What's Your Input? [article] That's One Fancy Hammer! [article] Saying Hello to Unity and Android [article]

0
0
3779

Packt

30 Apr 2015

17 min read

Welcome to the Spring Framework

Packt

30 Apr 2015

17 min read

0
0
9351

Packt

30 Apr 2015

12 min read

Angular 2.0

Packt

30 Apr 2015

12 min read

Angular 2.0 was officially announced in ng-conference on October 2014. Angular 2.0 will not be a major update to the previous version, but it is a complete rewrite of the entire framework and will include major changes. In this article by Mohammad Wadood Majid, coauthor of the book Mastering AngularJS for .NET Developers, we will learn the following topics: Why Angular 2.0 Design and features of Angular 2.0 AtScript Routing solution Dependency injection Annotations Instance scope Child injector Data binding and templating (For more resources related to this topic, see here.) Why Angular 2.0 AngularJS is one of the most popular open source frameworks available for client-side web application development. From the last few years, AngularJS's adaption and community support has been remarkable. The current AngularJS Version 1.3 is stable and used by many developers. There are over 1600 applications inside Google that use AngularJS 1.2 or 1.3. In the last few years, the Web has changed significantly, such as in the past, it was very difficult to build a cross-browser application; however, today's browsers are more consistent in their DOM implementations and the Web will continue to change. Angular 2.0 will address the following concerns: Mobile: Angular 2.0 will focus on mobile application development. Modular: Different modules will be removed from the core AngularJS, which will result in a better performance. Angular 2.0 will provide us the ability to pick the module parts we need. Modern: Angular 2.0 will include ECMAScript 6 (ES6). ECMAScript is a scripting language standard developed by Ecma International. It is widely used in client-side scripting, such as JavaScript, JScript, and ActionScript on the Web. Performance: AngularJS was developed around 5 years ago and it was not developed for developers. It was a tool targeting developers to quickly create persistent HTML forms. However, over time, it has been used to build more complex applications. The Angular 1.x team worked over the years to make changes to the current design, allowing it to continue to be relevant as needed for modern web applications. However, there are limits to improve the current AngularJS framework. A number of these limits are related to the performance that results to the current binding and template infrastructure. In order to fix these problems, a new infrastructure is required. In these days, the modern browser already supports some of the features of ES6, but the final implementation in progress will be available in 2015. With new features, developers will be able to describe their own views (template element) and package them for distribution to other developers (HTML imports). When in 2015 all these new features are available in all the browsers, the developer will be able to create as many endeavors for reusable components as required to resolve common problems. However, most of the frameworks, such as AngularJS 1.x, are not prepared, the data binding of the AngularJS 1.x framework works on the assumption of a small number of known HTML elements. In order to take advantage of the new components, an implementation in Angular is required. Design and features of AngularJS 2.0 The current AngularJS framework design in an amalgamation of the changing Web and the general computing landscape; however, it still needs some changes. The current Angular 1.x framework cannot work with new web components, as it lends itself to mobile applications and pushes its own module and class API against the standards. To answer these issues, the AngularJS team is coming up with the AngularJS 2.0 framework. AngularJS 2.0 is a reimaging of AngularJS 1.x for the modern web browser. The following are the changes in Angular 2.0: AtScript AtScript is a language used to develop AngularJS 2.0, which is a superset of ES6. It's managed by the Traceur compiler with ES6 to produce the ES5 code and it will use the TypeScript's syntax to generate runtime type proclamations instead of compile time checks. However, the developer will still be able to use the plain JavaScript (ES5) instead to using AtScript to write AngularJS 2.0 applications. The following is an example of an AtScript code: import {Component} from 'angulat'; import {server} from './server'; @Componenet({selector: 'test'}) export class MyNewComponent { constructor(serve:Server){ this.sever=server } } In the preceding code, the import and the class come from ES6. There are constructor functions and a server parameter that specifies a type. In AtScript, this type is used to generate a runtime type assertion. The reference is stored, so that the dependency injection framework can use it. The @Component annotation is a metadata annotation. When we decorate some code within @Component, the compiler generates code that instantiates the annotation and stores it in a known location, so that it can be accessed by the AngularJS 2.0 framework. Routing solution In AngularJS 1.x, the routing was designed to handle a few simple cases. As the framework grew, more features were added to it. AngularJS 2.0 includes the following basic routing features, but will still be able to extend: JSON route configuration Optional convention over configuration Static, parameterized, and splat route patterns URL resolver Query string Push state or hash change Navigation model Document title updates 404 route handling Location service History manipulation Child router Screen activate: canActivate activate deactivate Dependency Injection The main feature of AngularJS 1.x was Dependency Injection (DI). It is very easy to used DI and follow the divide and conquer software development approach. In this way, the complex problems can be abstracted together and the applications that are developed in this way can be assembled at runtime with the use of DI. However, there are few issues in the AngularJS 1.x framework. First, the DI implementation was associated with minification; DI was dependant on parsing parameter names from functions, and whenever the names were changed, they were no longer matching with the services, controllers, and other components. Second, the missing features, which are more common to advance server-side DI features, are available in .NET and Java. These two features add constraints to scope control and child injectors. Annotations With the use of AtScript in the AngularJS 2.0 framework, a way to relate metadata with any function was introduced. The formatting data for metadata with AtScript is strong in the face of minification and is easy to write by pointer with ES5. The instance scope In the AngularJS framework 1.x, in the DI container, all the instances were singletons. The same is the case with AngularJS 2.0 by default. However, to get different behavior, we need to use services, providers, constants, and so on. The following code can be used to create a new instance every time the DI. It will become more useful if you create your own scope identifiers for use in the combination with child injectors, as shown: @TransientScope Export class MyClass{…} The child injector The child injector is a major new feature in AngularJS 2.0. The child injector is inherited from its parent; however, the child injector has the ability to override the parents at the child level. Using this new feature, we can call certain type of objects in the application that will be automatically overridden in various scopes. For example, when a new route has a child route ability, each child route creates its own child injector. This allows each route to inherit from parent routes or to override those services during different navigation scenarios. Data binding and templating The data binding and template are considered a single unit while developing the application. In other words, the data binding and templates go hand in hand while writing an application with the AngularJS framework. When we bind the DOM, the HTML is handed to the template compiler. The compiler goes across the HTML to find out any directives, binding expressions, event handlers, and so on. All of the data is extracted from the DOM to data structures, which can be used to instantiate the template. During this phase, some processing is done on the data, for example, parsing the binding expression. Every node that contains the instructions is tagged with the class to cache the result of the process, so that work does not need to be repeated. Dynamic loading Dynamic loading was missing in AngularJS 1.x. It is very hard to add new directives or controllers at runtime. However, dynamic loading is added to Angular 2.0. Whenever any template is compiled, not only is the compiler provided with a template, but it is also provided with a component definition. The component definition contains metadata of directives, filters, and so on. This confirms that the necessary dependencies are loaded before the template gets managed by the compiler. Directives Directives in the AngularJS framework are meant to extend the HTML. In AngularJS 1.x, the Directive Definition Object (DDO) is used to create directives. In AngularJS 2.0, the directives are made simpler. There are three types of directives in AngularJS 2.0: The component directive: This is a collection of a view and a controller to create custom components. It can be used as an HTML element as well as a router that can map routes to the components. The decorator directive: Use this directive to decorate the HTML element with additional behavior, such as ng-show. The template directive: This directive transforms HTML into a reusable template. The directive developer can control how the template is instantiated and inserted into the DOM, such as ng-if and ng-repeat. The controller in AngularJS 2.0 is not a part of the component. However, the component contains the view and controller, where view is an HTML and controller is JavaScript. In AngularJS 2.0, the developer creates a class with some annotations, as shown in the following code: @dirComponent({ Selector: 'divTabContainter' Directives:[NgRepeat] }) Export class TabContainer{ constructor (panes:Query<Pane>){ this.panes=panes } select(selectPane:Pane){…} } In the preceding code, the controller of the component is a class. The dependencies are injected automatically into the constructor because the child injectors will be used. It can get access to any service up to the DOM hierarchy as well as it will local to service element. It can be seen in the preceding code that Query is injected. This is a special collection that is automatically synchronized with the child elements and lets us know when anything is added or removed. Templates In the preceding section, we created a dirTabContainer directive using AngularJS 2.0. The following code shows how to use the preceding directive in the DOM: <template> <div class="border"> <div class="tabs"> <div [ng-repeat|pane]="panes" class="tab" (^click)="select(pane)"> <img [src]="pane.icon"><span>${pane.name}</span> </div> </div> <content> </content> </div> </template> As you can see in the preceding code, in the <img [src]="pane.icon"><span>${pane.name}</span> image tag, the src attribute is surrounded with [], which tells us that the attribute has binding express. When we see ${}, it means that there is an expression that should be interpolated in the content. These bindings are unidirectional from the model or controller to the view. If you see div in the preceding <div [ng-repeat|pane]="panes" class="tab" (^click)="select(pane)"> template code, it is noticeable that ng-repeat is a template directive and is included with | and the pane word, where pane is the local variable. (^click) indicates that there is an event handler, where ^ means that the event is not a directory attached to the DOM, rather, we let it bubble and will be handled at the document level. In the following code example, we will compare the code of the Angular framework 1.x and AngularJS 2.0; let's create a hello world example for this demonstration. The following code is used to enable the write option in the AngularJS framework 1.x: var module = angular.module("example", []); module.controller("FormExample", function() { this.username = "World"; }); <div ng-controller="FormExample as ctrl"> <input ng-model="ctrl.username"> Hello {{ctrl.username}}! </div> The following code is used to write in the AngluraJS framework 1.x: @Component({ selector: 'form-example' }) @Template({ // we are binding the input element to the control object // defined in the component's class inline: '<input [control]="username">Hello {{username.value}}!', directives: [forms] }) class FormExample { constructor() { this.username = new Control('World'); } } In the preceding code example, TypeScript 1.5 is used, which will support the metadata annotations. However, the preceding code can be written in the ES5/ES6 JavaScript. More information on annotations can be found in the annotation guide at https://docs.google.com/document/d/1uhs-a41dp2z0NLs-QiXYY-rqLGhgjmTf4iwBad2myzY/edit#heading=h.qbaubqkoiqds. Here are some explanations from TypeScript 1.5: Form behavior cannot be unit tested without compiling the associated template. This is required because certain parts of the application behavior are contained in the template. We want to enable the dynamically generated data-driven forms in AngularJS 2.0 although it is present in AngularJS 1.x. This is because in Angular 1.x, this is not easy. The difficulty to reason your template statically arises because the ng-model directive was built using a generic two-way data binding. An atomic form that can easily be validated or reverted to its original state is required, which is missing from AngularJS 1.x. Although AngularJS 2.0 uses an extra level of indirection, it grants major benefits. The control object decouples form behavior from the template, so that you can test it in isolation. Tests are simpler to write and faster to execute. Summary In this article, we introduced the Angular 2.0 framework; it may not be a major update to the previous version, but it is a complete rewrite of the entire framework and will include breaking changes. We also talked about certain AngularJS 2.0 changes. AngularJS 2.0 will hopefully be released by the end of 2015. Resources for Article: Further resources on this subject: Setting Up The Rig [article] AngularJS Project [article] Working with Live Data and AngularJS [article]

0
0
4466

Packt

30 Apr 2015

15 min read

Machine Learning

Packt

30 Apr 2015

15 min read

0
0
2629

article-image-less-external-applications-and-frameworks

Packt

30 Apr 2015

11 min read

Less with External Applications and Frameworks

Packt

30 Apr 2015

11 min read

In this article by Bass Jobsen, author of the book Less Web Development Essentials - Second Edition, we will cover the following topics: WordPress and Less Using Less with the Play framework, AngularJS, Meteor, and Rails (For more resources related to this topic, see here.) WordPress and Less Nowadays, WordPress is not only used for weblogs, but it can also be used as a content management system for building a website. The WordPress system, written in PHP, has been split into the core system, plugins, and themes. The plugins add additional functionalities to the system, and the themes handle the look and feel of a website built with WordPress. They work independently of each other and are also independent of the theme. The theme does not depend on plugins. WordPress themes define the global CSS for a website, but every plugin can also add its own CSS code. The WordPress theme developers can use Less to compile the CSS code of the themes and the plugins. Using the Sage theme by Roots with Less Sage is a WordPress starter theme. You can use it to build your own theme. The theme is based on HTML5 Boilerplate (http://html5boilerplate.com/) and Bootstrap. Visit the Sage theme website at https://roots.io/sage/. Sage can also be completely built using Gulp. More information about how to use Gulp and Bower for the WordPress development can be found at https://roots.io/sage/docs/theme-development/. After downloading Sage, the Less files can be found at assets/styles/. These files include Bootstrap's Less files. The assets/styles/main.less file imports the main Bootstrap Less file, bootstrap.less. Now, you can edit main.less to customize your theme. You will have to rebuild the Sage theme after the changes you make. You can use all of the Bootstrap's variables to customize your build. JBST with a built-in Less compiler JBST is also a WordPress starter theme. JBST is intended to be used with the so-called child themes. More information about the WordPress child themes can be found at https://codex.wordpress.org/Child_Themes. After installing JBST, you will find a Less compiler under Appearance in your Dashboard pane, as shown in the following screenshot: JBST's built-in Less compiler in the WordPress Dashboard The built-in Less compiler can be used to fully customize your website using Less. Bootstrap also forms the skeleton of JBST, and the default settings are gathered by the a11y bootstrap theme mentioned earlier. JBST's Less compiler can be used in the following different ways: First, the compiler accepts any custom-written Less (and CSS) code. For instance, to change the color of the h1 elements, you should simply edit and recompile the code as follows: h1 {color: red;} Secondly, you can edit Bootstrap's variables and (re)use Bootstrap's mixins. To set the background color of the navbar component and add a custom button, you can use the code block mentioned here in the Less compiler: @navbar-default-bg: blue; .btn-colored { .button-variant(blue;red;green); } Thirdly, you can set JBST's built-in Less variables as follows: @footer_bg_color: black; Lastly, JBST has its own set of mixins. To set a custom font, you can edit the code as shown here: .include-custom-font(@family: arial,@font-path, @path: @custom-font-dir, @weight: normal, @style: normal); In the preceding code, the parameters mentioned were used to set the font name (@family) and the path name to the font files (@path/@font-path). The @weight and @style parameters set the font's properties. For more information, visit https://github.com/bassjobsen/Boilerplate-JBST-Child-Theme. More Less code blocks can also be added to a special file (wpless2css/wpless2css.less or less/custom.less); these files will give you the option to add, for example, a library of prebuilt mixins. After adding the library using this file, the mixins can also be used with the built-in compiler. The Semantic UI WordPress theme The Semantic UI, as discussed earlier, offers its own WordPress plugin. The plugin can be downloaded from https://github.com/ProjectCleverWeb/Semantic-UI-WordPress. After installing and activating this theme, you can use your website directly with the Semantic UI. With the default setting, your website will look like the following screenshot: Website built with the Semantic UI WordPress theme WordPress plugins and Less As discussed earlier, the WordPress plugins have their own CSS. This CSS will be added to the page like a normal style sheet, as shown here: <link rel='stylesheet' id='plugin-name' href='//domain/wp-content/plugin-name/plugin-name.css?ver=2.1.2' type='text/css' media='all' /> Unless a plugin provides the Less files for their CSS code, it will not be easy to manage its styles with Less. The WP Less to CSS plugin The WP Less to CSS plugin, which can be found at http://wordpress.org/plugins/wp-less-to-css/, offers the possibility of styling your WordPress website with Less. As seen earlier, you can enter the Less code along with the built-in compiler of JBST. This code will then be compiled into the website's CSS. This plugin compiles Less with the PHP Less compiler, Less.php. Using Less with the Play framework The Play framework helps you in building lightweight and scalable web applications by using Java or Scala. It will be interesting to learn how to integrate Less with the workflow of the Play framework. You can install the Play framework from https://www.playframework.com/. To learn more about the Play framework, you can also read, Learning Play! Framework 2, Andy Petrella, Packt Publishing. To read Petrella's book, visit https://www.packtpub.com/web-development/learning-play-framework-2. To run the Play framework, you need JDK 6 or later. The easiest way to install the Play framework is by using the Typesafe activator tool. After installing the activator tool, you can run the following command: > activator new my-first-app play-scala The preceding command will install a new app in the my-first-app directory. Using the play-java option instead of the play-scala option in the preceding command will lead to the installation of a Java-based app. Later on, you can add the Scala code in a Java app or the Java code in a Scala app. After installing a new app with the activator command, you can run it by using the following commands: cd my-first-app activator run Now, you can find your app at http://localhost:9000. To enable the Less compilation, you should simply add the sbt-less plugin to your plugins.sbt file as follows: addSbtPlugin("com.typesafe.sbt" % "sbt-less" % "1.0.6") After enabling the plugin, you can edit the build.sbt file so as to configure Less. You should save the Less files into app/assets/stylesheets/. Note that each file in app/assets/stylesheets/ will compile into a separate CSS file. The CSS files will be saved in public/stylesheets/ and should be called in your templates with the HTML code shown here: <link rel="stylesheet" href="@routes.Assets.at("stylesheets/main.css")"> In case you are using a library with more files imported into the main file, you can define the filters in the build.sbt file. The filters for these so-called partial source files can look like the following code: includeFilter in (Assets, LessKeys.less) := "*.less" excludeFilter in (Assets, LessKeys.less) := "_*.less" The preceding filters ensure that the files starting with an underscore are not compiled into CSS. Using Bootstrap with the Play framework Bootstrap is a CSS framework. Bootstrap's Less code includes many files. Keeping your code up-to-date by using partials, as described in the preceding section, will not work well. Alternatively, you can use WebJars with Play for this purpose. To enable the Bootstrap WebJar, you should add the code shown here to your build.sbt file: libraryDependencies += "org.webjars" % "bootstrap" % "3.3.2" When using the Bootstrap WebJar, you can import Bootstrap into your project as follows: @import "lib/bootstrap/less/bootstrap.less"; AngularJS and Less AngularJS is a structural framework for dynamic web apps. It extends the HTML syntax, and this enables you to create dynamic web views. Of course, you can use AngularJS with Less. You can read more about AngularJS at https://angularjs.org/. The HTML code shown here will give you an example of what repeating the HTML elements with AngularJS will look like: <!doctype html> <html ng-app> <head> <title>My Angular App</title> </head> <body ng-app> <ul> <li ng-repeat="item in [1,2,3]">{{ item }}</li> </ul> <script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.3.12/& angular.min.js"></script> </body> </html> This code should make your page look like the following screenshot: Repeating the HTML elements with AngularJS The ngBoilerplate system The ngBoilerplate system is an easy way to start a project with AngularJS. The project comes with a directory structure for your application and a Grunt build process, including a Less task and other useful libraries. To start your project, you should simply run the following commands on your console: > git clone git://github.com/ngbp/ngbp > cd ngbp > sudo npm -g install grunt-cli karma bower > npm install > bower install > grunt watch And then, open ///path/to/ngbp/build/index.html in your browser. After installing ngBoilerplate, you can write the Less code into src/less/main.less. By default, only src/less/main.less will be compiled into CSS; other libraries and other codes should be imported into this file. Meteor and Less Meteor is a complete open-source platform for building web and mobile apps in pure JavaScript. Meteor focuses on fast development. You can publish your apps for free on Meteor's servers. Meteor is available for Linux and OS X. You can also install it on Windows. Installing Meteor is as simple as running the following command on your console: > curl https://install.meteor.com | /bin/sh You should install the Less package for compiling the CSS code of the app with Less. You can install the Less package by running the command shown here: > meteor add less Note that the Less package compiles every file with the .less extension into CSS. For each file with the .less extension, a separate CSS file is created. When you use the partial Less files that should only be imported (with the @import directive) and not compiled into the CSS code itself, you should give these partials the .import.less extension. When using the CSS frameworks or libraries with many partials, renaming the files by adding the .import.less extension will hinder you in updating your code. Also running postprocess tasks for the CSS code is not always possible. Many packages for Meteor are available at https://atmospherejs.com/. Some of these packages can help you solve the issue with using partials mentioned earlier. To use Bootstrap, you can use the meteor-bootstrap package. The meteor-bootstrap package can be found at https://github.com/Nemo64/meteor-bootstrap. The meteor-bootstrap package requires the installation of the Less package. Other packages provide you postprocsess tasks, such as autoprefixing your code. Ruby on rails and Less Ruby on Rails, or Rails, for short is a web application development framework written in the Ruby language. Those who want to start developing with Ruby on Rails can read the Getting Started with Rails guide, which can be found at http://guides.rubyonrails.org/getting_started.html. In this section, you can read how to integrate Less into a Ruby on Rails app. After installing the tools and components required for starting with Rails, you can launch a new application by running the following command on your console: > rails new blog Now, you should integrate Less with Rails. You can use less-rails (https://github.com/metaskills/less-rails) to bring Less to Rails. Open the Gemfile file, comment on the sass-rails gem, and add the less-rails gem, as shown here: #gem 'sass-rails', '~> 5.0' gem 'less-rails' # Less gem 'therubyracer' # Ruby Then, create a controller called welcome with an action called index by running the following command: > bin/rails generate controller welcome index The preceding command will generate app/views/welcome/index.html.erb. Open app/views/welcome/index.html.erb and make sure that it contains the HTML code as shown here: <h1>Welcome#index</h1> <p>Find me in app/views/welcome/index.html.erb</p> The next step is to create a file, app/assets/stylesheets/welcome.css.less, with the Less code. The Less code in app/assets/stylesheets/welcome.css.less looks as follows: @color: red; h1 { color: @color; } Now, start a web server with the following command: > bin/rails server Finally, you can visit the application at http://localhost:3000/. The application should look like the example shown here: The Rails app Summary In this article, you learned how to use Less WordPress, Play, Meteor, AngularJS, Ruby on Rails. Resources for Article: Further resources on this subject: Media Queries with Less [article] Bootstrap 3 and other applications [article] Getting Started with Bootstrap [article]

0
0
5392

article-image-extracting-data-physically-dd

Packt

30 Apr 2015

10 min read

Extracting data physically with dd

Packt

30 Apr 2015

10 min read

In this article by Rohit Tamma and Donnie Tindall, authors of the book Learning Android Forensics, we will cover physical data extraction using free and open source tools wherever possible. The majority of the material covered in this article will use the ADB methods. (For more resources related to this topic, see here.) The dd command should be familiar to any examiner who has done traditional hard drive forensics. The dd command is a Linux command-line utility used by definition to convert and copy files, but is frequently used in forensics to create bit-by-bit images of entire drives. Many variations of the dd commands also exist and are commonly used, such as dcfldd, dc3dd, ddrescue, and dd_rescue. As the dd command is built for Linux-based systems, it is frequently included on Android platforms. This means that a method for creating an image of the device often already exists on the device! The dd command has many options that can be set, of which only forensically important options are listed here. The format of the dd command is as follows: dd if=/dev/block/mmcblk0 of=/sdcard/blk0.img bs=4096 conv=notrunc,noerror,sync if: This option specifies the path of the input file to read from. of: This option specifies the path of the output file to write to. bs: This option specifies the block size. Data is read and written in the size of the block specified, defaults to 512 bytes if not specified. conv: This option specifies the conversion options as its attributes: notrunc: This option does not truncate the output file. noerror: This option continues imaging if an error is encountered. sync: In conjunction with the noerror option, this option writes x00 for blocks with an error. This is important for maintaining file offsets within the image. Do not mix up the if and of flags, this could result in overwriting the target device! A full list of command options can be found at http://man7.org/linux/man-pages/man1/dd.1.html. Note that there is an important correlation between the block size and the noerror and sync flags: if an error is encountered, x00 will be written for the entire block that was read (as determined by the block size). Thus, smaller block sizes result in less data being missed in the event of an error. The downside is that, typically, smaller block sizes result in a slower transfer rate. An examiner will have to decide whether a timely or more accurate acquisition is preferred. Booting into recovery mode for the imaging process is the most forensically sound method. Determining what to image When imaging a computer, an examiner must first find what the drive is mounted as; /dev/sda, for example. The same is true when imaging an Android device. The first step is to launch the ADB shell and view the /proc/partitions file using the following command: cat /proc/partitions The output will show all partitions on the device: In the output shown in the preceding screenshot, mmcblk0 is the entirety of the flash memory on the device. To image the entire flash memory, we could use /dev/blk/mmcblk0 as the input file flag (if) for the dd command. Everything following it, indicated by p1- p29, is a partition of the flash memory. The size is shown in blocks, in this case the block size is 1024 bytes for a total internal storage size of approximately 32 GB. To obtain a full image of the device's internal memory, we would run the dd command with mmcblk0 as the input file. However, we know that most of these partitions are unlikely to be forensically interesting; we're most likely only interested in a few of them. To view the corresponding names for each partition, we can look in the device's by-name directory. This does not exist on every device, and is sometimes in a different path, but for this device it is found at /dev/block/msm_sdcc.1/by-name. By navigating to that directory and running the ls -al command, we can see to where each block is symbolically linked as shown in the following screenshot: If our investigation was only interested in the userdata partition, we now know that it is mmcblk0p28, and could use that as the input file to the dd command. If the by-name directory does not exist on the device, it may not be possible to identify every partition on the device. However, many of them can still be found by using the mount command within the ADB shell. Note that the following screenshot is from a different device that does not contain a by-name directory, so the data partition is not mmcblk0p28: On this device, the data partition is mmcblk0p34. If the mount command does not work, the same information can be found using the cat /proc/mounts command. Other options to identify partitions depending on the device are the cat /proc/mtd or cat /proc/yaffs commands; these may work on older devices. Newer devices may include an fstab file in the root directory (typically called fstab.<device>) that will list mountable partitions. Writing to an SD card The output file of the dd command can be written to the device's SD card. This should only be done if the suspect SD card can be removed and replaced with a forensically sterile SD to ensure that the dd command's output is not overwriting evidence. Obviously, if writing to an SD card, ensure that the SD card is larger than the partition being imaged. On newer devices, the /sdcard partition is actually a symbolic link to /data/media. In this case, using the dd command to copy the /data partition to the SD card won't work, and could corrupt the device because the input file is essentially being written to itself. To determine where the SD card is symbolically linked to, simply open the ADB shell and run the ls -al command. If the SD card partition is not shown, the SD likely needs to be mounted in recovery mode using the steps shown in this article, Extracting Data Logically from Android Devices. In the following example, /sdcard is symbolically linked to /data/media. This indicates that the dd command's output should not be written to the SD card. In the example that follows, the /sdcard is not a symbolic link to /data, so the dd command's output can be used to write the /data partition image to the SD card: On older devices, the SD card may not even be symbolically linked. After determining which block to read and to where the SD card is symbolically linked, image the /data partition to the /sdcard, using the following command: dd if=/dev/block/mmcblk0p28 of=/sdcard/data.img bs=512 conv=notrunc,noerror,sync Now, an image of the /data partition exists on the SD card. It can be pulled to the examiner's machine with the ADB pull command, or simply read from the SD card. Writing directly to an examiner's computer with netcat If the image cannot be written to the SD card, an examiner can use netcat to write the image directly to their machine. The netcat tool is a Linux-based tool used for transferring data over a network connection. We recommend using a Linux or a Mac computer for using netcat as it is built-in, though Windows versions do exist. The examples below were done on a Mac. Installing netcat on the device Very few Android devices, if any, come with netcat installed. To check, simply open the ADB shell and type nc. If it returns saying nc is not found, netcat will have to be installed manually on the device. Netcat compiled for Android can be found at many places online. We have shared the version we used at http://sourceforge.net/projects/androidforensics-netcat/files/. If we look back at the results from our mount command in the previous section, we can see that the /dev partition is mounted as tmpfs. The Linux term tmpfs means that the partition is meant to appear as an actual filesystem on the device, but is truly only stored in RAM. This means we can push netcat here without making any permanent changes to the device using the following command on the examiner's computer: adb push nc /dev/Examiner_Folder/nc The command should have created the Examiner_Folder in /dev, and nc should be in it. This can be verified by running the following command in the ADB shell: ls /dev/Examiner_Folder Using netcat Now that the netcat binary is on the device, we need to give it permission to execute from the ADB shell. This can be done as follows: chomd +x /dev/Examiner_Folder/nc We will need two terminal windows open with the ADB shell open in one of them. The other will be used to listen to the data being sent from the device. Now we need to enable port forwarding over ADB from the examiner's computer: adb forward tcp:9999 tcp:9999 9999 is the port we chose to use for netcat; it can be any arbitrary port number between 1023 and 65535 on a Linux or Mac system (1023 and below are reserved for system processes, and require root permission to use). Windows will allow any port to be assigned. In the terminal window with ADB shell, run the following command: dd if=/dev/block/mmcblk0p34 bs=512 conv=notrunc,noerror,sync | /dev/Examiner_Folder/nc –l –p 9999 mmcblk0p34 is the user data partition on this device, however, the entire flash memory or any other partition could also be imaged with this method. In most cases, it is best practice to image the entirety of the flash memory in order to acquire all possible data from the device. Some commercial forensic tools may also require the entire memory image, and may not properly handle an image of a single partition. In the other terminal window, run: nc 127.0.0.1 9999 > data_partition.img The data_partition.img file should now be created in the current directory of the examiner's computer. When the data is finished transferring, netcat in both terminals will terminate and return to the command prompt. The process can take a significant amount of time depending on the size of the image. Summary This article discussed techniques used for physically imaging internal memory or SD cards and some of the common problems associated with dd are as follows: Usually pre-installed on device May not work on MTD blocks Does not obtain Out-of-Band area Additionally, each imaging technique can be used to either save the image on the device (typically on the SD card), or used with netcat to write the file to the examiner's computer: Writing to SD card: Easy, doesn't require additional binaries to be pushed to the device Familiar to most examiners Cannot be used if SD card is symbolically linked to the partition being imaged Cannot be used if the entire memory is being imaged Using netcat: Usually requires yet another binary to be pushed to the device Somewhat complicated, must follow steps exactly Works no matter what is being imaged May be more time consuming than writing to the SD Resources for Article: Further resources on this subject: Reversing Android Applications [article] Introduction to Mobile Forensics [article] Processing the Case [article]

0
0
43871

Packt

29 Apr 2015

16 min read

Algorithmic Trading

Packt

29 Apr 2015

16 min read

In this article by James Ma Weiming, author of the book Mastering Python for Finance , we will see how algorithmic trading automates the systematic trading process, where orders are executed at the best price possible based on a variety of factors, such as pricing, timing, and volume. Some brokerage firms may offer an application programming interface (API) as part of their service offering to customers who wish to deploy their own trading algorithms. For developing an algorithmic trading system, it must be highly robust and handle any point of failure during the order execution. Network configuration, hardware, memory management and speed, and user experience are some factors to be considered when designing a system in executing orders. Designing larger systems inevitably add complexity to the framework. As soon as a position in a market is opened, it is subjected to various types of risk, such as market risk. To preserve the trading capital as much as possible, it is important to incorporate risk management measures to the trading system. Perhaps the most common risk measure used in the financial industry is the value-at-risk (VaR) technique. We will discuss the beauty and flaws of VaR, and how it can be incorporated into our trading system that we will develop in this article. In this article, we will cover the following topics: An overview of algorithmic trading List of brokers and system vendors with public API Choosing a programming language for a trading system Setting up API access on Interactive Brokers (IB) trading platform Using the IbPy module to interact with IB Trader WorkStation (TWS) Introduction to algorithmic trading In the 1990s, exchanges had already begun to use electronic trading systems. By 1997, 44 exchanges worldwide used automated systems for trading futures and options with more exchanges in the process of developing automated technology. Exchanges such as the Chicago Board of Trade (CBOT) and the London International Financial Futures and Options Exchange (LIFFE) used their electronic trading systems as an after-hours complement to traditional open outcry trading in pits, giving traders 24-hour access to the exchange's risk management tools. With improvements in technology, technology-based trading became less expensive, fueling the growth of trading platforms that are faster and powerful. Higher reliability of order execution and lower rates of message transmission error deepened the reliance of technology by financial institutions. The majority of asset managers, proprietary traders, and market makers have since moved from the trading pits to electronic trading floors. As systematic or computerized trading became more commonplace, speed became the most important factor in determining the outcome of a trade. Quants utilizing sophisticated fundamental models are able to recompute fair values of trading products on the fly and execute trading decisions, enabling them to reap profits at the expense of fundamental traders using traditional tools. This gave way to the term high-frequency trading (HFT) that relies on fast computers to execute the trading decisions before anyone else can. HFT has evolved into a billion-dollar industry. Algorithmic trading refers to the automation of the systematic trading process, where the order execution is heavily optimized to give the best price possible. It is not part of the portfolio allocation process. Banks, hedge funds, brokerage firms, clearing firms, and trading firms typically have their servers placed right next to the electronic exchange to receive the latest market prices and to perform the fastest order execution where possible. They bring enormous trading volumes to the exchange. Anyone who wishes to participate in low-latency, high-volume trading activities, such as complex event processing or capturing fleeting price discrepancies, by acquiring exchange connectivity may do so in the form of co-location, where his or her server hardware can be placed on a rack right next to the exchange for a fee. The Financial Information Exchange (FIX) protocol is the industry standard for electronic communications with the exchange from the private server for direct market access (DMA) to real-time information. C++ is the common choice of programming language for trading over the FIX protocol, though other languages, such as .NET framework common language and Java can be used. Before creating an algorithmic trading platform, you would need to assess various factors, such as speed and ease of learning before deciding on a specific language for the purpose. Brokerage firms would provide a trading platform of some sort to their customers for them to execute orders on selected exchanges in return for the commission fees. Some brokerage firms may offer an API as part of their service offering to technically inclined customers who wish to run their own trading algorithms. In most circumstances, customers may also choose from a number of commercial trading platforms offered by third-party vendors. Some of these trading platforms may also offer API access to route orders electronically to the exchange. It is important to read the API documentation beforehand to understand the technical capabilities offered by your broker and to formulate an approach in developing an algorithmic trading system. List of trading platforms with public API The following table lists some brokers and trading platform vendors who have their API documentation publicly available: Broker/vendor URL Programming languages supported Interactive Brokers https://www.interactivebrokers.com/en/index.php?f=1325 C++, Posix C++, Java, and Visual Basic for ActiveX E*Trade https://developer.etrade.com Java, PHP, and C++ IG http://labs.ig.com/ REST, Java, FIX, and Microsoft .NET Framework 4.0 Tradier https://developer.tradier.com Java, Perl, Python, and Ruby TradeKing https://developers.tradeking.com Java, Node.js, PHP, R, and Ruby Cunningham trading systems http://www.ctsfutures.com/wiki/T4%20API%2040.MainPage.ashx Microsoft .NET Framework 4.0 CQG http://cqg.com/Products/CQG-API.aspx C#, C++, Excel, MATLAB, and VB.NET Trading technologies https://developer.tradingtechnologies.com Microsoft .NET Framework 4.0 OANDA http://developer.oanda.com REST, Java, FIX, and MT4 Which is the best programming language to use? With many choices of programming languages available to interface with brokers or vendors, the question that comes naturally to anyone starting out in algorithmic trading platform development is: which language should I use? Well, the short answer is that there is really no best programming language. How your product will be developed, the performance metrics to follow, the costs involved, latency threshold, risk measures, and the expected user interface are pieces of the puzzle to be taken into consideration. The risk manager, execution engine, and portfolio optimizer are some major components that will affect the design of your system. Your existing trading infrastructure, choice of operating system, programming language compiler capability, and available software tools poses further constraints on the system design, development, and deployment. System functionalities It is important to define the outcomes of your trading system. An outcome could be a research-based system that might be more concerned with obtaining high-quality data from data vendors, performing computations or running models, and evaluating a strategy through signal generation. Part of the research component might include a data-cleaning module or a backtesting interface to run a strategy with theoretical parameters over historical data. The CPU speed, memory size, and bandwidth are factors to be considered while designing our system. Another outcome could be an execution-based system that is more concerned with risk management and order handling features to ensure timely execution of multiple orders. The system must be highly robust and handle any point of failure during the order execution. As such, network configuration, hardware, memory management and speed, and user experience are some factors to be considered when designing a system in executing orders. A system may contain one or more of these functionalities. Designing larger systems inevitably add complexity to the framework. It is recommended that you choose one or more programming languages that can address and balance the development speed, ease of development, scalability, and reliability of your trading system. Algorithmic trading with Interactive Brokers and IbPy In this section, we will build a working algorithmic trading platform that will authenticate with Interactive Brokers (IB) and log in, retrieve the market data, and send orders. IB is one of the most popular brokers in the trading community and has a long history of API development. There are plenty of articles on the use of the API available on the Web. IB serves clients ranging from hedge funds to retail traders. Although the API does not support Python directly, Python wrappers such as IbPy are available to make the API calls to the IB interface. The IB API is unique to its own implementation, and every broker has its own API handling methods. Nevertheless, the documents and sample applications provided by your broker would demonstrate the core functionality of every API interface that can be easily integrated into an algorithmic trading system if designed properly. Getting Interactive Brokers' Trader WorkStation The official page for IB is https://www.interactivebrokers.com. Here, you can find a wealth of information regarding trading and investing for retail and institutional traders. In this section, we will take a look at how to get the Trader WorkStation X (TWS) installed and running on your local workstation before setting up an algorithmic trading system using Python. Note that we will perform simulated trading on a demonstration account. If your trading strategy turns out to be profitable, head to the OPEN AN ACCOUNT section of the IB website to open a live trading account. Rules, regulations, market data fees, exchange fees, commissions, and other conditions are subjected to the broker of your choice. In addition, market conditions are vastly different from the simulated environment. You are encouraged to perform extensive testing on your algorithmic trading system before running on live markets. The following key steps describe how to install TWS on your local workstation, log in to the demonstration account, and set it up for API use: From IB's official website, navigate to TRADING, and then select Standalone TWS. Choose the installation executable that is suitable for your local workstation. TWS runs on Java; therefore, ensure that Java runtime plugin is already installed on your local workstation. Refer to the following screenshot: When prompted during the installation process, choose Trader_WorkStation_X and IB Gateway options. The Trader WorkStation X (TWS) is the trading platform with full order management functionality. The IB Gateway program accepts and processes the API connections without any order management features of the TWS. We will not cover the use of the IB Gateway, but you may find it useful later. Select the destination directory on your local workstation where TWS will place all the required files, as shown in the following screenshot: When the installation is completed, a TWS shortcut icon will appear together with your list of installed applications. Double-click on the icon to start the TWS program. When TWS starts, you will be prompted to enter your login credentials. To log in to the demonstration account, type edemo in the username field and demouser in the password field, as shown in the following screenshot: Once we have managed to load our demo account on TWS, we can now set up its API functionality. On the toolbar, click on Configure: Under the Configuration tree, open the API node to reveal further options. Select Settings. Note that Socket port is 7496, and we added the IP address of our workstation housing our algorithmic trading system to the list of trusted IP addresses, which in this case is 127.0.0.1. Ensure that the Enable ActiveX and Socket Clients option is selected to allow the socket connections to TWS: Click on OK to save all the changes. TWS is now ready to accept orders and market data requests from our algorithmic trading system. Getting IbPy – the IB API wrapper IbPy is an add-on module for Python that wraps the IB API. It is open source and can be found at https://github.com/blampe/IbPy. Head to this URL and download the source files. Unzip the source folder, and use Terminal to navigate to this directory. Type python setup.py install to install IbPy as part of the Python runtime environment. The use of IbPy is similar to the API calls, as documented on the IB website. The documentation for IbPy is at https://code.google.com/p/ibpy/w/list. A simple order routing mechanism In this section, we will start interacting with TWS using Python by establishing a connection and sending out a market order to the exchange. Once IbPy is installed, import the following necessary modules into our Python script: from ib.ext.Contract import Contractfrom ib.ext.Order import Orderfrom ib.opt import Connection Next, implement the logging functions to handle calls from the server. The error_handler method is invoked whenever the API encounters an error, which is accompanied with a message. The server_handler method is dedicated to handle all the other forms of returned API messages. The msg variable is a type of an ib.opt.message object and references the method calls, as defined by the IB API EWrapper methods. The API documentation can be accessed at https://www.interactivebrokers.com/en/software/api/api.htm. The following is the Python code for the server_handler method: def error_handler(msg):print "Server Error:", msgdef server_handler(msg):print "Server Msg:", msg.typeName, "-", msg We will place a sample order of the stock AAPL. The contract specifications of the order are defined by the Contract class object found in the ib.ext.Contract module. We will create a method called create_contract that returns a new instance of this object: def create_contract(symbol, sec_type, exch, prim_exch, curr):contract = Contract()contract.m_symbol = symbolcontract.m_secType = sec_typecontract.m_exchange = exchcontract.m_primaryExch = prim_exchcontract.m_currency = currreturn contract The Order class object is used to place an order with TWS. Let's define a method called create_order that will return a new instance of the object: def create_order(order_type, quantity, action):order = Order()order.m_orderType = order_typeorder.m_totalQuantity = quantityorder.m_action = actionreturn order After the required methods are created, we can then begin to script the main functionality. Let's initialize the required variables: if __name__ == "__main__":client_id = 100order_id = 1port = 7496tws_conn = None Note that the client_id variable is our assigned integer that identifies the instance of the client communicating with TWS. The order_id variable is our assigned integer that identifies the order queue number sent to TWS. Each new order requires this value to be incremented sequentially. The port number has the same value as defined in our API settings of TWS earlier. The tws_conn variable holds the connection value to TWS. Let's initialize this variable with an empty value for now. Let's use a try block that encapsulates the Connection.create method to handle the socket connections to TWS in a graceful manner: try:# Establish connection to TWS.tws_conn = Connection.create(port=port,clientId=client_id)tws_conn.connect()# Assign error handling function.tws_conn.register(error_handler, 'Error')# Assign server messages handling function.tws_conn.registerAll(server_handler)finally:# Disconnect from TWSif tws_conn is not None:tws_conn.disconnect() The port and clientId parameter fields define this connection. After the connection instance is created, the connect method will try to connect to TWS. When the connection to TWS has successfully opened, it is time to register listeners to receive notifications from the server. The register method associates a function handler to a particular event. The registerAll method associates a handler to all the messages generated. This is where the error_handler and server_handler methods declared earlier will be used for this occasion. Before sending our very first order of 100 shares of AAPL to the exchange, we will call the create_contract method to create a new contract object for AAPL. Then, we will call the create_order method to create a new Order object, to go long 100 shares. Finally, we will call the placeOrder method of the Connection class to send out this order to TWS: # Create a contract for AAPL stock using SMART order routing.aapl_contract = create_contract('AAPL','STK','SMART','SMART','USD')# Go long 100 shares of AAPLaapl_order = create_order('MKT', 100, 'BUY')# Place order on IB TWS.tws_conn.placeOrder(order_id, aapl_contract, aapl_order) That's it! Let's run our Python script. We should get a similar output as follows: Server Error: <error id=-1, errorCode=2104, errorMsg=Market data farmconnection is OK:ibdemo>Server Response: error, <error id=-1, errorCode=2104, errorMsg=Marketdata farm connection is OK:ibdemo>Server Version: 75TWS Time at connection:20141210 23:14:17 CSTServer Msg: managedAccounts - <managedAccounts accountsList=DU15200>Server Msg: nextValidId - <nextValidId orderId=1>Server Error: <error id=-1, errorCode=2104, errorMsg=Market data farmconnection is OK:ibdemo>Server Msg: error - <error id=-1, errorCode=2104, errorMsg=Market datafarm connection is OK:ibdemo>Server Error: <error id=-1, errorCode=2107, errorMsg=HMDS data farmconnection is inactive but should be available upon demand.demohmds>Server Msg: error - <error id=-1, errorCode=2107, errorMsg=HMDS data farmconnection is inactive but should be available upon demand.demohmds> Basically, what the error messages say is that there are no errors and the connections are OK. Should the simulated order be executed successfully during market trading hours, the trade will be reflected in TWS: The full source code of our implementation is given as follows: """ A Simple Order Routing Mechanism """from ib.ext.Contract import Contractfrom ib.ext.Order import Orderfrom ib.opt import Connectiondef error_handler(msg):print "Server Error:", msgdef server_handler(msg):print "Server Msg:", msg.typeName, "-", msgdef create_contract(symbol, sec_type, exch, prim_exch, curr):contract = Contract()contract.m_symbol = symbolcontract.m_secType = sec_typecontract.m_exchange = exchcontract.m_primaryExch = prim_exchcontract.m_currency = currreturn contractdef create_order(order_type, quantity, action):order = Order()order.m_orderType = order_typeorder.m_totalQuantity = quantityorder.m_action = actionreturn orderif __name__ == "__main__":client_id = 1order_id = 119port = 7496tws_conn = Nonetry:# Establish connection to TWS.tws_conn = Connection.create(port=port,clientId=client_id)tws_conn.connect()# Assign error handling function.tws_conn.register(error_handler, 'Error')# Assign server messages handling function.tws_conn.registerAll(server_handler)# Create AAPL contract and send orderaapl_contract = create_contract('AAPL','STK','SMART','SMART','USD')# Go long 100 shares of AAPLaapl_order = create_order('MKT', 100, 'BUY')# Place order on IB TWS.tws_conn.placeOrder(order_id, aapl_contract, aapl_order)finally:# Disconnect from TWSif tws_conn is not None:tws_conn.disconnect() Summary In this article, we were introduced to the evolution of trading from the pits to the electronic trading platform, and learned how algorithmic trading came about. We looked at some brokers offering API access to their trading service offering. To help us get started on our journey in developing an algorithmic trading system, we used the TWS of IB and the IbPy Python module. In our first trading program, we successfully sent an order to our broker through the TWS API using a demonstration account. Resources for Article: Prototyping Arduino Projects using Python Python functions – Avoid repeating code Pentesting Using Python

0
0
13995

Ensuring Five-star Rating in the MarketPlace

Installation and Upgrade

Symmetric Messages and Asynchronous Messages (Part 1)

Git Teaches – Great Tools Don't Make Great Craftsmen

Using Firebase: Learn how and why to use Firebase

Getting started with Codeception

Hadoop Monitoring and its aspects

Using your smart watch to control networked LEDs

Saying Hello to Unity and Android

Welcome to the Spring Framework

Trending Topics

Angular 2.0

Machine Learning

Less with External Applications and Frameworks

Extracting data physically with dd

Algorithmic Trading

Create a Free Account To Continue Reading

Sign in to activate your 7-day free access