How-To Tutorials

Bootstrap your app There are two ways to bootstrap your ZF2 app. The default is less flexible but handles the entire configuration, and the manual is really flexible but you have to take care of everything. The goal of the bootstrap is to provide to the application, ZendMvcApplication, with all the components and dependencies needed to successfully handle a request. A Zend Framework 2 application relies on the following six components: Configuration array ServiceManager instance EventManager instance ModuleManager instance Request object Response object As these are the pillars of a ZF2 application, we will take a look at how these components are configured to bootstrap the app. To begin with, we will see how the components interact from a high perspective and then we will jump into details of how each one works. When a new request arrives to our application, ZF2 needs to set up the environment to be able to fulfill it. This process implies reading configuration files and creating the required objects and services; attach them to the events that are going to be used and finally create the request object based on the request data. Once we have the request object, ZF2 will tell the router to do his job and will inspect the request object to determine who is responsible for processing the data. Once a controller and action has been identified as the one in charge of the request, ZF2 dispatches it and gives the controller/action the control of the program in order to execute the code that will interpret the request and will do something with it. This can be from accepting an uploaded image to showing a sign-up form and also changing data on an external database. When the controller processes the data, sometimes a view object is generated to encapsulate the data that we should send to the client who made the request, and a response object is created. After we have a response object, ZF2 sends it to the browser and the request ends. Now that we have seen a very simple overview of the lifecycle of a request we will jump into the details of how each object works, the options available and some examples of each one. Configuration array Let's dissect the first component of the list by taking a look at the index.php file: chdir(dirname(__DIR__)); // Setup autoloading require 'init_autoloader.php'; // Run the application! ZendMvcApplication::init(require 'config/application.config.php')->run(); As you can see we only do three things. The first thing is we change the current folder for the convenience of making everything relative to the root folder. Then we require the autoloader file; we will examine this file later. Finally, we initialize a ZendMvcApplication object by passing a configuration file and only then does the run method get called. The configuration file looks like the following code snippet: return array( 'modules' => array( 'Application', ), 'module_listener_options' => array( 'config_glob_paths' => array( 'config/autoload/{,*.}{global,local}.php', ), 'module_paths' => array( './module', './vendor', ), ), ); This file will return an array containing the configuration options for the application. Two options are used: modules and module_listener_options. As ZF2 uses a module organization approach, we should add the modules that we want to use on the application here. The second option we are using is passed as configuration to the ModuleManager object. The config_glob_path array is used when scanning the folders in search of config files and the module_paths array is used to tell ModuleManager a set of paths where the module resides. ZF2 uses a module approach to organize files. A module can contain almost anything, simple PHP files, view scripts, images, CSS, JavaScript, and so on. This approach will allow us to build reusable blocks of functionality and we will adhere to this while developing our project. PSR-0 and autoloaders Before continuing with the key components, let's take a closer look at the init_autoloader.php file used in the index.php file. As is stated on the first block comment, this file is more complicated than it's supposed to be. This is because ZF2 will try to set up different loading mechanisms and configurations. if (file_exists('vendor/autoload.php')) { $loader = include 'vendor/autoload.php'; } The first thing is to check if there is an autoload.php file inside the vendor folder; if it's found, we will load it. This is because the user might be using composer, in which case composer will provide a PSR-0 class loader. Also, this will register the namespaces defined by composer on the loader. PSR-0 is an autoloading standard proposed by the PHP Framework Interop Group (http://www.php-fig.org/) that describes the mandatory requirements for autoloader interoperability between frameworks. Zend Framework 2 is one of the projects that adheres to it. if (getenv('ZF2_PATH')) { $zf2Path = getenv('ZF2_PATH'); } elseif (get_cfg_var('zf2_path')) { $zf2Path = get_cfg_var('zf2_path'); } elseif (is_dir('vendor/ZF2/library')) { $zf2Path = 'vendor/ZF2/library'; } In the next section we will try to get the path of the ZF2 files from different sources. We will first try to get it from the environment, if not, we'll try from a directive value in the php.ini file. Finally, if the previous methods fail the code, we will try to check whether a specific folder exists inside the vendor folder. if ($zf2Path) { if (isset($loader)) { $loader->add('Zend', $zf2Path); } else { include $zf2Path . '/Zend/Loader/AutoloaderFactory.php'; ZendLoaderAutoloaderFactory::factory(array( 'ZendLoaderStandardAutoloader' => array( 'autoregister_zf' => true ) )); } } Finally, if the framework is found by any of these methods, based on the existence of the composer autoloader, the code will just add the Zend namespace or will instantiate an internal autoloader, ZendLoaderAutoloader, and use it as a default. As you can see, there are multiple ways to set up the autoloading mechanism on ZF2 and at the end what matters is which one you prefer, as all of them in essence will behave the same. ServiceManager After all this execution of code, we arrive at the last section of the index.php file where we actually instantiate the ZendMvcApplication object. As we said, there are two methods of creating an instance of ZendMvcApplication. In the default approach, we call the static method init of the class by passing an optional configuration as the first parameter. This method will take care of instantiating a new ServiceManager object, storing the configuration inside, loading the modules specified in the configuration, and getting a configured ZendMvcApplication. ServiceManager is a service/object locator that implements the Service Locator design pattern; its responsibility is to retrieve other objects. $serviceManager = new ServiceManager( new ServiceServiceManagerConfig($smConfig) ); $serviceManager->setService('ApplicationConfig', $configuration); $serviceManager->get('ModuleManager')->loadModules(); return $serviceManager->get('Application')->bootstrap(); As you can see, the init method calls the bootstrap() method of the ZendMvcApplication instance. Service Locator is a design pattern used in software development to encapsulate the process of obtaining other objects. The concept is based on a central repository that stores the objects and also knows how to create them if required. EventManager This component is designed to provide multiple functionalities. It can be used to implement simple observer patterns, and also can be used to do aspect-oriented design or even create event-driven architectures. The basic operations you can do over these components is attaching and detaching listeners to named events, trigger events, and interrupting the execution of listeners when an event is fired. Let's see a couple of examples on how to attach to an event and how to fire them: //Registering an event listener $events = new EventManager(); $events->attach(array('EVENT_NAME'), $callback); //Triggering an event $events->trigger('EVENT_NAME', $this, $params); Inside the bootstrap method of ZendMvcApplication, we are registering the events of RouteListener, DispatchListener, and ViewManager. After that, the code is instantiating a new custom event called MvcEvent that will be used as the target when firing events. Finally, this piece of code will fire the bootstrap event. ModuleManager Zend Framework 2 introduces a completely redesigned ModuleManager. This new module has been built with simplicity, flexibility, and reuse in mind. These modules can hold everything from PHP to images, CSS, library code, views, and so on. The responsibility of this component in the bootstrap process of an app is loading the available modules specified by the config file. This is accomplished by the following code line located in the init method of ZendMvcApplication: $serviceManager->get('ModuleManager')->loadModules(); This line, when executed, will retrieve the list of modules located at the config file and will load each module. Each module has to contain a file called Module.php with the initialization of the components of the module if needed. This will allow the module manager to retrieve the configuration of the module. Let's see the usual content of this file: namespace MyModule; class Module { public function getAutoloaderConfig() { return array( 'ZendLoaderClassMapAutoloader' => array( __DIR__ . '/autoload_classmap.php', ), 'ZendLoaderStandardAutoloader' => array( 'namespaces' => array( __NAMESPACE__ => __DIR__ . '/src/' . __NAMESPACE__, ), ), ); } public function getConfig() { return include __DIR__ . '/config/module.config.php'; } } As you can see we are defining a method called getAutoloaderConfig() that provides the configuration for the autoloader to ModuleManager. The last method getConfig() is used to provide the configuration of the module to ModuleManager; for example, this will contain the routes handled by the module. Request object This object encapsulates all data related to a request and allows the developer to interact with the different parts of a request. This object is used in the constructor of ZendMvcApplication and also is set inside MvcEvent to be able to retrieve when some events are fired. Response object This object encapsulates all the parts of an HTTP response and provides the developer with a fluent interface to set all the response data. This object is used in the same way as the request object. Basically it is instantiated on the constructor and added to MvcEvent to be able to interact with it across all the events and classes. The request object As we said, the request object will encapsulate all the data related to a request and provide the developer with a fluent API to access the data. Let's take a look at the details of the request object in order to understand how to use it and what it can offer to us: use ZendHttpRequest; $string = "GET /foo HTTP/1.1rnrnSome Content"; $request = Request::fromString($string); $request->getMethod(); $request->getUri(); $request->getUriString(); $request->getVersion(); $request->getContent(); This example comes directly from the documentation and shows how a request object can be created from a string, and then access some data related with the request using the methods provided. So, every time we need to know something related to the request, we will access this object to get the data we need. If we check the code on ZendHttpPhpEnvironmentRequest.php, the first thing we can notice is that the data is populated on the constructor using the superglobal arrays. All this data is processed and then populated inside the object to be able to expose it in a standard way using methods. To manipulate the URI of the request you can get/set the data with three methods, two getters and one setter. The only difference with the getters is that one returns a plain string and the other returns an HttpUri object. getUri() and getUriString() setUri() To retrieve the data passed in the request, there are a few specialized methods depending on the data you want to get: getQuery() getPost() getFiles() getHeader() and getHeaders() About the request method, the object has a general way to know the method used, returning a string or nine specialized functions that will test specific methods based on the RFC 2616, which defines the standard methods for an HTTP request. getMethod() isOptions() isGet() isHead() isPost() isPut() isDelete() isTrace() isConnect() isPatch() Finally, two more methods are available in this object that will test special requests such as AJAX and requests made by a flash object. isXmlHttpRequest() isFlashRequest() Notice that the data stored on the superglobal arrays when populated on the object are converted from an Array to a Parameters object. The Parameters object lives in the Stdlib section of ZF2, a folder where common objects can be found and used across the framework. In this case, the Parameters class is an extension of ArrayObject and implements ParametersInterface that will bring ArrayAccess, Countable, Serializable, and Traversable functionality to the parameters stored inside the object. The goal with this object is to provide a common interface to access data stored in the superglobal arrays. This expands the ways you can interact with the data in an object-oriented approach.

(For more resources related to this topic, see here.) So, if Cassandra is so good at everything, why not everyone drop whatever database they are using and jump start with Cassandra? This is a natural question. Some applications require strong ACID compliance, such as a booking system. If you are a person who goes by statistics, you'd ask how Cassandra fares with other existing data stores. TilmannRabl et al in their paper, Solving Big Data Challenges for Enterprise Application Performance Management (http://vldb.org/pvldb/vol5/p1724_tilmannrabl_vldb2012.pdf), told that, "In terms of scalability, there is a clear winner throughout our experiments. Cassandra achieves the highest throughput for the maximum number of nodes in all experiments with a linear increasing throughput from one to 12 nodes. This comes at the price of a high write and read latency. Cassandra's performance is best for high insertion rates." If you go through the paper, Cassandra wins in almost all the criteria. Equipped with proven concepts of distributed computing, made to reliably serve from commodity servers, and simple and easy maintenance, Cassandra is one of the most scalable, fastest, and very robust NoSQL database. So, the next natural question is what makes Cassandra so blazing fast? Let us dive deeper into the Cassandra architecture. Cassandra architecture Cassandra is a relative latecomer in the distributed data-store war. It takes advantage of two proven and closely similar data-store mechanisms, namely Google BigTable, a distributed storage system for structured data, 2006 (http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//archive/bigtable-osdi06.pdf [2006]), and Amazon Dynamo, Amazon's highly available key-value store, 2007 (http://www.read.seas.harvard.edu/~kohler/class/cs239-w08/decandia07dynamo.pdf [2007]). Figure 2.3: Read throughputs shows linear scaling of Cassandra Like BigTable, it has tabular data presentation. It is not tabular in the strictest sense. It is rather a dictionary-like structure where each entry holds another sorted dictionary/map. This model is more powerful than the usual key-value store and it is named as column family. The properties such as Eventual Consistency and decentralization are taken from Dynamo. For now, assume a column family as a giant spreadsheet, such as MS Excel. But unlike spreadsheets, each row is identified by a row key with a number (token), and unlike spreadsheets, each cell can have its own unique name within the row. The columns in the rows are sorted by this unique column name. Also, since the number of rows is allowed to be very large (1.7*(10)^38), we distribute the rows uniformly across all the available machines by dividing the rows in equal token groups. These rows create a Keyspace. Keyspace is a set of all the tokens (row IDs). Ring representation Cassandra cluster is denoted as a ring. The idea behind this representation is to show token distribution. Let's take an example. Assume that there is a partitioner that generates tokens from zero to 127 and you have four Cassandra machines to create a cluster. To allocate equal load, we need to assign each of the four nodes to bear an equal number of tokens. So, the first machine will be responsible for tokens one to 32, the second will hold 33 to 64, the third, 65 to 96, and the fourth, 97 to 127 and 0. If you mark each node with the maximum token number that it can hold, the cluster looks like a ring. (Figure 2.22) Partitioner is the hash function that determines the range of possible row keys. Cassandra uses a partitioner to calculate the token equivalent to a row key (row ID). Figure 2.4: Token ownership and distribution in a balanced Cassandra ring When you start to configure Cassandra, one thing that you may want to set is the maximum token number that a particular machine could hold. This property can be set in the Cassandra.yaml file as initial_token. One thing that may confuse a beginner is that the value of the initial token is what the last token owns. Be aware that nodes can be rebalanced and these tokens can be changed as the new nodes join or old nodes get discarded. This is the initial token because this is just the initial value, and it may be changed later. How Cassandra works Diving into various components of Cassandra without having a context is really a frustrating experience. It does not makes sense why you are studying SSTable, MemTable, and Log Structured Merge (LSM) tree without being able to see how they fit into functionality and performance guarantees that Cassandra gives. So, first, we will see Cassandra's write and read mechanism. It is possible that some of the terms that we encounter during this discussion may not be immediately understandable. A rough overview of the Cassandra components is as shown in the following figure: Figure 2.5: Main components of the Cassandra service The main class of Storage Layer is StorageProxy. It handles all the requests. Messaging Layer is responsible for internode communications like gossip. Apart from this, process-level structures keep a rough idea about the actual data containers and where they live. There are four data buckets that you need to know. MemTable is a hash table-like structure that stays in memory. It contains actual column data. SSTable is the disk version of MemTables. When MemTables are full, SSTables are persisted to the hard disk. Bloom filters is a probabilistic data structure that lives in memory. It helps Cassandra to quickly detect which SSTable does not have the requested data. CommitLog is the usual commit log that contains all the mutations that are to be applied. It lives on the disk and helps to replay uncommitted changes. With this primer, we can start looking into how write and read works in Cassandra. We will see more explanation later. Write in action To write, clients need to connect to any of the Cassandra nodes and send a write request. This node is called as the coordinator node. When a node in Cassandra cluster receives a write request, it delegates it to a service called StorageProxy. This node may or may not be the right place to write the data to. The task of StorageProxy is to get the nodes (all the replicas) that are responsible to hold the data that is going to be written. It utilizes a replication strategy to do that. Once the replica nodes are identified, it sends the RowMutation message to them, the node waits for replies from these nodes, but it does not wait for all the replies to come. It only waits for as many responses as are enough to satisfy the client's minimum number of successful writes defined by ConsistencyLevel. So, the following figure and steps after that show all that can happen during a write mechanism: Figure 2.6: A simplistic representation of the write mechanism. The figure on the left represents the node-local activities on receipt of the write request If FailureDetector detects that there aren't enough live nodes to satisfy ConsistencyLevel, the request fails. If FailureDetector gives a green signal, but writes time-out after the request is sent due to infrastructure problems or due to extreme load, StorageProxy writes a local hint to replay when the failed nodes come back to life. This is called hinted handoff. One might think that hinted handoff may be responsible for Cassandra's eventual consistency. But it's not entirely true. If the coordinator node gets shut down or dies due to hardware failure and hints on this machine cannot be forwarded, eventual consistency will not occur. The Anti-entropy mechanism is responsible for consistency rather than hinted handoff. Anti-entropy makes sure that all replicas are in sync. If the replica nodes are distributed across datacenters, it will be a bad idea to send individual messages to all the replicas in other datacenters. It rather sends the message to one replica in each datacenter with a header instructing it to forward the request to other replica nodes in that datacenter. Now, the data is received by the node that should actually store that data. The data first gets appended to CommitLog, and pushed to a MemTable for the appropriate column family in the memory. When MemTable gets full, it gets flushed to the disk in a sorted structure named SSTable. With lots of flushes, the disk gets plenty of SSTables. To manage SSTables, a compaction process runs. This process merges data from smaller SSTables to one big sorted file. Read in action Similar to a write case, when StorageProxy of the node that a client is connected to gets the request, it gets a list of nodes containing this key based on Replication Strategy. StorageProxy then sorts the nodes based on their proximity to itself. The proximity is determined by the Snitch function that is set up for this cluster. Basically, there are the following types of Snitch: SimpleSnitch: A closer node is the one that comes first when moving clockwise in the ring. (A ring is when all the machines in the cluster are placed in a circular fashion with each having a token number. When you walk clockwise, the token value increases. At the end, it snaps back to the first node.) AbstractNetworkTopologySnitch: Implementation of the Snitch function works like this: nodes on the same rack are closest. The nodes in the same datacenter but in different rack are closer than those in other datacenters, but farther than the nodes in the same rack. Nodes in different datacenters are the farthest. To a node, the nearest node will be the one on the same rack. If there is no node on the same rack, the nearest node will be the one that lives in the same datacenter, but on a different rack. If there is no node in the datacenter, any nearest neighbor will be the one in the other datacenter. DynamicSnitch: This Snitch determines closeness based on recent performance delivered by a node. So, a quick-responding node is perceived closer than a slower one, irrespective of their location closeness or closeness in the ring. This is done to avoid overloading a slow-performing node. Now that we have the list of nodes that have desired row keys, it's time to pull data from them. The coordinator node (the one that the client is connected to) sends a command to the closest node to perform read (we'll discuss local read in a minute) and return the data. Now, based on ConsistencyLevel, other nodes will send a command to perform a read operation and send just the digest of the result. If we have Read Repair (discussed later) enabled, the remaining replica nodes will be sent a message to compute the digest of the command response. Let's take an example: say you have five nodes containing a row key K (that is, replication factor (RF) equals 5). Your read ConsistencyLevel is three. Then the closest of the five nodes will be asked for the data. And the second and third closest nodes will be asked to return the digest. We still have two left to be queried. If read Repair is not enabled, they will not be touched for this request. Otherwise, these two will be asked to compute digest. The request to the last two nodes is done in the background, after returning the result. This updates all the nodes with the most recent value, making all replicas consistent. So, basically, in all scenarios, you will have a maximum one wrong response. But with correct read and write consistency levels, we can guarantee an up-to-date response all the time. Let's see what goes within a node. Take a simple case of a read request looking for a single column within a single row. First, the attempt is made to read from MemTable, which is rapid-fast since there exists only one copy of data. This is the fastest retrieval. If the data is not found there, Cassandra looks into SSTable. Now, remember from our earlier discussion that we flush MemTables to disk as SSTables and later when compaction mechanism wakes up, it merges those SSTables. So, our data can be in multiple SSTables. Figure 2.7: A simplified representation of the read mechanism. The bottom image shows processing on the read node. Numbers in circles shows the order of the event. BF stands for Bloom Filter Each SSTable is associated with its Bloom Filter built on the row keys in the SSTable. Bloom Filters are kept in memory, and used to detect if an SSTable may contain (false positive) the row data. Now, we have the SSTables that may contain the row key. The SSTables get sorted in reverse chronological order (latest first). Apart from Bloom Filter for row keys, there exists one Bloom Filter for each row in the SSTable. This secondary Bloom Filter is created to detect whether the requested column names exist in the SSTable. Now, Cassandra will take SSTables one by one from younger to older. And use the index file to locate the offset for each column value for that row key and the Bloom filter associated with the row (built on the column name). On Bloom filter being positive for the requested column, it looks into the SSTable file to read the column value. Note that we may have a column value in other yet-to-be-read SSTables, but that does not matter, because we are reading the most recent SSTables first, and any value that was written earlier to it does not matter. So, the value gets returned as soon as the first column in the most recent SSTable is allocated. Components of Cassandra We have gone through how read and write takes place in highly distributed Cassandra clusters. It's time to look into individual components of it a little deeper. Messaging service Messaging service is the mechanism that manages internode socket communication in a ring. Communications, for example, gossip, read, read digest, write, and so on, processed via a messaging service, can be assumed as a gateway messaging server running at each node. To communicate, each node creates two socket connections per node. This implies that if you have 101 nodes, there will be 200 open sockets on each node to handle communication with other nodes. The messages contain a verb handler within them that basically tells the receiving node a couple of things: how to deserialize the payload message and what handler to execute for this particular message. The execution is done by the verb handlers (sort of an event handler). The singleton that orchestrates the messaging service mechanism is org.apache.cassandra.net.MessagingService. Gossip Cassandra uses the gossip protocol for internode communication. As the name suggests, the protocol spreads information in the same way an office rumor does. It can also be compared to a virus spread. There is no central broadcaster, but the information (virus) gets transferred to the whole population. It's a way for nodes to build the global map of the system with a small number of local interactions. Cassandra uses gossip to find out the state and location of other nodes in the ring (cluster). The gossip process runs every second and exchanges information with at the most three other nodes in the cluster. Nodes exchange information about themselves and other nodes that they come to know about via some other gossip session. This causes all the nodes to eventually know about all the other nodes. Like everything else in Cassandra, gossip messages have a version number associated with it. So, whenever two nodes gossip, the older information about a node gets overwritten with a newer one. Cassandra uses an Anti-entropy version of gossip protocol that utilizes Merkle trees (discussed later) to repair unread data. Implementation-wise the gossip task is handled by the org.apache.cassandra.gms.Gossiper class. Gossiper maintains a list of live and dead endpoints (the unreachable endpoints). At every one-second interval, this module starts a gossip round with a randomly chosen node. A full round of gossip consists of three messages. A node X sends a syn message to a node Y to initiate gossip. Y, on receipt of this syn message, sends an ack message back to X. To reply to this ack message, X sends an ack2 message to Y completing a full message round. Figure 2.8: Two nodes gossiping Failure detection Failure detection is one of the fundamental features of any robust and distributed system. A good failure detection mechanism implementation makes a fault-tolerant system such as Cassandra. The failure detector that Cassandra uses is a variation of The ϕ accrual failure detector (2004) by Xavier Défago et al. (The phi accrual detector research paper is available at http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.106.3350.) The idea behind a FailureDetector is to detect a communication failure and take appropriate actions based on the state of the remote node. Unlike traditional failure detectors, phi accrual failure detector does not emit a Boolean alive or dead (true or false, trust or suspect) value. Instead, it gives a continuous value to the application and the application is left to decide the level of severity and act accordingly. This continuous suspect value is called phi (ϕ). Partitioner Cassandra is a distributed database management system. This means it takes a single logical database and distributes it over one or more machines in the database cluster. So, when you insert some data in Cassandra, it assigns each row to a row key; and based on that row key, Cassandra assigns that row to one of the nodes that's responsible for managing it. Let's try to understand this. Cassandra inherits the data model from Google's BigTable (BigTable research paper can be found at http://research.google.com/archive/bigtable.html.). This means we can roughly assume that the data is stored in some sort of a table that has an unlimited number of columns (not unlimited, Cassandra limits the maximum number of columns to be two billion) with rows binded with a unique key, namely, row key. Now, your terabytes of data on one machine will be restrictive from multiple points of views. One is disk space, another being limited parallel processing, and if not duplicated, a source of single point of failure. What Cassandra does is, it defines some rules to slice data across rows and assigns which node in the cluster is responsible for holding which slice. This task is done by a partitioner. There are several types of partitioners to choose from. In short, Cassandra (as of Version 1.2) offers three partitioners as follows: RandomPartitioner: It uses MD5 hashing to distribute data across the cluster. Cassandra 1.1.x and precursors have this as the default partitioner. Murmur3Partitioner: It uses Murmur hash to distribute the data. It performs better than RandomPartitioner. It is the default partitioner from Cassandra Version 1.2 onwards. ByteOrderPartitioner: Keeps keys distributed across the cluster by key bytes. This is an ordered distribution, so the rows are stored in lexical order. This distribution is commonly discouraged because it may cause a hotspot. Replication Cassandra runs on commodity hardware, and works reliably in network partitions. However, this comes with a cost: replication. To avoid data inaccessibility in case a node goes down or becomes unavailable, one must replicate data to more than one node. Replication brings features such as fault tolerance and no single point of failure to the system. Cassandra provides more than one strategy to replicate the data, and one can configure the replication factor while creating keyspace. Log Structured Merge tree Cassandra (also, HBase) is heavily influenced by Log Structured Merge (LSM). It uses an LSM tree-like mechanism to store data on a disk. The writes are sequential (in append fashion) and the data storage is contiguous. This makes writes in Cassandra superfast, because there is no seek involved. Contrast this with an RBDMS system that is based on the B+ Tree (http://en.wikipedia.org/wiki/B%2B_tree) implementation. LSM tree advocates the following mechanism to store data: note down the arriving modification into a log file (CommitLog), push the modification/new data into memory (MemTable) for faster lookup, and when the system has gathered enough updates in memory or after a certain threshold time, flush this data to a disk in a structured store file (SSTable). The logs corresponding to the updates that are flushed can now be discarded. Figure 2.12: Log Structured Merge (LSM) Trees CommitLog One of the promises that Cassandra makes to the end users is durability. In conventional terms (or in ACID terminology), durability guarantees that a successful transaction (write, update) will survive permanently. This means once Cassandra says write successful that means the data is persisted and will survive system failures. It is done the same way as in any DBMS that guarantees durability: by writing the replayable information to a file before responding to a successful write. This log is called the CommitLog in the Cassandra realm. This is what happens. Any write to a node gets tracked by org.apache.cassandra.db.commitlog.CommitLog, which writes the data with certain metadata into the CommitLog file in such a manner that replaying this will recreate the data. The purpose of this exercise is to ensure there is no data loss. If due to some reason the data could not make it into MemTable or SSTable, the system can replay the CommitLog to recreate the data. MemTable MemTable is an in-memory representation of column family. It can be thought of as a cached data. MemTable is sorted by key. Data in MemTable is sorted by row key. Unlike CommitLog, which is append-only, MemTable does not contain duplicates. A new write with a key that already exists in the MemTable overwrites the older record. This being in memory is both fast and efficient. The following is an example: Write 1: {k1: [{c1, v1}, {c2, v2}, {c3, v3}]} In CommitLog (new entry, append): {k1: [{c1, v1},{c2, v2}, {c3, v3}]} In MemTable (new entry, append): {k1: [{c1, v1}, {c2, v2}, {c3, v3}]} Write 2: {k2: [{c4, v4}]} In CommitLog (new entry, append): {k1: [{c1, v1}, {c2, v2}, {c3, v3}]} {k2: [{c4, v4}]} In MemTable (new entry, append): {k1: [{c1, v1}, {c2, v2}, {c3, v3}]} {k2: [{c4, v4}]} Write 3: {k1: [{c1, v5}, {c6, v6}]} In CommitLog (old entry, append): {k1: [{c1, v1}, {c2, v2}, {c3, v3}]} {k2: [{c4, v4}]} {k1: [{c1, v5}, {c6, v6}]} In MemTable (old entry, update): {k1: [{c1, v5}, {c2, v2}, {c3, v3}, {c6, v6}]} {k2: [{c4, v4}]} Cassandra Version 1.1.1 uses SnapTree (https://github.com/nbronson/snaptree) for MemTable representation, which claims it to be "... a drop-in replacement for ConcurrentSkipListMap, with the additional guarantee that clone() is atomic and iteration has snapshot isolation." See also copy-on-write and compare-and-swap (http://en.wikipedia.org/wiki/Copy-on-write, http://en.wikipedia.org/wiki/Compare-and-swap). Any write gets written first to CommitLog and then to MemTable. SSTable SSTable is a disk representation of the data. MemTables gets flushed to disk to immutable SSTables. All the writes are sequential, which makes this process fast. So, the faster the disk speed, the quicker the flush operation. The SSTables eventually get merged in the compaction process and the data gets organized properly into one file. This extra work in compaction pays off during reads. SSTables have three components: Bloom filter, index files, and datafiles. Bloom filter Bloom filter is a litmus test for the availability of certain data in storage (collection). But unlike a litmus test, a Bloom filter may result in false positives: that is, it says that a data exists in the collection associated with the Bloom filter, when it actually does not. A Bloom filter never results in a false negative. That is, it never states that a data is not there while it is. The reason to use Bloom filter, even with its false-positive defect, is because it is superfast and its implementation is really simple. Cassandra uses Bloom filters to determine whether an SSTable has the data for a particular row key. Bloom filters are unused for range scans, but they are good candidates for index scans. This saves a lot of disk I/O that might take in a full SSTable scan, which is a slow process. That's why it is used in Cassandra, to avoid reading many, many SSTables, which can become a bottleneck. Index files Index files are companion files of SSTables. The same as Bloom filter, there exists one index file per SSTable. It contains all the row keys in the SSTable and its offset at which the row starts in the datafile. At startup, Cassandra reads every 128th key (configurable) into the memory (sampled index). When the index is looked for a row key (after Bloom filter hinted that the row key might be in this SSTable), Cassandra performs a binary search on the sampled index in memory. Followed by a positive result from the binary search, Cassandra will have to read a block in the index file from the disk starting from the nearest value lower than the value that we are looking for. Datafiles Datafiles are the actual data. They contain row keys, metadata, and columns (partial or full). Reading data from datafiles is just one disk seek followed by a sequential read, as offset to a row key is already obtained from the associated index file. Compaction As mentioned earlier, a read require may require Cassandra to read across multiple SSTables to get a result. This is wasteful, costs multiple (disk) seeks, may require a conflict resolution, and if there are too many, SSTables were created. To handle this problem, Cassandra has a process in place, namely, compaction. Compaction merges multiple SSTable files into one. Off the shelf, Cassandra offers two types of compaction mechanism: Size Tiered Compaction Strategy and Level Compaction Strategy. The compaction process starts when the number of SSTables on disk reaches a certain threshold (N: configurable). Although the merge process is a little I/O intensive, it benefits in the long term with a lower number of disk seeks during reads. Apart from this, there are a few other benefits of compaction as follows: Removal of expired tombstones (Cassandra v0.8+) Merging row fragments Rebuilds primary and secondary indexes Tombstones Cassandra is a complex system with its data distributed among CommitLogs, MemTables, and SSTables on a node. The same data is then replicated over replica nodes. So, like everything else in Cassandra, deletion is going to be eventful. Deletion, to an extent, follows an update pattern except Cassandra tags the deleted data with a special value, and marks it as a tombstone. This marker helps future queries, compaction, and conflict resolution. Let's step further down and see what happens when a column from a column family is deleted. A client connected to a node (coordinator node, but it may not be the one holding the data that we are going to mutate), issues a delete command for a column C, in a column family CF. If the consistency level is satisfied, the delete command gets processed. When a node, containing the row key, receives a delete request, it updates or inserts the column in MemTable with a special value, namely, tombstone. The tombstone basically has the same column name as the previous one; the value is set to UNIX epoch. The timestamp is set to what the client has passed. When a MemTable is flushed to SSTable, all tombstones go into it as any regular column. On the read side, when the data is read locally on the node and it happens to have multiple versions of it in different SSTables, they are compared and the latest value is taken as the result of reconciliation. If a tombstone turns out to be a result of reconciliation, it is made a part of the result that this node returns. So, at this level, if a query has a deleted column, this exists in the result. But the tombstones will eventually be filtered out of the result before returning it back to the client. So, a client can never see a value that is a tombstone. Hinted handoff When we last talked about durability, we observed Cassandra provides CommitLogs to provide write durability. This is good. But what if the node, where the writes are going to be, is itself dead? No communication will keep anything new to be written to the node. Cassandra, inspired by Dynamo, has a feature called hinted handoff. In short, it's the same as taking a quick note locally that X cannot be contacted. Here is the mutation (operation that requires modification of the data such as insert, delete, and update) M that will be required to be replayed when it comes back. The coordinator node (the node which the client is connected to) on receipt of a mutation/write request, forwards it to appropriate replicas that are alive. If this fulfills the expected consistency level, write is assumed successful. The write requests to a node that does not respond to a write request or is known to be dead (via gossip) and is stored locally in the system.hints table. This hint contains the mutation. When a node comes to know via gossip that a node is recovered, it replays all the hints it has in store for that node. Also, every 10 minutes, it keeps checking any pending hinted handoffs to be written. Why worry about hinted handoff when you have written to satisfy consistency level? Wouldn't it eventually get repaired? Yes, that's right. Also, hinted handoff may not be the most reliable way to repair a missed write. What if the node that has hinted handoff dies? This is a reason why we do not count on hinted handoff as a mechanism to provide consistency (except for the case of the consistency level, ANY) guarantee; it's a single point of failure. The purpose of hinted handoff is, one, to make restored nodes quickly consistent with the other live ones; and two, to provide extreme write availability when consistency is not required. Read repair and Anti-entropy Cassandra promises eventual consistency and read repair is the process which does that part. Read repair, as the name suggests, is the process of fixing inconsistencies among the replicas at the time of read. What does that mean? Let's say we have three replica nodes A, B, and C that contain a data X. During an update, X is updated to X1 in replicas A and B. But it is failed in replica C for some reason. On a read request for data X, the coordinator node asks for a full read from the nearest node (based on the configured Snitch) and digest of data X from other nodes to satisfy consistency level. The coordinator node compares these values (something like digest(full_X) == digest_from_node_C). If it turns out that the digests are the same as the digests of full read, the system is consistent and the value is returned to the client. On the other hand, if there is a mismatch, full data is retrieved and reconciliation is done and the client is sent the reconciled value. After this, in background, all the replicas are updated with the reconciled value to have a consistent view of data on each node. See Figure 2.1 as shown: Figure 2.16: Image showing read repair dynamics. 1. Client queries for data x, from a node C (coordinator). 2. C gets data from replicas R1, R2, and R3; reconciles. 3. Sends reconciled data to client. 4. If there is a mismatch across replicas, repair is invoked. Merkle tree Merkle tree (A digital signature Based On A Conventional Encryption Function by Merkle, R. (1988), available at http://www.cse.msstate.edu/~ramkumar/merkle2.pdf.) is a hash tree where leaves of the tree hashes hold actual data in a column family and non-leaf nodes hold hashes of their children. The unique advantage of Merkle tree is a whole subtree can be validated just by looking at the value of the parent node. So, if nodes on two replica servers have the same hash values, then the underlying data is consistent and there is no need to synchronize. If one node passes the whole Merkle tree of a column family to another node, it can determine all the inconsistencies. Figure 2.17: Merkle tree to determine mismatch in hash values at parent nodes due to the difference in underlying data Summary By now, you are familiar with all the nuts and bolts of Cassandra. It is understandable that it may be a lot to take in for someone new to NoSQL systems. It is okay if you do not have complete clarity at this point. As you start working with Cassandra, tweaking it, experimenting with it, and going through the Cassandra mailing list discussions or talks, you will start to come across stuff that you have read in this article and it will start to make sense, and perhaps you may want to come back and refer to this article to improve clarity. It is not required to understand this article fully to be able to write queries, set up clusters, maintain clusters, or do anything else related to Cassandra. A general sense of this article will take you far enough to work extremely well with Cassandra-based projects. Resources for Article: Further resources on this subject: Apache Cassandra: Libraries and Applications [Article] Apache Felix Gogo [Article] Migration from Apache to Lighttpd [Article]

(For more resources related to this topic, see here.) John Kirkland is an awesome "accidental" SQL Server DBA for Red Speed Bicycle LLC—a growing bicycle startup based in the United States. The company distributes bikes, bicycle parts, and accessories to various distribution points around the world. To say that they are performing well financially is an understatement. They are booming! They've been expanding their business to Canada, Australia, France, and the United Kingdom in the last three years. The company has upgraded their SQL Server 2000 database recently to the latest version of SQL Server 2012. Linda, from the Finance Group, asked John if they can migrate their Microsoft Access Reports into the SQL Server 2012 Reporting Services. John installed SSRS 2012 in a native mode. He decided to build the reports from the ground up so that the report development process would not interrupt the operation in the Finance Group. There is only one caveat; John has never authored any reports in SQL Server Reporting Services (SSRS) before. Let's give John a hand and help him build his reports from the ground up. Then, we'll see more of his SSRS adventures as we follow his journey throughout this article. Here's the first report requirement for John: a simple table that shows all the sales transactions in their database. Linda wants to see a report with the following data: Date Sales Order ID Category Subcategory Product Name Unit Price Quantity Line Total We will build our report, and all succeeding reports in this article, using the SQL Server Data Tools (SSDT). SSDT is Visual Studio shell which is an integrated environment used to build SQL Server database objects. You can install SSDT from the SQL Server installation media. In June 2013, Microsoft released SQL Server Data Tools-Business Intelligence (SSDTBI). SSDTBI is a component that contains templates for SQL Server Analysis Services (SSAS), SQL Server Integration Services (SSIS), and SQL Server Reporting Services (SSRS) for Visual Studio 2012. SSDTBI replaced Business Intelligence Development Studio (BIDS) from the previous versions of SQL Server. You have two options in creating your SSRS reports: SSDT or Visual Studio 2012. If you use Visual Studio, you have to install the SSDTBI templates. Let's create a new solution and name it SSRS2012Blueprints. For the following exercises, we're using SSRS 2012 in native mode. Also, make a note that we're using the AdventureWorks2012 Sample database all throughout this article unless otherwise indicated. You can download the sample database from CodePlex. Here's the link: http://msftdbprodsamples.codeplex.com/releases/view/55330. Defining a data source for the project Now, let's define a shared data source and shared dataset for the first report. A shared dataset and data source can be shared among the reports within the project: Right-click on the Shared Data Sources folder under the SSRS2012Bueprints solution in the Solution Explorer window, as shown in the following illustration. If the Solution Explorer window is not visible, access it by navigating to Menu | View | Solution Explorer, or press Ctrl + Alt + L: Select Add New Data Source which displays the Shared Data Source Properties window. Let's name our data source DS_SSRS2012Blueprint. For this demonstration, let's use the wizard to create the connection string. As a good practice, I use the wizard for setting up connection strings for my data connections. Aside from convenience, I'm quite confident that I'm getting the right connections that I want. Another option for setting the connection is through the Connection Properties dialog box, as shown in the next screenshot. Clicking on the Edit button next to the connection string box displays the Connection Properties dialog box: Shared versus embedded data sources and datasets: as a good practice, always use shared data sources and shared datasets where appropriate. One characteristic of a productive development project is using reusable objects as much as possible. For the connection, one option is to manually specify the connection string as shown: Data Source=localhost;Initial Catalog=AdventureWorks2012 We may find this option as a convenient way of creating our data connections. But if you're new to the report environment you're currently working on, you may find setting up the connection string manually more cumbersome than setting it up through the wizard. Always test the connection before saving your data source. After testing, click on the OK buttons on both dialog boxes. Defining the dataset for the project Our next step is to create the shared dataset for the project. Before doing that, let's create a stored procedure named dbo.uspSalesDetails. This is going to be the query for our dataset. Download the T-SQL codes included in this article if you haven't done so already. We're going to use the T-SQL file named uspSalesDetails_Ch01.sql for this article. We will use the same stored procedure for this whole article, unless otherwise indicated. Right-click on the Shared Datasets folder in Solution Explorer, just like we did when we created the data source. That displays the Shared Datasets Properties dialog. Let's name our dataset ds_SalesDetailReport. We use the query type stored procedure, and select or type uspSalesDetails on the Select or enter stored procedure name drop-down combo box. Click on OK when you're done: Before we work on the report itself, let's examine our dataset. In the Solution Explorer window, double-click on the dataset ds_SalesDetailReport.rsd, which displays the Shared Dataset Properties dialog box. Notice that the fields returned by our stored procedure have been automatically detected by the report designer. You can rename the field as shown: Ad-hoc Query (Text Query Type) versus Stored Procedure: as a good practice, always use a stored procedure where a query is used. The primary reason for this is that a stored procedure is compiled into a single execution plan. Using stored procedures will also allow you to modify certain elements of your reports without modifying the actual report. Creating the report file Now, we're almost ready to build our first report. We will create our report by building it from scratch by performing the following steps: Going back to the Solution Explorer window, right-click on the Reports folder. Please take note that selecting the Add New Report option will initialize Report Wizard. Use the wizard to build simple tabular or matrix reports. Go ahead if you want to try the wizard but for the purpose of our demonstration, we'll skip the wizard. Select Add, instead of Add New Report, then select New Item: Selecting New Item displays the Add New Item dialog box as shown in the following screenshot. Choose the Report template (default report template) in the template window. Name the report SalesDetailsReport.rdl. Click on the Add button to add the report to our project: Clicking on the Add button displays the empty report in the report designer. It looks similar to the following screenshot: Creating a parameterized report You may have noticed that the stored procedure we created for the shared dataset is parameterized. It has the following parameters: It's a good practice to test all the queries on the database just to make sure we get the datasets that we need. Doing so will eliminate a lot of data quality issues during report execution. This is also the best time to validate all our data. We want our report consumers to have the correct data that is needed for making critical decisions. Let's execute the stored procedure in SQL Server Management Studio (SSMS) and take a look at the execution output. We want to make sure that we're getting the results that we want to have on the report. Now, we add a dataset to our report based on the shared dataset that we had previously created: Right-click on the Datasets folder in the Report Data window. If it's not open, you can open it by navigating to Menu | View | Report Data, or press Ctrl + Alt + D: Selecting Add Dataset displays the Dataset Properties. Let's name our report dataset tblSalesReport. We will use this dataset as the underlying data for the table element that we will create to hold our report data. Indicate that we want to use a shared dataset. A list of the project shared datasets is displayed. We only have one at this point, which is the ds_SalesDetailsReport. Let's select that one, then click on OK. Going back to the Report Data window, you may notice that we now have more objects under the Parameters and Datasets folders. Switch to the Toolbox window. If you don't see it, then go to Menu | View | Toolbox, or press Ctrl + Alt + X. Double-click or drag a table to the empty surface of the designer. Let's add more columns to the table to accommodate all eight dataset fields. Click on the table, then right-click on the bar on the last column and select Insert Column | Right. To add data to the report, let's drag each element from the dataset to their own cell at the table data region. There are three data regions in SSRS: table, matrix, and list. In SSRS 2012, a fourth data region has been added but you can't see that listed anywhere. It's called tablix. Tablix is not shown as an option because it is built into those three data regions. What we're doing in the preceding screenshot is essentially dragging data into the underlying tablix data region. But how can I add my parameters into the report? you may ask. Well, let's switch to the Preview tab. We should now see our parameters already built into the report because we specified them in our stored procedure. Our report should look similar to the following screenshot:

(For more resources related to this topic, see here.) Easy audio grouping Pro Tools offers an interesting shortcut to assist the process. To create a new audio group, the simplest way is to follow these steps: Select the tracks to be grouped. Press option + shift (Mac) or Alt + Shift (Windows) and click on the track output. Select New Track and create a mono or stereo auxiliary track. The bus name and routing will be automatically assigned. We can later select or show the corresponding tracks associated with any type of input or output by right-clicking on any track input or output. This also works for hardware sends. Multiple track outputs In Pro Tools, a track can have as many outputs as you like (as long as you have available voices); it can even be routed to all available outputs simultaneously. Hold the control key (Mac) or Start key (Windows) and click on the track's output to add another output. When a track is routed to multiple outputs, a + button is displayed like this: Routing to multiple outputs; this track is assigned to Mixbus and other outputs There are also key modifiers that we can use. Key modifier Mac Windows Add the desired output to all the tracks control + option Start + Alt Add the desired output to the selected tracks control + option + shift Start + Alt + Shift Stemming with multiple outputs Unfortunately, there is no easy way of creating stems or multitrack bounces automatically with Pro Tools 10; the process remains mainly manual. Pro Tools 11 addresses some of these issues in the next section, but for now, assuming that our master fader track contains processing, our first and only stemming option is to solo the appropriate tracks and use the bounce to disk function. If we have no processing for our master output fader, we can bounce inside the session instead by assigning multiple outputs to tracks or buses by grouping them and creating auxiliary sends. Using the key modifiers defined previously, you can quickly select tracks and assign their outputs or auxiliary sends to new audio tracks for recording. To stem tracks to a new audio track without affecting their current routing, use the following methods. Using multiple outputs Select the tracks. Hold control + option + shift (Mac) or Start + Alt + Shift (Windows). Click on New track from the track output menu. Create the appropriate audio track and record. Using auxiliary sends Select the tracks. Hold option + shift (Mac) or Alt + Shift (Windows). Click on N ew track from the track output menu. Create the appropriate audio track and record. We can select the desired recording format from the session window accessible from Setup | Session . If you're recording the track at anything other than a 32-bit float, make sure your levels do not exceed 0 dBfs. We can also use complex mixing routing techniques to our advantage, as this will give us many summing points from which we can easily create the New track stems. We will discuss more on how to increase routing complexity to create more summing points inside the mix. Sound layering Using auxiliary tracks is the best way to perform parallel processing in Pro Tools, but multiple output routing can also be very useful for sound layering, keeping your track sends free for other effects. By sending the output to multiple auxiliary tracks, we can very easily add textures and effects. As an example, I like to layer my basses this way with different distortions and other parallel processing. To me, sound layering is more suited to multiple outputs because their relative level mostly stays the same. Stemming with Pro Tools 11 Pro Tools 11 brings new offline bouncing capabilities with added simultaneous bounce from internal busses or physical outputs. Inside the Bounce to disk window, click on the + button to add another source to a maximum of 16. Each source can have a different file format, from mono to multichannel audio. Because we can bounce physical outputs, Pro Tools 11 also allows for stemming tracks that are processed using master fader inserts. These new features added to offline bounces have drastically improved the Pro Tools 10 workflow, saving us a lot of time and leaving us with the following two main stemming options: While working with multiple outputs, we can source our stem from the physical outputs directly While summing inside Pro Tools, it becomes a lot more relevant to group as many tracks as we can to create available bounce sources Increasing mixing complexity One of the biggest advantages of mixing inside a DAW over traditional analogue console setups is its expanded routing capabilities, giving us far more control over the signal. In this section, I'd like to discuss how to use buses to create more complex mixes, that is, creating extra summing points for several uses and comparing the digital approach over a traditional frontend analog mixing console setup. As we saw previously, Pro Tools 10 introduced a new 64-bit floating point mixer, allowing for almost unlimited headroom and greater precision, which made digital summing even more digitally perfect. This means that it is entirely up to the engineer to create his sound textures through recording or digital processing. Pro Tools 11 took this quest for digital perfection even further with a full 64-bit path. On the other hand, analog is technically flawed and has always been; every single piece of circuitry will color the signal in various amounts, but these are those imperfections that many of us came to love and successfully or not tried to recreate in the digital domain. A digital mixer will sum signals with a far lower harmonic distortion. Applying different distortions is therefore the key to restore an analog feel to the mix. The first step toward this approach is to increase the mixer's complexity to be able to apply different types of distortions in many more places. Audio groups or buses are the most basic routing idea. As an example, working with drums, I would group all the kicks together, the snare together, the toms together, the overheads together, the rooms together, and so on. Those subbuses will then be bused to a master drum bus. This allows us to do the following: Process multiple microphones at a time Bounce stems more easily later Have greater control over my individual drum sounds To me, routing is the heart of digital mixing and making it as complex as possible begins to open many new creative doors. But we can go a step further. My drum bus will be first bused to my rhythm bus, then to my instrumental bus, and finally to my mix bus. This introduces more advanced groupings based on other criteria such as rhythm or instrumental. This might not seem very useful at first, but it will make you think about the track in different ways, giving you control at every stages of the mix. Here is an example of what a complex mix would look like: An example of a more complex routing diagram I might not use all the possibilities all the time, and this is only a small example. But this helped me very much over the years to create better depth in my mixes and also considerably speed up my workflow while creating stem mixes. I have instant access to many more summing points in my mixes, allowing me to group and differentiate even more sounds from each other. The previous diagram only shows how to increase a mix complexity by busing tracks into each other, but these summing points are also open doors to some very effective parallel processing. Default output bus options While working with multiple buses, it becomes quite hard to systematically remember to update the track output to the user's Mixbus. We can change the default track's Pro Tools's default bus output through Preferences | I/O | Output | Default Output Bus . Along with changing the default bus output, there are a few key modifiers worth remembering to assign tracks to the desired output or the bus. Key modifier Mac Windows Assign all the tracks to the same I/O option Alt Assign all the selected tracks to the same I/O option + shift Alt + Shift Assign all the tracks' I/O incrementally command + option Ctrl + Alt Assign all the selected tracks' I/O incrementally command + option + shift Ctrl + Alt + Shift Greater control over parallel processing Parallel processing is widely used for mixing. It allows adding to the signal while preserving the original content. It usually sounds a lot better to mix this way rather than process everything on groups; drums can particularly benefit from this technique. The main issue with group processing is that it tends to make everything sound very clinical and lifeless, especially if too much dynamic treatment is applied. Digital dynamic processors especially do not compare well to their analog counterpart when it comes to attenuating levels without taking life out of the recording. For these reasons, I tend to complement my mixes with a lot of parallel processing to increase harmonics, depth, loudness, and focus frequencies rather than cutting or boosting them too much. Parallel processing allows me to get a richer sound. Creating parallel processing requires extra buses. We can use the track sends, multiple outputs, or duplicate the track to achieve quick results. As an example, using track sends in pre fader mode, I can quickly create a perfect copy of my signal independent of my main mix level to process and automate creatively. Select the tracks and press command + G (Mac) or Ctrl + G (Windows) to create a new group. Inside the Create group window, disable Follow Global and select Mix or Edit/Mix and the appropriate send and settings you would like to link.In my example, I will only link Volume and Mute . Now that we have created the group, a neat thing to do would be to copy the existing track volume levels to the send. To do so, perform the following steps: Press command + option + H (Mac) or Ctrl + Alt + H (Windows). The Copy to Send window appears. Select the correct destination send and the parameter to copy. We can copy everything, even automation, to have the exact same mix across the entire song. Now with the tracks still selected, display any send window, hold option + shift (Mac) or Alt + Shift (Windows), and click on PRE to make the sends pre fader and have the auxiliary mix independent of the main mix. Having copied the current mix to the send, we can not only use it as a perfect parallel copy, but now we also have the choice to alter the balance between the current mix and the send to better suit the type of parallel processing we will be applying. Trying to improve the low end might not require too much high frequency information, so I might change the balance to pull down the appropriate tracks. On the other hand, trying to improve my high end might benefit from pulling down low frequency content. Copying mix to send is also very useful while setting up cue mixes for recording. Using the copy to send function is very rewarding while working with parallel processing, but the default mixer's send view does not look very user friendly. Pro Tools offers an alternate way of displaying the send information inside the mixer window. We can focus on a particular send to display mini fader sends and level information. To do so, let's navigate to View | Sends A-E or View | Sends F-J and select the particular send you would like to display. We can also press command + click (Mac) or Ctrl + click (Windows) from the left-hand side of the send in the mixer view. We can display a maximum of two sends at a time in Pro Tools 10 and all of them simultaneously in Pro Tools 11, giving a great matrix view for parallel processing or headphone mixes. This new functionality is accessible from View | Expanded Sends . Pro Tools 11 Expanded Sends view, showing Sends A-E We now have a nice send display with linked parameters: To change the level of all the grouped tracks, keeping their relative level with the group active, just move one of the send fader To change one send value without affecting the entire group, hold control (Mac) or Start (Windows) and move the appropriate fader Advanced side chain Side chaining is one of the most commonly used techniques for music production, sound design, and mixing. Pro Tools allow for some interesting side chaining possibilities, allowing us to not only mix many different tracks and process them, but also to play with their timing by manipulating the automatic delay compensation. For bass I will stay with my drum recording and take the classic example of compressing the bass track according to the kick. By doing so, we make the groove tighter, reinforcing the impression that the bass player is locked to the drum groove. To do so, I could use the Bomb Factory BF76 plugin on the bass with my side chain activated on bus 15 carrying the kick. Once it is set up, every time the kick hits, the bass track gain will be pulled down, enhancing the groove and also giving more space in the mix for the "kick drum," but that is not all. Since Pro Tools uses buses to create external side chains, we can send as many signals as we'd like to control the bass accordingly. As an example, I could have the kick not only during the verses but could also add some of my percussion to the side chain mix. Tweaking the side chain balance mix can give you very natural results and give the impression that the bass player is grooving more with the entire performance. When we group multiple signals together into a side chain, we can also alter their frequency spectrum or even their dynamics to create different compression behaviors. Because my side chain information is contained on a bus, I can create a master fader or auxiliary track to process the audio before it reaches my plugin. While working with recorded drums, it might be useful to gate the side chain bus a lot more aggressively, since it cannot be heard. We can also boost one of the low kick frequencies and cut the high end to avoid hats, crashes, and so on triggering unwanted compressions. To do so, we could boost the low frequencies first and then gate the result more aggressively. Summary In this article, we learned how to use internal routing and automations to increase creative output. Resources for Article: Further resources on this subject: Audio Playback [Article] Automating the Audio Parameters – How it Works [Article] Getting Started with Adobe Premiere Pro CS6 Hotshot [Article]

(For more resources related to this topic, see here.) Playing audio files JUCE provides a sophisticated set of classes for dealing with audio. This includes: sound file reading and writing utilities, interfacing with the native audio hardware, audio data conversion functions, and a cross-platform framework for creating audio plugins for a range of well-known host applications. Covering all of these aspects is beyond the scope of this article, but the examples in this section will outline the principles of playing sound files and communicating with the audio hardware. In addition to showing the audio features of JUCE, in this section we will also create the GUI and autogenerate some other aspects of the code using the Introjucer application. Creating a GUI to control audio file playback Create a new GUI application Introjucer project of your choice, selecting the option to create a basic window. In the Introjucer application, select the Config panel, and select Modules in the hierarchy. For this project we need the juce_audio_utils module (which contains a special Component class for configuring the audio device hardware); therefore, turn ON this module. Even though we created a basic window and a basic component, we are going to create the GUI using the Introjucer application. Navigate to the Files panel and right-click (on the Mac, press control and click) on the Source folder in the hierarchy, and select Add New GUI Component… from the contextual menu. When asked, name the header MediaPlayer.h and click on Save. In the Files hierarchy, select the MediaPlayer.cpp file. First select the Class panel and change the Class name from NewComponent to MediaPlayer. We will need four buttons for this basic project: a button to open an audio file, a Play button, a Stop button, and an audio device settings button. Select the Subcomponents panel, and add four TextButton components to the editor by right-clicking to access the contextual menu. Space the buttons equally near the top of the editor, and configure each button as outlined in the table as follows: Purpose member name name text background (normal) Open file openButton open Open... Default Play/pause file playButton play Play Green Stop playback stopButton stop Stop Red Configure audio settingsButton settings Audio Settings... Default Arrange the buttons as shown in the following screenshot: For each button, access the mode pop-up menu for the width setting, and choose Subtracted from width of parent. This will keep the right-hand side of the buttons the same distance from the right-hand side of the window if the window is resized. There are more customizations to be done in the Introjucer project, but for now, make sure that you have saved the MediaPlayer.h file, the MediaPlayer.cpp file, and the Introjucer project before you open your native IDE project. Make sure that you have saved all of these files in the Introjucer application; otherwise the files may not get correctly updated in the file system when the project is opened in the IDE. In the IDE we need to replace the MainContentComponent class code to place a MediaPlayer object within it. Change the MainComponent.h file as follows: #ifndef __MAINCOMPONENT_H__#define __MAINCOMPONENT_H__#include "../JuceLibraryCode/JuceHeader.h"#include "MediaPlayer.h"class MainContentComponent : public Component{public:MainContentComponent();void resized();private:MediaPlayer player;};#endif Then, change the MainComponent.cpp file to: #include "MainComponent.h"MainContentComponent::MainContentComponent(){addAndMakeVisible (&player);setSize (player.getWidth(),player.getHeight());}void MainContentComponent::resized(){player.setBounds (0, 0, getWidth(), getHeight());} Finally, make the window resizable in the Main.cpp file, and build and run the project to check that the window appears as expected. Adding audio file playback support Quit the application and return to the Introjucer project. Select the MediaPlayer.cpp file in the Files panel hierarchy and select its Class panel. The Parent classes setting already contains public Component. We are going to be listening for state changes from two of our member objects that are ChangeBroadcaster objects. To do this, we need our MediaPlayer class to inherit from the ChangeListener class. Change the Parent classes setting such that it reads: public Component, public ChangeListener Save the MediaPlayer.h file, the MediaPlayer.cpp file, and the Introjucer project again, and open it into your IDE. Notice in the MediaPlayer.h file that the parent classes have been updated to reflect this change. For convenience, we are going to add some enumerated constants to reflect the current playback state of our MediaPlayer object, and a function to centralize the change of this state (which will, in turn, update the state of various objects, such as the text displayed on the buttons). The ChangeListener class also has one pure virtual function, which we need to add. Add the following code to the [UserMethods] section of MediaPlayer.h: //[UserMethods]-- You can add your own custom methods...enum TransportState {Stopped,Starting,Playing,Pausing,Paused,Stopping};void changeState (TransportState newState);void changeListenerCallback (ChangeBroadcaster* source);//[/UserMethods] We also need some additional member variables to support our audio playback. Add these to the [UserVariables] section: //[UserVariables] -- You can add your own custom variables...AudioDeviceManager deviceManager;AudioFormatManager formatManager;ScopedPointer<AudioFormatReaderSource> readerSource;AudioTransportSource transportSource;AudioSourcePlayer sourcePlayer;TransportState state;//[/UserVariables] The AudioDeviceManager object will manage our interface between the application and the audio hardware. The AudioFormatManager object will assist in creating an object that will read and decode the audio data from an audio file. This object will be stored in the ScopedPointer<AudioFormatReaderSource> object. The AudioTransportSource object will control the playback of the audio file and perform any sampling rate conversion that may be required (if the sampling rate of the audio file differs from the audio hardware sampling rate). The AudioSourcePlayer object will stream audio from the AudioTransportSource object to the AudioDeviceManager object. The state variable will store one of our enumerated constants to reflect the current playback state of our MediaPlayer object. Now add some code to the MediaPlayer.cpp file. In the [Constructor] section of the constructor, add following two lines: playButton->setEnabled (false);stopButton->setEnabled (false); This sets the Play and Stop buttons to be disabled (and grayed out) initially. Later, we enable the Play button once a valid file is loaded, and change the state of each button and the text displayed on the buttons, depending on whether the file is currently playing or not. In this [Constructor] section you should also initialize the AudioFormatManager as follows: formatManager.registerBasicFormats(); This allows the AudioFormatManager object to detect different audio file formats and create appropriate file reader objects. We also need to connect the AudioSourcePlayer, AudioTransportSource and AudioDeviceManager objects together, and initialize the AudioDeviceManager object. To do this, add the following lines to the [Constructor] section: sourcePlayer.setSource (&transportSource);deviceManager.addAudioCallback (&sourcePlayer);deviceManager.initialise (0, 2, nullptr, true); The first line connects the AudioTransportSource object to the AudioSourcePlayer object. The second line connects the AudioSourcePlayer object to the AudioDeviceManager object. The final line initializes the AudioDeviceManager object with: The number of required audio input channels (0 in this case). The number of required audio output channels (2 in this case, for stereo output). An optional "saved state" for the AudioDeviceManager object (nullptr initializes from scratch). Whether to open the default device if the saved state fails to open. As we are not using a saved state, this argument is irrelevant, but it is useful to set this to true in any case. The final three lines to add to the [Constructor] section to configure our MediaPlayer object as a listener to the AudioDeviceManager and AudioTransportSource objects, and sets the current state to Stopped: deviceManager.addChangeListener (this);transportSource.addChangeListener (this);state = Stopped; In the buttonClicked() function we need to add some code to the various sections. In the [UserButtonCode_openButton] section, add: //[UserButtonCode_openButton] -- add your button handler...FileChooser chooser ("Select a Wave file to play...",File::nonexistent,"*.wav");if (chooser.browseForFileToOpen()) {File file (chooser.getResult());readerSource = new AudioFormatReaderSource(formatManager.createReaderFor (file), true);transportSource.setSource (readerSource);playButton->setEnabled (true);}//[/UserButtonCode_openButton] When the openButton button is clicked, this will create a FileChooser object that allows the user to select a file using the native interface for the platform. The types of files that are allowed to be selected are limited using the wildcard *.wav to allow only files with the .wav file extension to be selected. If the user actually selects a file (rather than cancels the operation), the code can call the FileChooser::getResult() function to retrieve a reference to the file that was selected. This file is then passed to the AudioFormatManager object to create a file reader object, which in turn is passed to create an AudioFormatReaderSource object that will manage and own this file reader object. Finally, the AudioFormatReaderSource object is connected to the AudioTransportSource object and the Play button is enabled. The handlers for the playButton and stopButton objects will make a call to our changeState() function depending on the current transport state. We will define the changeState() function in a moment where its purpose should become clear. In the [UserButtonCode_playButton] section, add the following code: //[UserButtonCode_playButton] -- add your button handler...if ((Stopped == state) || (Paused == state))changeState (Starting);else if (Playing == state)changeState (Pausing);//[/UserButtonCode_playButton] This changes the state to Starting if the current state is either Stopped or Paused, and changes the state to Pausing if the current state is Playing. This is in order to have a button with combined play and pause functionality. In the [UserButtonCode_stopButton] section, add the following code: //[UserButtonCode_stopButton] -- add your button handler...if (Paused == state)changeState (Stopped);elsechangeState (Stopping);//[/UserButtonCode_stopButton] This sets the state to Stopped if the current state is Paused, and sets it to Stopping in other cases. Again, we will add the changeState() function in a moment, where these state changes update various objects. In the [UserButtonCode_settingsButton] section add the following code: //[UserButtonCode_settingsButton] -- add your button handler...bool showMidiInputOptions = false;bool showMidiOutputSelector = false;bool showChannelsAsStereoPairs = true;bool hideAdvancedOptions = false;AudioDeviceSelectorComponent settings (deviceManager,0, 0, 1, 2,showMidiInputOptions,showMidiOutputSelector,showChannelsAsStereoPairs,hideAdvancedOptions);settings.setSize (500, 400);DialogWindow::showModalDialog(String ("Audio Settings"),&settings,TopLevelWindow::getTopLevelWindow (0),Colours::white,true); //[/UserButtonCode_settingsButton] This presents a useful interface to configure the audio device settings. We need to add the changeListenerCallback() function to respond to changes in the AudioDeviceManager and AudioTransportSource objects. Add the following to the [MiscUserCode] section of the MediaPlayer.cpp file: //[MiscUserCode] You can add your own definitions...void MediaPlayer::changeListenerCallback (ChangeBroadcaster* src){if (&deviceManager == src) {AudioDeviceManager::AudioDeviceSetup setup;deviceManager.getAudioDeviceSetup (setup);if (setup.outputChannels.isZero())sourcePlayer.setSource (nullptr);elsesourcePlayer.setSource (&transportSource);} else if (&transportSource == src) {if (transportSource.isPlaying()) {changeState (Playing);} else {if ((Stopping == state) || (Playing == state))changeState (Stopped);else if (Pausing == state)changeState (Paused);}}}//[/MiscUserCode] If our MediaPlayer object receives a message that the AudioDeviceManager object changed in some way, we need to check that this change wasn't to disable all of the audio output channels, by obtaining the setup information from the device manager. If the number of output channels is zero, we disconnect our AudioSourcePlayer object from the AudioTransportSource object (otherwise our application may crash) by setting the source to nullptr. If the number of output channels becomes nonzero again, we reconnect these objects. If our AudioTransportSource object has changed, this is likely to be a change in its playback state. It is important to note the difference between requesting the transport to start or stop, and this change actually taking place. This is why we created the enumerated constants for all the other states (including transitional states). Again we issue calls to the changeState() function depending on the current value of our state variable and the state of the AudioTransportSource object. Finally, add the important changeState() function to the [MiscUserCode] section of the MediaPlayer.cpp file that handles all of these state changes: void MediaPlayer::changeState (TransportState newState){if (state != newState) {state = newState;switch (state) {case Stopped:playButton->setButtonText ("Play");stopButton->setButtonText ("Stop");stopButton->setEnabled (false);transportSource.setPosition (0.0);break;case Starting:transportSource.start();break;case Playing:playButton->setButtonText ("Pause");stopButton->setButtonText ("Stop");stopButton->setEnabled (true);break;case Pausing:transportSource.stop();break;case Paused:playButton->setButtonText ("Resume");stopButton->setButtonText ("Return to Zero");break;case Stopping:transportSource.stop();break;}}} After checking that the newState value is different from the current value of the state variable, we update the state variable with the new value. Then, we perform the appropriate actions for this particular point in the cycle of state changes. These are summarized as follows: In the Stopped state, the buttons are configured with the Play and Stop labels, the Stop button is disabled, and the transport is positioned to the start of the audio file. In the Starting state, the AudioTransportSource object is told to start. Once the AudioTransportSource object has actually started playing, the system will be in the Playing state. Here we update the playButton button to display the text Pause, ensure the stopButton button displays the text Stop, and we enable the Stop button. If the Pause button is clicked, the state becomes Pausing, and the transport is told to stop. Once the transport has actually stopped, the state changes to Paused, the playButton button is updated to display the text Resume and the stopButton button is updated to display Return to Zero. If the Stop button is clicked, the state is changed to Stopping, and the transport is told to stop. Once the transport has actually stopped, the state changes to Stopped (as described in the first point). If the Return to Zero button is clicked, the state is changed directly to Stopped (again, as previously described). When the audio file reaches the end of the file, the state is also changed to Stopped. Build and run the application. You should be able to select a .wav audio file after clicking the Open... button, play, pause, resume, and stop the audio file using the respective buttons, and configure the audio device using the Audio Settings… button. The audio settings window allows you to select the input and output device, the sample rate, and the hardware buffer size. It also provides a Test button that plays a tone through the selected output device. Summary This article has covered a few of the techniques for dealing with audio files in JUCE. The article has given only an introduction to get you started; there are many other options and alternative approaches, which may suit different circumstances. The JUCE documentation will take you through each of these and point you to related classes and functions. Resources for Article: Further resources on this subject: Quick start – media files and XBMC [Article] Audio Playback [Article] Automating the Audio Parameters – How it Works [Article]

(For more resources related to this topic, see here.) OpenCart is an e-commerce cart application built with its own in-house framework which uses Model-View-Controller (MVC) language pattern thus each module in OpenCart also follows the MVCL patterns. Controller creates logics and gathers data from Model and pass it to display them in the view OpenCart modules have admin/ and catalog/ folders and files in admin/ folder helps in controlling setting of module and files in catalog/ folder handles the presentation layer (front-end). Learning how to clone and write codes for Opencart Modules We assume that you already know PHP and have already installed the OpenCart and familiar with the OpenCart backend and frontend as well as some coding knowledge with PHP. You are going to create Hello World module which just has one input box at the admin, settings for the module and same content are shown in front end. First step on module creation is using a unique name, so there will not be conflict with other modules. The same unique name is used to create the file name and class name to extend controller and model. There are generally 6-8 files that need to be created for each module, and they follow a similar structure. If there is interaction with the database tables, we have to create two extra models. The following screenshot shows the hierarchy of files and folder of OpenCart module. So now you know the basic directory structure of OpenCart module. The file structure is divided into two sections admin and catalog. Admin folders and files deal with the setting of the modules and data handling while the catalog folders and files handles the frontend. Let's start with an easy way to make the module. You are going to make the duplicate of the default google_talk module of OpenCart and change it to the Hello World module. We are using the Dreamweaver to work with files. Changes made at the admin folder Go to admin/controller/module/ and copy google_talk.php and paste in the same folder and rename it to helloworld.php and open it to your favorite text editor, then find the following lines: classControllerModuleGoogleTalk extends Controller { Change the class name to classControllerModuleHelloworld extends Controller { Now look for google_talk and replace all with helloworld as shown in the following screenshot: Then, save the file Go to admin/language/english/module/ and copy google_talk.php and paste in the same folder and rename it to helloworld.php and open it. Then look for the following line of code: $_['entry_code'] = 'Google Talk Code:<br /> <span class="help">Goto <a href="http://www.google.com/talk/service/badge/New" target="_blank"> <u>Create a Google Talk chatback badge</u> </a> and copy &amp; paste the generated code into the text box. </span>'; Replace with the following line of code: $_['entry_code'] = 'Hello World Content'; Then again find Google Talk and replace all with Hello World, and save the file. Go to admin/view/template/module/ and copy the google_talk.tpl file and paste in the same folder and rename it to helloworld.tpl and open it and then find google_talk and replace it with helloworld then save it. Changes made at the catalog folder Go to catalog/controller/module/ and copy the google_talk.php file and paste in the same folder and rename it to helloworld.php and open it and look for the following line: class ControllerModuleGoogleTalk extends Controller { change the class name to class ControllerModuleHelloworld extends Controller { Now look for google_talk and replace all with helloworld and save it. Go to catalog/language/english/module/ and copy the google_talk.php file and paste in the same folder and rename it to helloworld.php and open it and then look for Live Chat and replace it with Hello World then save it. Go to catalog/view/theme/default/template/module/ and copy the google_talk.tpl file and paste in the same folder and rename it to helloworld.tpl. With the preceding files and codes change, our Hello World module is ready to be installed. Now log in to the admin section and go to Extensions | Module, look for the Hello World and click on [install] then click on [Edit]. Then type the content that you would like to show at the frontend in Hello World Content field after that click on the Add Module button and provide the setting as per your need and click on Save. Understanding the global Library methods used in OpenCart OpenCart has many predefined methods which can be called anywhere like in the controller or in the model and as well as in the view template files as per the need. You can find system level library files at system/library/. We have defined all the library functions so that it is easy for programmers to use it. For example: $this->customer->login($email, $password, $override = false) Log a customer in. It checks for the customer username and password if $override is passed false, else only for current logged in status and the e-mail. If it finds the correct entry then the cart entry, wishlist entries are retrieved. As well as customer ID, first name, last name, e-mail, telephone, fax, newsletter subscription status, customer group ID, and address ID are globally accessible for the customer. It also updates the customer IP address from where it logs in. Developing and customizing modules, pages, order totals, shipping, and payments extensions in OpenCart We describe the featured module of OpenCart, create feedback module and tips module, and describe and show how codes work and are managed. We helped to learn how to make pages and modules in OpenCart as well as let visualize the uses of database structure; how data are saved as per language, as per store so it helps OpenCart programmers to understand and follow the patterns of Opencart coding style. We describe codes how form works, how to list out the data from database, how edit works in module, and how they are saved. Show them how to code shipping module in OpenCart as well as total order modules and payment modules. We have outlined how templates, models, and controllers work for extensions. Summary In this way we have learned how to clone and write codes for OpenCart modules and the changes made at the admin and catalog folder. We also learned the global library methods used in OpenCart. Also, covered all ways to code the OpenCart extensions. Resources for Article: Further resources on this subject: Upgrading OpenCart [Article] OpenCart FAQs [Article] OpenCart: Layout Structure [Article]

(For more resources related to this topic, see here.) Working with lights Before starting to explain about lights, we need to learn how to create and manipulate them. Once you know how to handle them, you will be ready to start learning the who-is-who in the lightning stage. Let's start with the basics. Adding a light Lights are handled in modo just like regular items. You can move, rotate, and scale them, and of course, tweak their properties. By default, a newly created scene has got a default light already. You can use it, change its type, or add as many as you need. In order to add a new light you should go to the Item List tab, and then click on the Add Item button. In the drop-down menu go to Lights, then choose the type of light you want. The other way to do this is by using the top menu. Navigate to Item | Create Light and choose the type you want. Setting the type of a light You can always change the type of the light just created (or change an existent light). In the Item List tab, right-click the light you want to change, and from the menu click on Change Type, then choose the type you want for your light. Placing lights As said previously, lights are like all other regular items. So you can move, rotate, and scale them as you need. You will have the following two ways of placing a light: Direct manipulation: Working in item mode, click on the light on any of the viewports—or directly in the Item List tab— and use the corresponding tools (W for moving, R for scaling, or Y for rotating). Subjective manipulation: A more interesting and practical way to move a light is by changing the viewport to light view mode. Once you change it, your view will be literally inside the light, so the direction you are facing will be the direction of the light. In this view, use your standard viewport controls to orientate the light. Enabling/disabling lights Usually, there are occasions when you need to turn off a light, or a number of them. The first thought would be turning its intensity value to zero, but there is a more practical way to temporarily disable a light. If you take a look at the items list, you will see a column on the left of the panel showing a little eye icon. That column shows the visibility state of each item. The eye means that it's visible, and you can click on that icon to totally disable the light (or any item, in fact), and click on it again to enable it back. Of course you can do rest of the basic operations with the lights, as with other kinds of items including enabling/disabling them, grouping them in a single folder, and so on.

(For more resources related to this topic, see here.) The Social Networks Analysis Social Networks Analysis (SNA) is not new, sociologists have been using it for a long time to study human relationships (sociometry), to find communities and to simulate how information or a disease is spread in a population. With the rise of social networking sites such as Facebook, Twitter, LinkedIn, and so on. The acquisition of large amounts of social network data is easier. We can use SNA to get insight about customer behavior or unknown communities. It is important to say that this is not a trivial task and we will come across sparse data and a lot of noise (meaningless data). We need to understand how to distinguish between false correlation and causation. A good start is by knowing our graph through visualization and statistical analysis. Social networking sites bring us the opportunities to ask questions that otherwise are too hard to approach, because polling enough people is time-consuming and expensive. In this article, we will obtain our social network's graph from Facebook (FB) website in order to visualize the relationships between our friends. Finally we will create an interactive visualization of our graph using D3.js. Getting ready The easiest method to get our friends list is by using a third-party application. Netvizz is a Facebook app developed by Bernhard Rieder, which allows exporting social graph data to gdf and tab formats. Netvizz may export information about our friends such as gender, age, locale, posts, and likes. In order to get our social graph from Netvizz we need to access the link below and giving access to your Facebook profile. https://apps.facebook.com/netvizz/ As is shown in the following screenshot, we will create a gdf file from our personal friend network by clicking on the link named here in the Step 2. Then we will download the GDF (Graph Modeling Language) file. Netvizz will give us the number of nodes and edges (links); finally we will click on the gdf file link, as we can see in the following screenshot: The output file myFacebookNet.gdf will look like this: nodedef>name VARCHAR,label VARCHAR,gender VARCHAR,locale VARCHAR,agerankINT23917067,Jorge,male,en_US,10623931909,Haruna,female,en_US,10535702006,Joseph,male,en_US,104503839109,Damian,male,en_US,103532735006,Isaac,male,es_LA,102. . .edgedef>node1 VARCHAR,node2 VARCHAR23917067,3570200623917067,62939583723917067,74734348223917067,75560507523917067,1186286815. . . In the following screenshot we may see the visualization of the graph (106 nodes and 279 links). The nodes represent my friends and the links represent how my friends are connected between them. Transforming GDF to JSON In order to work with the graph in the web with d3.js, we need to transform our gdf file to json format. Firstly, we need to import the libraries numpy and json. import numpy as npimport json The numpy function, genfromtxt, will obtain only the ID and name from the nodes.csv file using the usecols attribute in the 'object' format. nodes = np.genfromtxt("nodes.csv",dtype='object',delimiter=',',skip_header=1,usecols=(0,1)) Then, the numpy function, genfromtxt, will obtain links with the source node and target node from the links.csv file using the usecols attribute in the 'object' format. links = np.genfromtxt("links.csv",dtype='object',delimiter=',',skip_header=1,usecols=(0,1)) The JSON format used in the D3.js Force Layout graph implemented in this article requires transforming the ID (for example, 100001448673085) into a numerical position in the list of nodes. Then, we need to look for each appearance of the ID in the links and replace them by their position in the list of nodes. for n in range(len(nodes)):for ls in range(len(links)):if nodes[n][0] == links[ls][0]:links[ls][0] = nif nodes[n][0] == links[ls][1]:links[ls][1] = n Now, we need to create a dictionary "data" to store the JSON file. data ={} Next, we need to create a list of nodes with the names of the friends in the format as follows: "nodes": [{"name": "X"},{"name": "Y"},. . .] and add it to thedata dictionary.lst = []for x in nodes:d = {}d["name"] = str(x[1]).replace("b'","").replace("'","")lst.append(d)data["nodes"] = lst Now, we need to create a list of links with the source and target in the format as follows: "links": [{"source": 0, "target": 2},{"source": 1, "target":2},. . .] and add it to the data dictionary.lnks = []for ls in links:d = {}d["source"] = ls[0]d["target"] = ls[1]lnks.append(d)data["links"] = lnks Finally, we need to create the file, newJson.json, and write the data dictionary in the file with the function dumps of the json library. with open("newJson.json","w") as f:f.write(json.dumps(data)) The file newJson.json will look as follows: {"nodes": [{"name": "Jorge"},{"name": "Haruna"},{"name": "Joseph"},{"name": "Damian"},{"name": "Isaac"},. . .],"links": [{"source": 0, "target": 2},{"source": 0, "target": 12},{"source": 0, "target": 20},{"source": 0, "target": 23},{"source": 0, "target": 31},. . .]} Graph visualization with D3.js D3.js provides us with the d3.layout.force() function that use the Force Atlas layout algorithm and help us to visualize our graph. First, we need to define the CSS style for the nodes, links, and node labels. <style>.link {fill: none;stroke: #666;stroke-width: 1.5px;}.node circle{fill: steelblue;stroke: #fff;stroke-width: 1.5px;}.node text{pointer-events: none;font: 10px sans-serif;}</style> Then, we need to refer the d3js library. <script src = "http://d3js.org/d3.v3.min.js"></script> Then, we need to define the width and height parameters for the svg container and include into the body tag. var width = 1100,height = 800var svg = d3.select("body").append("svg").attr("width", width).attr("height", height); Now, we define the properties of the force layout such as gravity, distance, and size. var force = d3.layout.force().gravity(.05).distance(150).charge(-100).size([width, height]); Then, we need to acquire the data of the graph using the JSON format. We will configure the parameters for nodes and links. d3.json("newJson.json", function(error, json) {force.nodes(json.nodes).links(json.links).start(); For a complete reference about the d3js Force Layout implementation, visit the link https://github.com/mbostock/d3/wiki/Force-Layout. Then, we define the links as lines from the json data. var link = svg.selectAll(".link").data(json.links).enter().append("line").attr("class", "link");var node = svg.selectAll(".node").data(json.nodes).enter().append("g").attr("class", "node").call(force.drag); Now, we define the node as circles of size 6 and include the labels of each node. node.append("circle").attr("r", 6);node.append("text").attr("dx", 12).attr("dy", ".35em").text(function(d) { return d.name }); Finally, with the function, tick, run step-by-step the force layout simulation. force.on("tick", function(){link.attr("x1", function(d) { return d.source.x; }).attr("y1", function(d) { return d.source.y; }).attr("x2", function(d) { return d.target.x; }).attr("y2", function(d) { return d.target.y; });node.attr("transform", function(d){return "translate(" + d.x + "," + d.y + ")";})});});</script> In the image below we can see the result of the visualization. In order to run the visualization we just need to open a Command Terminal and run the following Python command or any other web server. >>python –m http.server 8000 Then you just need to open a web browser and type the direction http://localhost:8000/ForceGraph.html. In the HTML page we can see our Facebook graph with a gravity effect and we can interactively drag-and-drop the nodes. All the codes and datasets of this article may be found in the author github repository in the link below.https://github.com/hmcuesta/PDA_Book/tree/master/Chapter10 Summary In this article we developed our own social graph visualization tool with D3js, transforming the data obtained from Netvizz with GDF format into JSON. Resources for Article: Further resources on this subject: GNU Octave: Data Analysis Examples [Article] Securing data at the cell level (Intermediate) [Article] Analyzing network forensic data (Become an expert) [Article]

(For more resources related to this topic, see here.) Controller The Leap::Controller class is a liaison between the controller and your code. Whenever you wish to do anything at all with the device you must first go through your controller. From a controller instance we can interact with the device configuration, detected displays, current and past frames, and set up event handling with our listener subclass. Config An instance of the Config class can be obtained from a controller. It provides a key/value interface to modify the operation of the Leap device and driver behavior. Some of the options available are: Robust mode : Somewhat slower frame processing but works better with less light. Low resource mode : Less accurate and responsive tracking, but uses less CPU and USB bandwidth. Tracking priority : Can prioritize either precision of tracking data or the rate at which data is sampled (resulting in approximately 4x data frame-rate boost), or a balance between the two (approximately 2x faster than the precise mode). Flip tracking : Allows you to use the controller with the USB cable coming out of either side. This setting simply flips the positive and negative coordinates on the X-axis. Screen A controller may have one or more calibratedScreens, which are computer displays in the field of view of the controller, which have a known position and dimensions. Given a pointable direction and a screen we can determine what the user is pointing at. Math Several math-related functions and types such as Leap::Vector, Leap::Matrix, and Leap::FloatArray are provided by LeapMath.h. All points in space, screen coordinates, directions, and normal are returned by the API as three-element vectors representing X, Y, and Z coordinates or unit vectors. Frame The real juicy information is stored inside each Frame. A Frame instance represents a point in time in which the driver was able to generate an updated view of its world and detect where screens, your hands, and pointables are. Hand At present the only body parts you can use with the controller are your hands. Given a frame instance we can inspect the number of hands in the frame, their position and rotation, normal vectors, and gestures. The hand motion API allows you to compare two frames and determine if the user has performed a translation, rotation, or scaling gesture with their hands in that time interval. The methods we can call to check for these interactions are: Leap::Hand::translation(sinceFrame): Translation (also known as movement) returned as a Leap::Vector including the direction of the movement of the hand and the distance travelled in millimeters. Leap::Hand::rotationMatrix(sinceFrame), ::rotationAxis(sinceFrame), ::rotationAngle(sinceFrame, axisVector): Hand rotation, either described as a rotation matrix, vector around an axis or float angle around a vector between –π and π radians (that's -180° to 180° for those of you who are a little rusty with your trigonometry). Leap::Hand::scaleFactor(sinceFrame): Scaling represents the distance between two hands. If the hands are closer together in the current frame compared to sinceFrame, the return value will be less than 1.0 but greater than 0.0. If the hands are further apart the return value will be greater than 1.0 to indicate the factor by which the distance has increased. Pointable A Hand also can contain information about Pointable objects that were recognized in the frame as being attached to the hand. A distinction is made between the two different subclasses of pointable objects, Tool, which can be any slender, long object such as a chopstick or a pencil, and Finger, whose meaning should be apparent. You can request either fingers or tools from a Hand, or a list of pointables to get both if you don't care. Finger positioning Suppose we want to know where a user's fingertips are in space. Here's a short snippet of code to output the spatial coordinates of the tips of the fingers on a hand that is being tracked by the controller: if (frame.hands().empty()) return; const Leap::Hand firstHand = frame.hands()[0]; const Leap::FingerList fingers = firstHand.fingers(); Here we obtain a list of the fingers on the first hand of the frame. For an enjoyable diversion let's output the locations of the fingertips on the hand, given in the Leap coordinate system: for (int i = 0; i < fingers.count(); i++) { const Leap::Finger finger = fingers[i]; std::cout << "Detected finger " << i << " at position (" << finger.tipPosition().x << ", " << finger.tipPosition().y << ", " << finger.tipPosition().z << ")" << std::endl; } This demonstrates how to get the position of the fingertips of the first hand that is recognized in the current frame. If you hold three fingers out the following dazzling output is printed: Detected finger 0 at position (-119.867, 213.155, -65.763) Detected finger 1 at position (-90.5347, 208.877, -61.1673) Detected finger 2 at position (-142.919, 211.565, -48.6942) While this is clearly totally awesome, the exact meaning of these numbers may not be immediately apparent. For points in space returned by the SDK the Leap coordinate system is used. Much like our forefathers believed the Earth to be the cornerstone of our solar system, your Leap device has similar notions of centricity. It measures locations by their distance from the Leap origin, a point centered on the top of the device. Negative X values represent a point in space to the left of the device, positive values are to the right. The Z coordinates work in much the same way, with positive values extending towards the user and negative values in the direction of the display. The Y coordinate is the distance from the top of the device, starting 25 millimeters above it and extending to about 600 millimeters (two feet) upwards. Note that the device cannot see below itself, so all Y coordinates will be positive. An example of cursor control By now we are feeling pretty saucy, having diligently run the sample code thus far and controlling our computer in a way never before possible. While there is certain utility and endless amusement afforded by printing out finger coordinates while waving your hands in the air and pretending to be a magician, there are even more exciting applications waiting to be written, so let's continue onwards and upwards. Until computer-gesture interaction is commonplace, pretending to be a magician while you test the functionality of Leap SDK is not recommended in public places such as coffee shops. In some cultures it is considered impolite to point at people. Fortunately your computer doesn't have feelings and won't mind if we use a pointing gesture to move its cursor around (you can even use a customarily offensive finger if you so choose). In order to determine where to move the cursor, we must first locate the position on the display that the user is pointing at. To accomplish this we will make use of the screen calibration and detection API in the SDK. If you happen to leave your controller near a computer monitor it will do its best to try and determine the location and dimensions of the monitor by looking for a large, flat surface in its field of view. In addition you can use the complementary Leap calibration functionality to improve its accuracy if you are willing to take a couple of minutes to point at various dots on your screen. Note that once you have calibrated your screen, you should ensure that the relative positions of the Leap and the screen do not change. Once your controller has oriented itself within your surroundings, hands and display, you can ask your trusty controller instance for a list of detected screens: // get list of detected screens const Leap::ScreenList screens = controller.calibratedScreens(); // make sure we have a detected screen if (screens.empty()) return;const Leap::Screen screen = screens[0]; We now have a screen instance that we can use to find out the physical location in space of the screen as well as its boundaries and resolution. Who cares about all that though, when we can use the SDK to compute where we're pointing to with the intersect() method? // find the first finger or tool const Leap::Frame frame = controller.frame(); const Leap::HandList hands = frame.hands(); if (hands.empty()) return; const Leap::PointableList pointables = hands[0].pointables(); if (pointables.empty()) return; const Leap::Pointable firstPointable = pointables[0]; // get x, y coordinates on the first screen const Leap::Vector intersection = screen.intersect( firstPointable, true, // normalize 1.0f // clampRatio ); The vector intersection contains what we want to know here; the pixel pointed at by our pointable. If the pointable argument to intersect() is not actually pointing at the screen then the return value will be (NaN, NaN, NaN). NaN stands for not a number . We can easily check for the presence of non-finite values in a vector with the isValid() method: if (! intersection.isValid()) return; // print intersection coordinates std::cout << "You are pointing at (" << intersection.x << ", " << intersection.y << ", " << intersection.z << ")" << std::endl; Prepare to be astounded when you point at the middle of your screen and the transfixing message You are pointing at (0.519522, 0.483496, 0) is revealed. Assuming your screen resolution is larger than one pixel on either side, this output may be somewhat unexpected, so let's talk about what screen.intersect(const Pointable &pointable, bool normalize, float clampRatio=1.0f) is returning. The intersect() method draws an imaginary ray from the tip of pointable extending in the same direction as your finger or tool and returns a three-element vector containing the coordinates of the point of intersection between the ray and the screen. If the second parameter normalize is set to false then intersect() will return the location in the leap coordinate system. Since we have no interest in the real world we have set normalize to true, which causes the coordinates of the returned intersection vector to be fractions of the screen width and height. When intersect() returns normalized coordinates, (0, 0, 0) is considered the bottom-left pixel, and (1, 1, 0) is the top-right pixel. It is worth noting that many computer graphics coordinate systems define the top-left pixel as (0, 0) so use caution when using these coordinates with other libraries. There is one last (optional) parameter to the intersect() method, clampRatio, which is used to expand or contract the boundaries of the area at which the user can point, should you want to allow pointing beyond the edges of the screen. Now that we have our normalized screen position, we can easily work out the pixel coordinate in the direction of the user's rude gesticulations: unsigned int x = screen.widthPixels() * intersection.x; // flip y coordinate to standard top-left origin unsigned int y = screen.heightPixels() * (1.0f - intersection.y); std::cout << "You are offending the pixel at (" << x << ", " << y << std::endl; Since intersection.x and intersection.y are fractions of the screen dimensions, simply multiply by the boundary sizes to get our intersection coordinates on the screen. We'll go ahead and leave out the Z-coordinate since it's usually (OK, always) zero. Now for the coup de grace —moving the cursor location, here's how to do it on Mac OS X: CGPoint destPoint = CGPointMake(x, y); CGDisplayMoveCursorToPoint(kCGDirectMainDisplay, de.stPoint); You will need to #include <CoreGraphics/CoreGraphics.h> and link it ( –framework CoreGraphics) to make use of CGDisplayMoveCursorToPoint(). Now all of our hard efforts are rewarded, and we can while away the rest of our days making the cursor zip around with nothing more than a twitch of the finger. At least until our arm gets tired. After a few seconds (or minutes, for the easily-amused) it may become apparent that the utility of such an application is severely limited, as we can't actually click on anything. So maybe you shouldn't throw your mouse away just yet, but read on if you are ready to escape from the shackles of such an antiquated input device. Summary In this article, we learned about the major components of the Leap SDK. We went through the various components of the Leap SDK. Resources for Article: Further resources on this subject: Kinect in Motion – An Overview [Article] Getting started with Kinect for Windows SDK Programming [Article] Getting Started with Kinect [Article]

(For more resources related to this topic, see here.) Creating a short-answer question For this task, we will create a card to host an interface for a short-answer question. This type of question allows the user to input their answers via the keyboard. Evaluating this type of answer can be especially challenging, since there could be several correct answers and users are prone to make spelling mistakes. Engage Thrusters Create a new card and name it SA. Copy the Title label from the Main card and paste it onto the new SA card. This will ensure the title label field has a consistent format and location. Copy the Question label from the TF card and paste it onto the new SA card. Copy the Progress label from the TF card and paste it onto the new SA card. Copy the Submit button from the Sequencing card and paste it onto the new SA card. Drag a text entry field onto the card and make the following modifications: Change the name to answer. Set the size to 362 by 46. Set the location to 237, 185. Change the text size to 14. We are now ready to program our interface. Enter the following code at the card level: on preOpenCard global qNbr, qArray # Section 1 put 1 into qNbr # Section 2 put "" into fld "question" put "" into fld "answer" put "" into fld "progress" # Section 3 put "What farm animal eats shrubs, can be eaten, and are smaller than cows?" into qArray["1"]["question"] put "goat" into qArray["1"]["answer"] -- put "What is used in pencils for writing?" into qArray["2"]["question"] put "lead" into qArray["2"]["answer"] -- put "What programming language are you learning" into qArray["3"] ["question"] put "livecode" into qArray["3"]["answer"] end preOpenCard In section 1 of this code, we reset the question counter (qNbr) variable to 1. Section 2 contains the code to clear the question, answer, and progress fields. Section 3 populates the question/answer array (qArray). As you can see, this is the simplest array we have used. It only contains a question and answer pairing for each row. Our last step for the short answer question interface is to program the Submit button. Here is the code for that button: on mouseUp global qNbr, qArray local tResult # Section 1 if the text of fld "answer" contains qArray[qNbr]["answer"] then put "correct" into tResult else put "incorrect" into tResult end if #Section 2 switch tResult case "correct" if qNbr < 3 then answer "Very Good." with "Next" titled "Correct" else answer "Very Good." with "Okay" titled "Correct" end if break case "incorrect" if qNbr < 3 then answer "The correct answer is: " & qArray[qNbr]["answer"] & "." with "Next" titled "Wrong Answer" else answer "The correct answer is: " & qArray[qNbr]["answer"] & "." with "Okay" titled "Wrong Answer" end if break end switch # Section 3 if qNbr < 3 then add 1 to qNbr nextQuestion else go to card "Main" end if end mouseUp Our Submit button script is divided into three sections. The first section (section 1) checks to see if the answer contained in the array (qArray) is part of the answer the user entered. This is a simple string comparison and is not case sensitive. Section 2 of this button's code contains a switch statement based on the local variable tResult. Here, we provide the user with the actual answer if they do not get it right on their own. The final section (section 3) navigates to the next question or to the main card, depending upon which question set the user is on. Objective Complete - Mini Debriefing We have successfully coded our short answer quiz card. Our approach was to use a simple question and data input design with a Submit button. Your user interface should resemble the following screenshot: Creating a picture question card Using pictures as part of a quiz, poll, or other interface can be fun for the user. It might also be more appropriate than simply using text. Let's create a card that uses pictures as part of a quiz. Engage Thrusters Create a new card and name it Pictures. Copy the Title label from the Main card and paste it onto the new Pictures card. This will ensure the title label field has a consistent format and location. Copy the Question label from the TF card and paste it onto the new Pictures card. Copy the Progress label from the TF card and paste it onto the new Pictures card. Drag a Rectangle Button onto the card and make the following customizations: Change the name to picture1. Set the size to 120 x 120. Set the location to 128, 196. Drag a second Rectangle Button onto the card and make the following customizations: Change the name to picture2. Set the size to 120 x 120. Set the location to 336, 196. Upload the following listed files into your mobile application's Image Library. This LiveCode function is available by selecting the Development pull-down menu, then selecting Image Library. Near the bottom of the Image Library dialog is an Import File button. Once your files are uploaded, take note of the ID numbers assigned by LiveCode: q1a1.png q1a2.png q2a1.png q2a2.png q3a1.png q3a2.png With our interface fully constructed, we are now ready to add LiveCode script to the card. Here is the code you will enter at the card level: on preOpenCard global qNbr, qArray # Section 1 put 1 into qNbr set the icon of btn "picture1" to empty set the icon of btn "picture2" to empty # Section 2 put "" into fld "question" put "" into fld "progress" # Section 3 put "Which puppy is real?" into qArray["1"]["question"] put "2175" into qArray["1"]["pic1"] put "2176" into qArray["1"]["pic2"] put "q1a1" into qArray["1"]["answer"] -- put "Which puppy looks bigger?" into qArray["2"]["question"] put "2177" into qArray["2"]["pic1"] put "2178" into qArray["2"]["pic2"] put "q2a2" into qArray["2"]["answer"] -- put "Which scene is likely to make her owner more upset?" into qArray["3"]["question"] put "2179" into qArray["3"]["pic1"] put "2180" into qArray["3"]["pic2"] put "q3a1" into qArray["3"]["answer"] end preOpenCard In section 1 of this code, we set the qNbr to 1. This is our question counter. We also ensure that there is no image visible in the two buttons. We do this by setting the icon of the buttons to empty. In section 2, we empty the contents of the two onscreen fields (Question and Progress). In the third section, we populate the question set array (qArray). Each question has an answer that corresponds with the filename of the images you added to your stack in the previous step. The ID numbers of the six images you uploaded are also added to the array, so you will need to refer to your notes from step 7. Our next step is to program the picture1 and picture2 buttons. Here is the code for the picture1 button: on mouseUp global qNbr, qArray # Section 1 if qArray[qNbr]["answer"] contains "a1" then if qNbr < 3 then answer "Very Good." with "Next" titled "Correct" else answer "Very Good." with "Okay" titled "Correct" end if else if qNbr < 3 then answer "That is not correct." with "Next" titled "Wrong Answer" else answer "That is not correct." with "Okay" titled "Wrong Answer" end if end if # Section 2 if qNbr < 3 then add 1 to qNbr nextQuestion else go to card "Main" end if end mouseUp In section 1 of our code, we check to see if the answer from the array contains a1, which indicates that the picture on the left is the correct answer. Based on the answer evaluation, one of two text feedbacks is provided to the user. The name of the button on the feedback dialog is either Next or Okay, depending upon which question set the user is currently on. The second section of this code routes the user to either the main card (if they finished all three questions) or to the next question. Copy the code you entered in the picture1 button and paste it onto the picture2 button. Only one piece of code needs to change. On the first line of the section 1 code, change the string from a1 to a2. That line of code should be as follows: if qArray[qNbr]["answer"] contains "a2" then Objective Complete - Mini Debriefing In just 9 easy steps, we created a picture-based question type that uses images we uploaded to our stack's image library and a question set array. Your final interface should look similar to the following screenshot: Adding navigational scripting In this task, we will add scripts to the interface buttons on the Main card. Engage Thrusters Navigate to the Main card. Add the following script to the true-false button: on mouseUp set the disabled of me to true go to card "TF" end mouseUp Add the following script to the m-choice button: on mouseUp set the disabled of me to true go to card "MC" end mouseUp Add the following script to the sequence button: on mouseUp set the disabled of me to true go to card "Sequencing" end mouseUp Add the following script to the short-answer button: on mouseUp set the disabled of me to true go to card "SA" end mouseUp Add the following script to the pictures button: on mouseUp set the disabled of me to true go to card "Pictures" end mouseUp The last step in this task is to program the Reset button. Here is the code for that button: on mouseUp global theScore, totalQuestions, totalCorrect # Section 1 set the disabled of btn "true-false" to false set the disabled of btn "m-choice" to false set the disabled of btn "sequence" to false set the disabled of btn "short-answer" to false set the disabled of btn "pictures" to false # Section 2 set the backgroundColor of grc "progress1" to empty set the backgroundColor of grc "progress2" to empty set the backgroundColor of grc "progress3" to empty set the backgroundColor of grc "progress4" to empty set the backgroundColor of grc "progress5" to empty # Section3 put 0 into theScore put 0 into totalQuestions put 0 into totalCorrect put theScore & "%" into fld "Score" end mouseUp There are three sections to this code. In section 1, we are enabling each of the buttons. In the second section, we are clearing out the background color of each of the five progress circles in the bottom-center of the screen. In the final section, section 3, we reset the score and the score display. Objective Complete - Mini Debriefing That is all there was to this task, seven easy steps. There are no visible changes to the mobile application's interface. Summary In this article, we saw how to create a couple of quiz apps for mobile such as short-answer questions and picture card questions. Resources for Article: Further resources on this subject: Creating and configuring a basic mobile application [Article] Creating mobile friendly themes [Article] So, what is XenMobile? [Article]

(For more resources related to this topic, see here.) What is Slick2D? Slick2D is a multi-platform library for two dimensional game development that sits upon the LWJGL(Light-Weight Java Game Library). Slick2D simplifies the processes of game development such as game loop, rendering, updating, frame setup, and state-based game creation. It also offers some features that LWJGL does not, such as particle emitters and integration with Tiled (a map editor). Developers of all skill levels can enjoy Slick2D, as it offers a degree of simplicity that you can't find in most libraries. This simplicity not only makes it a great library for programmers but artists as well, who may not have the technical knowledge to create games in other libraries. Downloading the Slick2D and LWJGL files The Slick2D and LWJGL jar files, plus the LWJGL native files, are needed to create a Slick2D game project. The only system requirement for Slick2D is a Java JDK. To get the files, we perform the following steps: Obtaining the LWJGL files: Navigate to http://www.lwjgl.org/download.php. Download the most recent stable build. The .zip file will include both the LWJGL jar file and the native files. (This .zip file will be referenced as lwjgl.zip file.) Obtaining the Slick2D files: Due to hosting issues, the Slick2D files are being hosted by a community member at http://slick.ninjacave.com. If this site is not available, follow the alternative instructions at step 3. Click on Download. Alternative method of obtaining the Slick2D files: Navigate to https://bitbucket.org/kevglass/slick. Download the source. Build the ant script located at slick/trunk/Slick/build.xml Build it in eclipse or command line using $ ant. Setting up an eclipse project We will utilize the Eclipse IDE that can be found at http://www.eclipse.org/ when working with Slick2D in this article. You may, however, utilize other options. Perform the following these steps to set up a Slick2D project: Navigate to File | New | Java Project. Name your project and click on Finish. Create a new folder in your project and name it lib. Add two subfolders named jars and native. Place both lwjgl.jar and slick.jar in the jars subfolder inside our eclipse project. Take all the native files from lwjgl.zip and place them in the native subfolder. Copy the contents of the subfolders inside native from lwjgl.zip not the subfolders themselves. Right-click on project then click on Properties. Click on Java Build Path and navigate to the Libraries tab. Add both the jars from the project. Select and expand lwjgl.jar from the Libraries tab and click on Native library location: (None) then click on Edit and search the workspace for the native's folder. Native files The native files included in lwjgl.zip are platform-specific libraries that allow the developers to make one game that will work on all of the different platforms. What if I want my game to be platform-specific? No real benefit exists to being platform-specific with Slick2D. In the foregoing tutorial, we will establish a game as a multi-platform game. However, if you want your game to be platform-specific, you can make it platform-specific. In the previous tutorial (step 6) we took the content of each operating system's folder and put that content into our native folder. If, instead, you desire to make your game platform-specific, then instead of copying the contents of these folders, you would copy the entire folder as illustrated as follows: When defining the natives for LWJGL (step 10 in previous example), simply point towards the operating system of your choice. Summary In this article we learned tons of important things necessary to create a project in Slick2D. So far we covered: Downloading the necessary library files Setting up a project (platform-specific or multi-platform) Native files Resources for Article: Further resources on this subject: HTML5 Games Development: Using Local Storage to Store Game Data [Article] Adding Sound, Music, and Video in 3D Game Development with Microsoft Silverlight 3: Part 2 [Article] Adding Finesse to Your Game [Article]

(For more resources related to this topic, see here.) Planning the audio In Camtasia Studio, we can stack multiple audio tracks on top of each other. While this is a useful and powerful way to build a soundtrack, it can lead to a cluttered audio output if we do not plan ahead. Audio tracks can be used for a wide range of purposes. It's best to storyboard audio to avoid creating a confusing audio mix. If we consider how each audio track will be used before we begin to overlay each file on the timeline, we can visualize the end result and resist the temptation to layer too many audio effects on top of each other. The importance of consistency Producing professional video in Camtasia Studio comes down to consistency and detail. The more consistent we are, the more professional the result will be. The more we pay attention to detail, the more professional the result is. By being consistent in our use of audio effects, we can avoid creating unintentional distractions or misleading the viewer. For example, if we choose to use a ping sound to represent a mouse click, we should make sure that all mouse clicks use the same ping sound so that the viewer understands and associates the sound with the action. A note on background music When deciding what audio we want in our video, we should always think about our target audience and the type of message we are trying to deliver. Never use background music unless it adds to the video content. For example, background music can be a useful way of engaging our viewer, but if we are delivering an important health and safety message, or delivering a quiz, a backing track may be distracting. If our audience are the staff in customer-facing departments, we may not want to include audio tracks at all. We wouldn't want the sound from our videos to be audible to a customer. Types of audio There are three main types of audio we can add to our video: Voice-over tracks Background music Sound effects Preparing to record a voice-over Various factors affect the quality and consistency of voice-over recordings. In Camtasia Studio, we can add effects but it's best to get the source audio right in the first instance. The factors are given as follows: We often don't pay attention to the qualities and tones in our own voices, but they can and do change. From day to day, your tone of voice can subtly change. Air temperature, illness, or mood can affect the way your voice sounds in a recording. In addition, the environment we use to record a voice-over can have a dramatic effect on the end result. Some rooms will give your voice natural reverb; others will sound very dead. The equipment we use will affect the recording. For example, different microphones will produce different results. When we prepare for a voice-over recording, we must aim to keep our voice, environment, and equipment as stable and consistent as possible. That means we should aim to record the voice-over in one session so that we can control all these factors. We may choose a different person to provide the voice-over. Again, we should take a consistent approach in how we use their voice. Voice-over recording is always a long process and involves trial, error, and multiple takes. We should allow more time than we feel is strictly necessary. Many recordings inevitably overrun. If any sections of the recording are questionable, we should aim to record all of the alternatives in the same session for a seamless result. The studio environment Most Camtasia Studio users do not have access to a professional recording studio. This need not be a problem. We can use practically any quiet room to record our voice-over, although there are some basic pointers that will improve the result. When choosing a studio location, consider the following: Ambient noise: Try to record in quiet environment. If we can use an empty room where there are no passers by or devices making any noise, this will make our recording clearer. Choose a room away from potential sources of noise (busy corridors, main roads, and so on). Noise leakage: Ensure that any doors and windows are closed to minimize noise pollution from outside the room and outside the building. Equipment noise: Ensure that all unnecessary programs on the PC are closed to prevent any unwanted sounds or alerts. End any background tasks, such as email checkers or task schedulers, and ensure any instant messaging software is closed or in offline mode. Positioning: Experiment with placing the microphone in different places around the room. The acoustics of a room can greatly affect the quality of a recording and taking time to find the best place for the microphone will help. For efficiency, we can test the audio quality quickly by wearing headphones while speaking into the microphone. Consider posture: Standing up opens up the diaphragm and improves the sound of our voice when we record. Avoid recording while seated, and hold any notes or papers at eye level to maintain a constant tone. Using scripts When it comes to voice-over recording, a well-prepared script is the most important piece of preparation we can do. Working from a script is far simpler than attempting to make up our narration as we go along. It helps to maintain a good pace in the video and greatly reduces the need for multiple takes, making recording far more efficient. Creating a script need not be time-consuming. If we have already planned out and recorded our video track, writing a script will be far simpler. Writing an effective script The script you write should support the action in the video and maintain a healthy pace. There are a number of tips we can bear in mind to do this. These tips are given as follows: Sync audio with video: Plan the script to coincide with any actions we take in the video. This may mean incorporating pauses into the script to allow a certain on-screen action to complete. Be flexible: We may need to go back and lengthen a section of video to incorporate the voice-over and explanation. It is better to do this than rush the voice-over and attempt to force it to fit. Use basic copywriting techniques: We should consider the message in the video and use the appropriate style. For example, if we are describing a process, we would want to use the active voice. In an internal company update, we may want to adopt a more conversational tone. Be direct and concise: A short and simple statement is far easier to process than a long, drawn out argument. We should always test our script prior to the recording session. We should also be prepared to re-write and hone the content. Reading a script aloud is a useful way of estimating its length and picking out any awkward phrases that do not flow. We will save time if we perfect the script before we sit down in front of the microphone. Recording equipment Most laptop computers have a built in microphone, as do some desktop computers. While these microphones are perfectly adequate for video or audio chats and other casual uses, we should not use them to create Camtasia Studio recordings. Although the quality may be good, and the audio may be clear, these microphones often pick up a large amount of ambient noise, such as the fans inside the computer. Additionally, the audio captured using built-in microphones often require processing and amplification, which can degrade its quality. Camtasia Studio has a range of editing tools that can help you to tweak your audio recording. However, processing should always be a last resort. The more we use a tool to process our voice-over, the more the source material is prone to being distorted. If we have better quality source material, we will not need to rely on these features; this will make the editing process much simpler. When working in Camtasia Studio, it is preferable to invest in a good quality external microphone. Basic microphones are inexpensive and offer considerably better audio recording than built-in microphones. Choosing a microphone External microphones are very affordable. Unless you have specific need for a professional-standard microphone, we recommend a USB microphone. Many of these microphones are sold as podcasting microphones and are perfectly adequate for use in Camtasia Studio. There are two main types of external microphone: Consider a lapel microphone if you plan to operate the computer as you record or present to the camera while you are speaking. Lapel microphones clip on to your clothing and leave your hands free. If you are more comfortable working at a desk, a microphone with a sturdy tripod stand will be a good investment. An external microphone with built in noise cancellation can give us a degree of control at the recording stage, rather than having to edit out noise later. A good stand will give us a greater degree of flexibility when it comes to microphone placement. How to set up an external microphone We can set up the external microphone before we begin recording by following the given steps: Navigate to Tools | Voice Narration. The Voice Narration screen is displayed. Click on Audio setup wizard.... The Audio Setup Wizard screen is displayed. Select the Audio device, as shown in the following screenshot. Summary In this article, we have looked at a range of ways to improve the quality of the audio in our Camtasia Studio projects. We have considered voice-over recording techniques, equipment, editing, sound effects, and background music. Resources for Article: Further resources on this subject: Editing attributes [Article] Basic Editing [Article] Video Editing in Blender using Video Sequence Editor: Part 1 [Article]

(For more resources related to this topic, see here.) Metrics to verify before CPSM deployment Until now, you have learned about the most obvious requirements to install CPSM; now, in the upcoming session, we will have a look at how to verify essentials for CPSM deployment. Verification of environment prerequisites We will look at the core components that should essentially be verified right at the outset before the installation. The first component that needs to be verified is the Active Directory (AD) schema, which is necessary to accommodate Citrix CloudPortal Services Manager. As you are aware, the operation can be performed using the Microsoft Exchange installation tools. The following steps need to be performed: Open the command prompt on your planned Exchange server. Then execute the following command: setup /p /on:OrganizationName The second component that needs to be cross-checked is whether DNS aliases have been configured. Citrix CloudPortal Services Manager uses DNS aliases to discover the servers where the platform modules will be positioned. For this, the following steps need to be performed: On AD, create CNAME records. There should be one record against each of your servers as shown in the following table: Server EX Name Database server CORTEXSQL Provisioning server CORTEXPROVISIONING Web server CORTEXWEB Reporting Services CORTEXREPORTS Use the Citrix CloudPortal Services Manager Setup utility to verify the preceding items. The utility probes our settings and if it is positive, displays a green check mark next to each confirmed item. If it is negative, the Setup utility shows a Validate button, so you can execute the checks over again. Perform the following steps: From your file cluster or from the installation media, execute Setup.exe. On the CloudPortal Services Manager splash screen, click on Get Started. On the Choice Deployment Task screen, choose Install CloudPortal Services Manager. In the CloudPortal Services Manager screen, choose check environment prerequisites. The Prepare Environment screen displays the status of the verified items. As the next step, we will now create the system database. The heart of the deployment is the Config.xml file, which will be useful throughout the wizard run-through. How to deploy SQL Server and Reporting Services For Cloud IT providers, it is recommended that they use the SQL Server deployment and Reporting Services. This should be done in a dedicated cluster for high availability, especially when providing for multiple consumers. With regards to installation, configuration, and performance tuning of SQL Server and Reporting Services, please refer to http://technet.microsoft.com/en-us/library/ms143219(v=sql.105).aspx. The next step is to create the DB. We have to perform this activity post deployment of SQL Server and SQL Server Reporting Services. The system databases are created using the Services Manager Configuration Tool, which is installed as a part of this process. Perform the following steps: From the source location where the installation media is located, execute the Setup.exe file. On the CloudPortal Services Manager splash screen, click on Get Started. On the Choose Deployment Task screen, choose Install CloudPortal Services Manager. On the Install CloudPortal Services Manager screen, choose Deploy Server Roles & Primary Location. On the Deploy Server Roles & Primary Location screen, choose Create System Databases. Now let us install the Citrix CloudPortal Services Manager Configuration Tool: When prompted, click on Install to deploy the Configuration utility. On the License Agreement screen, read and accept the license agreement and then select commit next. On the Ready to install screen, click on Install. The setup utility installs the Configuration Tool and the prerequisites that are required as well. Now, let us click on Finish to continue creating the system databases. The next step of the installation is to create a Configuration File screen. Browse to the directory where you want to store the Config.xml file and provide a filename. Then click on Next. Now, let us go to the Create Primary Databases screen and configure the following information about the SQL Server that will store system configuration information: Server address: This is used to specify the DB server using the DNS alias, IP address, or the FQDN. Server port: This is used to declare the port number used by SQL Server. The port for a default instance of SQL Server is 1433. Authentication mode: This is used to choose whether to apply Integrated Windows and SQL or SQL authentication. By default, Integrated is chosen. (Mixed Mode is recommended to be used). Connect as: This is used to declare Consumer name and password of the SQL administrator Consumer (Super account). Fields are accessible when we choose the SQL authentication mode for our installation. Auto-create SQL logins: This checkbox is available only if we want the required SQL Server Consumer accounts to be created automatically. If you do not choose this checkbox, we can later provide the login details manually on the Configure Database Logins screen. Run through the Test Connection to make sure the Configuration utility can make contact with the SQL Server and then click on Next. On the Configure Database Logins screen, proceed with Generate IDs chosen if you want passwords created automatically for CortexProp, OLMReports, and OLM DB accounts. Clear this choice if you want to provide the passwords for these accounts. CortexProp, OLM DB, and OLMReports accounts are formed to make sure the cross-domain right of entry is available to the server DBs. On the Summary screen, assess the DB configuration in sequence. If you want to change anything, click on Back to return to the suitable configuration screen. Upon completion of the entire configuration as per the guideline, go ahead and click on Commit. The Applying Configuration screen displays the progress. After the server DBs are effectively created, click on Finish. After the system databases are created, you can install Provisioning Directory Web Service and the web platform server roles on the other servers. Installation of the CPSM role using GUI By now you would have crystal clear understanding of the system requirements for a CPSM installation. In order to start the installation using GUI, we need to perform the following activity on the server you will be using to host each server role you planned: Deploy and configure the Reporting server role after the primary location has been configured. If you deploy Reporting Services before the primary location has been configured, configuration of Reporting Services fails. From the source location where the installation media is located, execute the Setup.exe file. On the Setup Tool splash screen, click on Get Started. On the Choose Deployment Task screen, choose Install CloudPortal Services Manager and click on Next. Now on the Install CloudPortal Services Manager screen, choose Deploy Server Roles & Primary Location and click on Next. Now on the Deploy Server Roles & Primary Location screen, choose Install Server Roles and click on Next. Now on the License Agreement screen, agree to the license agreement and then click on Next. On the Choose Server Roles screen, choose the roles to install and then click on Next. On the Review Prerequisites screen, evaluate the prerequisite objects that will be deployed and then click on Next. On the Ready to install screen, evaluate the chosen roles and prerequisites that will be deployed. Click on Install. The Deploying Server Roles screen shows the installation of the prerequisites and the chosen roles, and the outcome. On the Deployment Complete screen, click on Finish. Summary This article serves as a brief reference for readers to understand about the system, to verify the essentials, and install and configure CPSM using GUI and CLI. Resources for Article: Further resources on this subject: Content Switching using Citrix Security [Article] Managing Citrix Policies [Article] Getting Started with the Citrix Access Gateway Product Family [Article]

(For more resources related to this topic, see here.) Showing and hiding elements of the UI We'll start easy with a simple function that you are certainly going to need if you build even a moderately complex application. Those of you who have been doing extra credit exercises and/or experimenting with your own applications will probably have already wished for this or, indeed, have already found it. conditionalPanel() allows you to show/hide UI elements based on other selections within the UI. The function takes a condition (in JavaScript, but the form and syntax will be familiar from many languages) and a UI element, and displays the UI only when the condition is true. This is actually used a couple of times in the advanced GA application and indeed in all the applications I've ever written of even moderate complexity. The following is a simpler example (from ui.R, of course, in the first section, within sidebarPanel()), which allows users who request a smoothing line to decide what type they want: conditionalPanel(condition = "input.smoother == true",selectInput("linearModel", "Linear or smoothed",list("lm", "loess"))) As you can see, the condition appears very R/Shiny-like, except with the "." operator familiar to JavaScript users in place of "$", and with "true" in lower case. This is a very simple but powerful way of making sure that your UI is not cluttered with irrelevant material. Giving names to tabPanel elements In order to further streamline the UI, we're going to hide the hour selector when the monthly graph is displayed and the date selector when the hourly graph is displayed. The difference is illustrated in the following screenshot with side-by-side pictures, hourly figures UI on the left-hand side and monthly figures on the right-hand side: In order to do this, we're going to have to first give the tabs of the tabbed output names. This is done as follows (with the new code in bold): tabsetPanel(id ="theTabs",tabPanel("Summary", textOutput("textDisplay"),value = "summary"),tabPanel("Monthly figures",plotOutput("monthGraph"), value = "monthly"),tabPanel("Hourly figures",plotOutput("hourGraph"), value = "hourly")) As you can see, the whole panel is given an ID (theTabs), and then each tabPanel is also given a name (summary, monthly, and hourly). They are referred to in the server.R file very simply as input$theTabs. Let's have a quick look at a chunk of code in server.R that references the tab names; this code makes sure that we subset based on date only when the date selector is actually visible, and by hour only when the hour selector is actually visible. Our function to calculate and pass data now looks like the following (new code again bolded): passData <- reactive({if(input$theTabs != "hourly"){analytics <- analytics[analytics$Date %in%seq.Date(input$dateRange[1], input$dateRange[2],by = "days"),]}if(input$theTabs != "monthly"){analytics <- analytics[analytics$Hour %in%as.numeric(input$minimumTime) :as.numeric(input$maximumTime),]}analytics <- analytics[analytics$Domain %in%unlist(input$domainShow),]analytics}) As you can see, subsetting by month is carried out only when the date display is visible (that is, when the hourly tab is not shown), and vice versa. Finally, we can make our changes to ui.R to remove parts of the UI based on tab selection: conditionalPanel(condition = "input.theTabs != 'hourly'",dateRangeInput(inputId = "dateRange",label = "Date range",start = "2013-04-01",max = Sys.Date())),conditionalPanel(condition = "input.theTabs != 'monthly'",sliderInput(inputId = "minimumTime",label = "Hours of interest- minimum",min = 0,max = 23,value = 0,step = 1),sliderInput(inputId = "maximumTime",label = "Hours of interest- maximum",min = 0,max = 23,value = 23,step = 1)) Note the use in the latter example of two UI elements within the same conditionalPanel() call; it is worth noting that it helps you keep your code clean and easy to debug. Reactive user interfaces Another trick you will definitely want up your sleeve at some point is a reactive user interface. This enables you to change your UI (for example, the number or content of radio buttons) based on reactive functions. For example, consider an application that I wrote related to survey responses across a broad range of health services in different areas. The services are related to each other in quite a complex hierarchy, and over time, different areas and services respond (or cease to exist, or merge, or change their name...), which means that for each time period the user might be interested in, there would be a totally different set of areas and services. The only sensible solution to this problem is to have the user tell you which area and date range they are interested in and then give them back the correct list of services that have survey responses within that area and date range. The example we're going to look at is a little simpler than this, just to keep from getting bogged down in too much detail, but the principle is exactly the same and you should not find this idea too difficult to adapt to your own UI. We are going to imagine that your users are interested in the individual domains from which people are accessing the site, rather than just have them lumped together as the NHS domain and all others. To this end, we will have a combo box with each individual domain listed. This combo box is likely to contain a very high number of domains across the whole time range, so we will let users constrain the data by date and only have the domains that feature in that range return. Not the most realistic example, but it will illustrate the principle for our purposes. Reactive user interface example – server.R The big difference is that instead of writing your UI definition in your ui.R file, you place it in server.R, and wrap it in renderUI(). Then all you do is point to it from your ui.R file. Let's have a look at the relevant bit of the server.R file: output$reacDomains <- renderUI({domainList = unique(as.character(passData()$networkDomain))selectInput("subDomains", "Choose subdomain", domainList)}) The first line takes the reactive dataset that contains only the data between the dates selected by the user and gives all the unique values of domains within it. The second line is a widget type we have not used yet which generates a combo box. The usual id and label arguments are given, followed by the values that the combo box can take. This is taken from the variable defined in the first line. Reactive user interface example – ui.R The ui.R file merely needs to point to the reactive definition as shown in the following line of code (just add it in to the list of widgets within sidebarPanel()): uiOutput("reacDomains") You can now point to the value of the widget in the usual way, as input$subDomains. Note that you do not use the name as defined in the call to renderUI(), that is, reacDomains, but rather the name as defined within it, that is, subDomains. Summary It's a relatively small but powerful toolbox with which you can build a vast array of useful and intuitive applications with comparatively little effort. This article looked at fine-tuning the UI using conditionalPanel() and observe(), and changing our UI reactively. Resources for Article: Further resources on this subject: Fine Tune the View layer of your Fusion Web Application [Article] Building tiny Web-applications in Ruby using Sinatra [Article] Spring Roo 1.1: Working with Roo-generated Web Applications [Article]

(For more resources related to this topic, see here.) What all developers want is to do their job as fast as they can without sacrificing the quality of their work. IntelliJ has a large range of features that will reduce the time spent in development. But, to achieve the best performance that IntelliJ can offer, it is important that you understand the IDE and adapt some of your habits. In this article, we will navigate through the features that can help you do your job even faster. You will understand IntelliJ's main elements and how they work, and beyond this, learn how IntelliJ can help you organize your activities and the files you are working on. To further harness IntelliJ's abilities, you will also learn how to manage plugins and see a short list of plugins that can help you. Identifying and understanding window elements Before we start showing you techniques you can use to improve your performance using IntelliJ, you need to identify and understand the visual elements present in the main window of the IDE. Knowing these elements will help you find what you want faster. The following screenshot shows the IntelliJ main window: The main window can be divided into seven parts as shown in the previous screenshot: The main menu contains options that you can use to do tasks such as creating projects, refactoring, managing files in version control, and more. The main toolbar element contains some essential options. Some buttons are shown or hidden depending on the configuration of the project; version control buttons are an example of this. The Navigation Bar is sometimes a quick and good alternative to navigate easily and fast through the project files. Tool tabs are shown on both sides of the screen and at the bottom of IntelliJ. They represent the tools that are available for the project. Some tabs are available only when facets are enabled in the project (e.g. the Persistence tab). When the developer clicks on a tool tab, a window appears. These windows will present the project in different perspectives. The options available in each tool window will provide the developer with a wide range of development tasks. The editor is where you can write your code. The Status Bar indicates the current IDE state and provides some options to manipulate the environment. For example, you can hide the tool tabs by clicking on the icon at the bottom-left of the window. In almost all elements, there are context menus available. These menus will provide extra options that may complement and ease your work. For example, the context menu, available in the tool bar, provides an option to hide itself and another to customize the menu and toolbars. You will notice that some tool tabs have numbers. These numbers are used in conjunction with the Alt key to access the tool window you want quickly, Alt + 1, for example, will open the Project tool window. Each tool window will have different options; some will present search facilities, others will show specific options. They use a common structure: a title bar, a toolbar, and the content pane. Some tool windows don't have a toolbar and, in others, the options in the title bar may vary. However, all of them will have at least two buttons in the rightmost part of the title bar: a gear and a small bar with an arrow. The first button is used to configure some properties of the tool and the second will just minimize the window. The following screenshot shows some options in the Database tool: The options available under the gear button icon generally differ from tool to tool. However, in the drop-down list, you will find four common options: Pinned, Docked, Floating, and Split modes. As you may have already imagined, these options are used to define how the tool window will be shown. The Pinned mode is very useful when it is unmarked; using this, when you focus on code editor you don't lose time minimizing the tool window. Identifying and understanding code editor elements The editor provides some elements that can facilitate navigation through the code and help identify problems in it. In the following screenshot, you can see how the editor is divided: The editor area, as you probably know, is where you edit your source code. The gutter area is where different types of information about the code is shown, simply using icons or special marks like breakpoints and ranges. The indicators used here aren't used to just display information; you can perform some actions depending on the indicator, such as reverting changes or navigating through the code. The smart completion popup, as you've already seen, provides assistance to the developer in accordance with the current context. The document tabs area is where the tabs of each opened document are available. The type of document is identified by an icon and the color in the name of the file shows its status in version control: blue stands for "modified", green for "new", red for "not in VCS", and black for "not changed". This component has a context menu that provides some other facilities as well. The marker bar is positioned to the right-hand side of the IDE and its goal is to show the current status of the code. At the top, the square mark can be green for when your code is OK, yellow for warnings that are not critical, and red for compilation errors, respectively. Below the square situated on top of the IDE this element can have other colored marks used to help the developer go directly to the desired part of the code. Sometimes, while you are coding, you may notice a small icon floating near the cursor; this icon represents that there are some intentions available that could help you: indicates that IntelliJ proposes a code modification that isn't totally necessary. It covers warning corrections to code improvement. indicates an intention action that can be used but doesn't provide any improvement or code correction. indicates there is a quick fix available to correct an eminent code error. indicates that the alert for the intention is disabled but the intention is still available. The following figure shows the working intention: Intention actions can be grouped in four categories listed as follows: Create from usage is the kind of intention action that proposes the creation of code depending on the context. For example, if you enter a method name that doesn't exist, this intention will recognize it and propose the creation of the method. Quick fixes is the type of intention that responds to code mistakes, such as wrong type usage or missing resources. Micro refactoring is the kind of intention that is shown when the code is syntactically correct; however, it could be improved (for readability for example). Fragment action is the type of intention used when there are string literals of an injected language; this type of injection can be used to permit you to edit the corresponding sentence in another editor. Intention actions can be enabled or disabled on-the-fly or in the Intention section in the configuration dialog; by default, all intentions come activated. Adding intentions is possible only after installing plugins for them or creating your own plugin. If you prefer, you can use the Alt + Enter shortcut to invoke the intentions popup. Summary As you have seen in this article, IntelliJ provides a wide range of functionalities that will improve your development speed. More important than knowing all the shortcuts IntelliJ offers, is to understand what is possible do with them and when to use a feature. Resources for Article: Further resources on this subject: NetBeans IDE 7: Building an EJB Application [Article] JBI Binding Components in NetBeans IDE 6 [Article] Smart Processes Using Rules [Article]