How-To Tutorials

(For more resources on PHP, see here.) The problem Dealing with sessions can be confusing, and is also a source of security loopholes. So we want our CMS framework to provide basic mechanisms that are robust. We want them to be easy to use by more application-oriented software. To achieve these aims, we need to consider: The need for sessions and their working The pitfalls that can introduce vulnerabilities Efficiency and scalability considerations Discussion and considerations To see what is required for our session handling, we shall first review the need for them and consider how they work in a PHP environment. Then the vulnerabilities that can arise through session handling will be considered. Web crawlers for search engines and more nefarious activities can place a heavy and unnecessary load on session handling, so we shall look at ways to avoid this load. Finally, the question of how best to store session data is studied. Why sessions? The need for continuity was mentioned when we first discussed users. But it is worth reviewing the requirement in a little more detail. If Tim Berners-Lee and his colleagues had known all the developments that would eventually occur in the internet world, maybe the Web would have been designed differently. In particular, the basic web transport protocol HTTP might not have treated each request in isolation. But that is hindsight, and the Web was originally designed to present information in a computer-independent way. Simple password schemes were sufficient to control access to specific pages. Nowadays, we need to cater for complex user management, or to handle things like shopping carts, and for these we need continuity. Many people have recognized this, and introduced the idea of sessions. The basic idea is that a session is a series of requests from an individual website visitor, and the session provides access to enduring information that is available throughout the session. The shopping cart is an obvious example of information being retained across the requests that make up a session. PHP has its own implementation of sessions, and there is no point reinventing the wheel, so PHP sessions are the obvious tool for us to use to provide continuity. How sessions work There are three main choices which have been available for handling continuity: Adding extra information to the URI Using cookies Using hidden fields in the form sent to the browser All of them can be used at times. Which of them is most suitable for handling sessions? PHP uses either of the first two alternatives. Web software often makes use of hidden variables, but they do not offer a neat way to provide an unobtrusive general mechanism for maintaining continuity. In fact, whenever hidden variables are used, it is worth considering whether session data would be a better alternative. For reasons discussed in detail later, we shall consider only the use of cookies, and reject the URI alternative. There was a time when there were lots of scary stories about cookies, and people were inclined to block them. While there will always be security issues associated with web browsing, the situation has changed, and the majority of sites now rely on cookies. It is generally considered acceptable for a site to demand the use of cookies for operations such as user login or for shopping carts and purchase checkout. The PHP cookie-based session mechanism can seem obscure, so it is worth explaining how it works. First we need to review the working of cookies. A cookie is simply a named piece of data, usually limited to around 4,000 bytes, which is stored by the browser in order to help the web server to retain information about a user. More strictly, the connection is with the browser, not the user. Any cookie is tied to a specific website, and optionally to a particular part of the website, indicated by a path. It also has a life time that can be specified explicitly as a duration; a zero duration means that the cookie will be kept for as long as the browser is kept open, and then discarded. The browser does nothing with cookies, except to save and then return them to the server along with requests. Every cookie that relates to the particular website will be sent if either the cookie is for the site as a whole, or the optional path matches the path to which the request is being sent. So cookies are entirely the responsibility of the server, but the browser helps by storing and returning them. Note that, since the cookies are only ever sent back to the site that originated them, there are constraints on access to information about other sites that were visited using the same browser. In a PHP program, cookies can be written by calling the set_cookie function, or implicitly through session handling. The name of the cookie is a string, and the value to be stored is also a string, although the serialize function can be used to make more structured data into a string for storage as a cookie. Take care to keep the cookies within the size limit. PHP makes available the cookies that have been sent back by the browser in the $_COOKIES super-global, keyed by their names. Apart from any cookies explicitly written by code, PHP may also write a session cookie. It will do so either as a result of calls to session handling functions, or because the system has been configured to automatically start or resume a session for each request. By default, session cookies do not use the option of setting an expiry time, but can be deleted when the browser is closed down. Commonly, browsers keep this type of cookie in memory so that they are automatically lost on shutdown. Before looking at what PHP is doing with the session cookie, let's note that there is an important general consideration for writing cookies. In the construction of messages between the server and the browser, cookies are part of the header. That means rules about headers must be obeyed. Headers must be sent before anything else, and once anything else has been sent, it is not permitted to send more headers. So, in the case of server to browser communication, the moment any part of the XHTML has been written by the PHP program, it is too late to send a header, and therefore too late to write a cookie. For this reason, a PHP session is best started early in the processing. The only purpose PHP has in writing a session cookie is to allocate a unique key to the session, and retrieve it again on the next request. So the session cookie is given an identifying name, and its value is the session's unique key. The session key is usually called the session ID, and is used by PHP to pick out the correct set of persistent values that belong to the session. By default, the session name is PHPSESSID but it can, in most circumstances, be changed by calling the PHP function session_name prior to starting the session. Starting, or more often restarting, a session is done by calling session_start, which returns the session ID. In a simple situation, you do not need the session ID, as PHP places any existing session data in another superglobal, $_SESSION. In fact, we will have a use for the session ID as you will soon see. The $_SESSION super-global is available once session_start has been called, and the PHP program can store whatever data it chooses in it. It is an array, initially empty, and naturally the subscripts need to be chosen carefully in a complex system to avoid any clashes. The neat part of the PHP session is that provided it is restarted each time with session_start, the $_SESSION superglobal will retain any values assigned during the handling of previous requests. The data is thus preserved until the program decides to remove it. The only exception to this would be if the session expired, but in a default configuration, sessions do not expire automatically. Later in this article, we will look at ways to deliberately kill sessions after a determinate period of inactivity. As it is only the session ID that is stored in the cookie, rules about the timing of output do not apply to $_SESSION, which can be read or written at any time after session_start has been called. PHP stores the contents of $_SESSION at the end of processing or on request using the PHP function session_write_close. By default, PHP puts the data in a temporary file whose name includes the session ID. Whenever the session data is stored, PHP retrieves it again at the next session_start. Session data does not have to be stored in temporary files, and PHP permits the program to provide its own handling routines. We will look at a scheme for storing the session data in a database later in the article. Avoiding session vulnerabilities So far, the option to pass the session ID as part of the URI instead of as a cookie has not been considered. Looking at security will show why. The main security issue with sessions is that a cracker may find out the session ID for a user, and then hijack that user's session. Session handling should do its best to guard against that happening. PHP can pass the session ID as part of the URI. This makes it especially vulnerable to disclosure, since URIs can be stored in all kinds of places that may not be as inaccessible as we would like. As a result, secure systems avoid the URI option. It is also undesirable to find links appearing in search engines that include a session ID as part of the URI. These two points are enough to rule out the URI option for passing session ID. It can be prevented by the following PHP calls: ini_set('session.use_cookies', 1);ini_set('session.use_only_cookies', 1); These calls force PHP to use cookies for session handling, an option that is now considered acceptable. The extent to which the site will function without cookies depends on what a visitor can do with no continuity of data—user login will not stick, and anything like a shopping cart will not be remembered. It is best to avoid the default name of PHPSESSID for the session cookie, since that is something that a cracker could look for in the network traffic. One step that can be taken is to create a session name that is the MD5 hash of various items of internal information. This makes it harder but not impossible to sniff messages to find out a session ID, since it is no longer obvious what to seek—the well known name of PHPSESSID is not used. It is important for the session ID to be unpredictable, but we rely on PHP to achieve that. It is also desirable that the ID be long, since otherwise it might be possible for an attacker to try out all possible values within the life of a session. PHP uses 32 hexadecimal digits, which is a reasonable defense for most purposes. The other main vulnerability apart from session hijacking is called session fixation. This is typically implemented by a cracker setting up a link that takes the user to your site with a session already established, and known to the cracker. An important security step that is employed by robust systems is to change the session ID at significant points. So, although a session may be created as soon as a visitor arrives at the site, the session ID is changed at login. This technique is used by Amazon among others so that people can browse for items and build up a shopping cart, but on purchase a fresh login is required. Doing this reduces the available window for a cracker to obtain, and use, the session ID. It also blocks session fixation, since the original session is abandoned at critical points. It is also advisable to change the ID on logout, so although the session is continued, its data is lost and the ID is not the same. It is highly desirable to provide logout as an option, but this needs to be supplemented by time limits on inactive sessions. A significant part of session handling is devoted to keeping enough information to be able to expire sessions that have not been used for some time. It also makes sense to revoke a session that seems to have been used for any suspicious activity. Ideally, the session ID is never transmitted unencrypted, but achieving this requires the use of SSL, and is not always practical. It should certainly be considered for high security applications. Search engine bots One aspect of website building is, perhaps unexpectedly, the importance of handling the bots that crawl the web. They are often gathering data for search engines, although some have more dubious goals, such as trawling for e-mail addresses to add to spam lists. The load they place on a site can be substantial. Sometimes, search engines account for half or more of the bandwidth being used by a site, which certainly seems excessive. If no action is taken, these bots can consume significant resources, often for very little advantage to the site owner. They can also distort information about the site, such as when the number of current visitors is displayed but includes bots in the counts. Matters are made worse by the fact that bots will normally fail to handle cookies. After all, they are not browsers and have no need to implement support for cookies. This means that every request by a bot is separate from every other, as our standard mechanism for linking requests together will not work. If the system starts a new session, it will have to do this for every new request from a bot. There will never be a logout from the bot to terminate the session, so each bot-related session will last for the time set for automatic expiry. Clearly it is inadvisable to bar bots, since most sites are anxious to gain search engine exposure. But it is possible to build session handling so as to limit the workload created by visitors who do not permit cookies, which will mostly be bots. When we move into implementation techniques, the mechanisms will be demonstrated. Session data and scalability We could simply let PHP take care of session data. It does that by writing a serialized version of any data placed into $_SESSION into a file in a temporary directory. Each session has its own file. But PHP also allows us to implement our own session data handling mechanism. There are a couple of good reasons for using that facility, and storing the information in the database. One is that we can analyze and manage the data better, and especially limit the overhead of dealing with search engine bots. The other is that by storing session data in the database, we make it feasible for the site to be run across multiple servers. There may well be other issues before that can be achieved, but providing session continuity is an essential requirement if load sharing is to be fully effective. Storing session data in a database is a reliable solution to this issue. Arguments against storing session data in a database include questions about the overhead involved, constraints on database performance, or the possibility of a single point of failure. While these are real issues, they can certainly be mitigated. Most database engines, including MySQL, have many options for building scalable and robust systems. If necessary, the database can be spread across multiple computers linked by a high speed network, although this should never be done unless it is really needed. Design of such a system is outside the scope of this article, but the key point is that the arguments against storing session data in a database are not particularly strong.

In this article by Emilien Kenler and Federico Razzoli, author of the book MariaDB Essentials, he has explained in brief about transactions and operators. (For more resources related to this topic, see here.) Understanding transactions A transaction is a sequence of SQL statements that are grouped into a single logical operation. Its purpose is to guarantee the integrity of data. If a transaction fails, no change will be applied to the databases. If a transaction succeeds, all the statements will succeed. Take a look at the following example: START TRANSACTION; SELECT quantity FROM product WHERE id = 42; UPDATE product SET quantity = quantity - 10 WHERE id = 42; UPDATE customer SET money = money - 0(SELECT price FROM product WHERE id = 42) WHERE id = 512; INSERT INTO product_order (product_id, quantity, customer_id) VALUES (42, 10, 512); COMMIT; We haven't yet discussed some of the statements used in this example. However, they are not important to understand transactions. This sequence of statements occur when a customer (whose id is 512) ordered a product (whose id is 42). As a consequence, we need to execute the following suboperations in our database: Check whether the desired quantity of products is available. If not, we should not proceed Decrease the available quantity of items for the product that is being bought Decrease the amount of money in the online account of our customer Register the order so that the product is delivered to our customer These suboperations form a more complex operation. When a session is executing this operation, we do not want other connections to interfere. Consider the following scenarios: Connection checks how many products with the ID 42 are available. Only one is available, but it is enough. Immediately after, the connection B checks the availability of the same product. It finds that one is available. Connection A decreases the quantity of the product. Now, it is 0. Connection B decreases the same number. Now, it is -1. Both connections create an order. Two persons will pay for the same product; however, only one is available. This is something we definitely want to avoid. However, there is another situation that we want to avoid. Imagine that the server crashes immediately after the customer's money is deducted. The order will not be written to the database, so the customer will end up paying for something he will not receive. Fortunately, transactions prevent both these situations. They protect our database writes in two ways: During a transaction, relevant data is locked or copied. In both these cases, two connections will not be able to modify the same rows at the same time. The writes will not be made effective until the COMMIT command is issued. This means that if the server crashes during the transaction, all the suboperations will be rolled back. We will not have inconsistent data (such as a payment for a product that will not be delivered). In this example, the transaction starts when we issue the START TRANSACTION command. Then, any number of operations can be performed. The COMMIT command makes the changes effective. This does not mean that if a statement fails with an error, the transaction is always aborted. In many cases, the application will receive an error and will be free to decide whether the transaction should be aborted or not. To abort the current transaction, an application can execute the ROLLBACK command. A transaction can consist of only one statement. This perfectly makes sense because the server could crash in the middle of the statement's execution. The autocommit mode In many cases, we don't want to group multiple statements in a transaction. When a transaction consists of only one statement, sending the START TRANSACTION and COMMIT statements can be annoying. For this reason, MariaDB has the autocommit mode. By default, the autocommit mode is ON. Unless a START TRANSACTION command is explicitly used, the autocommit mode causes an implicit commit after each statement. Thus, every statement is executed in a separated transaction by default. When the autocommit mode is OFF, a new transaction implicitly starts after each commit, and the COMMIT command needs be issued explicitly. To turn the autocommit ON or OFF, we can use the @@autocommit server variable as follows: follows: MariaDB [mwa]> SET @@autocommit = OFF; Query OK, 0 rows affected (0.00 sec) MariaDB [mwa]> SELECT @@autocommit; +--------------+ | @@autocommit | +--------------+ | 0 | +--------------+ 1 row in set (0.00 sec) Transaction's limitations in MariaDB Transaction handling is not implemented in the core of MariaDB; instead, it is left to the storage engines. Many storage engines, such as MyISAM or MEMORY, do not implement it at all. Some of the transactional storage engines are: InnoDB; XtraDB; TokuDB. In a sense, Aria tables are partially transactional. Although Aria ignores commands such as START TRANSACTION, COMMIT, and ROLLBACK, each statement is somewhat a transaction. In fact, if it writes, modifies, or deletes multiple rows, the operation completely succeeds or fails, which is similar to a transaction. Only statements that modify data can be used in a transaction. Statements that modify a table structure (such as ALTER TABLE) implicitly commit the current transaction. Sometimes, we may not be sure if a transaction is active or not. Usually, this happens because we are not sure if autocommit is set to ON or not or because we are not sure if the latest statement implicitly committed a transaction. In these cases, the @in_transaction variable can help us. Its value is 1 if a transaction is active and 0 if it is not. Here is an example: MariaDB [mwa]> START TRANSACTION; Query OK, 0 rows affected (0.00 sec) MariaDB [mwa]> SELECT @@in_transaction; +------------------+ | @@in_transaction | +------------------+ | 1 | +------------------+ 1 row in set (0.00 sec) MariaDB [mwa]> DROP TABLE IF EXISTS t; Query OK, 0 rows affected, 1 warning (0.00 sec) MariaDB [mwa]> SELECT @@in_transaction; +------------------+ | @@in_transaction | +------------------+ | 0 | +------------------+ 1 row in set (0.00 sec) InnoDB is optimized to execute a huge number of short transactions. If our databases are busy and performance is important to us, we should try to avoid big transactions in terms of the number of statements and execution time. This is particularly true if we have several concurrent connections that read the same tables. Working with operators In our examples, we have used several operators, such as equals (=), less-than and greater-than (<, >), and so on. Now, it is time to discuss operators in general and list the most important ones. In general, an operator is a sign that takes one or more operands and returns a result. Several groups of operators exist in MariaDB. In this article, we will discuss the main types: Comparison operators; String operators; Logical operators; Arithmetic operators. Comparison operators A comparison operator checks whether there is a certain relation between its operands. If the relationship exists, the operator returns 1; otherwise, it returns 0. For example, let's take the equality operator that is probably used the most: 1 = 1 -- returns 1: the equality relationship exists 1 = 0 -- returns 0: no equality relationship here In MariaDB, 1 and 0 are used in many contexts to indicate whether something is true or false. In fact, MariaDB does not have a Boolean data type, so TRUE and FALSE are merely used as aliases for 1 and 0: TRUE = 1 -- returns 1 FALSE = 0 -- returns 1 TRUE = FALSE -- returns 0 In a WHERE clause, a result of 0 or NULL prevents a row to be shown. All the numeric results other than 0, including negative numbers, are regarded as true in this context. Non-numeric values other than NULL need to be converted to numbers in order to be evaluated by the WHERE clause. Non-numeric strings are converted to 0, whereas numeric strings are treated as numbers. Dates are converted to nonzero numbers.Consider the following example: WHERE 1 -- is redundant; it shows all the rows WHERE 0 -- prevents all the rows from being shown Now, let's take a look at the following MariaDB comparison operators: Operator Description Example = This specifies equality A = B != This indicates inequality A != B <>  This is the synonym for != A <> B <  This denotes less than A < B >  This indicates greater than A > B <= This refers to less than or equals to A <= B >= This specifies greater than or equals to A >= B IS NULL This indicates that the operand is NULL A IS NULL IS NOT NULL The operand is not NULL A IS NOT NULL <=> This denotes that the operands are equal, or both are NULL A <=> B BETWEEN ... AND This specifies that the left operand is within a range of values A BETWEEN B AND C NOT BETWEEN ... AND This indicates that the left operand is outside the specified range A NOT BETWEEN B AND C IN This denotes that the left operand is one of the items in a given list A IN (B, C, D) NOT IN This indicates that the left operand is not in the given list A NOT IN (B, C, D) Here are a couple of examples: SELECT id FROM product WHERE price BETWEEN 100 AND 200; DELETE FROM product WHERE id IN (100, 101, 102); Special attention should be paid to NULL values. Almost all the preceding operators return NULL if any of their operands is NULL. The reason is quite clear, that is, as NULL represents an unknown value, any operation involving a NULL operand returns an unknown result. However, there are some operators specifically designed to work with NULL values. IS NULL and IS NOT NULL checks whether the operand is NULL. The <=> operator is a shortcut for the following code: a = b OR (a IS NULL AND b IS NULL) String operators MariaDB supports certain comparison operators that are specifically designed to work with string values. This does not mean that other operators does not work well with strings. For example, A = B perfectly works if A and B are strings. However, some particular comparisons only make sense with text values. Let's take a look at them. The LIKE operator and its variants This operator is often used to check whether a string starts with a given sequence of characters, if it ends with that sequence, or if it contains the sequence. More generally, LIKE checks whether a string follows a given pattern. Its syntax is: <string_value> LIKE <pattern> The pattern is a string that can contain the following wildcard characters: _ (underscore) means: This specifies any character %: This denotes any sequence of 0 or more characters There is also a way to include these characters without their special meaning: the _ and % sequences represent the a_ and a% characters respectively. For example, take a look at the following expressions: my_text LIKE 'h_' my_text LIKE 'h%' The first expression returns 1 for 'hi', 'ha', or 'ho', but not for 'hey'. The second expression returns 1 for all these strings, including 'hey'. By default, LIKE is case insensitive, meaning that 'abc' LIKE 'ABC' returns 1. Thus, it can be used to perform a case insensitive equality check. To make LIKE case sensitive, the following BINARY keyword can be used: my_text LIKE BINARY your_text The complement of LIKE is NOT LIKE, as shown in the following code: <string_value> NOT LIKE <pattern> Here are the most common uses for LIKE: my_text LIKE 'my%' -- does my_text start with 'my'? my_text LIKE '%my' -- does my_text end with 'my'? my_text LIKE '%my%' -- does my_text contain 'my'? More complex uses are possible for LIKE. For example, the following expression can be used to check whether mail is a valid e-mail address: mail LIKE '_%@_%.__%' The preceding code snippet checks whether mail contains at least one character, a '@' character, at least one character, a dot, at least two characters in this order. In most cases, an invalid e-mail address will not pass this test. Using regular expressions with the REGEXP operator and its variants Regular expressions are string patterns that contain a meta character with special meanings in order to perform match operations and determine whether a given string matches the given pattern or not. The REGEXP operator is somewhat similar to LIKE. It checks whether a string matches a given pattern. However, REGEXP uses regular expressions with the syntax defined by the POSIX standard. Basically, this means that: Many developers, but not all, already know their syntax REGEXP uses a very expressive syntax, so the patterns can be much more complex and detailed REGEXP is much slower than LIKE; this should be preferred when possible The regular expressions syntax is a complex topic, and it cannot be covered in this article. Developers can learn about regular expressions at www.regular-expressions.info. The complement of REGEXP is NOT REGEXP. Logical operators Logical operators can be used to combine truth expressions that form a compound expression that can be true, false, or NULL. Depending on the truth values of its operands, a logical operator can return 1 or 0. MariaDB supports the following logical operators: NOT; AND; OR; XOR The NOT operator NOT is the only logical operator that takes one operand. It inverts its truth value. If the operand is true, NOT returns 0, and if the operand is false, NOT returns 1. If the operand is NULL, NOT returns NULL. Here is an example: NOT 1 -- returns 0 NOT 0 -- returns 1 NOT 1 = 1 -- returns 0 NOT 1 = NULL -- returns NULL NOT 1 <=> NULL -- returns 0 The AND operator AND returns 1 if both its operands are true and 0 in all other cases. Here is an example: 1 AND 1 -- returns 1 0 AND 1 -- returns 0 0 AND 0 -- returns 0 The OR operator OR returns 1 if at least one of its operators is true or 0 if both the operators are false. Here is an example: 1 OR 1 -- returns 1 0 OR 1 -- returns 1 0 OR 0 -- returns 0 The XOR operator XOR stands for eXclusive OR. It is the least used logical operator. It returns 1 if only one of its operators is true or 0 if both the operands are true or false. Take a look at the following example: 1 XOR 1 -- returns 0 1 XOR 0 -- returns 1 0 XOR 1 --returns 1 0 XOR 0 -- returns 0 A XOR B is the equivalent of the following expression: (A OR B) AND NOT (A AND B) Or: (NOT A AND B) OR (A AND NOT B) Arithmetic operators MariaDB supports the operators that are necessary to execute all the basic arithmetic operations. The supported arithmetic operators are: + for additions - for subtractions * for multiplication / for division Depending on the MariaDB configuration, remember that a division by 0 raises an error or returns NULL. In addition, two more operators are useful for divisions: DIV: This returns the integer part of a division without any decimal part or reminder MOD or %: This returns the reminder of a division Here is an example: MariaDB [(none)]> SELECT 20 DIV 3 AS int_part, 20 MOD 3 AS modulus; +----------+---------+ | int_part | modulus | +----------+---------+ | 6 | 2 | +----------+---------+ 1 row in set (0.00 sec) Operators precedence MariaDB does not blindly evaluate the expression from left to right. Every operator has a given precedence. The And operators that is evaluated before another one is said to have a higher precedence. In general, arithmetic and string operators have a higher priority than logical operators. The precedence of arithmetic operators reflect their precedence in common mathematical expressions. It is very important to remember the precedence of logical operators (from the highest to the lowest): NOT AND XOR OR MariaDB supports many operators, and we did not discuss all of them. Also, the exact precedence can slightly vary depending on the MariaDB configuration. The complete precedence can be found in the MariaDB KnowledgeBase, at https://mariadb.com/kb/en/mariadb/documentation/functions-and-operators/operator-precedence/. Parenthesis can be used to force MariaDB to follow a certain order. They are also useful when we do not remember the exact precedence of the operators that we will use, as shown in the following code: (NOT (a AND b)) OR c OR d Summary In this article you learned about the basic transactions and operators. Resources for Article: Further resources on this subject: Set Up MariaDB [Article] Installing MariaDB on Windows and Mac OS X [Article] Building a Web Application with PHP and MariaDB – Introduction to caching [Article]

The first section will focus on configuring the application and its components so that the application can be deployed. The focus of the second section will be a real world application that will be developed using the steps described in the article on Developing the MVC components and in this article. That sets the agenda for this discussion. Using Spring MVC – Configuring the Application There are four main steps in configuring of the application. They are: Configure the DispatcherServlet Configure the Controller Configure the View Configure the Build Script The first step will be same for any application that is built using Spring MVC. The other three steps change according to the components that have been developed for the application. Here are the details. Configure the DispatcherServlet The first step is to tell the Application server that all the requests for this (Spring MVC based) application need to be routed to Spring MVC. This is done by setting up the DispatcherServlet. The reason for setting up DispatcherServlet is that it acts as the entry point to the Spring MVC and thus to the application.  Since the DispatcherServlet interacts with the application as a whole (instead of individual components), its configuration or setting up at application level. And any setup that needs to be done at the application level is done by making the required entries in the web.xml. The entries required  in the web.xml can be divided into the following: Servlet mapping URL mapping The former specifies the details of the servlet and the latter specifies how the servlet is related to a URL. Here are the details. Servlet mapping Servlet mapping is akin to declaring a variable. It is through servlet mapping that Application Server knows which servlets of the application it needs to support.  Servlet mapping, in essence, assigns a name to a servlet class that can be reference throughout web.xml. To set up the DispatcherServlet, first it has to be mapped to a name. that can be done using <servlet-name> and <servlet-class> tags that are the child nodes of the <servlet> tag. The following statement maps the DispatcherServlet to the name "dispatcher". <servlet> <servlet-name> dispatcher </servlet-name> <servlet-class> org.springframework.web.servlet.DispatcherServlet </servlet-class> </servlet> Since the DispatcherServlet needs to be loaded on the startup of the Application Server instead of the loading when a request arrives, the optional node <load-on-startup> with value of 1 is also required. The modified <servlet> tag will be: <servlet> <servlet-name> dispatcher </servlet-name> <servlet-class> org.springframework.web.servlet.DispatcherServlet </servlet-class> <load-on-startup>1</load-on-startup> </servlet> Next step is to map the URL to the servlet name so that the requests can be routed to the DispatcherServlet. URL mapping Once the servlet has been mapped, the next step is to map the servlet name with a URL so that the requests for that particular URL can be passed on to the application via the DispatcherServlet. That can be done using the <servlet-name> and <url-pattern> nodes of the <servlet-mapping> node. The <servlet-name> is used to refer the name that was mapped with the DispatcherServlet class. The <url-pattern> is used to map a URL pattern with a servlet name so that when a request arrives matching the URL pattern, Application Server can redirect it to the mapped servlet. To map the DispatcherServlet with a URL pattern the <servlet-mapping> tag will be: <servlet-mapping> <servlet-name>dispatcher</servlet-name> <url-pattern>*.html</url-pattern> </servlet-mapping> With this configuration/setting up of DispatcherServlet is complete. One point to keep in mind is that the URL pattern can be any pattern of one’s choice. However, it’s a common practice to use *.html for DispatcherServlet and *do for ActionServlet (Struts 1.x). Next step is to configure the View and Controller components of the application. Mapping the Controller By setting up the DispatcherServlet, the routing of requests to the application will be taken care of by the Application Server. However, unless the individual controllers of the application are setup/configured, the Framework would not know which controller to be called once the DispatcherServlet receives the request. The configuration of the Controller as well as the View components is done in the Spring MVC configuration file. The name of the configuration file is dependent on the name of the DispatcherServlet in web.xml, which is of the form <DispatcherServlet_name-servlet>.xml. So if the DispacherServlet is mapped to the name dispatcher, then the name of the configuration file will be dispatcher-servlet.xml. The file will reside in WEB-INF folder of the application. Everything in Spring Framework is a bean. Controllers are no exceptions. Controllers are configured as beans using the <bean> child tag of <beans> tag. A Controller is mapped by providing the URL of the request as the name attribute and complete qualified name of the Controller class as the value of the class attribute. For example, if the request URL is say, http://localhost/test/hello.html, then the name attribute will have /hello.html and the value attribute will have the fully qualified class name say, org.me.HelloWorldController. The following statements depicts the same: <bean name="/hello.html" class=" org.me.HelloWorldController "/> One point to keep in mind is that the "/" in the bean name represents the relative path. In other words, /hello.html means that hello.html is directly under http://localhost/test. If hello.html was under another directory say, jsp which, in turn was directly under the application, then the name attribute will be /jsp/hello.html. Let us move onto configuring the Views.

(For more resources related to this topic, see here.) The Hadoop platform Hadoop can be used for a lot of things. However, when you break it down to its core parts, the primary features of Hadoop are Hadoop Distributed File System (HDFS) and MapReduce. HDFS stores read-only files by splitting them into large blocks and distributing and replicating them across a Hadoop cluster. Two services are involved with the filesystem. The first service, the NameNode acts as a master and keeps the directory tree of all file blocks that exist in the filesystem and tracks where the file data is kept across the cluster. The actual data of the files is stored in multiple DataNode nodes, the second service. MapReduce is a programming model for processing large datasets with a parallel, distributed algorithm in a cluster. The most prominent trait of Hadoop is that it brings processing to the data; so, MapReduce executes tasks closest to the data as opposed to the data travelling to where the processing is performed. Two services are involved in a job execution. A job is submitted to the service JobTracker, which first discovers the location of the data. It then orchestrates the execution of the map and reduce tasks. The actual tasks are executed in multiple TaskTracker nodes. Hadoop handles infrastructure failures such as network issues, node, or disk failures automatically. Overall, it provides a framework for distributed storage within its distributed file system and execution of jobs. Moreover, it provides the service ZooKeeper to maintain configuration and distributed synchronization. Many projects surround Hadoop and complete the ecosystem of available Big Data processing tools such as utilities to import and export data, NoSQL databases, and event/real-time processing systems. The technologies that move Hadoop beyond batch processing focus on in-memory execution models. Overall multiple projects, from batch to hybrid and real-time execution exist. MapReduce Massive parallel processing of large datasets is a complex process. MapReduce simplifies this by providing a design pattern that instructs algorithms to be expressed in map and reduce phases. Map can be used to perform simple transformations on data, and reduce is used to group data together and perform aggregations. By chaining together a number of map and reduce phases, sophisticated algorithms can be achieved. The shared nothing architecture of MapReduce prohibits communication between map tasks of the same phase or reduces tasks of the same phase. Communication that's required happens at the end of each phase. The simplicity of this model allows Hadoop to translate each phase, depending on the amount of data that needs to be processed into tens or even hundreds of tasks being executed in parallel, thus achieving scalable performance. Internally, the map and reduce tasks follow a simplistic data representation. Everything is a key or a value. A map task receives key-value pairs and applies basic transformations emitting new key-value pairs. Data is then partitioned and different partitions are transmitted to different reduce tasks. A reduce task also receives key-value pairs, groups them based on the key, and applies basic transformation to those groups. A MapReduce example To illustrate how MapReduce works, let's look at an example of a log file of total size 1 GB with the following format: INFO MyApp - Entering application. WARNING com.foo.Bar - Timeout accessing DB - Retrying ERROR com.foo.Bar - Did it again! INFO MyApp - Exiting application Once this file is stored in HDFS, it is split into eight 128 MB blocks and distributed in multiple Hadoop nodes. In order to build a MapReduce job to count the amount of INFO, WARNING, and ERROR log lines in the file, we need to think in terms of map and reduce phases. In one map phase, we can read local blocks of the file and map each line to a key and a value. We can use the log level as the key and the number 1 as the value. After it is completed, data is partitioned based on the key and transmitted to the reduce tasks. MapReduce guarantees that the input to every reducer is sorted by key. Shuffle is the process of sorting and copying the output of the map tasks to the reducers to be used as input. By setting the value to 1 on the map phase, we can easily calculate the total in the reduce phase. Reducers receive input sorted by key, aggregate counters, and store results. In the following diagram, every green block represents an INFO message, every yellow block a WARNING message, and every red block an ERROR message: Implementing the preceding MapReduce algorithm in Java requires the following three classes: A Map class to map lines into <key,value> pairs; for example, <"INFO",1> A Reduce class to aggregate counters A Job configuration class to define input and output types for all <key,value> pairs and the input and output files MapReduce abstractions This simple MapReduce example requires more than 50 lines of Java code (mostly because of infrastructure and boilerplate code). In SQL, a similar implementation would just require the following: SELECT level, count(*) FROM table GROUP BY level Hive is a technology originating from Facebook that translates SQL commands, such as the preceding one, into sets of map and reduce phases. SQL offers convenient ubiquity, and it is known by almost everyone. However, SQL is declarative and expresses the logic of a computation without describing its control flow. So, there are use cases that will be unusual to implement in SQL, and some problems are too complex to be expressed in relational algebra. For example, SQL handles joins naturally, but it has no built-in mechanism for splitting data into streams and applying different operations to each substream. Pig is a technology originating from Yahoo that offers a relational data-flow language. It is procedural, supports splits, and provides useful operators for joining and grouping data. Code can be inserted anywhere in the data flow and is appealing because it is easy to read and learn. However, Pig is a purpose-built language; it excels at simple data flows, but it is inefficient for implementing non-trivial algorithms. In Pig, the same example can be implemented as follows: LogLine = load 'file.logs' as (level, message); LevelGroup = group LogLine by level; Result = foreach LevelGroup generate group, COUNT(LogLine); store Result into 'Results.txt'; Both Pig and Hive support extra functionality through loadable user-defined functions (UDF) implemented in Java classes. Cascading is implemented in Java and designed to be expressive and extensible. It is based on the design pattern of pipelines that many other technologies follow. The pipeline is inspired from the original chain of responsibility design pattern and allows ordered lists of actions to be executed. It provides a Java-based API for data-processing flows. Developers with functional programming backgrounds quickly introduced new domain specific languages that leverage its capabilities. Scalding, Cascalog, and PyCascading are popular implementations on top of Cascading, which are implemented in programming languages such as Scala, Clojure, and Python. Introducing Cascading Cascading is an abstraction that empowers us to write efficient MapReduce applications. The API provides a framework for developers who want to think in higher levels and follow Behavior Driven Development (BDD) and Test Driven Development (TDD) to provide more value and quality to the business. Cascading is a mature library that was released as an open source project in early 2008. It is a paradigm shift and introduces new notions that are easier to understand and work with. In Cascading, we define reusable pipes where operations on data are performed. Pipes connect with other pipes to create a pipeline. At each end of a pipeline, a tap is used. Two types of taps exist: source, where input data comes from and sink, where the data gets stored. In the preceding image, three pipes are connected to a pipeline, and two input sources and one output sink complete the flow. A complete pipeline is called a flow, and multiple flows bind together to form a cascade. In the following diagram, three flows form a cascade: The Cascading framework translates the pipes, flows, and cascades into sets of map and reduce phases. The flow and cascade planner ensure that no flow or cascade is executed until all its dependencies are satisfied. The preceding abstraction makes it easy to use a whiteboard to design and discuss data processing logic. We can now work on a productive higher level abstraction and build complex applications for ad targeting, logfile analysis, bioinformatics, machine learning, predictive analytics, web content mining, and for extract, transform and load (ETL) jobs. By abstracting from the complexity of key-value pairs and map and reduce phases of MapReduce, Cascading provides an API that so many other technologies are built on. What happens inside a pipe Inside a pipe, data flows in small containers called tuples. A tuple is like a fixed size ordered list of elements and is a base element in Cascading. Unlike an array or list, a tuple can hold objects with different types. Tuples stream within pipes. Each specific stream is associated with a schema. The schema evolves over time, as at one point in a pipe, a tuple of size one can receive an operation and transform into a tuple of size three. To illustrate this concept, we will use a JSON transformation job. Each line is originally stored in tuples of size one with a schema: 'jsonLine. An operation transforms these tuples into new tuples of size three: 'time, 'user, and 'action. Finally, we extract the epoch, and then the pipe contains tuples of size four: 'epoch, 'time, 'user, and 'action. Pipe assemblies Transformation of tuple streams occurs by applying one of the five types of operations, also called pipe assemblies: Each: To apply a function or a filter to each tuple GroupBy: To create a group of tuples by defining which element to use and to merge pipes that contain tuples with similar schemas Every: To perform aggregations (count, sum) and buffer operations to every group of tuples CoGroup: To apply SQL type joins, for example, Inner, Outer, Left, or Right joins SubAssembly: To chain multiple pipe assemblies into a pipe To implement the pipe for the logfile example with the INFO, WARNING, and ERROR levels, three assemblies are required: The Each assembly generates a tuple with two elements (level/message), the GroupBy assembly is used in the level, and then the Every assembly is applied to perform the count aggregation. We also need a source tap to read from a file and a sink tap to store the results in another file. Implementing this in Cascading requires 20 lines of code; in Scala/Scalding, the boilerplate is reduced to just the following: TextLine(inputFile) .mapTo('line->'level,'message) { line:String => tokenize(line) } .groupBy('level) { _.size } .write(Tsv(outputFile)) Cascading is the framework that provides the notions and abstractions of tuple streams and pipe assemblies. Scalding is a domain-specific language (DSL) that specializes in the particular domain of pipeline execution and further minimizes the amount of code that needs to be typed. Cascading extensions Cascading offers multiple extensions that can be used as taps to either read from or write data to, such as SQL, NoSQL, and several other distributed technologies that fit nicely with the MapReduce paradigm. A data processing application, for example, can use taps to collect data from a SQL database and some more from the Hadoop file system. Then, process the data, use a NoSQL database, and complete a machine learning stage. Finally, it can store some resulting data into another SQL database and update a mem-cache application. Summary This article explains the core technologies used in the distributed model of Hadoop Resources for Article: Further resources on this subject: Analytics – Drawing a Frequency Distribution with MapReduce (Intermediate) [article] Understanding MapReduce [article] Advanced Hadoop MapReduce Administration [article]

In this article by Steve Buchanan, Steve Beaumont, Anders Asp, Dieter Gasser, and Andreas Baumgarten, the authors of the book Microsoft System Center 2016 Service Manager Cookbook - Second Edition, we will provide recipes to tailor SCSM to your environment. Specifically, we will cover the area of setting up the SLA functions of Service Manager with the following tasks: Creating priority queues Configuring business hours and non-working days Creating SLA metrics Creating SLOs (For more resources related to this topic, see here.) Introduction SLAs in ITIL© and IT Service Management terms allow two parties to set out an agreement on how a specific service will be delivered by one to the other. We will define how it will handle the tracking of Incidents and Service Requests against defined SLAs, how to view the progress of work items against these SLAs, and how to configure SCSM 2016 to alert users when work items are nearing, or have breached, these SLAs. As with most areas of configuration within Service Manager 2016, the organization must define its processes before implementing the Service Manager feature. Creating priority queues This recipe will define a number of queues related to your defined priority for work items such as incidents and service requests. These queues will then be mapped to Service Level Objectives (SLOs). How to do it... The following steps will guide you through the process of creating priority queues: Navigate to the Queues folder in the Library section of the Service Manager 2016 console. Choose Create Queue from the taskbar on the right-hand side of the console. Review the information on the Before You Begin screen of the Create Queue Wizard and click on Next. Enter a queue name that describes the queue. In this example, we will name it Incident SLA Queue – Priority 1 to describe a queue holding Incidents with a priority of 1. Then click on the … selection button next to the Work item type textbox: Use the filter box to scope the choices down to Incident Work Items, choose Incident, and then click OK: Choose your custom Incident Management Pack from the selection list and click on Next. Use the Search box under Available properties to drop the list down to Priority. Tick the box next to Priority and then click on Add: Change the criteria for [Trouble Ticket] Priority, using the drop-down list, to equals and supply the Priority value; in this example, we will give a value of 1. Click on Next: Review the Summary screen of the wizard and then click on Create. You have now successfully created a queue. Click on Close to complete the wizard. Repeat this process for each priority you want to link an SLO to. How it works... Creating a queue allows Service Manager to group similar work items that meet specified criteria, such as all Incidents with a priority of 1. Service Manager can use these queues to scope actions. Using this grouping of work items, we have a target to apply an SLO to. There's more... This recipe requires you to repeat the steps for each priority you would like to apply an SLO to. Repeat each step, but change key information such as the name, priority value, and description to reflect the priority you are creating the queue for. For example, for an Incident Priority 3 queue, make the changes as reflected in the following screenshots: Service Request queues Queues can be created to define any type of grouping of supported process work items in scope for SLA management. For example, you may wish to repeat this recipe for the Service Request process class. Repeat the recipe but select Service Request as the work item type in the wizard, and then choose the defining criteria for the queue related to the Service Request class: You can also use this recipe, but instead of defining the criteria for the queue based on priority, you could choose the category of the incident, say, Hardware: Further queue types If the incident class was extended to capture whether the affected user was a VIP, you would be able to define a VIP queue and give those work items matching that criteria a different resolution time SLA. Configuring business hours and non-working days This recipe will define the hours that your business offers IT services, which allows calculation of resolution and response times against SLAs. Getting ready For this recipe, it is required that you have already assessed the business hours that your IT services will offer to your organization, and that you have custom management packs in place to store your queue customizations. How to do it... The following steps will guide you through the process of configuring business hours and non-working days within Service Manager: Under Administration, expand Service Level Management and then click on Calendar. Under Tasks on the right-hand side of the screen, click on Create Calendar. Give the calendar a meaningful name; in this example, we have used Core Business Hours: Choose the relevant time zone. Place a check mark against all the days for which you offer services. Under each working day, enter a start time and an end time in the 00:00:00 format, for example, 8 am should be entered as 08:00:00: You can also specify the non-working days using the Holidays section; under the Holidays pane, click on Add. In the Add Holiday window that opens enter a name for the Holiday, for example, New Year’s Day. Either manually enter the date in the format relevant for your regional settings (for example, for the United Kingdom regional settings, use DD/MM/YYYY) or use the visual calendar by clicking on the button to the right of the date entry textbox: Click on OK for each holiday. Once all holidays have been added, click on OK to close the Create/Edit Calendar window. How it works... When you specify the business hours and non-working days, Service Manager will take these into consideration when calculating SLA metrics, such as resolution time and first response time for all work items that are affected by the calendar. There's more... A calendar on its own has no impact on service levels. The calendar is one part of the SLO configuration. Adding holidays in bulk Adding holidays manually can be a very time consuming process. Our co-author Anders Asp has automated the process using PowerShell to import a list of holidays. You can download the script and read about the process on the TechNet Gallery at http://gallery.technet.microsoft.com/Generate-SCSMHolidaysCSVps1-a32722ce. Creating SLA metrics Using SLA metrics in Service Manager, we can define what is measured within an SLA. For this recipe, we will show how to create a metric to measure the resolution time of an Incident. How to do it... The following steps will guide you through the process of creating SLA metrics in Service Manager: Under Administration, expand Service Level Management and then click on Metric. Under Tasks on the right-hand side of the screen, click on Create Metric. Supply a title for the metric. In this example, we will use Resolution Time and a description. Click on the Browse... button next to the class field and use the filter box in the Select a Class window that opens to select Incident. Click on OK. Use the drop-down list for Start Date and choose Created date. Use the drop-down list for End Date and choose Resolved date: Click on OK. How it works... Creating a metric defines what you want Service Manager to track, within your SLA definition. So, when an item falls outside the parameters, you can start a notification and escalation process. Creating SLOs This recipe will show you how to create a SLO, which is used within Service Manager to create the relationships between the queues, service levels, calendars, and metrics. The SLO will define the timings to trigger warnings or breaches of service levels. Getting ready To create an SLO, you will need to have already created the following: Queues that correspond to each service level Metrics to measure differences in the start and end times of an incident A calendar to define business working hours You will also need custom management packs in place to store your SLO customizations. How to do it... The following steps will guide you through the process of creating SLOs within Service Manager: Under Administration, expand Service Level Management and then click on Service Level Objectives. Under Tasks on the right-hand side of the screen, click on Create Service Level Objective. Review the Before You Begin information and then click on Next. Provide a title and description relevant to the Service Level Objective you are creating. For this recipe, we will create an SLO for a Priority 1 Incident, and so we will set this SLO's Title to Incident Resolution Time SLO - Priority 1 with a meaningful description. Click on the Browse... button next to the Class textbox and use the filter box in the Select a Class window that opens to select Incident. Click on OK. Use the drop-down list under the Management pack heading to select your custom management pack for storing SLA related customizations to. If you are planning to use this SLO immediately then leave the Enabled checkbox ticked. Only untick this if you plan to create/stage SLOs before setting up SLA functions: Click on Next. In this recipe, use the queue named Incident SLA Queue – Priority 1: Click on Next. On the Service Level Criteria screen, choose the Calendar that you want to associate this SLO with. Under Metric, use the drop-down list to select the time metric you wish to measure against. Following along with the examples, select the Resolution Time metric. Define the target time period before a breach would occur for this metric by entering a value under target. For our Priority 1 Resolution, enter 4 Hours to define the time period before an incident would change to a breach SLA status. Define the target time period before a warning would occur for this metric by entering a value under Warning threshold. For our Priority 1 Resolution, enter 2 Hours to define the time period before an incident would change to a warning SLA status: Click on Next. Review the Information on the Summary page, and when ready, click on Create. Once the SLO has been created and a successful message is displayed, click on Close. How it works... When you configure a SLO, you're pulling together three components, queues, calendars, and metrics. These three components are defined and illustrated as follows: Queues: Work items this SLO will be applied to Calendar: Days and hours that services are offered on Metric: Items that are measured Summary In this article, we saw how to create priority queues from the Service Manager console. Priority queues are mapped to SLOs, which we also learned how to create. To make our setup more precise, we configured business hours and non-working days. We also looked at the useful and time-saving feature of adding holidays in bulk. SLA metrics are an important tool for analyzing how well or not a business is meeting its SLAs and defining the criteria that are considered when measuring that performance. To this end, we looked at how to create SLA metrics. Resources for Article: Further resources on this subject: Features of Dynamics GP [article] Configuring Endpoint Protection in Configuration Manager [article] Managing Application Configuration [article]

(For more resources related to this topic, see here.) Step 1 – fresh start In a new folder called, for example, 1_PubSub_Chat, let's open our editor of choice and create here a file called pubsub_chat.js. Also, make sure that you have a terminal window open and you moved into the newly created project directory. Step 2 – creating the TCP server TCP servers are called net servers in Vert.x. Creating and using a net server is really similar to HTTP servers: Obtain the vertx bridge object to access the framework features: var vertx = require('vertx'); /* 1 */var netServer = vertx.createNetServer(); /* 2 */netServer.listen(1234); /* 3 */ Ask Vert.x to create a TCP server (called NetServer in Vert.x). Actually, start the server by telling it to listen on TCP port 1234. Let's test whether this works. This time we need another terminal to run the telnet command: $ telnet localhost 1234 The terminal should now be connected and waiting to send/receive characters. If you have "connection refused" errors, make sure the server is running. Step 3 – adding a connect handler Now, we need to place a block of code to be executed as soon as a client connects: Define a handler function. This function will be called every time a client connects to the server: var vertx = require('vertx')var server = vertx.createNetServer().connectHandler(function(socket) {// Composing a client address stringaddr = socket.remoteAddress();addr = addr.ipaddress + addr.port;socket.write('Welcome to the chat ' + addr + '!');}).listen(1234) A NetServer connect handler accepts the socket object as a parameter; this object is our gateway to reading, writing, or closing the connection to a client. Use the socket object to write a greeting to the newly connected clients. If we test this one as in step 2 (Step 2 – creating the TCP server), we see that the server now welcomes us with a message containing an identifier of the client as its origin host and origin port. Step 4 – adding a data handler We just learned how to execute a block of code at the moment in which the client connects. However now we are interested in doing something else at the time when we receive new data from a client connection. The socket object we used in the previous step for writing data back to the client, accepts a handler function too: the data handler. Let's add one: Add a data handler function to the socket object. This is going to be called every time the client sends a new string of data: var vertx = require('vertx') var server = vertx.createNetServer().connectHandler( function(socket) { // Composing a client address string addr = socket.remoteAddress(); addr = addr.ipaddress + addr.port; socket.write('Welcome to the chat ' + addr + '!'); socket.dataHandler(function(data) { var now = new Date(); now = now.getHours() + ':' + now.getMinutes() + ':' + now.getSeconds(); var msg = now + ' <' + addr + '> ' + data; socket.write(msg); }) }).listen(1234) React to the new data event by writing the same data back to the socket (plus a prefix). What we have now is a sort of an echo server, which returns back to the sender the same message with a prefix string. Step 5 – adding the event bus magic The base requirement of a chat server is that every time a client sends a message, the rest of the connected clients should receive it. We will use event bus, the messaging service provided by the framework, to send (publish) received messages to a broadcast address. Each client will subscribe to the address upon connection and receive other clients' messages from there: Add a data handler function to the socket object: var vertx = require('vertx') var server = vertx.createNetServer().connectHandler( function(socket) { // Composing a client address string addr = socket.remoteAddress(); addr = addr.ipaddress + addr.port; socket.write('Welcome to the chat ' + addr + '!'); vertx.eventBus.registerHandler('broadcast_address', function(event){ socket.write(event); }); socket.dataHandler(function(data) { var now = new Date(); now = now.getHours() + ':' + now.getMinutes() + ':' + now.getSeconds(); var msg = now + ' <' + addr + '> ' + data; vertx.eventBus.publish('broadcast_address', msg); }) }).listen(1234) As soon as a client connects, they listen to the event bus for new data to be published on the address broadcast_address. When a client sends a string of characters to the server, this data is published to the broadcast address, triggering a handler function that writes the string to all the clients' sockets. The chat server is now complete! To try it out, just open three terminals: Terminal 1: $ vertx run pubsub_chat.js Terminal 2: $ telnet localhost 1234 Terminal 3: $ telnet localhost 1234 Now, we have a server and two clients running and connected. Type something in terminal 2 or 3 and see the message being broadcast to both the other windows: $ telnet localhost 1234Trying ::1...Connected to localhost.Escape character is '^]'.Hello from terminal two!13:6:56 <0:0:0:0:0:0:0:155991> Hello from terminal two!13:7:24 <0:0:0:0:0:0:0:155992> Hi there, here's terminal three!13:7:56 <0:0:0:0:0:0:0:155992> Great weather today! Step 6 – Organizing a more complex project Since Vert.x is a polyglot platform, we can choose to write an application (or a part of it) in either of the many supported languages. The granularity of the language choice is at verticle level. It's important to give a good architecture to a non-trivial project from the beginning. Follow this list of generic principles to avoid performance bottlenecks or the need for massive refactoring in the future: Wrap synchronous libraries or legacy code inside a worker verticle (or a module). This will keep the blocking code away from the event loop threads. Divide the problem in isolated domains and write a verticle to handle each of them (for example, database persistor verticle, web server verticle, authenticator verticle, and cache manager verticle). Use a startup verticle. This will be the single entry point to the application. Its responsibilities will be to: Validate the configuration file Programmatically deploy other verticles in the correct order Decide how many instances of a verticle to create (the decision might depend on the environment: for example, the amount of available processors) Register periodic tasks Summary: In this article, we learned in a step-wise procedure how we can create an Internet Relay Chat using the TCP server, and interconnect the server with the clients using an event bus, and enable different types of communication between them. Resources for Article: Further resources on this subject: Getting Started with Zombie.js [Article] Building a Chat Application [Article] Accessing and using the RDF data in Stanbol [Article]

(For more resources on Alfresco, see here.) The wiki page The wiki page list displays a summary of all pages created for the current site's wiki. A browsing pane on the left side of the page enables you to populate the wiki list with a subset of the wiki content. Selecting a page in the wiki list displays the entire wiki page in the page view.   When you enter the wiki page, the page view displays the wiki main page. Click Wiki Page List to display the wiki list.   You can create, delete, view details for, and edit a wiki page in both views. To perform some of the operations on a wiki page like renaming a page, you must be on the wiki page.   How to access the wiki page Accessing the wiki page gives you access to all wiki content. Within the wiki you can create, delete, rename, and edit the wiki pages. You can perform most actions from both the wiki list and the page view.   To access the wiki page, perform the following steps: Enter the Marketing Site . On the banner, click Wiki. The wiki page appears, displaying the main wiki page for this site in the page view. In this view, the actions you can perform on the current wiki page appear as buttons beneath the banner (New Page, Delete , Rename) and as links at the top right of the content area ( View Page, Edit Page, Details): (Click on the image to enlarge.) Click Wiki Page List to display the wiki list. The wiki list displays a summary of all pages in the wiki for the current site. In this view, the actions you can perform on a wiki page appear as buttons beneath the banner (only New Page is available in this view) and as links to the right of each page summary. Browsing the wiki pages Assuming that you are on the wiki page of the Marketing Site , the browsing pane to the left of the wiki page list enables you to filter the wiki content to make locating specific wiki pages more manageable. The wiki page list displays all wiki pages in the current site, organized chronologically. The most recent page appears at the top of the list. The browsing pane on the left side of the page enables you to display a subset of the wiki content by selecting a specific view or a tag. Whether browsing by view or tag, the wiki list displays a summary of all pages matching the selected option. The summary includes the wiki page name, author, creation and modification dates, a sample of the content, and the tags associated with the page. The browsing pane provides the following views: Recently Modified: In this view, the user can see the pages modified recently, for instance pages modified in the past seven days. All: This displays all pages created in the wiki for the current site. Recently Added: This displays the pages created recently, like in the past seven days. My Pages: In this view, the user can see the pages created by himself/herself. Tags: This displays all tags currently associated with one or more wiki pages. By clicking on an individual tag, the user can see all the wiki pages tagged with that particular tag. To browse the wiki pages perform the following steps: Click Wiki Page List on the page view to navigate to the wiki list, if it is not already displayed. Select the desired option in the browsing pane. In the Pages list, click the view representing the wiki pages you want to display. The wiki list displays all pages in the current site that correspond to the selection. In the Tags list, click the tag of interest. The wiki list displays all pages in the current site associated with the selected tag. n the wiki list, click a wiki page title to display that full page. The selected wiki page appears in the page view. Creating the wiki main page When any new site is created with a wiki page, it contains a main wiki page, which will be, by default, empty. And this can be considered as an introductory page for the site wiki. To add content to the wiki main page perform the following steps: Click Main Page beneath the banner if the main page is not already displayed. Click Edit Page. Type the content for the main page in the Text box. Use the formatting features provided to markup the content, insert bulleted and numbered lists, and include hyperlinks: Optionally, add tags for the main page. Type tags in the box provided and click Add. To add multiple tags, separate each tag with a space. Click Choose from popular tags in this site to display the existing tags used in this site. Click a tag to associate it with the main page. The newly associated tags appear beneath the Text box. Click a tag to remove it. Click Save. The page view displays the main page. Creating a new wiki page You can create a new wiki page from both the wiki list and the page view. To create a new wiki page perform the following steps: Click New Page to create the wiki page. Enter a Title for the page. When your title contains a disallowed character, like some special characters, such as quotation marks, periods, colons, and asterisks (*), the Save button is disabled. Type the wiki page content in the Text box. Use the formatting features provided to markup the content, insert bulleted and numbered lists, and include hyperlinks. To create a link to another wiki page, type Page Name. If the page indicated does not exist, Share creates it; however, this is an empty wiki page and it will not appear in the wiki list until you select Edit Page and provide content for it. Optionally, add tags for the wiki page. Type a tag or multiple tags, separated by a space, in the box provided and click Add. Click Choose from popular tags in this site> to display the existing tags used in this site. Click a tag to associate it with the new wiki page. The newly associated tags appear beneath the Text> box. Click a tag to remove it. Click Save. The new wiki page appears as users will see it. Click Wiki Page List to return to the wiki list: Editing a wiki page Adding and editing wiki content happens in a simple browser. You can perform this task from both the wiki list and the page view. To edit a wiki page perform the following steps: Click Edit for the page you want to modify. If the wiki page you want to edit is already open in the page view, simply click Edit Page. The content of the selected wiki page displays in an editing box. Make the required changes to the content. To create a link to another wiki page, type Page Name. If the page indicated does not exist, Share creates it; however, this is an empty wiki page and it will not appear in the wiki list until you select Edit Page and provide content for it. To edit the tags for this wiki page perform the following steps: To add a new tag, type a tag in the box provided and click Add. To add multiple tags, separate each tag with a space. To add an existing tag, click Choose from popular tags in this site to display the tags used in this site. Click a tag to associate it with the wiki page. To remove an existing tag from the set beneath the Text box, click the tag you want to remove. Click Save. The page view displays the updated wiki page. Renaming a wiki page You can rename a wiki page in the page view. When you rename a wiki page, the updated page retains the page history. Share creates a new page with the original page name; this page links to the updated page. To rename a wiki page perform the following steps: Click the title of the page that you want to rename. Click Rename. The Rename page opens. Type the new name for the wiki page. Click Save. The page view reflects the name change and this wiki page retains the page history of the original page. Share creates a new page with the original page name, which links to the updated page. Deleting a wiki page Delete a wiki page you no longer want to appear in the wiki for the current site. You can perform this task from both the wiki list and the page view. To delete a wiki page perform the following steps: In the wiki list, locate the wiki page you want to delete. Click Delete for that page. If the wiki page you want to delete is already open in the page view, simply click Delete on that page. A message prompts you to confirm the deletion. Click Yes to delete the current wiki page View the details for a wiki page to see the version history, view the tags associated with the page, and list the wiki pages that link to the selected page. You can view the page details from both the wiki list and the page view. To view the wiki page details perform the following steps: Click Details for the page whose details you want to view. If the wiki page you want to view is already open in the page view, simply click Details on that page. The page view displays the wiki page content in an expanded form to include the Version History, Tags, and Linked Pages. From this page, you can click Edit Page to make any necessary changes. Click View Page to hide the details. The wiki page displays in the page view: For more details on the Alfresco Share wiki feature, refer to the link: http://docs.alfresco.com/4.0/ index.jsp?topic=%2Fcom.alfresco.enterprise. doc%2Fconcepts%2Fwiki-intro.html. The Blog page Site members can create, edit, and add comments to blog postings. The Blog page enables you to add commentary, descriptions of events, and other material related to your site. The postings can be saved as drafts and then, when ready can be published. The blogs can be published as an internal blog and also to an external blogging site, including WordPress and TypePad. We will discuss more about how you can configure your blog to get published on external blog sites from Alfresco Share in a later section of this article. How to access a Blog page Accessing the Blog page enables you to view all published blog posts for the Marketing Site . Within the blog, you can create new posts, as well as edit, publish, and delete posts you created. You can also add comments to existing blog posts. To access the Blog page perform the following steps: Enter the Marketing Site. On the banner, click Blog. The Blog page appears. The default view, Latest, is selected and the post list displays a summary of the posts created or edited in the past seven days. The summary includes the post title, date and time the post was published (if applicable), the author who created the post, a sample of the content, the number of replies to the post, and the tags associated with the post. The Previous and Next navigation buttons help move through multiple pages of posts. Click Simple List> to display only the title of the posts. Click Detailed List to display the summary view. Browsing the blog The browsing feature in the Blog page enables you to filter the blog posts. This allows you to more easily navigate the blog content. A subset of the blog posts by selecting a specific view, a period of time (month), or a tag is displayed on the left hand browsing pane. The Posts list in the browsing pane provides the following views/filters: All: In this view, the user can see all posts in the blogs. Latest: This view displays only the posts created, or modified, recently like in the past seven days. My Drafts: Here, the user can see the posts created by himself/herself that are saved as drafts (not yet published) My Published: This view displays the posts created and published, either externally or internally, by the current user. Published Externally: In this view, all posts that have been published to an external blog are displayed. Archives: This view organizes posts by month and year. This will have further sub lists of month and year. Tags: Here, the list displays all tags currently associated with one or more blog posts. By clicking on an individual tag, the user can see all the posts tagged with that particular tag. To browse the blog posts perform the following steps: Select the desired option in the browsing pane. In the Posts list, click the view representing the posts you want to view. The post list displays all blog posts in the current site that correspond to the selection. In the Archives list, click the month of interest. The post list displays all blog posts in the current site published during that month. In the Tags list, click the tag of interest. The post list displays all posts in the current site associated with the selected tag. Click Simple List to display only the title of the posts. Click Detailed List to display the summary view Creating a blog post Create a new blog post to add information or a commentary related to the current site. When you create a new post, you can save it as a draft without publishing it, or you can immediately publish it to the blog on the Marketing Site . You also have the option of publishing the post to both the internal blog and a configured external blog To create a new post perform the following steps: Click New Post. The Create Blog Post page appears. Type a Title for the post. Type the post content in the Text box. Use the formatting features provided to markup the content, insert bulleted and numbered lists, and include hyperlinks.Optionally, add tags for the blog post: Type a tag in the box provided and click Add. To add multiple tags, separate each tag with a space. Click Choose from popular tags in this site to display the existing tags used in this site. Click a tag to associate it with the new post: The newly associated tags appear beneath the Text box. Click a tag to remove it. To save or publish the new blog post perform the following steps: Click Save as Draft to save the post without publishing it. The post will not appear in the post list. Click Publish Internally to publish the post to the internal blog, making it available for other users of this site. Click Publish Internally and Externally to publish the post to both the internal and external blogs. The new post appears as users will see it. The text (Draft) appears after the title if the post remains unpublished. Click Go to Post List to return to the main view. Editing a blog post Only the Marketing Site Manager, and the user who created the blog post, can edit it to modify or add to its content. To edit an existing post perform the following steps: In the post list, locate the blog post you want to edit. Click Edit. The Edit Blog Post page appears displaying the selected post. Make the desired changes to the post title and content. To edit the tags for this post perform the following steps: To add a new tag or multiple tags, each separated with a space, type a tag in the box provided and click Add. To add an existing tag, click Choose from popular tags in this site to display the tags used in this site. Click a tag to associate it with the new blog post. To remove an existing tag from the set beneath the Text box, click the tag you want to remove: To save or publish the updated blog post perform the following steps: Click Update to save the changes. Click Publish Internally to save the changes and publish the post to the internal blog, making it available for other site members. This option appears only for draft posts. Click Update Internally and Publish Externally to publish the post to both the internal and external blogs. The updated post appears as users will see it. The text (updated) appears after the title. Click Go to Post List to return to the main view. Viewing a blog post Browsing the blog displays either a summary view or a simple list of the existing posts in the main window of the Blog page. Viewing a post allows you to see the full contents of the post. You can perform actions on a post from the main page, but viewing a post enables you to confirm that you have selected the correct post before performing irreversible actions, such as deleting and publishing the post. While the same actions are available in both views (Edit, Publish Externally, and Delete ), you must view a post to add a comment to it. To view a blog post perform the following steps: Browse the blog posts to locate the post you want to view. Click the title of the post or click Read beneath the post. If the main page displays posts in a simple list, you must click the title of the post. The post view displays the selected blog post, along with any related comments. Click Go to Post List to return to the main view. Deleting a blog post Delete a blog post to permanently remove it from the blog of the Marketing Site. Deleting a post also deletes all of its comments. Only a Site Manager and the user who created the blog post can delete it. To delete a post perform the following steps: In the post list, locate the blog post you want to delete. Click Delete. A message prompts you to confirm the deletion f the selected post. Click Delete. A message indicates the selected blog post has been deleted. Configuring the external blog In addition to creating blog posts and publishing them internally within the site, you can publish the blog posts to an existing WordPress or TypePad blog. To publish the site blogs externally, you must configure the external blog details. To configure an external blog perform the following steps: Click Configure External Blog. The Configure External Blog page appears. In the Type list, select the appropriate option: Select WordPress to configure the external blog to run on the WordPress blog publishing system. Select TypePad to configure the external blog to run on a TypePad blogging service. Complete the remaining fields with the details of the external blog. The User Name and Password provided must be the user details of the author, or administrator, responsible for posting the external blog. Click OK: Publishing a post to an external blog In addition to publishing a blog post to the internal blog of the Marketing Site , you can publish a post to a configured external blog. To publish a draft post externally, you must also publish it internally. To publish a post externally perform the following steps: In the post list, locate the blog post you want to publish externally. Click Publish Externally. Optionally, open the post to perform this action. A message indicates the selected blog post is published. Working with comments Adding comments to a blog post can help make a site blog interactive. People can comment on the post and provide their views or comments for any blogs written by other users. While all users with access to the Marketing Site can view the blog conversations, only site members can add comments. The number of replies added to a post is displayed in the post list beneath the post summary. You must view a post to add, view, and manage the related comments. Adding a comment to a post In the Blog page, you can add a comment to reply to any published blog post. To add a comment perform the following steps: In the post list, locate the blog post to which you want to add a comment. Click the post title to view the post. The post view displays the selected blog post, along with any related comments. Type the post comment in the Add comment box. Use the formatting features provided to markup the content, insert bulleted and numbered lists, and include hyperlinks. Click Create comment. The comment displays beneath the post. Click Go to post list to return to the main view: Editing a comment You can edit a blog comment to modify or add to its content. Only a Site Manager, and the user who created the comment, can edit it. To edit a comment perform the following steps: In the post list, locate the blog post containing the comment you want to edit. Click the post title to view the post. The post view displays the selected blog post and the related comments. Click Edit to the right of the comment you want to change. The Edit Comment box appears, displaying the selected comment. Make the desired changes to the comment. Click Update. The updated comment displays beneath the post. Click Go to Post List to return to the main view. Deleting a comment Delete a comment to permanently remove it from a blog post. Only a Site Manager, and the user who created the comment, can delete it. To delete a comment perform the following steps: In the post list, locate the blog post containing the comment you want to delete. Click the post title to view the post. The post view displays the selected blog post and the related comments. Click Delete. A message prompts you to confirm the deletion of the selected comment. Click Delete. A message indicates the selected comment has been deleted. Click Go to Post List to return to the main view.

In this article by James Denton, the author of the book OpenStack Networking Essentials, you can see that Neutron is a virtual networking service that allows users to define network connectivity and IP addressing for instances and other cloud resources using an application programmable interface (API). The Neutron API is made up of core elements that define basic network architectures and extensions that extend base functionality. Neutron accomplishes this by virtue of its data model that consists of networks, subnets, and ports. These objects help define characteristics of the network in an easily storable format. (For more resources related to this topic, see here.) These core elements are used to build a logical network data model using information that corresponds to layers 1 through 3 of the OSI model, shown in the following screenshot: For more information on the OSI model, check out the Wikipedia article at https://en.wikipedia.org/wiki/OSI_model. Neutron uses plugins and drivers to identify network features and construct the virtual network infrastructure based on information stored in the database. A core plugin, such as the Modular Layer 2 (ML2) plugin included with Neutron, implements the core Neutron API and is responsible for adapting the logical network described by networks, ports, and subnets into something that can be implemented by the L2 agent and IP address management system running on the hosts. The extension API, provided by service plugins, allows users to manage the following resources, among others: Security groups Quotas Routers Firewalls Load balancers Virtual private networks Neutron's extensibility means that new features can be implemented in the form of extensions and plugins that extend the API without requiring major changes. This allows vendors to introduce features and functionality that would otherwise not be available with the base API. The following diagram demonstrates at a high level how the Neutron API server interacts with the various plugins and agents responsible for constructing the virtual and physical network across the cloud: The previous diagram demonstrates the interaction between the Neutron API service, Neutron plugins and drivers, and services such as the L2 and L3 agents. As network actions are performed by users via the API, the Neutron server publishes messages to the message queue that are consumed by agents. L2 agents build and maintain the virtual network infrastructure, while L3 agents are responsible for building and maintaining Neutron routers and associated functionality. The Neutron API specifications can be found on the OpenStack wiki at https://wiki.openstack.org/wiki/Neutron/APIv2-specification. In the next few sections, we will look at some of the core elements of the API and the data models used to represent those elements. Networks A network is the central object of the Neutron v2.0 API data model and describes an isolated L2 segment. In a traditional infrastructure, machines are connected to switch ports that are often grouped together into virtual local area networks (VLANs) identified by unique IDs. Machines in the same network or VLAN can communicate with one another but cannot communicate with other networks in other VLANs without the use of a router. The following diagram demonstrates how networks are isolated from one another in a traditional infrastructure: Neutron network objects have attributes that describe the network type and the physical interface used for traffic. The attributes also describe the segmentation ID used to differentiate traffic between other networks connected to virtual switches on the underlying host. The following diagram shows how a Neutron network describes various Layer 1 and Layer 2 attributes: Traffic between instances on different hosts requires underlying connectivity between the hosts. This means that the hosts must reside on the same physical switching infrastructure so that VLAN-tagged traffic can pass between them. Traffic between hosts can also be encapsulated using L2-in-L3 technologies such as GRE or VXLAN. Neutron supports multiple L2 methods of segmenting traffic, including using 802.1q VLANs, VXLANs, GRE, and more, depending on the plugin and configured drivers and agents. Devices in the same network are in the same broadcast domain, even though they may reside on different hosts and attach to different virtual switches. Neutron network attributes are very important in defining how traffic between virtual machine instances should be forwarded between hosts. Network attributes The following table describes base attributes associated with network objects, and more details can be found at the Neutron API specifications wiki referenced earlier in this article: Attribute Type Required Default Notes id uuid-str N/A Auto generated The UUID for the network name string no None The human-readable name for the network admin_state_up boolean no True The administrative state of the network status string N/A Null Indicates whether the network is currently operational subnets list no Empty list The subnets associated with the network shared boolean no False Specifies whether the network can be accessed by any tenant tenant_id uuid-str no N/A The owner of the network Networks are typically associated with tenants or projects and are usable by any user that is a member of the same tenant or project. Networks can also be shared with all other projects or a subnet of projects using Neutron's role-based access control (RBAC) functionality. Neutron RBAC first became available in the Liberty release of OpenStack. For more information on using the RBAC features, check out my blog at the following URL: https://developer.rackspace.com/blog/A-First-Look-at-RBAC-in-the-Liberty-Release-of-Neutron/. Provider attributes One of the earliest extensions to the Neutron API is known as the provider extension. The provider network extension maps virtual networks to physical networks by adding additional network attributes that describe the network type, segmentation ID, and physical interface. The following table shows various provider attributes and their associated values: Attribute Type Required Options Default Notes provider:network_type string yes vlan,flat,local, vxlan,gre Based on the configuration   provider:segmentation_id int optional Depends on the network type Based on the configuration The segmentation ID range varies among L2 technologies provider:physical_network string optional Provider label Based on the configuration This specifies the physical interface used for traffic (flat or VLAN-only) All networks have provider attributes. However, because provider attributes specify particular network configuration settings and mappings, only users with the admin role can specify them when creating networks. Users without the admin role can still create networks, but the Neutron server, not the user, will determine the type of network created and any corresponding interface or segmentation ID. Additional attributes The external-net extension adds an attribute to networks that is used to determine whether or not the network can be used as the external, or gateway, network for a Neutron router. When set to true, the network becomes eligible for use as a floating IP pool when attached to routers. Using the Neutron router-gateway-set command, routers can be attached to external networks. The following table shows the external network attribute and its associated values: Attribute Type Required Default Notes router:external Boolean no false When true, the network is eligible for use as a floating IP pool when attached to a router Subnets In the Neutron data model, a subnet is an IPv4 or IPv6 address block from which IP addresses can be assigned to virtual machine instances and other network resources. Each subnet must have a subnet mask represented by a classless inter-domain routing (CIDR) address and must be associated with a network, as shown here: In the preceding diagram, three isolated VLAN networks each have a corresponding subnet. Instances and other devices cannot be attached to networks without an associated subnet. Instances connected to a network can communicate among one another but are unable to connect to other networks or subnets without the use of a router. The following diagram shows how a Neutron subnet describes various Layer 3 attributes in the OSI model: When creating subnets, users can specify IP allocation pools that limit which addresses in the subnet are available for allocation. Users can also define a custom gateway address, a list of DNS servers, and individual host routes that can be pushed to virtual machine instances using DHCP. The following table describes attributes associated with subnet objects: Attribute Type Required Default Notes id uuid-str n/a Auto Generated The UUID for the subnet network_id uuid-str Yes N/A The UUID of the associated network name string no None The human-readable name for the subnet ip_version int Yes 4 IP version 4 or 6 cidr string Yes N/A The CIDR address representing the IP address range for the subnet gateway_ip string or null no First address in CIDR The default gateway used by devices in the subnet dns_nameservers list(str) no None The DNS name servers used by hosts in the subnet allocation_pools list(dict) no Every address in the CIDR (excluding the gateway) The subranges of the CIDR available for dynamic allocation. tenant_id uuid-str no N/A The owner of the subnet enable_dhcp boolean no True This indicates whether or not DHCP is enabled for the subnet host_routes list(dict) no N/A Additional static routes Ports In the Neutron data model, a port represents a switch port on a logical switch that spans the entire cloud and contains information about the connected device. Virtual machine interfaces (VMIFs) and other network objects, such as router and DHCP server interfaces, are mapped to Neutron ports. The ports define both the MAC address and the IP address to be assigned to the device associated with them. Each port must be associated with a Neutron network. The following diagram shows how a port describes various Layer 2 attributes in the OSI model: The following table describes attributes associated with port objects: Attribute Type Required Default Notes id uuid-str n/a Auto generated The UUID for the subnet network_id uuid-str Yes N/A The UUID of the associated network name string no None The human-readable name for the subnet admin_state_up Boolean no True The administrative state of the port status string N/A N/A The current status of the port (for example, ACTIVE, BUILD, or DOWN) mac_address string no Auto generated The MAC address of the port fixed_ips list(dict) no Auto allocated The IP address(es) associated with the port device_id string no None The instance ID or other resource associated with the port device_owner string no None   tenant_id uuid-str no ID of tenant adding resource The owner of the port When Neutron is first installed, no ports exist in the database. As networks and subnets are created, ports may be created for each of the DHCP servers reflected by the logical switch model, seen here: As instances are created, a single port is created for each network interface attached to the instance, as shown here: A port can only be associated with a single network. Therefore, if an instance is connected to multiple networks, it will be associated with multiple ports. As instances and other cloud resources are created, the logical switch may scale to hundreds or thousands of ports over time, as shown in the following diagram: There is no limit to the number of ports that can be created in Neutron. However, quotas exist that limit tenants to a small number of ports that can be created. As the number of Neutron ports scale out, the performance of the Neutron API server and the implementation of networking across the cloud may degrade over time. It's a good idea to keep quotas in place to ensure a high-performing cloud, but the defaults and subsequent quota increases should be kept reasonable. The Neutron workflow In the standard Neutron workflow, networks must be created first, followed by subnets and then ports. The following sections describe the workflows involved with booting and deleting instances. Booting an instance Before an instance can be created, it must be associated with a network that has a corresponding subnet or a precreated port that is associated with a network. The following process documents the steps involved in booting an instance and attaching it to a network: The user creates a network. The user creates a subnet and associates it with the network. The user boots a virtual machine instance and specifies the network. Nova interfaces with Neutron to create a port on the network. Neutron assigns a MAC address and IP address to the newly created port using attributes defined by the subnet. Nova builds the instance's libvirt XML file containing local network bridge and MAC address information and starts the instance. The instance sends a DHCP request during boot, at which point the DHCP server responds with the IP address corresponding to the MAC address of the instance. If multiple network interfaces are attached to an instance, each network interface will be associated with a unique Neutron port and may send out DHCP requests to retrieve their respective network information. How the logical model is implemented Neutron agents are services that run on network and compute nodes and are responsible for taking information described by networks, subnets, and ports and using it to implement the virtual and physical network infrastructure. In the Neutron database, the relationship between networks, subnets, and ports can be seen in the following diagram: This information is then implemented on the compute node by way of virtual network interfaces, virtual switches or bridges, and IP addresses, as shown in the following diagram: In the preceding example, the instance was connected to a network bridge on a compute node that provides connectivity from the instance to the physical network. For now, it's only necessary to know how the data model is implemented into something that is usable. Deleting an instance The following process documents the steps involved in deleting an instance: The user destroys virtual machine instance. Nova interfaces with Neutron to destroy the ports associated with the instances. Nova deletes local instance data. The allocated IP and MAC addresses are returned to the pool. When instances are deleted, Neutron removes all virtual network connections from the respective compute node and removes corresponding port information from the database. Summary In this article, we looked at the basics of the Neutron API and its data model made up of networks, subnets, and ports. These objects were used to describe in a logical way how the virtual network is architected and implemented across the cloud. Resources for Article: Further resources on this subject: Introducing OpenStack Trove[article] Concepts for OpenStack[article] Monitoring OpenStack Networks[article]

In this article by Kurt Menke, GISP, Dr. Richard Smith Jr., GISP, Dr. Luigi Pirelli, Dr. John Van Hoesen, GISP, authors of the book Mastering QGIS, we'll have a look at how to geocode address-based date using QGIS and MMQGIS. (For more resources related to this topic, see here.) Geocoding addresses has many applications, such as mapping the customer base for a store, members of an organization, public health records, or incidence of crime. Once mapped, the points can be used in many ways to generate information. For example, they can be used as inputs to generate density surfaces, linked to parcels of land, and characterized by socio-economic data. They may also be an important component of a cadastral information system. An address geocoding operation typically involves the tabular address data and a street network dataset. The street network needs to have attribute fields for address ranges on the left- and right-hand side of each road segment. You can geocode within QGIS using a plugin named MMQGIS (http://michaelminn.com/linux/mmqgis/). MMQGIS has many useful tools. For geocoding, we will use the tools found in MMQGIS | Geocode. There are two tools there: Geocode CSV with Google/ OpenStreetMap and Geocode from Street Layer as shown in the following screenshot. The first tool allows you to geocode a table of addresses using either the Google Maps API or the OpenStreetMap Nominatim web service. This tool requires an Internet connection but no local street network data as the web services provide the street network. The second tool requires a local street network dataset with address range attributes to geocode the address data: How address geocoding works The basic mechanics of address geocoding are straightforward. The street network GIS data layer has attribute columns containing the address ranges on both the even and odd side of every street segment. In the following example, you can see a piece of the attribute table for the Streets.shp sample data. The columns LEFTLOW, LEFTHIGH, RIGHTLOW, and RIGHTHIGH contain the address ranges for each street segment: In the following example we are looking at Easy Street. On the odd side of the street, the addresses range from 101 to 199. On the even side, they range from 102 to 200. If you wanted to map 150 Easy Street, QGIS would assume that the address is located halfway down the even side of that block. Similarly, 175 Easy Street would be on the odd side of the street three quarters the way down the block. Address geocoding assumes that the addresses are evenly spaced along the linear network. QGIS should place the address point very close to its actual position, but due to variability in lot sizes not every address point will be perfectly positioned. Now that you've learned the basics, let's work through an example. Here we will geocode addresses using web services. The output will be a point shapefile containing all the attribute fields found in the source Addresses.csv file. An example – geocoding using web services Here are the steps for geocoding the Addresses.csv sample data using web services. Load the Addresses.csv and the Streets.shp sample data into QGIS Desktop. Open Addresses.csv and examine the table. These are addresses of municipal facilities. Notice that the street address (for example, 150 Easy Street) is contained in a single field. There are also fields for the city, state, and country. Since both Google and OpenStreetMap are global services, it is wise to include such fields so that the services can narrow down the geography. Install and enable the MMQGIS plugin. Navigate to MMQGIS | Geocode | Geocode CSV with Google/OpenStreetMap. The Web Service Geocode dialog window will open. Select Input CSV File (UTF-8) by clicking on Browse… and locating the delimited text file on your system. Select the address fields by clicking on the drop-down menu and identifying the Address Field, City Field, State Field, and Country Field fields. MMQGIS may identify some or all of these fields by default if they are named with logical names such as Address or State. Choose the web service. Name the output shapefile by clicking on Browse…. Name Not Found Output List by clicking on Browse…. Any records that are not matched will be written to this file. This allows you to easily see and troubleshoot any unmapped records. Click on OK. The status of the geocoding operation can be seen in the lower-left corner of QGIS. The word Geocoding will be displayed, followed by the number of records that have been processed. The output will be a point shapefile and a CSV file listing that addresses were not matched. Two additional attribute columns will be added to the output address point shapefile: addrtype and addrlocat. These fields provide information on how the web geocoding service obtained the location. These may be useful for accuracy assessment. Addrtype is the Google <type> element or the OpenStreetMap class attribute. This will indicate what kind of address type this is (highway, locality, museum, neighborhood, park, place, premise, route, train_station, university etc.). Addrlocat is the Google <location_type> element or OpenStreetMap type attribute. This indicates the relationship of the coordinates to the addressed feature (approximate, geometric center, node, relation, rooftop, way interpolation, and so on). If the web service returns more than one location for an address, the first of the locations will be used as the output feature. Use of this plugin requires an active Internet connection. Google places both rate and volume restrictions on the number of addresses that can be geocoded within various time limits. You should visit the Google Geocoding API website: (http://code.google.com/apis/maps/documentation/geocoding/) for more details, and current information and Google's terms of service. Geocoding via these web services can be slow. If you don't get the desired results with one service, try the other. Geocoding operations rarely have 100% success. Street names in the street shapefile must match the street names in the CSV file exactly. Any discrepancies between the name of a street in the address table, and the street attribute table will lower the geocoding success rate. The following image shows the results of geocoding addresses via street address ranges. The addresses are shown with the street network used in the geocoding operation: Geocoding is often an iterative process. After the initial geocoding operation, you can review the Not Found CSV file. If it's empty then all the records were matched. If it has records in it, compare them with the attributes of the streets layer. This will help you determine why those records were not mapped. It may be due to inconsistencies in the spelling of street names. It may also be due to a street centerline layer that is not as current as the addresses. Once the errors have been identified they can be corrected by editing the data, or obtaining a different street centreline dataset. The geocoding operation can be re-run on those unmatched addresses. This process can be repeated until all records are matched. Use the Identify tool to inspect the mapped points, and the roads, to ensure that the operation was successful. Never take a GIS operation for granted. Check your results with a critical eye. Summary This article introduced you to the process of address geocoding using QGIS and the MMQGIS plugin. Resources for Article: Further resources on this subject: Editing attributes [article] How Vector Features are Displayed [article] QGIS Feature Selection Tools [article]

In this article by Ankur Goel, author of the book, Neo4j Cookbook, we will cover the following recipes: Installing the Neo4j Spatial extension Importing the Esri shapefiles Importing the OpenStreetMap files Importing data using the REST API Creating a point layer using the REST API Finding geometries within the bounding box Finding geometries within a distance Finding geometries within a distance using Cypher By definition, any database that is optimized to store and query the data that represents objects defined in a geometric space is called a spatial database. Although Neo4j is primarily a graph database, due to the importance of geospatial data in today's world, the spatial extension has been introduced in Neo4j as an unmanaged extension. It gives you most of the facilities, which are provided by common geospatial databases along with the power of connectivity through edges, which Neo4j, as a graph database, provides. In this article, we will take a look at some of the widely used use cases of Neo4j as a spatial database, and you will learn how typical geospatial operations can be performed on it. Before proceeding further, you need to install Neo4j on your system using one of the recipies that follow here. The installation will depend on your system type. (For more resources related to this topic, see here.) Single node installation of Neo4j over Linux Neo4j is a highly scalable graph database that runs over all the common platforms; it can be used as is or can be embedded inside applications as well. The following recipe will show you how to set up a single instance of Neo4j over the Linux operating system. Getting ready Perform the following steps to get started with this recipe: Download the community edition of Neo4j from http://www.neo4j.org/download for the Linux platform: $ wget http://dist.neo4j.org/neo4j-community-2.2.0-M02-unix.tar.gz Check whether JDK/JRE is installed for your operating system or not by typing this in the shell prompt: $ echo $JAVA_HOME If this command throws no output, install Java for your Linux distribution and also set the JAVA_HOME path How to do it... Now, let's install Neo4j over the Linux operating system, which is simple, as shown in the following steps: Extract the TAR file using the following command: $ tar –zxvf neo4j-community-2.2.0-M02-unix.tar.gz $ ls Go to the bin directory under the root folder: $ cd <neo4j-community-2.2.0-M02>/bin/ Start the Neo4j graph database server: $ ./neo4j start Check whether Neo4j is running or not using the following command: $ ./neo4j status Neo4j can also be monitored using the web console. Open http://<ip>:7474/webadmin, as shown in the following screenshot: The preceding diagram is a screenshot of the web console of Neo4j through which the server can be monitored and different Cypher queries can be run over the graph database. How it works... Neo4j comes with prebuilt binaries over the Linux operating system, which can be extracted and run over. Neo4j comes with both web-based and terminal-based consoles, over which the Neo4j graph database can be explored. See also During installation, you may face several kind of issues, such as max open files and so on. For more information, check out http://neo4j.com/docs/stable/server-installation.html#linux-install. Single node installation of Neo4j over Windows Neo4j is a highly scalable graph database that runs over all the common platforms; it can be used as is or can be embedded inside applications. The following recipe will show you how to set up a single instance of Neo4j over the Windows operating system. Getting ready Perform the following steps to get started with this recipe: Download the Windows installer from http://www.neo4j.org/download. This has both 32- and 64-bit prebuilt binaries Check whether Java is installed for the operating system or not by typing this in the cmd prompt: echo %JAVA_HOME% If this command throws no output, install Java for your Windows distribution and also set the JAVA_HOME path How to do it... Now, let's install Neo4j over the Windows operating system, which is simple, as shown here: Run the installer by clicking on the downloaded file: The preceding screenshot shows the Windows installer running. After the installation is complete, when you run the software, it will ask for the database location. Carefully choose the location as the entire graph database will be stored in this folder: The preceding screenshot shows the Windows installer asking for the graph database's location. The Neo4j browser can be opened by entering http://localhost:7474/ in the browser. The following screenshot depicts Neo4j started over the Windows platform: How it works... Neo4j comes with prebuilt binaries over the Windows operating system, which can be extracted and run over. Neo4j comes with both web-based and terminal-based consoles, over which the Neo4j graph database can be explored. See also During installation, you might face several kinds of issues such as max open files and so on. For more information, check out http://neo4j.com/docs/stable/server-installation.html#windows-install. Single node installation of Neo4j over Mac OS X Neo4j is a highly scalable graph database that runs over all the common platforms; it can be used as in a mode and can also be embedded inside applications. The following recipe will show you how to set up a single instance of Neo4j over the OS X operating system. Getting ready Perform the following steps to get started with this recipe: Download the binary version of Neo4j from http://www.neo4j.org/download for the Mac OS X platform and the community edition as shown in the following command: $ wget http://dist.neo4j.org/neo4j-community-2.2.0-M02-unix.tar.gz Check whether Java is installed for the operating system or not by typing this over the cmd prompt: $ echo $JAVA_HOME If this command throws no output, install Java for your Mac OS X distribution and also set the JAVA_HOME path How to do it... Now, let's install Neo4j over the OS X operating system, which is very simple, as shown in the following steps: Extract the TAR file using the following command: $ tar –zxvf neo4j-community-2.2.0-M02-unix.tar.gz $ ls Go to the bin directory under the root folder: $ cd <neo4j-community-2.2.0-M02>/bin/ Start the Neo4j graph database server: $ ./neo4j start Check whether Neo4j is running or not using the following command: $ ./neo4j status How it works... Neo4j comes with prebuilt binaries over the OS X operating system, which can be extracted and run over. Neo4j comes with both web-based and terminal-based consoles, over which the Neo4j graph database can be explored. There's more… Neo4j over Mac OS X can also be installed using brew, which has been explained here. Run the following commands over the shell: $ brew update $ brew install neo4j After this, Neo4j can be started using the start option with the Neo4j command: $ neo4j start This will start the Neo4j server, which can be accessed from the default URL (http://localhost:7474). The installation can be reached using the following commands: $ cd /usr/local/Cellar/neo4j/ $ cd {NEO4J_VERSION}/libexec/ You can learn more about OS X installation from http://neo4j.com/docs/stable/server-installation.html#osx-install. Due to the limitation of content that can provided in this article, we assume you would already know how to perform the basic operations using Neo4j such as creating a graph, importing data from different formats into Neo4j, the common configurations used for Neo4j. Installing the Neo4j Spatial extension Neo4j Spatial is a library of utilities for Neo4j that facilitates the enabling of spatial operations on the data. Even on the existing data, geospatial indexes can be added and many geospatial operations can be performed on it. In this recipe, you will learn how to install the Neo4j Spatial extension. Getting ready The following steps will get you started with this recipe: Install Neo4j using the earlier recipes in this article. Install the dependencies listed in the pom.xml file for this project from https://github.com/neo4j-contrib/spatial/blob/master/pom.xml. Install Maven using the following command for your operating system: For Debian systems: apt-get install maven For Redhat/Centos systems: yum install apache-maven To install on a Windows-based system, please refer to https://maven.apache.org/guides/getting-started/windows-prerequisites.html. How to do it... Now, let's install the Neo4j Spatial plugin, which is very simple to do, by following these steps: Clone the GitHub repository for spatial extension: git clone git://github.com/neo4j/spatial spatial Move into the spatial directory: cd spatial Build the code using Maven. This will download all the dependencies, compile the library, run the tests, and install the artifacts in the local repository: mvn clean install Move the built artifact to the Neo4j plugin directory: unzip target/neo4j/neo4j-spatial-0.11-SNAPSHOT-server-plugin.zip $NEO4J_ROOT_DIR/plugins/ Restart the Neo4j graph database server: $NEO4J_ROOT_DIR/bin/neo4j restart Check whether the Neo4j Spatial plugin is properly installed or not: curl –L http://<neo4j_server_ip>:<port>/db/data If you are using Neo4j 2.2 or higher, then use the following command: curl --user neo4j:<password> http://localhost:7474/db/data/ The output will look like what is shown in the following screenshot, which shows the Neo4j Spatial plugin installed: How it works... Neo4j Spatial is a library of utilities that helps perform geospatial operations on the dataset, which is present in the Neo4j graph database. You can add geospatial indexes on the existing data and perform operations, such as data within a specified region or within some distance of point of interest. Neo4j Spatial comes as an unmanaged extension, which can be easily installed as well as removed from Neo4j. The extension does not interfere with any of the core functionality. There's more… To read more about Neo4j Spatial extension, we encourage users to visit the GitHub repository at https://github.com/neo4j-contrib/spatial. Also, it will be good to read about the Neo4j unmanaged extension in general (http://neo4j.com/docs/stable/server-unmanaged-extensions.html). Importing the Esri shapefiles The shapefile format is a popular geospatial vector data format for the Geographic Information System (GIS) software. It is developed and regulated by Esri as an open specification for data interoperability among Esri. It is very popular among GIS products, and many times, the data format is in Esri shapefiles. The main file is the .shp file, which contains the geometry data. The binary data file consists of a single, fixed-length header followed by variable-length data records. In this recipe, you will learn how to import the Esri shapefiles in the Neo4j graph database. Getting ready Perform the following steps to get started with this recipe: Install Neo4j using the earlier recipies in this article. Install the Neo4j Spatial plugin using the recipe Installing the Neo4j Spatial extension, from this article. Restart the Neo4j graph database server using the following command: $NEO4J_ROOT_DIR/bin/neo4j restart How to do it... Since the Esri shapefile format is, by default, supported by the Neo4j Spatial extension, it is very easy to import data using the Java API from it using the following steps: Download a sample .shp file from http://www.statsilk.com/maps/download-free-shapefile-maps. Execute the following commands: wget http://biogeo.ucdavis.edu/data/diva/adm/AFG_adm.zip unzip AFG_adm.zip mv AFG_adm1.* /data The ShapefileImporter method lets you import data from the Esri shapefile using the following code: GraphDatabaseService esri_database = new GraphDatabaseFactory().newEmbeddedDatabase(storeDir); try {    ShapefileImporter importer = new ShapefileImporter(esri_database); importer.importFile("/data/AFG_adm1.shp", "layer_afganistan");        } finally {            esri_database.shutdown(); } Using similar code, we can import multiple SHP files into the same layer or different layers, as shown in the following code snippet: File dir = new File("/data");      FilenameFilter filter = new FilenameFilter() {          public boolean accept(File dir, String name) {      return name.endsWith(".shp"); }};   File[] listOfFiles = dir.listFiles(filter); for (final File fileEntry : listOfFiles) {      System.out.println("FileEntry Directory "+fileEntry);    try {    importer.importFile(fileEntry.toString(), "layer_afganistan"); } catch(Exception e){    esri_database.shutdown(); }} How it works... The Neo4j Spatial extension natively supports the import of data in the shapefile format. Using the ShapefileImporter method, any SHP file can be easily imported into Neo4j. The ShapefileImporter method takes two arguments: the first argument is the path to the SHP files and the second is the layer in which it should be imported. There's more… We will encourage you to read more about shapefiles and layers in general; for this, please visit the following URLs for more information: http://en.wikipedia.org/wiki/Shapefile http://wiki.openstreetmap.org/wiki/Shapefiles http://www.gdal.org/drv_shapefile.html Importing the OpenStreetMap files OpenStreetMap is a powerhouse of data when it comes to geospatial data. It is a collaborative project to create a free, editable map of the world. OpenStreetMap provides geospatial data in the .osm file format. To read more about .osm files in general, check out http://wiki.openstreetmap.org/wiki/.osm. In this recipe, you will learn how to import the .osm files in the Neo4j graph database. Getting ready Perform the following steps to get started with this recipe: Install Neo4j using the earlier recipies in this article. Install the Neo4j Spatial plugin using the recipe Installing the Neo4j Spatial extension, from this article. Restart the Neo4j graph database server: $NEO4J_ROOT_DIR/bin/neo4j restart How to do it... Since the OSM file format is, by default, supported by the Neo4j Spatial extension, it is very easy to import data from it using the following steps: Download one sample .osm file from http://wiki.openstreetmap.org/wiki/Planet.osm#Downloading. Execute the following commands: wget http://download.geofabrik.de/africa-latest.osm.bz2 bunzip2 africa-latest.osm.bz2 mv africa-latest.osm /data The importfile method lets you import data from the .osm file, as shown in the following code snippet: OSMImporter importer = new OSMImporter("africa"); try {    importer.importFile(osm_database, "/data/botswana-latest.osm", false, 5000, true); } catch(Exception e){    osm_database.shutdown(); } importer.reIndex(osm_database,10000); Using similar code, we can import multiple OSM files into the same layer or different layers, as shown here: File dir = new File("/data");      FilenameFilter filter = new FilenameFilter() {          public boolean accept(File dir, String name) {      return name.endsWith(".osm"); }}; File[] listOfFiles = dir.listFiles(filter); for (final File fileEntry : listOfFiles) {      System.out.println("FileEntry Directory "+fileEntry);    try {importer.importFile(osm_database, fileEntry.toString(), false, 5000, true); importer.reIndex(osm_database,10000); } catch(Exception e){    osm_database.shutdown(); } How it works... This is slightly more complex as it requires two phases: the first phase requires a batch inserter performing insertions into the database, and the second phase requires reindexing of the database with the spatial indexes. There's more… We will encourage you to read more about the OSM file and the batch inserter in general; for this, visit the following URLs: http://en.wikipedia.org/wiki/OpenStreetMap http://wiki.openstreetmap.org/wiki/OSM_file_formats http://neo4j.com/api_docs/2.0.2/org/neo4j/unsafe/batchinsert/BatchInserter.html Importing data using the REST API The recipes that you have learned until now consist of Java code, which is used to import spatial data into Neo4j. However, by using any other programming language, such as Python or Ruby, spatial data can be easily imported into Neo4j using the REST interface. In this recipe, you will learn how to import geospatial data using the REST interface. Getting ready Perform the following steps to get started with this recipe: Install Neo4j using the earlier recipies in this article. Install the Neo4j Spatial plugin using the recipe Installing the Neo4j Spatial extension, from this article. Restart the Neo4j graph database server: $NEO4J_ROOT_DIR/bin/neo4j restart How to do it... Using the REST interface is a very simple three-stage process to import the geospatial data into the Neo4j graph database server. For the sake of simplicity, the code of the Python language has been used to explain this recipe, although you can also use curl for this recipe: Create the spatial index, as shown in the following code: # Create geom index url = http://<neo4j_server_ip>:<port>/db/data/index/node/ payload= {    "name" : "geom",    "config" : {        "provider" : "spatial",        "geometry_type" : "point",        "lat" : "lat",        "lon" : "lon"    } } Create nodes as lat/lng and data as properties, as shown in the following code: url = "http://<neo4j_server_ip>:<port>/db/data/node" payload = {'lon': 38.6, 'lat': 67.88, 'name': 'abc'} req = urllib2.Request(url) req.add_header('Content-Type', 'application/json') response = urllib2.urlopen(req, json.dumps(payload)) node = json.loads(response.read())['self'] Add the preceding created node to the geospatial index, as shown in the following code snippet: #add node to geom index url = "http://<neo4j_server_ip>:<port>/db/data/index/node/geom" payload = {'value': 'dummy', 'key': 'dummy', 'uri': node} req = urllib2.Request(url) req.add_header('Content-Type', 'application/json') response = urllib2.urlopen(req, json.dumps(payload)) print response.read() The data will look like what is shown in the following screenshot after the addition of a few more nodes; this screenshot depicts the Neo4j Spatial data that has been imported: The following screenshot depicts the properties of a single node, which has been imported into Neo4j: How it works... Adding geospatial data using the REST API is a three-step process, listed as follows: Create a geospatial index using an endpoint, by following this URL as a template: http://<neo4j_server_ip>:<port>/db/data/index/node/ Add a node to the Neo4j graph database using an endpoint, by following this URL as a template: http://<neo4j_server_ip>:<port>/db/data/node Add the created node to the geospatial index using the endpoint, by following this URL as a template: http://<neo4j_server_ip>:<port>/db/data/index/node/geom There's more… We encourage you to read more about the spatial REST interfaces in general (http://neo4j-contrib.github.io/spatial/). Creating a point layer using the REST API In this recipe, you will learn how to create a point layer using the REST API interface. Getting ready Perform the following steps to get started with this recipe: Install Neo4j using the earlier recipies in this article. Install the Neo4j Spatial plugin using the recipe Installing the Neo4j Spatial extension, from this article. Restart the Neo4j graph database server using the following command: $NEO4J_ROOT_DIR/bin/neo4j restart How to do it... In this recipe, we will use the http://<neo4j_server_ip>:/db/data/ext/ SpatialPlugin/graphdb/addSimplePointlayer endpoint to create a simple point layer. Let's add a simple point layer, as shown in the following code: "layer" : "geom", "lat"   : "lat" , "lon"   : "lon",url = "http://<neo4j_server_ip>:<port>//db/data/ext/SpatialPlugin/graphdb/addSimplePointlayer payload= { "layer" : "geom", "lat"   : "lat" , "lon"   : "lon", } r = requests.post(url, data=json.dumps(payload), headers=headers) The data will look like what is shown in the following screenshot; this screenshot shows the output of the create point in layer query: How it works... Creating a point in the layer query is based on the REST interface, which the Neo4j Spatial plugin already provides with it. There's more… We will encourage you to read more about spatial REST interfaces in general; to do this, visit http://neo4j-contrib.github.io/spatial/. Finding geometries within the bounding box In this recipe, you will learn how to find all the geometries within the bounding box using the spatial REST interface. Getting ready Perform the following steps to get started with this recipe: Install Neo4j using the earlier recipies in this article. Install the Neo4j Spatial plugin using the recipe Installing the Neo4j Spatial extension, from this article. Restart the Neo4j graph database server using the following command: $NEO4J_ROOT_DIR/bin/neo4j restart How to do it... In this recipe, we will use the following endpoint to find all the geometries within the bounding box:http://<neo4j_server_ip>:<port>/db/data/ext/SpatialPlugin/graphdb/findGeometriesInBBox. Let's find the all the geometries, using the following information: "minx" : 0.0, "maxx" : 100.0, "miny" : 0.0, "maxy" : 100.0 url = "http://<neo4j_server_ip>:<port>//db/data/ext/SpatialPlugin/graphdb payload= { "layer" : "geom", "minx" : 0.0, "maxx" : 100.3, "miny" : 0.0, "maxy" : 100.0 } r = requests.post(url, data=json.dumps(payload), headers=headers) The data will look like what is shown in the following screenshot; this screenshot shows the output of the bounding box query: How it works... Finding geometries in the bounding box is based on the REST interface, which the Neo4j Spatial plugin provides. The output of the REST call contains an array of the nodes, containing the node's id, lat/lng, and its incoming/outgoing relationships. In the preceding output, you can see node id54 returned as the output. There's more… We will encourage you to read more about spatial REST interfaces in general; to do this, visit http://neo4j-contrib.github.io/spatial/. Finding geometries within a distance In this recipe, you will learn how to find all the geometries within a distance using the spatial REST interface. Getting ready Perform the following steps to get started with this recipe: Install Neo4j using the earlier recipies in this article. Install the Neo4j Spatial plugin using the recipe Installing the Neo4j Spatial extension, from this article. Restart the Neo4j graph database server using the following command: $NEO4J_ROOT_DIR/bin/neo4j restart How to do it... In this recipe, we will use the following endpoint to find all the geometries within a certain distance: http://<neo4j_server_ip>:<port>/db/data/ext/SpatialPlugin/graphdb/findGeometriesWithinDistance. Let's find all the geometries between the specified distance using the following information: "pointX" : -116.67, "pointY" : 46.89, "distanceinKm" : 500, url = "http://<neo4j_server_ip>:<port>//db/data/ext/SpatialPlugin/graphdb/findGeometriesWithinDistance payload= { "layer" : "geom", "pointY" : 46.8625, "pointX" : -114.0117, "distanceInKm" : 500, } r = requests.post(url, data=json.dumps(payload), headers=headers) The data will look like what is shown in the following screenshot; this screenshot shows the output of a withinDistance query: How it works... Finding geometries within a distance is based on the REST interface that the Neo4j Spatial plugin provides. The output of the REST call contains an array of the nodes, containing the node's id, lat/lng, and its incoming/outgoing relationships. In the preceding output, we can see node id71 returned as the output. There's more… We encourage you to read more about the spatial REST interfaces in general (http://neo4j-contrib.github.io/spatial/). Finding geometries within a distance using Cypher In this recipe, you will learn how to find all the geometries within a distance using the Cypher query. Getting ready Perform the following steps to get started with this recipe: Install Neo4j using the earlier recipies in this article. Install the Neo4j Spatial plugin using the recipe Installing the Neo4j Spatial extension, from this article. Restart the Neo4j graph database server: $NEO4J_ROOT_DIR/bin/neo4j restart How to do it... In this recipe, we will use the following endpoint to find all the geometries within a certain distance: http://<neo4j_server_ip>:<port>/db/data/cipher Let's find all the geometries within a distance using a Cypher query: "pointX" : -116.67, "pointY" : 46.89, "distanceinKm" : 500, url = "http://<neo4j_server_ip>:<port>//db/data/cypher payload= { "query" : "START n=node:geom('withinDistance:[46.9163, -114.0905, 500.0]') RETURN n" } r = requests.post(url, data=json.dumps(payload), headers=headers) The data will look like what is shown in the following screenshot; this screenshot shows the output of the withinDistance query that uses Cypher: The following is the Cypher output in the Neo4j console: How it works... Cypher comes with a withinDistance query, which takes three parameters: lat, lon, and search distance. There's more… We will encourage you to read more about the spatial REST interfaces in general (http://neo4j-contrib.github.io/spatial/). Summary Developing Location-based Services with Neo4j, teaches you the most important aspect of today's data, location, and how to deal with it in Neo4j. You have learnt how to import geospatial data into Neo4j and run queries, such as proximity searches, bounding boxes, and so on. Resources for Article:   Further resources on this subject: Recommender systems dissected Components [article] Working with a Neo4j Embedded Database [article] Differences in style between Java and Scala code [article]

(For more resources on JSF, see here.) JSF declarative event handling Starting with JSF 2.0 the event system has been really improved and the declarative event handling is exposed through a tag, f:event, and an annotation, @NamedEvent. In this recipe, you will see how to work with these two and how to subscribe to events like preRenderComponent, PostAddToView , and so on. Getting ready We developed this recipe with NetBeans 6.8, JSF 2.0, and GlassFish v3. The JSF 2.0 classes were obtained from the NetBeans JSF 2.0 bundled library. How to do it... Starting with the f:event tag, we can say that this is a simple tag that should be fitted in the right place and configured with its two simple attributes. Speaking of fitting it in the right place, you should know that f:event can be placed in any component that you want—for example we put it in an h:inputText component: <h:inputText value="#{bean.number}"> <f:event type="preRenderComponent" listener="#{bean.initNumber}" /></h:inputText>... As you can see there are two attributes of the f:event tag , named type and listener. The value of the type attribute represents the name of the event for which to install a listener (in our example, we have used the preRenderComponent value—with other words, before the component is rendered). In the following table are the possible values, and the corresponding event type for which the listener action is registered. Value for type attribute Type of event sent to listener method preRenderComponent javax.faces.event.PreRenderComponentEvent postAddToView javax.faces.event.PostAddToViewEvent preValidate javax.faces.event.PreValidateEvent postValidate javax.faces.event.PostValidateEvent The listener attribute's value represents a MethodExpression pointing to a method that will be called when the listener's processEvent method would have been called. In our example, that method is named initNumber and it can be seen in the following managed bean: package beans;import javax.faces.bean.ManagedBean;import javax.faces.bean.SessionScoped;@ManagedBean@SessionScopedpublic class Bean { private String number = ""; public Bean() { } public String getNumber() { return number; } public void setNumber(String number) { this.number = number; } public void initNumber(){ setNumber("2010"); }} While f:event works only with predefined events, the @NamedEvent provides support for exposing custom events. The application developer can make a custom event available to the page authors using the @NamedEvent annotation . This annotation can be placed on custom events to register them with the runtime, making them available to f:event. When the application starts, JSF scans for a set of annotations, including @NamedEvent. If it is found on a class, the following logic is applied to get the name/names for the event: Get the unqualified class name Cut off the trailing "Event", if present Convert the first character to lower-case Prepend the package name to the lower-cased name The preceding four rules are ignored if the shortName attribute is specified. In this case JSF registers the event by that name. URLs based on specified navigation outcome One of the most requested features in JSF 2.0 was a nice and smooth mechanism for achieving bookmarkability of JSF pages. As you will see in this recipe, this mechanism is finally provided by JSF 2.0 and is a very robust and easy-to-use solution. Getting ready We have developed this recipe with NetBeans 6.8, JSF 2.0, and GlassFish v3. The JSF 2.0 classes were obtained from the NetBeans JSF 2.0 bundled library. How to do it... Now, let's get into the subject, and let's say that the JSF 2.0 bookmarkability mechanism is based on two new tags, named h:link and h:button. These tags will generate a URL based on the specified navigation outcome. In JSF 2.0, we can make use of implicit navigation, therefore the outcome can be defined in the view or using common navigation rules. OK, enough theory, let's see an example: ...<h:link outcome="page2" value="HelloToYou"> <f:param name="helloparam" value="#{bean.hello}"/></h:link>... In the previous example, we assume no navigation rule, therefore the outcome attribute indicates a navigation to page2.xhtml (the FacesServlet is mapped to *.xhtml). The value attribute indicates text that will be rendered as a link in the page. The f:param will add a query parameter to the generated URL. The result of this component will be: http://localhost:8080/ URLs_based_on_specified_navigation_outcome/ faces/page2.xhtml?helloparam=Adrian The Adrian value comes from a simple managed bean: package beans;import javax.enterprise.context.RequestScoped;import javax.faces.bean.ManagedBean;@ManagedBean@RequestScopedpublic class Bean { private String hello = "Adrian"; public Bean() { } public String getHello() { return hello; } public void setHello(String hello) { this.hello = hello; }} You can bookmark this page at any moment and conserve the URL. The h:button works in the same manner except that it renders a button instead of a link. How it works... Before the user uses the component—clicks on the hyperlink—the current view ID and the specified outcome are used to find the target view ID. Afterwards, it is translated into a bookmarkable URL and used as the hyperlink's target. Note that this is true even if the user never activates the component. The target view ID is placed in the attribute named outcome on the new bookmarkable component tags, h:link or/and h:button (those components inherit from a component class named UIOutcomeTarget). Notice that you are not targeting a view ID directly, but rather a navigation outcome, which may be interpreted as a view ID if the matching falls through to implicit navigation. We consider that this is a good place and time to point out some methods of creating the query string parameters, therefore we present them in the order that they are processed: Implicit query string parameter View parameter (the <f:metadata> of the target view ID) Nested <f:param> in UIOutcomeTarget (such as, <h:link>) Nested <view-param> in the navigation case <redirect> element in facesconfig.xml

(For more resources related to this topic, see here.) Mouse versus tablet drawing If you're accustomed to drawing traditionally with a pen or pencil, you will discover quite quickly that drawing with a mouse requires a different skillset. The way a mouse moves, the difference in control, and the lack of intimacy can really take some time getting used to. While initially overwhelming, it is possible to map your mind towards mouse drawing. A graphic tablet is like a digital drawing pad that allows you to sketch on screen using a utensil that resembles a pen or pencil. What's nice is that Anime Studio was built to work with certain graphic tablets, thus making Plug and Play easy. We will be creating cartoon assets with a mouse. This is the most universal way as most users have this accessory for their computer. In addition, we cover both freehand and point drawing styles. We will be majorly using point drawing. Learning about Wacom tablets Wacom is a very well-known brand of graphic tablets which work well with Anime Studio. This is because Smith Micro Software teamed up with Wacom while building Anime Studio to deliver seamless compatibility. What's great about Wacom tablets is that they correspond to the amount of pressure you apply to your lines. For instance, if you apply a lot of pressure at the start of a line and then end the line with light pressure, you will see a difference in width just as you would with a real pen or pencil. This option can be turned off in Anime Studio, but most artists welcome it. If you're interested in tablet drawing, Wacom has many different tablets varying in size and features. You can visit www.wacom.com for more details. The following is the image of a Wacom tablet: Understanding the basics of vector and raster graphics Before we begin drawing in Anime Studio, it's important to understand the differences between vector and raster graphics. Anime Studio allows you to output both types of graphics, and each has its strengths and weaknesses. Vector drawings are created whenever you use a drawing tool in Anime Studio. This is also the main format for Adobe Flash, Toon Boom, and Adobe Illustrator. Vector format is a popular choice and has been dominating the Internet cartoon scene for several years. The following image is an example of a vector image. Notice how all the lines retain a sharp quality. Vector graphics tend to have smaller file sizes compared to equivalent raster graphics. This not only makes streaming embedded Shockwave Flash (SWF) easier, but also keeps your project files lower in size, thus freeing up more space on your hard drive and cutting down on load times. Raster or bitmap images are made up of pixels. Common file types include JPEG, BMP, PNG, and GIF. Basically, images you take with your camera, found on the Internet (at least the majority of them), or created in Adobe Photoshop are raster graphics. Raster graphics can be imported into Anime Studio and used for different functions. While they can contain great detail, raster graphics have many disadvantages when it comes to animation. As they are pixel-based, if you enlarge or zoom into a raster graphic past its original size, you will lose the image's quality. They also tend to bloat project file sizes up; this is due to the pixels needing more information to display the image. Many artists do find raster images worthwhile; additionally, you have the ability to convert raster images into vector graphics if desired. This method is called tracing, and while it can be useful, it's definitely not 100 percent effective, especially when trying to make the image work with animation. The following image is an example of a raster graphic. Compare it to the previous vector image. Note how the raster graphic appears blurry or pixelated in comparison. Now, you must be wondering which image type is the best. There is really no right or wrong answer to this question. It all comes down to personal preference and what you plan to do with your cartoon. We will explore a few uses of bitmap images, but the primary focus will be on creating vector art through the drawing tools. Exploring the Draw and Fill tools As we start working with the drawing tools in this article, it would be best for you to have a new document loaded up so that we have room to play around. In order to do that, navigate to File | New. New documents always open with a vector layer on the right-hand side Layers Panel, labeled Layer 1. This is perfect for us as all of the drawing tools require a vector layer to be used. Some drawing tools have features that can be adjusted at the top of the Anime Studio window. We will refer to this area as the top bar. The drawing tools are located on the left-hand side of your screen by default. The tools we will be looking at are divided into two panels: Draw and Fill. If you go in order while learning these tools, it may make sense, but we're simply too free-spirited for that. We will be going back and forth between these tools as some of them directly benefit the usage of others. Drawing shapes and lines with the Add Point tool The Add Point tool allows us to create lines and shapes using a series of points. All of Anime Studio's tools work with a point system, but this tool arguably gives you the most control in this regard. Points can then be moved or deleted depending on your needs. The following screenshot shows the location of the Add Point tool on the toolbar. As you can see, it looks like a curved line with a point at the end. You can also press the A key on your keyboard to select the tool. To get started, perform the following steps: Go to the top of your toolbar and click on the Add Point tool. Next, you will find a few options just below your File menu at the top of the Anime Studio program window. This is your top bar area. Please make sure Auto-Weld and Auto-Fill are both selected (this will be indicated by a check mark next to the corresponding option). Autowelding ensures that the two points we are joining will snap or weld together. Autofilling ensures that once two points are joined together to complete an enclosed object, the drawing will fill in with the colors from your Style palette. Try deselecting these options and redoing this exercise later on, to see what happens! On the right-hand side of your screen is the Style palette. Right below the title, you will see two colors, each labeled with Fill and Stroke. Click on the Fill color swatch and select a color of your choice from the options given. With the Color Picker window, you have the ability to click on a color, adjust the color range, modify transparency, as well as adjust your colors numerically for precise control. Once you have selected your color, click on the OK button. Now, select the Stroke color swatch and repeat the preceding steps. Try to pick a different color than that of the fill. The following screenshot shows the Style palette and Color Picker: Move your cursor somewhere on the blank canvas. Click and hold down the left button of your mouse, drag in any direction, and release. You should now see two points connected with a link. Now, we are simply seeing an outline, or reference for this object. No physical line has been created yet. Place your cursor on one of the two points. When correctly placed, your Add Point drawing tool will be highlighted in green. Now, click and hold down the left button of the mouse and drag anywhere to add to your line. If you keep the left button of the mouse pressed and move the point around, you should notice that the placement of this point affects the line curvature from the other two points. If you don't like this effect, you can always select the Sharp Corners option on the top of your window to create perfectly straight lines from point to point. Release the left button of the mouse once you've found a spot for your point. By repeating the preceding steps, you can continue to add interconnecting points to create an object; complex or simple, the choice is yours. If you desire, you can add points in between other established points by simply clicking on the line that interconnects them. To complete your object, you must overlap one point over another. Click the left button of your mouse, hold it, and drag the mouse to your first point. Once the area is highlighted in green, release the mouse button and notice how the object fills in with the colors you have selected from the Style palette. Have a look at the image in the following screenshot for an example: The Add Point tool offers a lot of control and is popular with mouse users. It may take some time to get used to, but if you prefer precision, practice will definitely pay off. This tool will be used quite a bit when we start drawing our assets. However, there are other tools that can get the job done, which we will be exploring momentarily. Freestyle drawing with the Freehand tool The Freehand tool allows us to draw in Anime Studio as if we were using a pen or pencil. This tool is a favorite amongst tablet users as it allows for absolute freedom of movement. It offers benefits for mouse users as well, especially if they plan to create a sense of stroke width variation. Just keep in mind, even though you can draw freely with this tool, you will still be creating points to make up your lines and objects, just like the Add Point tool. Just note that since Version 10, points will be hidden when using freehand drawing tools, to make the workspace less cluttered. In order to view and edit the points, you will need to select the Transform Points tool. The Freehand tool is the first tool in the second row (it looks like a pencil). You can also use the F key on your keyboard to select this tool. For your reference, you can see the location of this tool in the following screenshot: For this exercise, you can keep the document you created for the Add Point tool open. If you need more room to draw, feel free to create a new document. If you would like to save the current document to work on later, go to File and click on Save before creating a new document. Now, let's start drawing! The following steps will guide you on freestyle drawing with the Freehand tool: Click on the Freehand tool. At the top, where you have your tool options, be sure that Auto-Weld, Auto-Fill, and Auto-Stroke are checked. Before trying this tool out, let's check out some of the other options we can adjust with the Freehand tool. At the top, to the left-hand side of the Auto-Fill and Auto-Stroke settings, is a button labeled Freehand Options. Click on the button and a new panel will appear, as shown in the following screenshot: The Variable line width options allow you to change how the Freehand tool acts according to the pressure from your graphic tablet utensil. You can choose None, which will create a line with a consistent width; Use pen pressure, which detects how hard you are pressing on your tablet when drawing and adjusts the width accordingly (hard for thick, soft for light); or Random, which will randomize the line width as you place the points down. These options will work with a mouse, with the exception of the Use pen pressure setting. In the same panel, you can also adjust the percentage of variation of line width. The higher the percentage, the more dramatic a shift you will have for your line widths. Finally, you can dictate if you want your freehand lines to taper at the start and end. This can be useful, especially if you're using a mouse and want to simulate the freehand pressure-sensitive look. Once you have picked the appropriate options, let's start drawing! Place your cursor on the canvas, preferably outside of the other object you drew with the Add Point tool, hold down your left mouse button, and drag to create a line. You will notice that whichever settings you picked in the Freehand Options panel will be reflected in your line. Since we have selected Auto-Weld and Auto-Fill, we can automatically create closed objects. Try drawing an oval with the Freehand tool. Your beginning and end points should snap together, creating an enclosed and filled-in object. You can view an example of a line and shape with the Freehand tool in the following screenshot: If you are drawing with a tablet or are familiar with traditional drawing methods, the Freehand tool may be a better choice over the Add Point tool. As we start to draw characters and props, the Add Point tool will be referred to a lot. However, don't be afraid to use the Freehand tool in its place if that's what you're more comfortable with. You can always combine these tools too. The more options you have, the better!

In this article by Andrew Fawcett, the author of Force.com Enterprise Architecture, we will use the declarative aspects of the platform to quickly build an initial version of an application, which will give you an opportunity to get some hands-on experience with some of the packaging and installation features that are needed to release applications to subscribers. We will also take a look at the facilities available to publish your application through Salesforce AppExchange (equivalent to the Apple App Store) and finally provide end user support. (For more resources related to this topic, see here.) We will then use this application as a basis for incrementally releasing new versions of the application to build our understanding of Enterprise Application Development. The following topics outline what we will achieve in this article: Required organizations Introducing the sample application Package types and benefits Creating your first managed package Package dependencies and uploading Introduction to AppExchange and creating listings Installing and testing your package Becoming a Salesforce partner and its benefits Licensing Supporting your application Customer metrics Trialforce and Test Drive Required organizations Several Salesforce organizations are required to develop, package, and test your application. You can sign up for these organizations at https://developer.salesforce.com/, though in due course, as your relationship with Salesforce becomes more formal, you will have the option of accessing their Partner Portal website to create organizations of different types and capabilities. We will discuss more on this later. It's a good idea to have some kind of naming convention to keep track of the different organizations and logins. Use the following table as a guide and create the following organizations via https://developer.salesforce.com/. As stated earlier, these organizations will be used only for the purposes of learning and exploring: Username Usage Purpose myapp@packaging.my.com Packaging Though we will perform initial work in this org, it will eventually be reserved solely for assembling and uploading a release. myapp@testing.my.com Testing In this org, we will install the application and test upgrades. You may want to create several of these in practice, via the Partner Portal website described later in this article. myapp@dev.my.com Developing Later, we will shift development of the application into this org, leaving the packaging org to focus only on packaging. You will have to substitute myapp and my.com (perhaps by reusing your company domain name to avoid naming conflicts) with your own values. Take time to take note of andyapp@packaging.andyinthecloud.com. The following are other organization types that you will eventually need in order to manage the publication and licensing of your application. Usage Purpose Production / CRM Org Your organization may already be using this org for managing contacts, leads, opportunities, cases, and other CRM objects. Make sure that you have the complete authority to make changes, if any, to this org since this is where you run your business. If you do not have such an org, you can request one via the Partner Program website described later in this article, by requesting (via a case) a CRM ISV org. Even if you choose to not fully adopt Salesforce for this part of your business, such an org is still required when it comes to utilizing the licensing aspects of the platform. AppExchange Publishing Org (APO) This org is used to manage your use of AppExchange. We will discuss this a little later in this article. This org is actually the same Salesforce org you designate as your production org, where you conduct your sales and support activities from. License Management Org (LMO) Within this organization, you can track who installs your application (as leads), the licenses you grant to them, and for how long. It is recommended that this is the same org as the APO described earlier. Trialforce Management Org (TMO) Trialforce is a way to provide orgs with your preconfigured application data for prospective customers to try out your application before buying. It will be discussed later in this article. Trialforce Source Org (TSO)   Typically, the LMO and APO can be the same as your primary Salesforce production org, which allows you to track all your leads and future opportunities in the same place. This leads to the rule of APO = LMO = production org. Though neither of them should be your actual developer or test orgs. you can work with Salesforce support and your Salesforce account manager to plan and assign these orgs. Introducing the sample application For this article, we will use the world of Formula1 motor car racing as the basis for a packaged application that we will build together. Formula1 is for me the motor sport that is equivalent to Enterprise applications software, due to its scale and complexity. It is also a sport that I follow, both of which helped me when building the examples that we will use. We will refer to this application as FormulaForce, though please keep in mind Salesforce's branding policies when naming your own application, as they prevent the use of the word "Force" in the company or product titles. This application will focus on the data collection aspects of the races, drivers, and their many statistics, utilizing platform features to structure, visualize, and process this data in both historic and current contexts. For this article, we will create some initial Custom Objects as detailed in the following table. Do not worry about creating any custom tabs just yet. You can use your preferred approach for creating these initial objects. Ensure that you are logged in to your packaging org. Object Field name and type Season__c Name (text) Race__c Name (text) Season__c (Master-Detail to Season__c) Driver__c Name Contestant__c Name (Auto Number, CONTESTANT-{00000000} ) Race__c (Master-Detail to Race__c) Driver__c (Lookup to Driver__c) The following screenshot shows the preceding objects within the Schema Builder tool, available under the Setup menu: Package types and benefits A package is a container that holds your application components such as Custom Objects, Apex code, Apex triggers, Visualforce pages, and so on. This makes up your application. While there are other ways to move components between Salesforce orgs, a package provides a container that you can use for your entire application or deliver optional features by leveraging the so-called extension packages. There are two types of packages, managed and unmanaged. Unmanaged packages result in the transfer of components from one org to another; however, the result is as if those components had been originally created in the destination org, meaning that they can be readily modified or even deleted by the administrator of that org. They are also not upgradable and are not particularly ideal from a support perspective. Moreover, the Apex code that you write is also visible for all to see, so your Intellectual Property is at risk. Unmanaged packages can be used for sharing template components that are intended to be changed by the subscriber. If you are not using GitHub and the GitHub Salesforce Deployment Tool (https://github.com/afawcett/githubsfdeploy), they can also provide a means to share open source libraries to developers. Features and benefits of managed packages Managed packages have the following features that are ideal for distributing your application. The org where your application package is installed is referred to as a subscriber org, since users of this org are subscribing to the services your application provides: Intellectual Property (IP) protection: Users in the subscriber org cannot see your Apex source code, although they can see your Visualforce pages code and static resources. While the Apex code is hidden, JavaScript code is not, so you may want to consider using a minify process to partially obscure such code. The naming scope: Your component names are unique to your package throughout the utilization of a namespace. This means that even if you have object X in your application, and the subscriber has an object of the same name, they remain distinct. You will define a namespace later in this article. The governor scope: Code in your application executes within its own governor limit scope (such as DML and SOQL governors that are subject to passing Salesforce Security Review) and is not affected by other applications or code within the subscriber org. Note that some governors such as the CPU time governor are shared by the whole execution context (discussed in a later article) regardless of the namespace. Upgrades and versioning: Once the subscribers have started using your application, creating data, making configurations, and so on, you will want to provide upgrades and patches with new versions of your application. There are other benefits to managed packages, but these are only accessible after becoming a Salesforce Partner and completing the security review process; these benefits are described later in this article. Salesforce provides ISVForce Guide (otherwise known as the Packaging Guide) in which these topics are discussed in depth; bookmark it now! The following is the URL for ISVForce Guide: http://login.salesforce.com/help/pdfs/en/salesforce_packaging_guide.pdf. Creating your first managed package Packages are created in your packaging org. There can be only one managed package being developed in your packaging org (though additional unmanaged packages are supported, it is not recommended to mix your packaging org with them). You can also install other dependent managed packages and reference their components from your application. The steps to be performed are discussed in the following sections: Setting your package namespace Creating the package and assigning it to the namespace Adding components to the package Setting your package namespace An important decision when creating a managed package is the namespace; this is a prefix applied to all your components (Custom Objects, Visualforce pages, and so on) and is used by developers in subscriber orgs to uniquely qualify between your packaged components and others, even those from other packages. The namespace prefix is an important part of the branding of the application since it is implicitly attached to any Apex code or other components that you include in your package. It can be up to 15 characters, though I personally recommend that you keep it less than this, as it becomes hard to remember and leads to frustrating typos if you make it too complicated. I would also avoid underscore characters as well. It is a good idea to have a naming convention if you are likely to create more managed packages in the future (in different packaging orgs). The following is the format of an example naming convention: [company acronym - 1 to 4 characters][package prefix 1 to 4 characters] For example, the ACME Corporation's Road Runner application might be named acmerr. When the namespace has not been set, the Packages page (accessed under the Setup menu under the Create submenu) indicates that only unmanaged packages can be created. Click on the Edit button to begin a small wizard to enter your desired namespace. This can only be done once and must be globally unique (meaning it cannot be set in any other org), much like a website domain name. The following screenshot shows the Packages page: Once you have set the namespace, the preceding page should look like the following screenshot with the only difference being the namespace prefix that you have used. You are now ready to create a managed package and assign it to the namespace. Creating the package and assigning it to the namespace Click on the New button on the Packages page and give your package a name (it can be changed later). Make sure to tick the Managed checkbox as well. Click on Save and return to the Packages page, which should now look like the following: Adding components to the package In the Packages page, click on the link to your package in order to view its details. From this page, you can manage the contents of your package and upload it. Click on the Add button to add the Custom Objects created earlier in this article. Note that you do not need to add any custom fields; these are added automatically. The following screenshot shows broadly what your Package Details page should look like at this stage: When you review the components added to the package, you will see that some components can be removed while other components cannot be removed. This is because the platform implicitly adds some components for you as they are dependencies. As we progress, adding different component types, you will see this list automatically grow in some cases, and in others, we must explicitly add them. Extension packages As the name suggests, extension packages extend or add to the functionality delivered by the existing packages they are based on, though they cannot change the base package contents. They can extend one or more base packages, and you can even have several layers of extension packages, though you may want to keep an eye on how extensively you use this feature, as managing inter-package dependency can get quite complex to manage, especially during development. Extension packages are created in pretty much the same way as the process you've just completed (including requiring their own packaging org), except that the packaging org must also have the dependent packages installed in it. As code and Visualforce pages contained within extension packages make reference to other Custom Objects, fields, Apex code, and Visualforce pages present in base packages. The platform tracks these dependencies and the version of the base package present at the time the reference was made. When an extension package is installed, this dependency information ensures that the subscriber org must have the correct version (minimum) of the base packages installed before permitting the installation to complete. You can also manage the dependencies between extension packages and base packages yourself through the Versions tab or XML metadata for applicable components. Package dependencies and uploading Packages can have dependencies on platform features and/or other packages. You can review and manage these dependencies through the usage of the Package detail page and the use of dynamic coding conventions as described here. While some features of Salesforce are common, customers can purchase different editions and features according to their needs. Developer Edition organizations have access to most of these features for free. This means that as you develop your application, it is important to understand when and when not to use those features. By default, when referencing a certain Standard Object, field, or component type, you will generate a prerequisite dependency on your package, which your customers will need to have before they can complete the installation. Some Salesforce features, for example Multi-Currency or Chatter, have either a configuration or, in some cases, a cost impact to your users (different org editions). Carefully consider which features your package is dependent on. Most of the feature dependencies, though not all, are visible via the View Dependencies button on the Package details page (this information is also available on the Upload page, allowing you to make a final check). It is a good practice to add this check into your packaging procedures to ensure that no unwanted dependencies have crept in. Clicking on this button, for the package that we have been building in this article so far, confirms that there are no dependencies. Uploading the release and beta packages Once you have checked your dependencies, click on the Upload button. You will be prompted to give a name and version to your package. The version will be managed for you in subsequent releases. Packages are uploaded in one of two modes (beta or release). We will perform a release upload by selecting the Managed - Released option from the Release Type field, so make sure you are happy with the objects created in the earlier section of this article, as they cannot easily be changed after this point. Once you are happy with the information on the screen, click on the Upload button once again to begin the packaging process. Once the upload process completes, you will see a confirmation page as follows: Packages can be uploaded in one of two states as described here: Release packages can be installed into subscriber production orgs and also provide an upgrade path from previous releases. The downside is that you cannot delete the previously released components and change certain things such as a field's type. Changes to the components that are marked global, such as Apex Code and Visualforce components, are also restricted. While Salesforce is gradually enhancing the platform to provide the ability to modify certain released aspects, you need to be certain that your application release is stable before selecting this option. Beta packages cannot be installed into subscriber production orgs; you can install only into Developer Edition (such as your testing org), sandbox, or Partner Portal created orgs. Also, Beta packages cannot be upgraded once installed; hence, this is the reason why Salesforce does not permit their installation into production orgs. The key benefit is in the ability to continue to change new components of the release, to address bugs and features relating to user feedback. The ability to delete previously-published components (uploaded within a release package) is in pilot. It can be enabled through raising a support case with Salesforce Support. Once you have understood the full implications, they will enable it. We have simply added some Custom Objects. So, the upload should complete reasonably quickly. Note that what you're actually uploading to is AppExchange, which will be covered in the following sections. If you want to protect your package, you can provide a password (this can be changed afterwards). The user performing the installation will be prompted for it during the installation process. Optional package dependencies It is possible to make some Salesforce features and/or base package component references (Custom Objects and fields) an optional aspect of your application. There are two approaches to this, depending on the type of the feature. Dynamic Apex and Visualforce For example, the Multi-Currency feature adds a CurrencyIsoCode field to the standard and Custom Objects. If you explicitly reference this field, for example in your Apex or Visualforce pages, you will incur a hard dependency on your package. If you want to avoid this and make it a configuration option (for example) in your application, you can utilize dynamic Apex and Visualforce. Extension packages If you wish to package component types that are only available in subscriber orgs of certain editions, you can choose to include these in extension packages. For example, you may wish to support Professional Edition, which does not support record types. In this case, create an Enterprise Edition extension package for your application's functionality, which leverages the functionality from this edition. Note that you will need multiple testing organizations for each combination of features that you utilize in this way, to effectively test the configuration options or installation options that your application requires. Introduction to AppExchange and listings Salesforce provides a website referred to as AppExchange, which lets prospective customers find, try out, and install applications built using Force.com. Applications listed here can also receive ratings and feedback. You can also list your mobile applications on this site as well. In this section, I will be using an AppExchange package that I already own. The package has already gone through the process to help illustrate the steps that are involved. For this reason, you do not need to perform these steps; they can be revisited at a later phase in your development once you're happy to start promoting your application. Once your package is known to AppExchange, each time you click on the Upload button on your released package (as described previously), you effectively create a private listing. Private listings are not visible to the public until you decide to make them so. It gives you the chance to prepare any relevant marketing details and pricing information while final testing is completed. Note that you can still distribute your package to other Salesforce users or even early beta or pilot customers without having to make your listing public. In order to start building a listing, you need to log in to AppExchange using the login details you designated to your AppExchange Publishing Org (APO). Go to www.appexchange.com and click on Login in the banner at the top-right corner. This will present you with the usual Salesforce login screen. Once logged in, you should see something like this: Select the Publishing Console option from the menu, then click on the Create New Listing button and complete the steps shown in the wizard to associate the packaging org with AppExchange; once completed, you should see it listed. It's really important that you consistently log in to AppExchange using your APO user credentials. Salesforce will let you log in with other users. To make it easy to confirm, consider changing the user's display name to something like MyCompany Packaging. Though it is not a requirement to complete the listing steps, unless you want to try out the process yourself a little further to see the type of information required, you can delete any private listings that you created after you complete this app. Installing and testing your package When you uploaded your package earlier in this article, you will receive an e-mail with a link to install the package. If not, review the Versions tab on the Package detail page in your packaging org. Ensure that you're logged out and click on the link. When prompted, log in to your testing org. The installation process will start. A reduced screenshot of the initial installation page is shown in the following screenshot; click on the Continue button and follow the default installation prompts to complete the installation: Package installation covers the following aspects (once the user has entered the package password if one was set): Package overview: The platform provides an overview of the components that will be added or updated (if this is an upgrade) to the user. Note that due to the namespace assigned to your package, these will not overwrite existing components in the subscriber org created by the subscriber. Connected App and Remote Access: If the package contains components that represent connections to the services outside of the Salesforce services, the user is prompted to approve these. Approve Package API Access: If the package contains components that make use of the client API (such as JavaScript code), the user is prompted to confirm and/or configure this. Such components will generally not be called much; features such as JavaScript Remoting are preferred, and they leverage the Apex runtime security configured post install. Security configuration: In this step, you can determine the initial visibility of the components being installed (objects, pages, and so on). Selecting admin only or the ability to select Profiles to be updated. This option predates the introduction of permission sets, which permit post installation configuration. If you package profiles in your application, the user will need to remember to map these to the existing profiles in the subscriber org as per step 2. This is a one-time option, as the profiles in the package are not actually installed, only merged. I recommend that you utilize permission sets to provide security configurations for your application. These are installed and are much more granular in nature. When the installation is complete, navigate to the Installed Packages menu option under the Setup menu. Here, you can see confirmation of some of your package details such as namespace and version, as well as any licensing details, which will be discussed later in this article. It is also possible to provide a Configure link for your package, which will be displayed next to the package when installed and listed on the Installed Packages page in the subscriber org. Here, you can provide a Visualforce page to access configuration options and processes for example. If you have enabled Seat based licensing, there will also be a Manage Licenses link to determine which users in the subscriber org have access to your package components such as tabs, objects, and Visualforce pages. Licensing, in general, is discussed in more detail later in this article. Automating package installation It is possible to automate some of the processes using the Salesforce Metadata API and associated tools, such as the Salesforce Migration Toolkit (available from the Tools menu under Setup), which can be run from the popular Apache Ant scripting environment. This can be useful if you want to automate the deployment of your packages to customers or test orgs. Options that require a user response such as the security configuration are not covered by automation. However, password-protected managed packages are supported. You can find more details on this by looking up the Installed Package component in the online help for the Salesforce Metadata API at https://www.salesforce.com/us/developer/docs/api_meta/. As an aid to performing this from Ant, a custom Ant task can be found in the sample code related to this article (see /lib/antsalesforce.xml). The following is a /build.xml Ant script to uninstall and reinstall the package. Note that the installation will also upgrade a package if the package is already installed. The following is the Ant script: <project name="FormulaForce" basedir="."> <!-- Downloaded from Salesforce Tools page under Setup --> <typedef uri="antlib:com.salesforce" resource="com/salesforce/antlib.xml" classpath="${basedir}/lib/ant-salesforce.jar"/> <!-- Import macros to install/uninstall packages --> <import file="${basedir}/lib/ant-salesforce.xml"/> <target name="package.installdemo"> <uninstallPackage namespace="yournamespace" username="${sf.username}" password="${sf.password}"/> <installPackage namespace="yournamespace" version="1.0" username="${sf.username}" password="${sf.password}"/> </target> </project> You can try the preceding example with your testing org by replacing the namespace attribute values with the namespace you entered earlier in this article. Enter the following commands, all on one line, from the folder that contains the build.xml file: ant package.installdemo -Dsf.username=testorgusername -Dsf.password=testorgpasswordtestorgtoken You can also use the Salesforce Metadata API to list packages installed in an org, for example, if you wanted to determine whether a dependent package needs to be installed or upgraded before sending an installation request. Finally, you can also uninstall packages if you wish. Becoming a Salesforce partner and benefits The Salesforce Partner Program has many advantages. The first place to visit is http://www.salesforce.com/partners/overview. You will want to focus on the areas of the site relating to being an Independent Software Vendor (ISV) partner. From here, you can click on Join. It is free to join, though you will want to read through the various agreements carefully of course. Once you wish to start listing a package and charging users for it, you will need to arrange billing details for Salesforce to take the various fees involved. Pay careful attention to the Standard Objects used in your package, as this will determine the license type required by your users and the overall cost to them in addition to your charges. Obviously, Salesforce would prefer your application to use as many features of the CRM application as possible, which may also be beneficial to you as a feature of your application, since it's an appealing immediate integration not found on other platforms, such as the ability to instantly integrate with accounts and contacts. If you're planning on using Standard Objects and are in doubt about the costs (as they do vary depending on the type), you can request a conversation with Salesforce to discuss this; this is something to keep in mind in the early stages. Once you have completed the signup process, you will gain access to the Partner Portal (your user will end with @partnerforce.com). You must log in to the specific site as opposed to the standard Salesforce login; currently, the URL is https://www.salesforce.com/partners/login. Starting from July 2014, the http://partners.salesforce.com URL provides access to the Partner Community. Logging in to this service using your production org user credentials is recommended. The following screenshot shows what the current Partner Portal home page looks like. Here you can see some of its key features: This is your primary place to communicate with Salesforce and also to access additional materials and announcements relevant to ISVs, so do keep checking often. You can raise cases and provide additional logins to other users in your organization, such as other developers who may wish to report issues or ask questions. There is also the facility to create test or developer orgs; here, you can choose the appropriate edition (Professional, Group, Enterprise, and others) you want to test against. You can also create Partner Developer Edition orgs from this option as well. These carry additional licenses and limits over the public's so-called Single Developer Editions orgs and are thus recommended for use only once you start using the Partner Portal. Note, however, that these orgs do expire, subject to either continued activity over 6 months or renewing the security review process (described in the following section) each year. Once you click on the create a test org button, there is a link on the page displayed that navigates to a table that describes the benefits, processes, and the expiry rules. Security review and benefits The following features require that a completed package release goes through a Salesforce-driven process known as the Security review, which is initiated via your listing when logged into AppExchange. Unless you plan to give your package away for free, there is a charge involved in putting your package through this process. However, the review is optional. There is nothing stopping you from distributing your package installation URL directly. However, you will not be able to benefit from the ability to list your new application on AppExchange for others to see and review. More importantly, you will also not have access to the following features to help you deploy, license, and support your application. The following is a list of the benefits you get once your package has passed the security review: Bypass subscriber org setup limits: Limits such as the number of tabs and Custom Objects are bypassed. This means that if the subscriber org has reached its maximum number of Custom Objects, your package will still install. This feature is sometimes referred to as Aloha. Without this, your package installation may fail. You can determine whether Aloha has been enabled via the Subscriber Overview page that comes with the LMA application, which is discussed in the next section. Licensing: You are able to utilize the Salesforce-provided License Management Application in your LMO (License Management Org as described previously). Subscriber support: With this feature, the users in the subscriber org can enable, for a specific period, a means for you to log in to their org (without exchanging passwords), reproduce issues, and enable much more detailed debug information such as Apex stack traces. In this mode, you can also see custom settings that you have declared as protected in your package, which are useful for enabling additional debug or advanced features. Push upgrade: Using this feature, you can automatically apply upgrades to your subscribers without their manual intervention, either directly by you or on a scheduled basis. You may use this for applying either smaller bug fixes that don't affect the Custom Objects or APIs or deploy full upgrades. The latter requires careful coordination and planning with your subscribers to ensure that changes and new features are adopted properly. Salesforce asks you to perform an automated security scan of your software via a web page (http://security.force.com/security/tools/forcecom/scanner). This service can be quite slow depending on how many scans are in the queue. Another option is to obtain the Eclipse plugin from the actual vendor CheckMarx at http://www.checkmarx.com, which runs the same scan but allows you to control it locally. Finally, for the ultimate confidence as you develop your application, Salesforce can provide a license to integrate it into your Continuous Integration (CI) build system. Keep in mind that if you make any callouts to external services, Salesforce will also most likely ask you and/or the service provider to run a BURP scanner, to check for security flaws. Make sure you plan a reasonable amount of time (at least 2–3 weeks, in my experience) to go through the security review process; it is a must to initially list your package, though if it becomes an issue, you have the option of issuing your package install URL directly to initial customers and early adopters. Licensing Once you have completed the security review, you are able to request through raising support cases via the Partner Portal to have access to the LMA. Once this is provided by Salesforce, use the installation URL to install it like any other package into your LMO. If you have requested a CRM for ISV's org (through a case raised within the Partner Portal), you may find the LMA already installed. The following screenshot shows the main tabs of the License Management Application once installed: In this section, I will use a package that I already own and have already taken through the process to help illustrate the steps that are involved. For this reason, you do not need to perform these steps. After completing the installation, return to AppExchange and log in. Then, locate your listing in Publisher Console under Uploaded Packages. Next to your package, there will be a Manage Licenses link. The first time after clicking on this link, you will be asked to connect your package to your LMO org. Once this is done, you will be able to define the license requirements for your package. The following example shows the license for a free package, with an immediately active license for all users in the subscriber org: In most cases, for packages that you intend to charge for, you would select a free trial rather than setting the license default to active immediately. For paid packages, select a license length, unless perhaps it's a one-off charge, and then select the license that does not expire. Finally, if you're providing a trial license, you need to consider carefully the default number of seats (users); users may need to be able to assign themselves different roles in your application to get the full experience. While licensing is expressed at a package level currently, it is very likely that more granular licensing around the modules or features in your package will be provided by Salesforce in the future. This will likely be driven by the Permission Sets feature. As such, keep in mind a functional orientation to your Permission Set design. The Manage Licenses link is shown on the Installed Packages page next to your package if you configure a number of seats against the license. The administrator in the subscriber org can use this page to assign applicable users to your package. The following screenshot shows how your installed package looks to the administrator when the package has licensing enabled: Note that you do not need to keep reapplying the license requirements for each version you upload; the last details you defined will be carried forward to new versions of your package until you change them. Either way, these details can also be completely overridden on the License page of the LMA application as well. You may want to apply a site-wide (org-wide) active license to extensions or add-on packages. This allows you to at least track who has installed such packages even though you don't intend to manage any licenses around them, since you are addressing licensing on the main package. The Licenses tab and managing customer licenses The Licenses tab provides a list of individual license records that are automatically generated when the users install your package into their orgs. Salesforce captures this action and creates the relevant details, including Lead information, and also contains contact details of the organization and person who performed the install, as shown in the following screenshot: From each of these records, you can modify the current license details to extend the expiry period or disable the application completely. If you do this, the package will remain installed with all of its data. However, none of the users will be able to access the objects, Apex code, or pages, not even the administrator. You can also re-enable the license at any time. The following screenshot shows the License Edit section: The Subscribers tab The Subscribers tab lists all your customers or subscribers (it shows their Organization Name from the company profile) that have your packages installed (only those linked via AppExchange). This includes their organization ID, edition (Developer, Enterprise, or others), and also the type of instance (sandbox or production). The Subscriber Overview page When you click on Organization Name from the list in this tab, you are taken to the Subscriber Overview page. This page is sometimes known as the Partner Black Tab. This page is packed with useful information such as the contact details (also seen via the Leads tab) and the login access that may have been granted (we will discuss this in more detail in the next section), as well as which of your packages they have installed, its current licensed status, and when it was installed. The following is a screenshot of the Subscriber Overview page: How licensing is enforced in the subscriber org Licensing is enforced in one of two ways, depending on the execution context in which your packaged Custom Objects, fields, and Apex code are being accessed from. The first context is where a user is interacting directly with your objects, fields, tabs, and pages via the user interface or via the Salesforce APIs (Partner and Enterprise). If the user or the organization is not licensed for your package, these will simply be hidden from view, and in the case of the API, return an error. Note that administrators can still see packaged components under the Setup menu. The second context is the type of access made from Apex code, such as an Apex trigger or controller, written by the customers themselves or from within another package. This indirect way of accessing your package components is permitted if the license is site (org) wide or there is at least one user in the organization that is allocated a seat. This condition means that even if the current user has not been assigned a seat (via the Manage Licenses link), they are still accessing your application's objects and code, although indirectly, for example, via a customer-specific utility page or Apex trigger, which automates the creation of some records or defaulting of fields in your package. Your application's Apex triggers (for example, the ones you might add to Standard Objects) will always execute even if the user does not have a seat license, as long as there is just one user seat license assigned in the subscriber org to your package. However, if that license expires, the Apex trigger will no longer be executed by the platform, until the license expiry is extended. Providing support Once your package has completed the security review, additional functionality for supporting your customers is enabled. Specifically, this includes the ability to log in securely (without exchanging passwords) to their environments and debug your application. When logged in this way, you can see everything the user sees in addition to extended Debug Logs that contain the same level of details as they would in a developer org. First, your customer enables access via the Grant Account Login page. This time however, your organization (note that this is the Company Name as defined in the packaging org under Company Profile) will be listed as one of those available in addition to Salesforce Support. The following screenshot shows the Grant Account Login page: Next, you log in to your LMO and navigate to the Subscribers tab as described. Open Subscriber Overview for the customer, and you should now see the link to Login as that user. From this point on, you can follow the steps given to you by your customer and utilize the standard Debug Log and Developer Console tools to capture the debug information you need. The following screenshot shows a user who has been granted login access via your package to their org: This mode of access also permits you to see protected custom settings if you have included any of those in your package. If you have not encountered these before, it's well worth researching them as they provide an ideal way to enable and disable debug, diagnostic, or advanced configurations that you don't want your customers to normally see. Customer metrics Salesforce has started to expose information relating to the usage of your package components in the subscriber orgs since the Spring '14 release of the platform. This enables you to report what Custom Objects and Visualforce pages your customers are using and more importantly those they are not. This information is provided by Salesforce and cannot be opted out by the customer. At the time of writing, this facility is in pilot and needs to be enabled by Salesforce Support. Once enabled, the MetricsDataFile object is available in your production org and will receive a data file periodically that contains the metrics records. The Usage Metrics Visualization application can be found by searching on AppExchange and can help with visualizing this information. Trialforce and Test Drive Large enterprise applications often require some consultation with customers to tune and customize to their needs after the initial package installation. If you wish to provide trial versions of your application, Salesforce provides a means to take snapshots of the results of this installation and setup process, including sample data. You can then allow prospects that visit your AppExchange listing or your website to sign up to receive a personalized instance of a Salesforce org based on the snapshot you made. The potential customers can then use this to fully explore the application for a limited duration until they sign up to be a paid customer from the trial version. Such orgs will eventually expire when the Salesforce trial period ends for the org created (typically 14 days). Thus, you should keep this in mind when setting the default expiry on your package licensing. The standard approach is to offer a web form for the prospect to complete in order to obtain the trial. Review the Providing a Free Trial on your Website and Providing a Free Trial on AppExchange sections of the ISVForce Guide for more on this. You can also consider utilizing the Signup Request API, which gives you more control over how the process is started and the ability to monitor it, such that you can create the lead records yourself. You can find out more about this in the Creating Signups using the API section in the ISVForce Guide. Alternatively, if the prospect wishes to try your package in their sandbox environment for example, you can permit them to install the package directly either from AppExchange or from your website. In this case, ensure that you have defined a default expiry on your package license as described earlier. In this scenario, you or the prospect will have to perform the setup steps after installation. Finally, there is a third option called Test Drive, which does not create a new org for the prospect on request, but does require you to set up an org with your application, preconfigure it, and then link it to your listing via AppExchange. Instead of the users completing a signup page, they click on the Test Drive button on your AppExchange listing. This logs them into your test drive org as a read-only user. Because this is a shared org, the user experience and features you can offer to users is limited to those that mainly read information. I recommend that you consider Trialforce over this option unless there is some really compelling reason to use it. When defining your listing in AppExchange, the Leads tab can be used to configure the creation of lead records for trials, test drives, and other activities on your listing. Enabling this will result in a form being presented to the user before accessing these features on your listing. If you provide access to trials through signup forms on your website for example, lead information will not be captured. Summary This article has given you a practical overview of the initial package creation process through installing it into another Salesforce organization. While some of the features discussed cannot be fully exercised until you're close to your first release phase, you can now head to development with a good understanding of how early decisions such as references to Standard Objects are critical to your licensing and cost decisions. It is also important to keep in mind that while tools such as Trialforce help automate the setup, this does not apply to installing and configuring your customer environments. Thus, when making choices regarding configurations and defaults in your design, keep in mind the costs to the customer during the implementation cycle. Make sure you plan for the security review process in your release cycle (the free online version has a limited bandwidth) and ideally integrate it into your CI build system (a paid facility) as early as possible, since the tool not only monitors security flaws but also helps report breaches in best practices such as lack of test asserts and SOQL or DML statements in loops. As you revisit the tools covered in this article, be sure to reference the excellent ISVForce Guide at http://www.salesforce.com/us/developer/docs/packagingGuide/index.htm for the latest detailed steps and instructions on how to access, configure, and use these features. Resources for Article: Further resources on this subject: Salesforce CRM Functions [Article] Force.com: Data Management [Article] Configuration in Salesforce CRM [Article]

Before we continue, it is worth pointing out that at the moment of adding the basic functionality you are more than likely using the administrative user (user number 1) for all the site's development needs. That is absolutely fine, but once the major changes to the site are completed, you should begin using a normal administrative user that has only the permissions required to complete your day-to-day tasks. The next section will highlight the general philosophy behind user access, which should make the reason for this clear. Planning an Access Policy When you think about how your site should work, focus in on what will be required of yourself, other community members, or even anonymous users. For instance: Will there be a team of moderators working to ensure that the content of the site conforms to the dictates of good taste and avoids material that is tantamount to hate speech, and so on? Will there be subject experts who are allowed to create and maintain their own content? How much will anonymous visitors be allowed to become involved, or will they be forced to merely window shop without being able to contribute? Some of you might feel that the site should grow organically with the community, and so you want to be extremely flexible in your approach. However, you can take it as given that Drupal's access policies are already flexible, given how easy it is to reconfigure, so it is good practice to start out with a sensible set of access rules, even if they are going to change over time. If you need to make modifications later, so be it, but at least there will be a coherent set of rules from the start. The first and foremost rule of security that can be applied directly to our situation is Grant a user permissions sufficient for completing the intended task, and no more! Our entire approach is going to be governed by this rule. With a bit of thought you should be able to see why this is so important. The last thing anyone wants is for an anonymous user to be able to modify the personal blog of a respected industry expert. This means that each type of user should have carefully controlled permissions that effectively block their ability to act outside the scope of their remit. One upshot of this is that it is better to create a larger number of specific roles, rather than create a generic role or two, and allow everyone to use those catch-all permissions. A role constitutes a number of permissions that define what actions any members of that role can and can't perform. We will explore roles in detail in the next section! Drupal gives us fine-grained control over what users can accomplish, and you should make good use of this facility. It may help to think of your access control using the following figure (this does not necessarily represent the actual roles on your site—it's just an example): The shaded region represents the total number of permissions available for the site. Contained within this set are the various roles that exist either by default, like the Anonymous users role, or those you create in order to cater for the different types of users the site will require—in this case, the Blog Writer users and Forum Moderator users roles. From the previous diagram you can see that the Anonymous users role has the smallest set of permissions because they have the smallest area of the total diagram. This set of permissions is totally encapsulated by the Forum Moderator users and Blog Writer users—meaning that forum moderators and blog writers can do everything an anonymous user does, and a whole lot more. Remember, it is not compulsory that forum moderators encapsulate all the permissions of the anonymous users. You can assign any permissions to any role—it's just that in this context it makes sense that a forum moderator should be able to do everything an anonymous user can and more. Of course, the blog writers have a slightly different remit. While they share some privileges in common with the forum administrators, they also have a few of their own. Your permissions as the primary or administrative user encompass the entire set, because there should be nothing that you cannot control. It is up to you to decide which roles are best for the site, but before attempting this it is important to ask: What are roles and how are they used in the first place? To answer this question, let's take a look at the practical side of things in more detail. Roles It may seem a bit odd that we are not beginning a practical look at access control with a discussion on users. After all, it is all about what users can and cannot do! The problem with immediately talking about users is that the focus of a single user is too narrow, and we can learn far more about controlling access by taking a more broad view using roles. Once we have learned everything there is to know about roles, actually working with users becomes a trivial matter. As mentioned, a user role in Drupal defines a set of rules that must be obeyed by all the users in that role. It may be helpful to think of a role as a character in a play. In a play, an actor must always be true to their character (in the same way a user must be faithful to their role in Drupal)—in other words, there is a defined way to behave and the character never deviates (no matter which actor portrays the character). Creating a role in Drupal is very easy. Click the User management link under Administer and select the Roles tab to bring up the following: As you can see, we have two roles already defined by default—the anonymous user and the authenticated user. It is not possible to change these, and so the Operations column is permanently set to locked. To begin with, the anonymous user (this is any user who is browsing the site without logging in) has very few permissions set, and you would more than likely want to keep it this way, despite the fact it is possible to give them any and all permissions. Similarly, the authenticated user, by default, has only a few more permissions than the anonymous user, and it is also sensible to keep these to a minimum. We will see in a little while how to go about deciding who should have which permissions. In order to add a new role, type in a name for the role and click Add role, and you're done. But what name do you want to add? That's the key question! If you are unsure about what name to use, then it is most likely you haven't defined the purpose of the role properly. To see how this is done, let's assume we require a forum moderator who will be a normal user in every way, except for the ability to work directly on the forums (to take some of the burden of responsibility off the administrator's hands) to create new topics, and to edit the content if necessary. To get the ball rolling, type in forum moderator and click Add role—actually, you might even want to be more specific and use something like conservation forum moderator if there will be teams of forum moderators—you get the general idea. Now the roles page should display the new role with the option to edit it, shown in the Operations column. Click edit role in order to change the name of the role or delete it completely. Alternatively, click edit permissions to deal with the permissions for this specific role (we discuss permissions in a moment so let's leave this for now). Our work is just beginning, because now we need to grant or deny the various permissions that the forum moderator role will need in order to successfully fulfill its purpose. New roles are not given any permission at all to begin with—this makes sense, because the last thing we want is to create a role only to find that it has the same permissions as the administrative user. Chances are you will need to add several roles depending on the needs of the site, so add at least a blogger user that can edit their own blog—we will need a few different types to play with later on. Let's move on and take a look at how to flesh out this new role by setting permissions. Permissions In order to work with permissions, click the Permissions link under User management and you should be presented with a screen much like the following (notice the new forum moderator role on the right-hand side of the page): As you can see, this page lists all of the available permissions down the left-hand column and allows you to enable or disable that permission by checking or un-checking boxes in the relevant column. It is easy enough to see that one traverses the list, selecting those permissions required for each role. What is not so easy is actually determining what should and shouldn't be enabled in the first place. Notice too that the permissions given in the list on the left-hand side pertain to specific modules. This means that if we change the site's setup by adding or removing modules, then we will also have to change the permissions on this page. Most times a module is added, you will need to ensure that the permissions are set as required for that module, because by default no permissions are granted. What else can we learn from the permissions page shown in the previous screenshot? Well, what does each permission precisely mean? There are quite a few verbs that allow for completely different actions. The following lists the more common, generic ones, although you might find one or two others crop up every now and then to cater for a specific module: administer: gives the user the ability to affect the function of a module. For example, granting administer rights to the locale module means that the user can add or remove languages, manage strings, and even export .po files. This permission should only ever be given to trusted users, and never to anonymous users. access: gives the user the ability to make use of a module without being able to affect it in any way. For example, granting access rights to the comment module allows a user to view comments without being able to delete, edit, or reply to them. create: gives the user the ability to create content of some sort. For example, granting rights to create stories allows users to do so, but does not also give them the ability to edit those stories. edit any/own: gives the user the ability to work with either anyone's content or specifically the content they have created—depending on whether edit any or edit own is selected. For example, granting edit own rights to the blog module means that the user can modify their own blogs at will. delete any/own: applies to content related modules such as Node and empowers users to remove either anyone's content or confine them to removing only content posted by themselves. For example, setting delete own blog entry allows users to take back any blog postings they may regret having published. There are also other module-specific permissions available, and it is recommended that you play around and understand any new permission(s) you set. Previously, assigning the edit own permission automatically provided the delete own permission. For added security, delete own permissions for individual core content types have been removed from all roles and should be assigned separately. How do we go about setting up the required permissions for the forum moderator user? If we look down the list of permissions shown on the Permission page, we see the following forum-related options (at the moment, the forum moderator permissions are those in the outermost column): Enabling these three options, and then testing out what new powers are made available, should quickly demonstrate that this is not quite what we want. If you are wondering how to actually test this out, you need to create a new user and then assign them to the forum moderator role. The following section on Users explains how to create new users and administer them properly. Jump ahead quickly and check that out so that you have a new user to work with if you are unsure how it is done. The following point might make your life a bit easier: Use two browsers to test out your site. The demo site's development machine has IE and Firefox. Keep one browser for the administrator and the other for anonymous or other users in order to test out changes. This will save you from having to log in and log out whenever testing new permissions. When testing out the new permissions one way or another, you will find that the forum moderator can access and work with all of the forums—assuming you have created any. However, notice that there are node module permissions available, which is quite interesting because most content in Drupal is actually a node. How will this affect the forum moderator? Disable the forum module permissions for the forum moderator user and then enable all the node options for the authenticated user before saving and logging out. Log back in as the forum administrator and it will be clear that despite having revoked the forum based options for this user, it is possible to post to or edit anything in the forum quite easily by selecting the Create content link in the main menu. Is this what you expected? It should be precisely what you expect because the forum moderator is an authenticated user, so they have acquired the permissions that came from the authenticated user. In addition, the forum posts are all nodes, and any authenticated user can add and edit nodes, so even though the forum moderator is not explicitly allowed to work with forums, through generic node permissions we get the same result: Defined roles are given the authenticated user permissions. Actually, the result is not entirely the same because the forum moderator can now also configure all the different types of content on the site, as well as edit any type of content including other people's blogs. This is most certainly undesirable, so log back in as the primary user and remove the node permissions (except the first one) from the authenticated user role. With that done, you can now spend some time building a fairly powerful and comprehensive role-based access control plan. As an addendum, you might find that despite having a goodly amount of control over who does what, there are some things that are not easily done without help from elsewhere. Users A single user account can be given as many or as few permissions as you like via the use of roles. Drupal users are not really anything unless they already have a role that defines the manner in which they can operate within the Drupal framework. Hence, we discussed roles first. Users can be created in two ways. The most common way is by registering on the site—if you haven't already, go ahead and register a new user on your site by clicking the Create new account link on the homepage just to test things out. Remember to supply a valid email address otherwise you won't be able to sign in properly. This will create an authenticated user, with any and all permissions that have been assigned to the authenticated user role. The second way is to use the administrative user to create a new user. In order to do so, log on as the administrative user and click on Users in User management under Administer. Select the Add user tab and follow the instructions on that page. For example, I created a new forum moderator user by ensuring that the relevant role was checked: You will need to supply Drupal with usernames, email addresses, and passwords. Once there are a couple of users to play around with, it's time to begin working with them. Administering Users The site's administrator is given complete access to the other users' account information. By clicking on the edit link shown to the right of each user account (under the Operations column heading) in the Users page under User management, it is possible to make any changes you require to a given user. Before we do though, it's worth noting that the administration page itself is fairly powerful in terms of being able to administer individual users or groups of users with relative ease: The upper box, Show only users where, allows you to specify several filter conditions to cut down the result set and make it more manageable. This will become more and more important as the site accumulates more and more users. Once the various filter options have been implemented, the Update options allow you to apply whatever changes are needed to the list of users selected (by checking the relevant checkbox next to their name). Having both broad, sweeping powers as well as fine-grained control over users is one of the most valuable facilities provided by Drupal, and you will no doubt become very familiar with this page in due course. Click on the edit link next to the forum moderator user and take a look at the Roles section. Notice that it is now possible to stipulate which roles this user belongs to. At present there are only two new roles to be assigned (yours might vary depending on which roles have been created on your setup): Whenever a user is added to another role, they obtain the combined permissions of these roles. With this in mind, you should go about delegating roles in the following fashion: Define the most basic user of the site by setting the anonymous user permissions. Set permissions for a basic authenticated user (i.e. any Tom, Dick or Harry that registers on your site). Create special roles by only adding the specific additional permissions that are required by that role, and no more. Don't re-assign permissions that the authenticated user already has. Create new users by combining whatever roles are required for their duties or needs.

(For more resources related to this topic, see here.) Time handling features are an important part of every BI system. Programming languages, database systems, they all incorporate various time-related functions and Microsoft SQL Server Analysis Services (SSAS) is no exception there. In fact, that's one of its main strengths. The MDX language has various time-related functions designed to work with a special type of dimension called the Time and its typed attributes. While it's true that some of those functions work with any type of dimension, their usefulness is most obvious when applied to time-type dimensions. An additional prerequisite is the existence of multi-level hierarchies, also known as user hierarchies, in which types of levels must be set correctly or some of the time-related functions will either give false results or will not work at all. In this article we're dealing with typical operations, such as year-to-date calculations, running totals, and jumping from one period to another. We go into detail with each operation, explaining known and less known variants and pitfalls. We will discuss why some time calculations can create unnecessary data for the periods that should not have data at all, and why we should prevent it from happening. We will then show you how to prevent time calculations from having values after a certain point in time. In most BI projects, there are always reporting requirements to show measures for today, yesterday, month-to-date, quarter-to-date, year-to-date, and so on. We have three recipes to explore various ways to calculate today's date, and how to turn it into a set and use MDX's powerful set operations to calculate other related periods. Calculating date and time spans is also a common reporting requirement. Calculating the YTD (Year-To-Date) value In this recipe we will look at how to calculate the Year-To-Date value of a measure, that is, the accumulated value of all dates in a year up to the current member on the date dimension. An MDX function YTD() can be used to calculate the Year-To-Date value, but not without its constraints. In this recipe, we will discuss the constraints when using the YTD() function and also the alternative solutions. Getting ready Start SQL Server Management Studio and connect to your SSAS 2012 instance. Click on the New Query button and check that the target database is Adventure Works DW 2012. In order for this type of calculation to work, we need a dimension marked as Time in the Type property, in the Dimension Structure tab of SSDT. That should not be a problem because almost every database contains at least one such dimension and Adventure Works is no exception here. In this example, we're going to use the Date dimension. We can verify in SSDT that the Date dimension's Type property is set to Time. See the following screenshot from SSDT: Here's the query we'll start from: SELECT{ [Measures].[Reseller Sales Amount] } ON 0,{ [Date].[Calendar Weeks].[Calendar Week].MEMBERS } ON 1FROM[Adventure Works] Once executed, the preceding query returns reseller sales values for every week in the database. How to do it... We are going to use the YTD() function, which takes only one member expression, and returns all dates in the year up to the specified member. Then we will use the aggregation function SUM() to sum up the Reseller Sales Amount. Follow these steps to create a calculated measure with YTD calculation: Add the WITH block of the query. Create a new calculated measure within the WITH block and name it Reseller Sales YTD. The new measure should return the sum of the measure Reseller Sales Amount using the YTD() function and the current date member of the hierarchy of interest. Add the new measure on axis 0 and execute the complete query: WITH MEMBER [Measures].[Reseller Sales YTD] AS Sum( YTD( [Date].[Calendar Weeks].CurrentMember ), [Measures].[Reseller Sales Amount] ) SELECT { [Measures].[Reseller Sales Amount], [Measures].[Reseller Sales YTD] } ON 0, { [Date].[Calendar Weeks].[Calendar Week].MEMBERS } ON 1 FROM [Adventure Works] The result will include the second column, the one with the YTD values. Notice how the values in the second column increase over time: How it works... The YTD() function returns the set of members from the specified date hierarchy, starting from the first date of the year and ending with the specified member. The first date of the year is calculated according to the level [Calendar Year] marked as Years type in the hierarchy [Calendar Weeks]. In our example, the YTD() value for the member Week 9 CY 2008 is a set of members starting from Week 1 CY 2008 and going up to that member because the upper level containing years is of the Years type. The set is then summed up using the SUM() function and the Reseller Sales Amount measure. If we scroll down, we'll see that the cumulative sum resets every year, which means that YTD() works as expected. In this example we used the most common aggregation function, SUM(), in order to aggregate the values of the measure throughout the calculated set. SUM() was used because the aggregation type of the Reseller Sales Amount measure is Sum. Alternatively, we could have used the Aggregate() function instead. More information about that function can be found later in this recipe. There's more... Sometimes it is necessary to create a single calculation that will work for any user hierarchy of the date dimension. In that case, the solution is to prepare several YTD() functions, each using a different hierarchy, cross join them, and then aggregate that set using a proper aggregation function (Sum, Aggregate, and so on). However, bear in mind that this will only work if all user hierarchies used in the expression share the same year level. In other words, that there is no offset in years among them (such as exists between the fiscal and calendar hierarchies in Adventure Works cube in 2008 R2). Why does it have to be so? Because the cross join produces the set intersection of members on those hierarchies. Sets are generated relative to the position when the year starts. If there is offset in years, it is possible that sets won't have an intersection. In that case, the result will be an empty space. Now let's continue with a couple of working examples. Here's an example that works for both monthly and weekly hierarchies: WITHMEMBER [Measures].[Reseller Sales YTD] ASSum( YTD( [Date].[Calendar Weeks].CurrentMember ) *YTD( [Date].[Calendar].CurrentMember ),[Measures].[Reseller Sales Amount] )SELECT{ [Measures].[Reseller Sales Amount],[Measures].[Reseller Sales YTD] } ON 0,{ [Date].[Calendar Weeks].[Calendar Week].MEMBERS } ON 1FROM[Adventure Works] If we replace [Date].[Calendar Weeks].[Calendar Week].MEMBERS with [Date].[Calendar].[Month].MEMBERS, the calculation will continue to work. Without the cross join part, that wouldn't be the case. Try it in order to see for yourself! Just be aware that if you slice by additional attribute hierarchies, the calculation might become wrong. In short, there are many obstacles to getting the time-based calculation right. It partially depends on the design of the time dimension (which attributes exist, which are hidden, how the relations are defined, and so on), and partially on the complexity of the calculations provided and their ability to handle various scenarios. A better place to define time-based calculation is the MDX script. There, we can define scoped assignments, but that's a separate topic which will be covered later in the recipe, Using utility dimension to implement time-based calculations. In the meantime, here are some articles related to that topic: http://tinyurl.com/MoshaDateCalcs http://tinyurl.com/DateToolDim Inception-To-Date calculation A similar calculation is the Inception-To-Date calculation in which we're calculating the sum of all dates up to the current member, that is, we do not perform a reset at the beginning of every year. In that case, the YTD() part of the expression should be replaced with this: Null : [Date].[Calendar Weeks].CurrentMember Using the argument in the YTD() function The argument of the YTD() function is optional. When not specified, the first dimension of the Time type in the measure group is used. More precisely, the current member of the first user hierarchy with a level of type Years. This is quite convenient in the case of a simple Date dimension; a dimension with a single user hierarchy. In the case of multiple hierarchies or a role-playing dimension, the YTD() function might not work, if we forget to specify the hierarchy for which we expect it to work. This can be easily verified. Omit the [Date].[Calendar Weeks].CurrentMember part in the initial query and see that both columns return the same values. The YTD() function is not working anymore. Therefore, it is best to always use the argument in the YTD() function. Common problems and how to avoid them In our example we used the [Date].[Calendar Weeks] user hierarchy. That hierarchy has the level Calendar Year created from the same attribute. The type of attribute is Years, which can be verified in the Properties pane of SSDT: However, the Date dimension in the Adventure Works cube has fiscal attributes and user hierarchies built from them as well. The fiscal hierarchy equivalent to [Date].[Calendar Weeks] hierarchy is the [Date].[Fiscal Weeks] hierarchy. There, the top level is named Fiscal Year, created from the same attribute. This time, the type of the attribute is FiscalYear, not Year. If we exchange those two hierarchies in our example query, the YTD() function will not work on the new hierarchy. It will return an error: The name of the solution is the PeriodsToDate() function. YTD() is in fact a short version of the PeriodsToDate() function, which works only if the Year type level is specified in a user hierarchy. When it is not so (that is, some BI developers tend to forget to set it up correctly or in the case that the level is defined as, let's say, FiscalYear like in this test), we can use the PeriodsToDate() function as follows: MEMBER [Measures].[Reseller Sales YTD] ASSum( PeriodsToDate( [Date].[Fiscal Weeks].[Fiscal Year],[Date].[Fiscal Weeks].CurrentMember ),[Measures].[Reseller Sales Amount] ) PeriodsToDate() might therefore be used as a safer variant of the YTD() function. YTD() and future dates It's worth noting that the value returned by a SUM-YTD combination is never empty once a value is encountered in a particular year. Only the years with no values at all will remain completely blank for all their descendants. In our example with the [Calendar Weeks] hierarchy, scrolling down to the Week 23 CY 2008, you will see that this is the last week that has reseller sales. However, the Year-To-Date value is not empty for the rest of the weeks for year 2008, as shown in the following screenshot: This can cause problems for the descendants of the member that represents the current year (and future years as well). The NON EMPTY keyword will not be able to remove empty rows, meaning we'll get YTD values in the future. We might be tempted to use the NON_EMPTY_BEHAVIOR operator to solve this problem but it wouldn't help. Moreover, it would be completely wrong to use it, because it is only a hint to the engine which may or may not be used. It is not a mechanism for removing empty values. In short, we need to set some rows to null, those positioned after the member representing today's date. We'll cover the proper approach to this challenge in the recipe, Finding the last date with data. Calculating the YoY (Year-over-Year) growth (parallel periods) This recipe explains how to calculate the value in a parallel period, the value for the same period in a previous year, previous quarter, or some other level in the date dimension. We're going to cover the most common scenario – calculating the value for the same period in the previous year, because most businesses have yearly cycles. A ParallelPeriod() is a function that is closely related to time series. It returns a member from a prior period in the same relative position as a specified member. For example, if we specify June 2008 as the member, Year as the level, and 1 as the lag, the ParallelPeriod() function will return June 2007. Once we have the measure from the prior parallel period, we can calculate how much the measure in the current period has increased or decreased with respect to the parallel period's value. Getting ready Start SQL Server Management Studio and connect to your SSAS 2012 instance. Click on the New Query button, and check that the target database is Adventure Works DW 2012. In this example we're going to use the Date dimension. Here's the query we'll start from: SELECT{ [Measures].[Reseller Sales Amount] } ON 0,{ [Date].[Fiscal].[Month].MEMBERS } ON 1FROM[Adventure Works] Once executed, the previous query returns the value of Reseller Sales Amount for all fiscal months. How to do it... Follow these steps to create a calculated measure with YoY calculation: Add the WITH block of the query. Create a new calculated measure there and name it Reseller Sales PP. The new measure should return the value of the measure Reseller Sales Amount measure using the ParallelPeriod() function. In other words, the definition of the new measure should be as follows: MEMBER [Measures].[Reseller Sales PP] As( [Measures].[Reseller Sales Amount],ParallelPeriod( [Date].[Fiscal].[Fiscal Year], 1,[Date].[Fiscal].CurrentMember ) ) Specify the format string property of the new measure to match the format of the original measure. In this case that should be the currency format. Create the second calculated measure and name it Reseller Sales YoY %. The definition of that measure should be the ratio of the current member's value against the parallel period member's value. Be sure to handle potential division by zero errors (see the recipe Handling division by zero errors). Include both calculated measures on axis 0 and execute the query, which should look like: WITHMEMBER [Measures].[Reseller Sales PP] As( [Measures].[Reseller Sales Amount],ParallelPeriod( [Date].[Fiscal].[Fiscal Year], 1,[Date].[Fiscal].CurrentMember ) ), FORMAT_STRING = 'Currency'MEMBER [Measures].[Reseller Sales YoY %] Asiif( [Measures].[Reseller Sales PP] = 0, null,( [Measures].[Reseller Sales Amount] /[Measures].[Reseller Sales PP] ) ), FORMAT_STRING = 'Percent'SELECT{ [Measures].[Reseller Sales Amount],[Measures].[Reseller Sales PP],[Measures].[Reseller Sales YoY %] } ON 0,{ [Date].[Fiscal].[Month].MEMBERS } ON 1FROM[Adventure Works] The result will include two additional columns, one with the PP values and the other with the YoY change. Notice how the values in the second column repeat over time and that YoY % ratio shows the growth over time: How it works... The ParallelPeriod() function takes three arguments, a level expression, an index, and a member expression, and all three arguments are optional. The first argument indicates the level on which to look for that member's ancestor, typically the year level like in this example. The second argument indicates how many members to go back on the ancestor's level, typically one, as in this example. The last argument indicates the member for which the function is to be applied. Given the right combination of arguments, the function returns a member that is in the same relative position as a specified member, under a new ancestor. The value for the parallel period's member is obtained using a tuple which is formed with a measure and the new member. In our example, this represents the definition of the PP measure. The growth is calculated as the ratio of the current member's value over the parallel period member's value, in other words, as a ratio of two measures. In our example, that was YoY % measure. In our example we've also taken care of a small detail, setting the FORMAT_STRING to Percent. There's more... The ParallelPeriod() function is very closely related to time series, and typically used on date dimensions. However, it can be used on any type of dimension. For example, this query is perfectly valid: SELECT{ [Measures].[Reseller Sales Amount] } ON 0,{ ParallelPeriod( [Geography].[Geography].[Country],2,[Geography].[Geography].[State-Province].&[CA]&[US] ) } ON 1FROM[Adventure Works] The query returns Hamburg on rows, which is the third state-province in the alphabetical list of states-provinces under Germany. Germany is two countries back from the USA, whose member California, used in this query, is the third state-province underneath that country in the Geography.Geography user hierarchy. We can verify this by browsing the Geography user hierarchy in the Geography dimension in SQL Server Management Studio, as shown in the following screenshot. The UK one member back from the USA, has only one state-province: England. If we change the second argument to 1 instead, we'll get nothing on rows because there's no third state-province under the UK. Feel free to try it: All arguments of the ParallelPeriod() function are optional. When not specified, the first dimension of type Time in the measure group is used, more precisely, the previous member of the current member's parent. This can lead to unexpected results as discussed in the previous recipe. Therefore, it is recommended that you use all the arguments of the ParallelPeriod() function. ParallelPeriod is not a time-aware function The ParallelPeriod() function simply looks for the member from the prior period based on its relative position to its ancestor. For example, if your hierarchy is missing the first six months in the year 2005, for member January 2006, the function will find July 2005 as its parallel period (lagging by one year) because July is indeed the first month in the year 2005. This is exactly the case in Adventure Works DW SSAS prior to 2012. You can test the following scenario in Adventure Works DW SSAS 2008 R2. In our example we used the [Date].[Fiscal] user hierarchy. That hierarchy has all 12 months in every year which is not the case with the [Date].[Calendar] user hierarchy where there's only six months in the first year. This can lead to strange results. For example, if you search-replace the word "Fiscal" with the word "Calendar" in the query we used in this recipe, you'll get this as the result: Notice how the values are incorrect for the year 2006. That's because the ParallelPeriod() function is not a time-aware function, it merely does what it's designed for taking the member that is in the same relative position. Gaps in your time dimension are another potential problem. Therefore, always make the complete date dimensions, with all 12 months in every year and all dates in them, not just working days or similar shortcuts. Remember, Analysis Services isn't doing the date math. It's just navigating using the member's relative position. Therefore, make sure you have laid a good foundation for that. However, that's not always possible. There's an offset of six months between fiscal and calendar years, meaning if you want both of them as date hierarchies, you have a problem; one of them will not have all of the months in the first year. The solution is to test the current member in the calculation and to provide a special logic for the first year, fiscal or calendar; the one that doesn't have all months in it. This is most efficiently done with a scope statement in the MDX script. Another problem in calculating the YoY value is leap years. Calculating moving averages The moving average, also known as the rolling average, is a statistical technique often used in events with unpredictable short-term fluctuations in order to smooth their curve and to visualize the pattern of behavior. The key to get the moving average is to know how to construct a set of members up to and including a specified member, and to get the average value over the number of members in the set. In this recipe, we're going to look at two different ways to calculate moving averages in MDX. Getting ready Start SQL Server Management Studio and connect to your SSAS 2012 instance. Click on the New Query button and check that the target database is Adventure Works DW 2012. In this example we're going to use the Date hierarchy of the Date dimension. Here's the query we'll start from: SELECT{ [Measures].[Internet Order Count] } ON 0,{ [Date].[Date].[Date].MEMBERS} ON 1FROM[Adventure Works] Execute it. The result shows the count of Internet orders for each date in the Date.Date attribute hierarchy. Our task is to calculate the simple moving average (SMA) for dates in the year 2008 based on the count of orders in the previous 30 days. How to do it... We are going to use the LastPeriods() function with a 30 day moving window, and a member expression, [Date].[Date].CurrentMember, as two parameters, and also the AVG() function, to calculate the moving average of Internet order count in the last 30 days. Follow these steps to calculate moving averages: Add the WHERE part of the query and put the year 2006 inside using any available hierarchy. Add the WITH part and define a new calculated measure. Name it SMA 30. Define that measure using the AVG() and LastPeriods() functions. Test to see if you get a managed query similar to this. If so, execute it: WITHMEMBER [Measures].[SMA 30] ASAvg( LastPeriods( 30, [Date].[Date].CurrentMember ),[Measures].[Internet Order Count] )SELECT{ [Measures].[Internet Order Count],[Measures].[SMA 30] } ON 0,{ [Date].[Date].[Date].MEMBERS } ON 1FROM[Adventure Works]WHERE( [Date].[Calendar Year].&[2008] ) The second column in the result set will represent the simple moving average based on the last 30 days. Our final result will look like the following screenshot: How it works... The moving average is a calculation that uses the moving window of N items for which it calculates the statistical mean, that is, the average value. The window starts with the first item and then progressively shifts to the next one until the whole set of items is passed. The function that acts as the moving window is the LastPeriods() function. It returns N items, in this example, 30 dates. That set is then used to calculate the average orders using the AVG() function. Note that the number of members returned by the LastPeriods() function is equal to the span, 30, starting with the member that lags 30 - 1 from the specified member expression, and ending with the specified member. There's more... Another way of specifying what the LastPeriods() function does is to use a range of members with a range-based shortcut. The last member of the range is usually the current member of the hierarchy on an axis. The first member is the N-1th member moving backwards on the same level in that hierarchy, which can be constructed using the Lag(N-1) function. The following expression employing the Lag() function and a range-based shortcut is equivalent to the LastPeriods() in the preceding example: [Date].[Date].CurrentMember.Lag(29) : [Date].[Date].CurrentMember Note that the members returned from the range-based shortcut are inclusive of both the starting member and the ending member. We can easily modify the moving window scope to fit different requirements. For example, in case we need to calculate a 30-day moving average up to the previous member, we can use this syntax: [Date].[Date].CurrentMember.Lag(30) : [Date].[Date].PrevMember The LastPeriods() function is not on the list of optimized functions on this web page: http://tinyurl.com/Improved2008R2. However, tests show no difference in duration with respect to its range alternative. Still, if you come across a situation where the LastPeriods() function performs slowly, try its range alternative. Finally, in case we want to parameterize the expression (for example, to be used in SQL Server Reporting Services), these would be generic forms of the previous expressions: [Date].[Date].CurrentMember.Lag( @span - @offset ) :[Date].[Date].CurrentMember.Lag( @offset ) And LastPeriods( @span, [Date].[Date].CurrentMember.Lag( @offset ) ) The @span parameter is a positive value which determines the size of the window. The @offset parameter determines how much the right side of the window is moved from the current member's position. This shift can be either a positive or negative value. The value of zero means there is no shift at all, the most common scenario. Other ways to calculate the moving averages The simple moving average is just one of many variants of calculating the moving averages. A good overview of a possible variant can be found in Wikipedia: http://tinyurl.com/WikiMovingAvg MDX examples of other variants of moving averages can be found in Mosha Pasumansky's blog article: http://tinyurl.com/MoshaMovingAvg Moving averages and the future dates It's worth noting that the value returned by the moving average calculation is not empty for dates in future because the window is looking backwards, so that there will always be values for future dates. This can be easily verified by scrolling down in our example using the LastPeriods() function, as shown in the following screenshot: In this case the NON EMPTY keyword will not be able to remove empty rows. We might be tempted to use NON_EMPTY_BEHAVIOR to solve this problem but it wouldn't help. Moreover, it would be completely wrong. We don't want to set all the empty rows to null, but only those positioned after the member representing today's date. We'll cover the proper approach to this challenge in the following recipes. Summary This article presents various time-related functions in MDX language that are designed to work with a special type of dimension called the Time and its typed attributes. Resources for Article: Further resources on this subject: What are SSAS 2012 dimensions and cube? [Article] Creating an Analysis Services Cube with Visual Studio 2008 - Part 1 [Article] Terms and Concepts Related to MDX [Article]