How-To Tutorials

In this article by Doug Bierer, author of the book PHP 7 Programming Cookbook, we will cover the following topics: (For more resources related to this topic, see here.) Authenticating with middleware Making inter-framework system calls Using middleware to cross languages Introduction As often happens in the IT industry, terms get invented, and then used and abused. The term middleware is no exception. Arguably the first use of the term came out of the Internet Engineering Task Force (IETF) in the year 2000. Originally, the term was applied to any software which operates between the transport (that is, TCP/IP) and the application layer. More recently, especially with the acceptance of PHP Standard Recommendation number 7 (PSR-7), middleware, specifically in the PHP world, has been applied to the web client-server environment. Authenticating with middleware One very important usage of middleware is to provide authentication. Most web-based applications need the ability to verify a visitor via username and password. By incorporating PSR-7 standards into an authentication class, you will make it generically useful across the board, so to speak, being secure that it can be used in any framework that provides PSR-7-compliant request and response objects. How to do it… We begin by defining a ApplicationAclAuthenticateInterface class. We use this interface to support the Adapter software design pattern, making our Authenticate class more generically useful by allowing a variety of adapters, each of which can draw authentication from a different source (for example, from a file, using OAuth2, and so on). Note the use of the PHP 7 ability to define the return value data type: namespace ApplicationAcl; use PsrHttpMessage { RequestInterface, ResponseInterface }; interface AuthenticateInterface { public function login(RequestInterface $request) : ResponseInterface; } Note that by defining a method that requires a PSR-7-compliant request, and produces a PSR-7-compliant response, we have made this interface universally applicable. Next, we define the adapter that implements the login() method required by the interface. We make sure to use the appropriate classes, and define fitting constants and properties. The constructor makes use of ApplicationDatabaseConnection: namespace ApplicationAcl; use PDO; use ApplicationDatabaseConnection; use PsrHttpMessage { RequestInterface, ResponseInterface }; use ApplicationMiddleWare { Response, TextStream }; class DbTable implements AuthenticateInterface { const ERROR_AUTH = 'ERROR: authentication error'; protected $conn; protected $table; public function __construct(Connection $conn, $tableName) { $this->conn = $conn; $this->table = $tableName; } The core login() method extracts the username and password from the request object. We then do a straightforward database lookup. If there is a match, we store user information in the response body, JSON-encoded: public function login(RequestInterface $request) : ResponseInterface { $code = 401; $info = FALSE; $body = new TextStream(self::ERROR_AUTH); $params = json_decode($request->getBody()->getContents()); $response = new Response(); $username = $params->username ?? FALSE; if ($username) { $sql = 'SELECT * FROM ' . $this->table . ' WHERE email = ?'; $stmt = $this->conn->pdo->prepare($sql); $stmt->execute([$username]); $row = $stmt->fetch(PDO::FETCH_ASSOC); if ($row) { if (password_verify($params->password, $row['password'])) { unset($row['password']); $body = new TextStream(json_encode($row)); $response->withBody($body); $code = 202; $info = $row; } } } return $response->withBody($body)->withStatus($code); } } Best practice Never store passwords in clear text. When you need to do a password match, use password_verify(), which negates the need to reproduce the password hash. The Authenticate class is a wrapper for an adapter class that implements AuthenticationInterface. Accordingly, the constructor takes an adapter class as an argument, as well as a string that serves as the key in which authentication information is stored in $_SESSION: namespace ApplicationAcl; use ApplicationMiddleWare { Response, TextStream }; use PsrHttpMessage { RequestInterface, ResponseInterface }; class Authenticate { const ERROR_AUTH = 'ERROR: invalid token'; const DEFAULT_KEY = 'auth'; protected $adapter; protected $token; public function __construct( AuthenticateInterface $adapter, $key) { $this->key = $key; $this->adapter = $adapter; } In addition, we provide a login form with a security token, which helps prevent Cross Site Request Forgery (CSRF) attacks: public function getToken() { $this->token = bin2hex(random_bytes(16)); $_SESSION['token'] = $this->token; return $this->token; } public function matchToken($token) { $sessToken = $_SESSION['token'] ?? date('Ymd'); return ($token == $sessToken); } public function getLoginForm($action = NULL) { $action = ($action) ? 'action="' . $action . '" ' : ''; $output = '<form method="post" ' . $action . '>'; $output .= '<table><tr><th>Username</th><td>'; $output .= '<input type="text" name="username" /></td>'; $output .= '</tr><tr><th>Password</th><td>'; $output .= '<input type="password" name="password" />'; $output .= '</td></tr><tr><th>&nbsp;</th>'; $output .= '<td><input type="submit" /></td>'; $output .= '</tr></table>'; $output .= '<input type="hidden" name="token" value="'; $output .= $this->getToken() . '" />'; $output .= '</form>'; return $output; } Finally, the login() method in this class checks whether the token is valid. If not, a 400 response is returned. Otherwise, the login() method of the adapter is called: public function login( RequestInterface $request) : ResponseInterface { $params = json_decode($request->getBody()->getContents()); $token = $params->token ?? FALSE; if (!($token && $this->matchToken($token))) { $code = 400; $body = new TextStream(self::ERROR_AUTH); $response = new Response($code, $body); } else { $response = $this->adapter->login($request); } if ($response->getStatusCode() >= 200 && $response->getStatusCode() < 300) { $_SESSION[$this->key] = json_decode($response->getBody()->getContents()); } else { $_SESSION[$this->key] = NULL; } return $response; } } How it works… Go ahead and define the classes presented in this recipe, summarized in the following table: Class Discussed in these steps ApplicationAclAuthenticateInterface 1 ApplicationAclDbTable 2 - 3 ApplicationAclAuthenticate 4 - 6 You can then define a chap_09_middleware_authenticate.php calling program that sets up autoloading and uses the appropriate classes: <?php session_start(); define('DB_CONFIG_FILE', __DIR__ . '/../config/db.config.php'); define('DB_TABLE', 'customer_09'); define('SESSION_KEY', 'auth'); require __DIR__ . '/../Application/Autoload/Loader.php'; ApplicationAutoloadLoader::init(__DIR__ . '/..'); use ApplicationDatabaseConnection; use ApplicationAcl { DbTable, Authenticate }; use ApplicationMiddleWare { ServerRequest, Request, Constants, TextStream }; You are now in a position to set up the authentication adapter and core class: $conn = new Connection(include DB_CONFIG_FILE); $dbAuth = new DbTable($conn, DB_TABLE); $auth = new Authenticate($dbAuth, SESSION_KEY); Be sure to initialize the incoming request, and set up the request to be made to the authentication class: $incoming = new ServerRequest(); $incoming->initialize(); $outbound = new Request(); Check the incoming class method to see if it is POST. If so, pass a request to the authentication class: if ($incoming->getMethod() == Constants::METHOD_POST) { $body = new TextStream(json_encode( $incoming->getParsedBody())); $response = $auth->login($outbound->withBody($body)); } $action = $incoming->getServerParams()['PHP_SELF']; ?> The display logic looks like this: <?= $auth->getLoginForm($action) ?> Here is the output from an invalid authentication attempt. Notice the 401 status code on the right. In this illustration, you could add a var_dump() of the response object. Here is a successful authentication: Making inter-framework system calls One of the primary reasons for the development of PSR-7 (and middleware) was a growing need to make calls between frameworks. It is of interest to note that the main documentation for PSR-7 is hosted by PHP Framework Interop Group (PHP-FIG). How to do it… The primary mechanism used in middleware inter-framework calls is to create a driver program that executes framework calls in succession, maintaining a common request and response object. The request and response objects are expected to represent PsrHttpMessageServerRequestInterface and PsrHttpMessageResponseInterface respectively. For the purposes of this illustration, we define a middleware session validator. The constants and properties reflect the session thumbprint, which is a term we use to incorporate factors such as the website visitor's IP address, browser, and language settings: namespace ApplicationMiddleWareSession; use InvalidArgumentException; use PsrHttpMessage { ServerRequestInterface, ResponseInterface }; use ApplicationMiddleWare { Constants, Response, TextStream }; class Validator { const KEY_TEXT = 'text'; const KEY_SESSION = 'thumbprint'; const KEY_STATUS_CODE = 'code'; const KEY_STATUS_REASON = 'reason'; const KEY_STOP_TIME = 'stop_time'; const ERROR_TIME = 'ERROR: session has exceeded stop time'; const ERROR_SESSION = 'ERROR: thumbprint does not match'; const SUCCESS_SESSION = 'SUCCESS: session validates OK'; protected $sessionKey; protected $currentPrint; protected $storedPrint; protected $currentTime; protected $storedTime; The constructor takes a ServerRequestInterface instance and the session as arguments. If the session is an array (such as $_SESSION), we wrap it in a class. The reason why we do this is in case we are passed a session object, such as JSession used in Joomla. We then create the thumbprint using the factors previously mentioned factors. If the stored thumbprint is not available, we assume this is the first time, and store the current print as well as stop time, if this parameter is set. We used md5() because it's a fast hash, and is not exposed externally and is therefore useful to this application: public function __construct( ServerRequestInterface $request, $stopTime = NULL) { $this->currentTime = time(); $this->storedTime = $_SESSION[self::KEY_STOP_TIME] ?? 0; $this->currentPrint = md5($request->getServerParams()['REMOTE_ADDR'] . $request->getServerParams()['HTTP_USER_AGENT'] . $request->getServerParams()['HTTP_ACCEPT_LANGUAGE']); $this->storedPrint = $_SESSION[self::KEY_SESSION] ?? NULL; if (empty($this->storedPrint)) { $this->storedPrint = $this->currentPrint; $_SESSION[self::KEY_SESSION] = $this->storedPrint; if ($stopTime) { $this->storedTime = $stopTime; $_SESSION[self::KEY_STOP_TIME] = $stopTime; } } } It's not required to define __invoke(), but this magic method is quite convenient for standalone middleware classes. As is the convention, we accept ServerRequestInterface and ResponseInterface instances as arguments. In this method we simply check to see if the current thumbprint matches the one stored. The first time, of course, they will match. But on subsequent requests, the chances are an attacker intent on session hijacking will be caught out. In addition, if the session time exceeds the stop time (if set), likewise, a 401 code will be sent: public function __invoke( ServerRequestInterface $request, Response $response) { $code = 401; // unauthorized if ($this->currentPrint != $this->storedPrint) { $text[self::KEY_TEXT] = self::ERROR_SESSION; $text[self::KEY_STATUS_REASON] = Constants::STATUS_CODES[401]; } elseif ($this->storedTime) { if ($this->currentTime > $this->storedTime) { $text[self::KEY_TEXT] = self::ERROR_TIME; $text[self::KEY_STATUS_REASON] = Constants::STATUS_CODES[401]; } else { $code = 200; // success } } if ($code == 200) { $text[self::KEY_TEXT] = self::SUCCESS_SESSION; $text[self::KEY_STATUS_REASON] = Constants::STATUS_CODES[200]; } $text[self::KEY_STATUS_CODE] = $code; $body = new TextStream(json_encode($text)); return $response->withStatus($code)->withBody($body); } We can now put our new middleware class to use. The main problems with inter-framework calls, at least at this point, are summarized here. Accordingly, how we implement middleware depends heavily on the last point: Not all PHP frameworks are PSR-7-compliant Existing PSR-7 implementations are not complete All frameworks want to be the "boss" As an example, have a look at the configuration files for Zend Expressive, which is a self-proclaimed PSR7 Middleware Microframework. Here is the file, middleware-pipeline.global.php, which is located in the config/autoload folder in a standard Expressive application. The dependencies key is used to identify middleware wrapper classes that will be activated in the pipeline: <?php use ZendExpressiveContainerApplicationFactory; use ZendExpressiveHelper; return [ 'dependencies' => [ 'factories' => [ HelperServerUrlMiddleware::class => HelperServerUrlMiddlewareFactory::class, HelperUrlHelperMiddleware::class => HelperUrlHelperMiddlewareFactory::class, // insert your own class here ], ], Under the middleware_pipline key you can identify classes that will be executed before or after the routing process occurs. Optional parameters include path, error, and priority: 'middleware_pipeline' => [ 'always' => [ 'middleware' => [ HelperServerUrlMiddleware::class, ], 'priority' => 10000, ], 'routing' => [ 'middleware' => [ ApplicationFactory::ROUTING_MIDDLEWARE, HelperUrlHelperMiddleware::class, // insert reference to middleware here ApplicationFactory::DISPATCH_MIDDLEWARE, ], 'priority' => 1, ], 'error' => [ 'middleware' => [ // Add error middleware here. ], 'error' => true, 'priority' => -10000, ], ], ]; Another technique is to modify the source code of an existing framework module, and make a request to a PSR-7-compliant middleware application. Here is an example modifying a Joomla! installation to include a middleware session validator. Next, add this code the end of the index.php file in the /path/to/joomla folder. Since Joomla! uses Composer, we can leverage the Composer autoloader: session_start(); // to support use of $_SESSION $loader = include __DIR__ . '/libraries/vendor/autoload.php'; $loader->add('Application', __DIR__ . '/libraries/vendor'); $loader->add('Psr', __DIR__ . '/libraries/vendor'); We can then create an instance of our middleware session validator, and make a validation request just before $app = JFactory::getApplication('site');: $session = JFactory::getSession(); $request = (new ApplicationMiddleWareServerRequest())->initialize(); $response = new ApplicationMiddleWareResponse(); $validator = new ApplicationSecuritySessionValidator( $request, $session); $response = $validator($request, $response); if ($response->getStatusCode() != 200) { // take some action } How it works… First, create the ApplicationMiddleWareSessionValidator test middleware class described in steps 2 - 5. Then you will need to go to getcomposer.org and follow the directions to obtain Composer. Next, build a basic Zend Expressive application, as shown next. Be sure to select No when prompted for minimal skeleton: cd /path/to/source/for/this/chapter php composer.phar create-project zendframework/zend-expressive-skeleton expressive This will create a folder /path/to/source/for/this/chapter/expressive. Change to this directory. Modify public/index.php as follows: <?php if (php_sapi_name() === 'cli-server' && is_file(__DIR__ . parse_url( $_SERVER['REQUEST_URI'], PHP_URL_PATH)) ) { return false; } chdir(dirname(__DIR__)); session_start(); $_SESSION['time'] = time(); $appDir = realpath(__DIR__ . '/../../..'); $loader = require 'vendor/autoload.php'; $loader->add('Application', $appDir); $container = require 'config/container.php'; $app = $container->get(ZendExpressiveApplication::class); $app->run(); You will then need to create a wrapper class that invokes our session validator middleware. Create a SessionValidateAction.php file that needs to go in the /path/to/source/for/this/chapter/expressive/src/App/Action folder. For the purposes of this illustration, set the stop time parameter to a short duration. In this case, time() + 10 gives you 10 seconds: namespace AppAction; use ApplicationMiddleWareSessionValidator; use ZendDiactoros { Request, Response }; use PsrHttpMessageResponseInterface; use PsrHttpMessageServerRequestInterface; class SessionValidateAction { public function __invoke(ServerRequestInterface $request, ResponseInterface $response, callable $next = null) { $inbound = new Response(); $validator = new Validator($request, time()+10); $inbound = $validator($request, $response); if ($inbound->getStatusCode() != 200) { session_destroy(); setcookie('PHPSESSID', 0, time()-300); $params = json_decode( $inbound->getBody()->getContents(), TRUE); echo '<h1>',$params[Validator::KEY_TEXT],'</h1>'; echo '<pre>',var_dump($inbound),'</pre>'; exit; } return $next($request,$response); } } You will now need to add the new class to the middleware pipeline. Modify config/autoload/middleware-pipeline.global.php as follows. Modifications are shown in bold: <?php use ZendExpressiveContainerApplicationFactory; use ZendExpressiveHelper; return [ 'dependencies' => [ 'invokables' => [ AppActionSessionValidateAction::class => AppActionSessionValidateAction::class, ], 'factories' => [ HelperServerUrlMiddleware::class => HelperServerUrlMiddlewareFactory::class, HelperUrlHelperMiddleware::class => HelperUrlHelperMiddlewareFactory::class, ], ], 'middleware_pipeline' => [ 'always' => [ 'middleware' => [ HelperServerUrlMiddleware::class, ], 'priority' => 10000, ], 'routing' => [ 'middleware' => [ ApplicationFactory::ROUTING_MIDDLEWARE, HelperUrlHelperMiddleware::class, AppActionSessionValidateAction::class, ApplicationFactory::DISPATCH_MIDDLEWARE, ], 'priority' => 1, ], 'error' => [ 'middleware' => [ // Add error middleware here. ], 'error' => true, 'priority' => -10000, ], ], ]; You might also consider modifying the home page template to show the status of $_SESSION. The file in question is /path/to/source/for/this/chapter/expressive/templates/app/home-page.phtml. Simply adding var_dump($_SESSION) should suffice. Initially, you should see something like this: After 10 seconds, refresh the browser. You should now see this: Using middleware to cross languages Except in cases where you are trying to communicate between different versions of PHP, PSR-7 middleware will be of minimal use. Recall what the acronym stands for: PHP Standards Recommendations. Accordingly, if you need to make a request to an application written in another language, treat it as you would any other web service HTTP request. How to do it… In the case of PHP 4, you actually have a chance in that there was limited support for object-oriented programming. There is not enough space to cover all changes, but we present a potential PHP 4 version of ApplicationMiddleWareServerRequest. The first thing to note is that there are no namespaces! Accordingly, we use a classname with underscores, _, in place of namespace separators: class Application_MiddleWare_ServerRequest extends Application_MiddleWare_Request implements Psr_Http_Message_ServerRequestInterface { All properties are identified in PHP 4 using the key word var: var $serverParams; var $cookies; var $queryParams; // not all properties are shown The initialize() method is almost the same, except that syntax such as $this->getServerParams()['REQUEST_URI'] was not allowed in PHP 4. Accordingly, we need to split this out into a separate variable: function initialize() { $params = $this->getServerParams(); $this->getCookieParams(); $this->getQueryParams(); $this->getUploadedFiles; $this->getRequestMethod(); $this->getContentType(); $this->getParsedBody(); return $this->withRequestTarget($params['REQUEST_URI']); } All of the $_XXX super-globals were present in later versions of PHP 4: function getServerParams() { if (!$this->serverParams) { $this->serverParams = $_SERVER; } return $this->serverParams; } // not all getXXX() methods are shown to conserve space The null coalesce" operator was only introduced in PHP 7. We need to use isset(XXX) ? XXX : ''; instead: function getRequestMethod() { $params = $this->getServerParams(); $method = isset($params['REQUEST_METHOD']) ? $params['REQUEST_METHOD'] : ''; $this->method = strtolower($method); return $this->method; } The JSON extension was not introduced until PHP 5. Accordingly, we need to be satisfied with raw input. We could also possibly use serialize() or unserialize() in place of json_encode() and json_decode(): function getParsedBody() { if (!$this->parsedBody) { if (($this->getContentType() == Constants::CONTENT_TYPE_FORM_ENCODED || $this->getContentType() == Constants::CONTENT_TYPE_MULTI_FORM) && $this->getRequestMethod() == Constants::METHOD_POST) { $this->parsedBody = $_POST; } elseif ($this->getContentType() == Constants::CONTENT_TYPE_JSON || $this->getContentType() == Constants::CONTENT_TYPE_HAL_JSON) { ini_set("allow_url_fopen", true); $this->parsedBody = file_get_contents('php://stdin'); } elseif (!empty($_REQUEST)) { $this->parsedBody = $_REQUEST; } else { ini_set("allow_url_fopen", true); $this->parsedBody = file_get_contents('php://stdin'); } } return $this->parsedBody; } The withXXX() methods work pretty much the same in PHP 4: function withParsedBody($data) { $this->parsedBody = $data; return $this; } Likewise, the withoutXXX() methods work the same as well: function withoutAttribute($name) { if (isset($this->attributes[$name])) { unset($this->attributes[$name]); } return $this; } } For websites using other languages, we could use the PSR-7 classes to formulate requests and responses, but would then need to use an HTTP client to communicate with the other website. Here is the example: $request = new Request( TARGET_WEBSITE_URL, Constants::METHOD_POST, new TextStream($contents), [Constants::HEADER_CONTENT_TYPE => Constants::CONTENT_TYPE_FORM_ENCODED, Constants::HEADER_CONTENT_LENGTH => $body->getSize()] ); $data = http_build_query(['data' => $request->getBody()->getContents()]); $defaults = array( CURLOPT_URL => $request->getUri()->getUriString(), CURLOPT_POST => true, CURLOPT_POSTFIELDS => $data, ); $ch = curl_init(); curl_setopt_array($ch, $defaults); $response = curl_exec($ch); curl_close($ch); Summary In this article, we learned about providing authentication to a system, to make calls between frameworks, and to make a request to an application written in another language. Resources for Article: Further resources on this subject: Middleware [Article] Building a Web Application with PHP and MariaDB – Introduction to caching [Article] Data Tables and DataTables Plugin in jQuery 1.3 with PHP [Article]

(For more resources related to this topic, see here.) How to do it... Anyone who can read the files that Redis uses to persist your dataset has a full copy of all your data. Worse, anyone who can write to those files can, with a minimal amount of effort and some patience, change the data that your Redis server contains. Both of these things are probably not what you want, and thankfully it isn't particularly difficult to prevent. All you have to do is prevent anyone but the user running your Redis server from accessing the data directory your Redis instance is using. The simplest way to achieve this is by changing the owner of the directory to the user who runs your Redis server, and then disallow all privileges to everyone else, like this: Determine the user under whom you are running your Redis instance. You can typically find this out by running ps caux |grep redis-server. The name in the first column is the user under which Redis is running. Determine the directory in which Redis is storing its files. If you don't already know this, you can ask Redis by running CONFIG GET dir from within redis-cli. Ensure that the user running your Redis instance owns its data directory: chown <redisuser> /path/to/redis/datadir Restrict permissions on the data directory so that only the owner can access it at all: chmod 0700 /path/to/redis/datadir It is important that you protect the Redis data directory, and not individual data files, because Redis is regularly rewriting those data files, and the permissions you choose won't necessarily be preserved on the next rewrite. It is also a good practice to restrict access to your redis.conf file, because in some cases it can contain sensitive data. This is simply achieved: chmod 0600 /path/to/redis.conf If you run your Redis using applications on a server which is shared with other people, your Redis instance is at pretty serious risk of abuse. The most common way of connecting to Redis is via TCP, which can only limit access based on the address connecting to it. On a shared server, that address is shared amongst everyone using it, so anyone else on the same server as you can connect to your Redis. Not cool! If, however, the programs that need to access your Redis server are on the same machine as the Redis server, there is another, more secure, method of connection called Unix sockets. A Unix socket looks like a file on disk, and you can control its permissions just like a file, but Redis can listen on it (and clients can connect to it), in a very similar way to a TCP socket. Enabling Redis to listen on a Unix socket is fairly straightforward: Set the port parameter to 0 in your Redis configuration file. This will tell Redis to not listen on a TCP socket. This is very important to prevent miscreants from still being able to connect to your Redis server while you're happily using a Unix socket. Set the unixsocket parameter in your Redis configuration file to a fully-qualified filename where you want the socket to exist. If your Redis server runs as the same user as your client programs (which is common in shared-hosting situations), I recommend making the name of the file redis.sock, in the same directory as your Redis dataset. So, if you keep your Redis data in /home/joe/redis, set unixsocket to /home/joe/redis/redis.sock. Set the unixsocketperm parameter in your Redis configuration file to 600, or a more relaxed permission set if you know what you're doing. Again, this assumes that your Redis server and Redis-using programs are running as the same user. If they're not, you'll probably need a dedicated group and things get a lot more complicated—and beyond the scope of what can be covered in this guide. Once you've changed those configuration parameters and restarted Redis, you should find that the file you specified for unixsocket has magically appeared, and you can no longer connect to Redis using TCP. All that remains to do now is to configure your Redis-using programs to connect using the Unix socket, which is something you should find how to do in the manual for your particular Redis client library or application. Configuring Redis to use Unix sockets is all well and good when it's practical, but what about if you need to connect to Redis over a network? In that case, you'll need to let Redis listen on a TCP socket, but you should at least limit the computers that can connect to it with a suitable firewall configuration. While the properly paranoid systems administrator runs their systems with a default deny firewalling policy, not everyone shares this philosophy. However, given that by default, anyone who can connect to your Redis server can do anything they want with it, you should definitely configure a firewall on your Redis servers to limit incoming TCP connections to those which are coming from machines that have a legitimate need to talk to your Redis server. While it won't protect you from all attacks, it will cut down significantly on the attack surface, which is an important part of a defense-in-depth security strategy. Unfortunately, it is hard to give precise commands to configure a firewall ruleset, because there are so many firewall management tools in common use on systems today. In the interest of addressing the greatest common factor, though, I'll provide a set of Linux iptables rules, which should be translatable to whatever means of managing your firewall (whether it be an iptables wrapper of some sort on Linux, or a pf-based system on a BSD). In all of the following commands, replace the word <port> with the TCP port that your Redis server listens on. Also, note that these commands will temporarily stop all traffic to your Redis instance, so you'll want to avoid doing this on a live server. Setting up your firewall in an init script is the best course of action. Insert a rule that will drop all traffic to your Redis server port by default: iptables -I INPUT -p tcp --dport <port> -j DROP For each IP address you want to allow to connect, run these two commands to let the traffic in: iptables -I INPUT -p tcp --dport <port> -s <clientIP> -j ACCEPT iptables -I OUTPUT -p tcp --sport <port> -d <clientIP> -j ACCEPT A firewall is great, but sometimes you can't trust everyone with access to a machine that needs to talk to your Redis instance. In that case, you can use authentication to provide a limited amount of protection against miscreants: Select a very strong password. Redis is not hardened against repeated password guessing, so you want to make this very long and very random. If you make the password too short, an attacker can just write a program that tries every possible password very quickly and guess your password that way, not cool! Thankfully, since humans should rarely be typing this password, it can be a complete jumble, and very long. I like the command pwgen -sy 32 1 for all my "generating very strong password" needs. Configure all your clients to authenticate against the server, by sending the following command when they first connect to the server: AUTH <password> Edit your Redis configuration file to include a line like this: requirepass "\:d!&!:Y<p'TXBI0"ys96rfH]lxaA7|E" If your selected password contains any double-quotes, you'll need to escape them with a backslash (so " would become "), as I've done in the preceding example. You'll also need to double any actual backslashes (so becomes \), again as I've done in the password of the preceding example. Let the configuration changes take effect by restarting Redis. The authentication password cannot be changed at runtime. If you don't need certain commands, or want to limit the use of certain commands to a subset of clients, you can use the rename-command configuration parameter. Like firewalling, restricting, or disabling commands reduces your attack surface, but is not a panacea. The simplest solution to the risk of a dangerous command is to disable it. For example, if you want to stop anyone from accidentally (or deliberately) nuking all the data in your Redis server with a single command, you might decide to disable the FLUSHDB and FLUSHALL commands, by putting the following in your Redis config file: rename-command FLUSHDB ""rename-command FLUSHALL "" This doesn't stop someone from enumerating all the keys in your dataset with KEYS * and then deleting them all one-by-one, but it does raise the bar somewhat. If you never wanted to delete keys (but, say, only let them expire) you could disable the DEL command; although all that would probably do is encourage the wily cracker to enumerate all your keys and run PEXPIRE 1 over them. Arms races are a terrible thing... While disabling commands entirely is great when it can be done, you sometimes need a particular command, but you'd prefer not to give access to it to absolutely everyone—commands that can cause serious problems if misused, such as CONFIG. For those cases, you can rename the command to something hard-to-guess, as shown in the following command: rename-command CONFIG somegiantstringnobodywouldguess It's important to not make the new name of the command something easy-to-guess. Like the AUTH command, which we discussed previously, someone who wanted to do bad things could easily write a program to repeatedly guess what you've renamed your commands to. For any environment in which you can't trust the network (which these days is pretty much everywhere, thanks to the NSA and the Cloud), it is important to consider the possibility of someone watching all your data as it goes over the wire. There's little point configuring authentication, or renaming commands, if an attacker can watch all your data and commands flow back and forth. The least-worst option we have for generically securing network traffic from eavesdropping is still the Secure Sockets Layer (SSL). Redis doesn't support SSL natively; however, through the magic of the stunnel program, we can create a secure tunnel between Redis clients and servers. The setup we will build will look like the following diagram: In order to set this up, you'll need to do the following: In your redis.conf, ensure that Redis is only listening on 127.0.0.1, by setting the bind parameter: bind 127.0.0.1 Create a private key and certificate, which stunnel will use to secure the network communications. First, create a private key and a certificate request, by running: openssl req -out /etc/ssl/redis.csr -keyout /etc/ssl/redis.key -nodes -newkey rsa:2048 This will ask you all sorts of questions which you can answer with whatever you like. Create the self-signed certificate itself, by running: openssl x509 -req -days 3650 -signkey /etc/ssl/redis.key -in /etc/ssl/redis.csr -out /etc/ssl/redis.crt Finally, stunnel expects to find the private key and the certificate in the same file, so we'll concatenate the two together into one file: cat /etc/ssl/redis.key /etc/ssl/redis.crt >/etc/ssl/redis.pem Now, we've got our SSL keys, we can start stunnel on the server side, configuring it to listen out for SSL connections, and forward them to our local Redis server: stunnel -d 46379 -r 6379 -p /etc/ssl/redis.pem If your local Redis instance isn't listening on port 6379, or if you'd like to change the public port that stunnel listens on, you can, of course, adjust the preceding command line to suit. Also, don't forget to open up your firewall for the port you're listening on! Once you run the preceding command, you should be returned to a command line pretty quickly, because stunnel runs in the background. Although you examine your listening ports with netstat -ltn, you will still find that port 46379 is listening. If that's the case, you're done configuring the server. On the client(s), the process is somewhat simpler, because you don't have to create a whole new key pair. However, you do need the certificate from the server, because you want to be able to verify that you're connecting to the right SSL-enabled service. There's little point in using SSL if an attacker can just set up a fake SSL service and trick you into connecting to it. To set up the client, do the following: Copy /etc/ssl/redis.crt from the server to the same location on the client. Start stunnel on the client, as shown in the following code snippet: stunnel -c -v 3 -A /etc/ssl/redis.crt -d 127.0.0.1:56379 -r 192.0.2.42:46379 Replace 192.0.2.42 with the IP address of your Redis server. Verify that stunnel is listening correctly by running netstat -ltn, and look for something listening on port 56379. Reconfigure your client to connect to 127.0.0.1:56379, rather than directly to the remote Redis server. Summary This article contains an assortment of quick enhancements that you can deploy to your systems to protect them from various threats, which are frequently encountered on the Internet today. Resources for Article: Further resources on this subject: Implementing persistence in Redis (Intermediate) [Article] Python Text Processing with NLTK: Storing Frequency Distributions in Redis [Article] Coding for the Real-time Web [Article]

Understanding user properties User properties are child object types which are available for the following object types in the Siebel Repository: Applet, Control, List Column Application Business Service Business Component, Field Integration Object, Integration Component, Integration Component Field View To view the User Property (or User Prop as it is sometimes abbreviated) object type we typically have to modify the list of displayed types for the Object Explorer window. This can be achieved by selecting the Options command in the View menu. In the Object Explorer tab of the Development Tools Options dialog, we can select the object types for display as shown in the following screenshot: In the preceding example, the Business Component User Prop type is enabled for display. After confirming the changes in the Development Tools Options dialog by clicking the OK button, we can for example navigate to the Account business component and review its existing user properties by selecting the Business Component User Prop type in the Object Explorer. The following screenshot shows the list of user properties for the Account business component: The screenshot also shows the standard Properties window on the right. This is to illustrate that a list of user properties, which mainly define a Name/Value pair, can be simply understood as an extension to an object type's usual properties which are accessible by means of the Properties window and represent Name/Value pairs as well.Because an additional user property is just a new record in the Siebel Repository, the list of user properties for a given parent record is theoretically infinite. This allows developers to define a rich set of business logic as a simple list of Name/Value pairs instead of having to write program code.The Name property of a user property definition must use a reserved name - and optional sequence number - as defined by Oracle engineering. The Value property must also follow the syntax defined for the special purpose of the user property. Because an additional user property is just a new record in the Siebel Repository, the list of user properties for a given parent record is theoretically infinite. This allows developers to define a rich set of business logic as a simple list of Name/Value pairs instead of having to write program code. The Name property of a user property definition must use a reserved name—and optional sequence number—as defined by Oracle engineering. The Value property must also follow the syntax defined for the special purpose of the user property. Did you know? The list of available names for a user property depends on the object type (for example Business Component) and the C++ class associated with the object definition. For example, the business component Account is associated with the CSSBCAccountSIS class which defines a different range of available user property names than other classes. Many user property names are officially documented in the Siebel Developer's Reference guide in the Siebel Bookshelf. We can find the guide online at the following URL: http://download.oracle.com/docs/cd/E14004_01/books/ToolsDevRef/ToolsDevRef_UserProps.html The user property names described in this guide are intended for use by custom developers. Any other user property which we may find in the Siebel Repository but which is not officially documented should be considered an internal user property of Oracle engineering. Because the internal user properties could change in a future version of Siebel CRM in both syntax and behavior without prior notice, it is highly recommended to use only user properties which are documented by Oracle. Another way to find out which user property names are made available by Oracle to customers is to click the dropdown icon in the Name property of a user property record. This opens the user property pick list which displays a wide range of officially documented user properties along with a description text. Multi-instance user properties Some user properties can be instantiated more than once. If this is the case a sequence number is used to generate a distinguished name. For example, the On Field Update Set user property used on business components uses a naming convention as displayed in the following screenshot: In the previous example, we can see four instances of the On Field Update Set user property distinguished by a sequential numeric suffix (1 to 4). Because it is very likely that Oracle engineers and custom developers add additional instances of the same user property while working on the next release, Oracle provides a customer allowance gap of nine instances for the next sequence number. In the previous example, a custom developer could continue the set of On Field Update Set user properties with a suffix of 13. By doing so, the custom developer will most likely avoid conflicts during an upgrade to a newer version of Siebel CRM. The Oracle engineer would continue with a suffix of five and upgrade conflicts will only occur when Oracle defines more than eight additional instances. The gap of nine also ensures that the sequence of multi-instance user properties is still functional when one or more of the user property records are marked as inactive. In the following sections, we will describe the most important user properties for the business and user interface layer. In addition, we will examine case study scenarios to identify best practices for using user properties to define specialized behavior of Siebel CRM applications. Business component and field user properties On the business layer of the Siebel Repository, user properties are widely used to control specialized behavior of business components and fields. The following table describes the most important user properties on the business component level. The Multiple Instances column contains Yes for all user properties which can be instantiated more than once per parent object: Source: Siebel Developer's Reference, Version 8.1: http://download.oracle.com/docs/cd/E14004_01/books/ToolsDevRef/booktitle.html

(For more resources related to this topic, see here.) Installing Drupal Commerce to an existing Drupal 7 website There are two approaches to installing Drupal Commerce; this recipe covers installing Drupal Commerce on an existing Drupal 7 website. Getting started You will need to download Drupal Commerce from http://drupal.org/project/ commerce. Download the most recent recommended release you see that couples with your Drupal 7 website's core version: You will also require the following modules to allow Drupal Commerce to function: Ctools: http://drupal.org/project/ctools Entity API: http://drupal.org/project/entity Views: http://drupal.org/project/views Rules: http://drupal.org/project/rules Address Field: http://drupal.org/project/addressfield How to do it... Now that you're ready, install Drupal Commerce by performing the following steps: Install the modules that Drupal Commerce depends on, first by copying the preceding module files into your Drupal site's modules directory, sites/all/modules. Install Drupal Commerce's modules next, by copying the files into the sites/all/ modules directory, so that they appear in the sites/all/modules/commerce directory. Enable the newly installed Drupal Commerce module in your Drupal site's administration panel (example.com/admin/modules if you've installed Drupal Commerce at example.com), under the Modules navigation option, by ensuring the checkbox to the left-hand side of the module name is checked. Now that Drupal Commerce is installed, a new menu option will appear in the administration navigation at the top of your screen when you are logged in as a user with administration permissions. You may need to clear the cache to see this. Navigate to Configuration | Development | Performance in the administration panel to do this. How it works... Drupal Commerce depends on a number of other Drupal modules to function, and by installing and enabling these in your website's administration panel you're on your way to getting your Drupal Commerce store off the ground. You can also install the Drupal Commerce modules via Drush (the Drupal Shell) too. For more information on Drush, see http://drupal.org/project/drush. Installing Drupal Commerce with Commerce Kickstart 2 Drupal Commerce requires quite a number of modules, and doing a basic installation can be quite time-consuming, which is where Commerce Kickstart 2 comes in. It packages Drupal 7 core and all of the necessary modules. Using Commerce Kickstart 2 is a good idea if you are building a Drupal Commerce website from scratch, and don't already have Drupal core installed. Getting started Download Drupal Commerce Kickstart 2 from its drupal.org project page at http://drupal.org/project/commerce kickstart. How to do it... Once you have decompressed the Commerce Kickstart 2 files to the location you want to install Drupal Commerce in, perform the following steps: Visit the given location in your web browser. For this example, it is assumed that your website is at example.com, so visit this address in your web browser. You'll see that you are presented with a welcome screen as shown in the following screenshot: Click the Let's Get Started button underneath this, and the installer moves to the next configuration option. Next, your server's requirements are checked to ensure Drupal can run in this environment. In the preceding screenshot you can see some common problems when installing Drupal that prevent installation. In particular, ensure that you create the /sites/ default/files directory in your Drupal installation and ensure it has permissions to allow Drupal to write to it (as this is where your website's images and files are stored). You will also need to copy the /sites/default/default.settings.php file to /sites/default/settings.php before you can start. Make sure this file is writeable by Drupal too (you'll secure it after installation is complete). Once these problems have been resolved, refresh the page and you will be taken to the Set up database screen. Enter the database username, password, and database name you want to use with Drupal, and click on Save and continue: The next step is the Install profile section, which can take some time as Drupal Commerce is installed for you. There's nothing for you to do here; just wait for installation to complete! You can now safely remove write permissions for the settings.php file in the /sites/default directory of your Drupal Commerce installation. The next step is Configure site. Enter the name of your new store and your e-mail address here, and provide a username and password for your Drupal Commerce administrator account. Don't forget to make a note of these as you'll need them to access your website later! Below these options, you can specify the country of your server and the default time zone. These are usually picked up from your server itself, but you may want to change them: Click on the Save and continue button to progress now; the next step is Configure store. Here you can set your Default store country field (if it's different from your server settings) and opt to install Drupal Commerce's demo, which includes sample content and a sample Drupal Commerce theme too: Further down on this screen, you're presented with more options. By checking the Do you want to be able to translate the interface of your store? field, Drupal Commerce provides you with an ability to translate your website for customers of different languages (for this simple store installation, leave this set to No). Finally, you can set the Default store currency field you wish to use, and whether you want Commerce Kickstart to set up a sales tax rule for your store (select which is more appropriate for your store, or leave it set to No sample tax rate for now): Click on Create and finish at the bottom of the screen. If you chose to install the demo store in the previous screen, you will have to wait as it is added for you. There are now options to allow Drupal to check for updates automatically, and to receive e-mails about security updates. Leave these both checked to help you stay on top of keeping your Drupal website secure and up-to-date. Wait as Commerce Kickstart installs everything Drupal Commerce requires to run. That's it! Your Drupal Commerce store is now up and running thanks to Commerce Kickstart 2. How it works... The Commerce Kickstart package includes Drupal 7 core and the Drupal Commerce module. By packaging these together, installation and initial configuration for your Drupal Commerce store is made much easier! Creating your first product Now that you've installed Drupal Commerce, you can start to add products to display to customers and start making money. In this recipe you will learn how to add a basic product to your Drupal Commerce store. Getting started Log in to your Drupal Commerce store's administration panel, and navigate to Products | Add a product: If you haven't, navigate to Site settings | Modules and ensure that the Commerce Kickstart Menu module is enabled for your store. Note the sample products from Drupal Kickstart's installation are displaying there. How to do it... To get started adding a product to your store, click on the Add product button and follow these steps: Click on the Product display. Product displays groups of multiple related product variations together for display on the frontend of your website. Fill in the form that appears, entering a suitable Title, using the Body field for the product's description, as well as filling in the SKU (stock keeping unit; a unique reference for this product) and Price fields. Ensure that the Status field is set to Active. You can also optionally upload an image for the product here: Optionally, you can assign the product to one of the pre-existing categories in the Product catalog tab underneath these fields, as well as a URL for it in the URL path settings tab: Click on the Save product button, and you've now created a basic product in your store. To view the product on the frontend of your store, you can navigate to the category listings if you imported Drupal Commerce's demo data, or else you can return to the Products menu and click on the name of the product in the Title column: You'll now see your product on the frontend of your Drupal Commerce store: How it works... In Drupal Commerce, a product can represent several things, listed as follows: A single product for sale (for example, a one-size-fits-all t-shirt) A variation of a product (for example, a medium-size t-shirt) An item that is not necessarily a purchase as such (for example, it may represent a donation to a charity) An intangible product which the site allows reservations for (for example, an event booking) Product displays (for example, a blue t-shirt) are used to group product variations (for example, a medium-sized blue t-shirt and a large-sized blue t-shirt), and display them on your website to customers. So, depending on the needs of your Drupal Commerce website, products may be displayed on unique pages, or multiple products might be grouped onto one page as a product display.

The goal of this article is to learn how to work with sprites and get to know their main properties. After reading this article, you will be able to add sprites to your games. In this article, we will cover the following topics: Setting up the initial project Sprites and their main properties Adding sprites to the scene Adding sprites as a child node of another sprite Manipulating sprites (moving, flipping, and so on) Performance considerations when working with many sprites Creating spritesheets and using the sprite batch node to optimize performance Using basic animation Creating the game project We could create many separate mini projects, each demonstrating a single Cocos2D aspect, but this way we won't learn how to make a complete game. Instead, we're going to create a game that will demonstrate every aspect of Cocos2D that we learn. The game we're going to make will be about hunting. Not that I'm a fan of hunting, but taking into account the material we need to cover and practically use in the game's code, a hunting game looks like the perfect candidate. The following is a screenshot from the game we're going to develop. It will have several levels demonstrating several different aspects of Cocos2D in action: Time for action – creating the Cocohunt Xcode project Let's start creating this game by creating a new Xcode project using the Cocos2D template, just as we did with HelloWorld project, using the following steps: Start Xcode and navigate to File | New | Project… to open the project creation dialog. Navigate to the iOS | cocos2d v3.x category on the left of the screen and select the cocos2d iOS template on the right. Click on the Next button. In the next dialog, fill out the form as follows: Product Name: Cocohunt Organization Name: Packt Publishing Company Identifier: com.packtpub Device Family: iPhone Click on the Next button and pick a folder where you'd like to save this project. Then, click on the Create button. Build and run the project to make sure that everything works. After running the project, you should see the already familiar Hello World screen, so we won't show it here. Make sure that you select the correct simulator version to use. This project will support iPhone, iPhone Retina (3.5-inch), iPhone Retina (4-inch), and iPhone Retina (4-inch, 64-bit) simulators, or an actual iPhone 3GS or newer device running iOS 5.0 or higher. What just happened? Now, we have a project that we'll be working on. The project creation part should be very similar to the process of creating the HelloWorld project, so let's keep the tempo and move on. Time for action – creating GameScene As we're going to work on this project for some time, let's keep everything clean and tidy by performing the following steps: First of all, let's remove the following files as we won't need them: HelloWorldScene.h HelloWorldScene.m IntroScene.h IntroScene.m We'll use groups to separate our classes. This will allow us to keep things organized. To create a group in Xcode, you should right-click on the root project folder in Xcode, Cocohunt in our case, and select the New Group menu option (command + alt + N). Refer to the following sceenshot: Go ahead and create a new group and name it Scenes. After the group is created, let's place our first scene in it. We're going to create a new Objective-C class called GameScene and make it a subclass of CCScene. Right-click on the Scenes group that we've just created and select the New File option. Right-clicking on the group and selecting New File instead of using File | New | File will place our new file in the selected group after creation. Select Cocoa Touch category on the left of the screen and the Objective-C class on the right. Then click on the Next button. In the next dialog, name the the class as GameScene and make it a subclass of the CCScene class. Then click on the Next button. Make sure that you're in the Cocohunt project folder to save the file and click on the Create button. You can create the Scenes folder while in the save dialog using New Folder button and save the GameScene class there. This way, the hierarchy of groups in Xcode will match the physical folders hierarchy on the disk. This is the way I'm going to do this so that you can easily find any file in the book's supporting file's projects. However, the groups and files organization within groups will be identical, so you can always just open the Cocohunt.xcodeproj project and review the code in Xcode. This should create the GameScene.h and GameScene.m files in the Scenes group, as you can see in the following screenshot: Now, switch to the AppDelegate.m file and remove the following header imports at the top: #import "IntroScene.h" #import "HelloWorldScene.h" It is important to remove these #import directives or we will get errors as we removed the files they are referencing. Import the GameScene.h header as follows: #import "GameScene.h" Then find the startScene: method and replace it with following: -(CCScene *)startScene { return [[GameScene alloc] init]; } Build and run the game. After the splash screen, you should see the already familiar black screen as follows: What just happened? We've just created another project using the Cocos2D template. Most of the steps should be familiar as we have already done them in the past. After creating the project, we removed the unneeded files generated by the Cocos2D template, just as you will do most of the time when creating a new project, since most of the time you don't need those example scenes in your project. We're going to work on this project for some time and it is best to start organizing things well right away. This is why we've created a new group to contain our game scene files. We'll add more groups to the project later. As a final step, we've created our GameScene scene and displayed it on the screen at the start of the game. This is very similar to what we did in our HelloWorld project, so you shouldn't have any difficulties with it.

Understanding common errors There are five main areas that cause the majority of problems for Joomla! sites. Understanding these areas and the common problems that occur with in each of them is a very important part of fixing them and thus, our site. Even though there is a practically unlimited number of potential issues and problems that can occur, there are certain problems which occur much more regularly than others. If we understand these main problems, we should be able to take care of many of the problems that will occur on our site without needing to resort to hiring people to fix them, or waiting for extension developers to provide support. The five areas are: PHP code JavaScript code CSS/HTML code Web server Database We will now look at the two most common error sources, PHP and JavaScript. PHP code Because PHP code is executed on the server, we usually have some control over the conditions that it is subject to. Most PHP errors originate from one of four sources: Incorrect extension parameters PHP code error PHP version Server settings Incorrect extension parameters It is often easy to misunderstand what the correct value for an extension parameter is, or if a particular parameter is required or not. These misunderstandings are behind a large number of PHP "errors" that developers experience when building a site. Diagnosis In a well-coded extension, putting the wrong information into a parameter shouldn't result in an error, but will usually result in the extension producing strange or unexpected output, or even no output at all. In a poorly coded extension, an incorrect parameter value will probably cause an error. These errors are often easy to spot, especially in modules, because our site will output everything it processed up until the point of the error, giving our page the appearance of being cut off. Some very minor errors may even result in the whole page, except for the error causing extension, being output correctly, and error messages appearing in the page, where the extension with the error was supposed to appear. A critical error, however, may cause the site to crash completely, and output only an error message. In extreme cases not even an error message will be output, and visitors will only see a white screen. The messages should always appear in our PHP log though. Fixing the problem Incorrect extension parameters are the easiest problems to fix, and are often solved simply by going through the parameter screens for the extensions on the page with the errors, and making sure they all have correct values. If they all look correct, then we may want to try changing some parameters to see if that fixes the issue. If this still doesn't work, then we have a genuine error. PHP code error Extension developers aren't perfect, and even the best ones can overlook or miss small issues in the code. This is especially true with large, complex extensions so please remember that even if an extension has PHP code error, it may not necessarily mean that the whole extension is poorly coded. Diagnosis Similar to incorrect extension parameters, a PHP coding error will usually result in a cut-off page, or a white screen, sometimes with an error message displayed, sometimes without. Whether an error message is displayed or not depends partly on the configuration of your server, and partly on how severe the error was. Some servers are configured to suppress error output of certain types of errors. Regardless of the screen output, all PHP errors should be output to the PHP log. So, if we get a white screen, or even get a normal screen but strange output, checking our PHP log can often help us to find the problem. PHP logs can reside in different places on differently configured servers, although it will almost always be in a directory called logs. We may also not have direct access to the log, again depending on our server host. We should ask our web hosting company's support staff for the location of our PHP log, if we can't easily find it. Some common error messages and causes are: Parse error: parse error, unexpected T_STRING in... This is usually caused by a missing semi-colon at the end of a line, or a missing double quote (") or end bracket ()) after we opened one. For quotes and semicolons, the problem is usually the line above the one reported in the error. For missing brackets, the error will sometimes occur at the end of the script, even though the problem code is much earlier in the script. Parse error: syntax error, unexpected $end in... We are most likely missing a closing brace (}) somewhere. Make sure that each open brace ({) we have has been closed with a closing brace (}). Parse error: syntax error, unexpected T_STRING, expecting ',' or ';' in... There may be double quotes within double quotes. They either need to be escaped, using a forward slash before the inside quote, or changed to single quotes. Fixing the problem Fixing a PHP code error is possible but can be difficult depending on the extension. Usually when there is a PHP code error, it will give a brief description of the error, and a line number. If nothing is being output at all, then we may need to turn error reporting up as described later. We will then go to the line specified to examine it and the lines around it to try and find our problem. If we can't find an obvious error, then it might be better to take the error back to the developer and ask them for support. PHP version The current version of PHP is 5.x.x and version 6.x is expected soon, but because many older, but still popular, applications only run on PHP version 4.x.x. It is still very common to find many Web hosting companies still using PHP 4 on their servers. This problem is even more unfortunate due to the fact that PHP 4 isn't even supported anymore by the PHP developers. In PHP 5, there are many new functions and features that don't exist in PHP 4. As a result, using these functions in an extension will cause it to error when run on a PHP 4 server. Diagnosis Diagnosing if we have the wrong PHP version is not obvious, as it will usually result in an error about an unknown function when the extension tries to call a function that doesn't exist in the version of PHP installed on our server. Sometimes, the error will not be that the function is unknown, but that the number of parameters we are sending it is incorrect if they were changed between PHP 4 and PHP 5. Fixing the problem The only real way to fix the problem is to upgrade our PHP version. Some web hosts offer PHP 4 or 5 as an option and it might be as simple as checking a box or clicking a button to turn on PHP 5. In case if our host doesn't offer PHP 5 at all, the only solution is to use a different extension or change our web host. This may actually be a good idea anyway, because if our host is still using an unsupported PHP version with no option to upgrade, then what other unsupported, out of date software is running those servers? Server settings One of the most common problems encountered by site owners in regards to server settings is file permissions. Many web hosting companies run Linux, which uses a three-part permission model, on their servers. Using this model, every file can have separate permissions set for: The user who owns the particular file Other users in the same user group as the owner Everyone else (in a web site situation this is mainly the site visitors) Each file also has three permissions that enable, or disable, certain actions on the file. These permissions are read, write, and execute. Permissions are usually expressed in one of two ways, first as single characters in a file listing or as a three digit number. For example, a file listing on a Linux server might look like this: drwxr-x--- 2 auser agroup 4096 Dec 28 04:09 tmp-rwxr-x--- 1 auser agroup 345 Sep 1 04:12 somefile.php-rwxr--r-- 1 auser agroup 345 Sep 1 04:12 foo The very first character to the left, a d or – in this case, indicates if this is a directory (the d) or a file (the -). The next nine characters indicate the permissions and who they apply to. The first three belong to the file owner, next three to those in the same group as the owner, and the final three to everyone else. The letters used are: r—read permission w—write permission x—execute permission A dash (-) indicates that this permission hasn't been given to those particular people. So in our example above, tmp.php can be read, written to, or executed by the owner (a user). It can be read or executed (but not written to) by other users in the same group (agroup) as the owner, but the file cannot be used at all by people outside the group.foo however, can be read by people in the owners group, and also read by everyone else, but it cannot be executed by them. As mentioned above, permissions are also often expressed as a three-digit number. Each of the digits represents the sum of the numbers that represent the permissions granted. For example: r = 4, w = 2, and x = 1. Adding these together gives us a number from 0-7, which can indicate the permission level. So a file with a permission level of 644 would translate as: 6 = 4 + 2 = rw4 = r4 = r or -rw-r--r-- in the first notation that we looked at. Most servers are set by default to one of the following: 644 -rw-r--r--755 -rwxr-xr-x775 -rwxrwxr-x All of this looks fine so far. The problems start to creep in depending on how the server runs their PHP. PHP can either be set up to run as the same user who owns all the files (usually our FTP user or hosting account owner), or it can be set up to run as a different user, but in the same group as the owner. Or it can be set up to be a completely different user and group, as illustrated here: The ideal setup, from a convenience point of view, is the first one where PHP is executed as the same user who owns the files. This setup should have no problems with permissions. But the ideal setup for a very security-conscious web host is the third one since the PHP engine can't be used to hack the web site files, or the server itself. A web server with this setup though used to have a difficult time running a Joomla! site. It was difficult because changing the server preferences requires that files be edited by the PHP user, uploading extensions means that folders and files need to be created by the PHP user, and so on. If the PHP engine isn't even in the same group as the file owner, then it gets treated the same as any site visitor and can usually only read, and probably execute files, but not change them. This prevents us from editing preferences or uploading new extensions. However, if we changed the files so that the PHP engine could edit and execute them (permission 777, for example) then anyone who can see our site on the internet can potentially edit and execute our files by themselves, making our site very vulnerable to being hacked by even a novice hacker. We should never give files or directories a permission of 777 (read, write, and execute to all three user types) because it is almost guaranteed that our site will be hacked eventually as a result. If, for some reason, we need to do it for testing, or because we need to in order to install extensions, then we should change it back as soon as possible. Diagnosis To spot this problem is relatively simple. If we can't edit our web site configuration, or install any extensions at all, then nine times out of ten, server permissions will be the problem. Fixing the problem We can start by asking our web host if they allow PHP to be run as CGI, or install suEXEC (technical terms for running it as the same user who owns the files) and if so, how do we set it up. If they don't allow this, then the next best situation is to enable the Joomla! FTP layer in our configuration. This will force Joomla! to log into our site as the FTP user, which is almost always the same user that uploaded the site files, and edit or install files. We can enable the FTP layer by going to the Site | Global Configuration page and then clicking on the server item in the menu below the heading. We can then enter the required information for the FTP layer on this screen. The FTP layer should only be used on Linux-based servers. More information about the FTP layer can be found in the official Joomla! documentation at http://help.joomla.org/content/view/1941/302/1/2/ If for some reason the FTP layer doesn't work, we only have two other options. We could change our web hosting provider as one option. Or, whenever we want to install a new extension or change our configuration, we need to change the permissions on our folders, perform our tasks, and then change the permissions back to their original settings.

(For more resources related to this topic, see here.) Case study description Weir & Bell Telecom is a manufacturer and retailer of mobile phones and devices that has experienced rapid growth, following a number of highly successful marketing campaigns, and aggressive growth on the high street and internet. It has a very strong line up of products that are considered niche products and that result in high profit margins. With competitors poised to release similar products, the company realizes that, in order to achieve its business objectives and thus maintain its leadership position, it needs to vastly improve and exploit its IT processes and infrastructure. Having grown rapidly, the IT department has not developed in line with the business strategy and the board has recently appointed a new Chief Information Officer (CIO) to oversee the IT department, and to develop an IT strategy that is closely aligned with the business strategy. The aim is to allow the organization to increase the efficiency and responsiveness of its supply chain processes by making effective use of digitized processes. Weir & Bell Telecom has been quite progressive in its IT development to date, and has been using SOA for a number of years. However, SOA has been employed tactically rather than strategically. Furthermore, the company had made some initial investment in Oracle Fusion Middleware products such as Oracle SOA Suite 11g, WebCenter, and Oracle Database. The new CIO has identified SOA as a key enabler to allow the IT department to respond in an agile manner to business and regulatory changes. The assessment The new CIO immediately commissioned a review of the existing SOA infrastructure and an assessment of the maturity of service delivery. The assessment was executed in three primary phases as depicted in the following diagram: In order to gather all the information in a structured way and to subsequently quantify the results in a meaningful fashion, the Oracle Maturity Model was employed. The model was used to quantify each capability domain using a tool created by the company. This allowed the CIO to obtain an accurate view of the "As-Is" situation and to develop and quantify a realistic "To-Be" model. The CIO set aside a period of three weeks (one week per phase) for the execution of the assessment, the elaboration of recommendations and the resulting SOA Roadmap. As the timeframe was very aggressive, it was agreed that the organization should commission the consultancy services of a niche player with a background in SOA and SOA Governance. The phases of the assessment consisted of the following: Evaluate the As-Is: This phase consisted of gathering current information from each capability domain and quantifying it as accurately as possible. The main deliverable of this phase was the assessment model populated with the As-Is data. Evaluate the To-Be: In this phase, a target maturity level was defined for each capability based upon realistic and viable assumptions. For example, a period of one year was assumed as the target period to achieve the desired maturity. It was considered that a longer period might prove too long in a rapidly evolving market and would result in W&B losing its competitive edge in the retail space. Elaborate the results: In this phase, the outcome of the maturity assessment was mapped to a SOA Roadmap by specifying how the gap between the As-Is and the To-Be could be addressed. The results The results of the SOA Maturity Assessment mentioned in the previous section were as shown in the following diagram: The consultancy conducting the maturity assessment identified a number of key challenges as follows: The IT department has grown immaturely while trying to constantly react to the demands placed on it by strong retail sales. Consequently, there has been a lack of overall architecture and governance, with IT developments being very tactical in nature. Each business unit within Weir & Bell Telecom has commissioned and developed its own IT systems with little regard for what had already been developed previously. Partly due to the lack of architecture and visibility over existing assets, many IT projects have been delivered late and there has been little reuse of existing assets. This has resulted in a poor perception of IT by key business stakeholders. Development teams have had little guidance; immature processes and an ill-defined software development lifecycle have resulted in estimates for development being at best optimistic and at worst poorly thought through. This has caused severe problems to the business as IT systems are continually developed late despite best intentions and lots of overtime. Poor estimation models and lack of architecture were identified as key contributors to poor planning, execution, and dependency management. SOA has not lived up to expectations with duplicate services being developed on a per-project basis with virtually no reuse of existing assets. This was attributed to the distinct lack of overall governance. In occasions, services were being overused, meaning used above the maximum throughput for which the service was designed to support. This resulted in performance penalties in the SOA infrastructure and therefore damaging the user experience. The assessment concluded that, in order to bring the current maturity level of 1.3 to the required level of 3.8, strong architecture and a new governance framework were required. This would result in other dependent disciplines being intrinsically matured (such as project management and support). The following diagram illustrates the identified health status of the different areas of Governance when compared to a Reference Governance Framework: Furthermore, the outcome of the assessment suggested that, because of a lack of overall strategy and almost nonexistent design-time and runtime governance, SOA initiatives had failed to deliver any measurable business benefits. The objectives The CIO responded to the results of the SOA Maturity Assessment by creating an enterprise architect team whose responsibility was to ensure that business needs were clearly reflected in all future IT development efforts, and that the underlying infrastructure was able to scale to meet business demands. The team was also responsible for rationalizing existing systems and services and for reducing or removing redundancy. One of the first actions for the architecture team was to create a reference architecture that specified a blueprint for all IT systems within Weir & Bell Telecom. The reference architecture supported IT agility by putting SOA at the heart of system development. This would allow the IT department to keep pace with the demands of the business moving forward and facilitate the introduction of new sales channels as identified in the business strategy for growth. The enterprise architecture team also set out clear objectives for SOA Governance. Moreover, it was concluded that, in order to achieve these objectives, appropriate tooling would be required to support the desired processes and responsibilities. While the targeted objectives of the framework were known (as it can be appreciated from the following diagram), the tools needed to support these objectives was yet to be determined. Invitations to tender were consequently sent out to the main software vendors and a selection process was undertaken to determine which could provide the infrastructure required to underpin the governance effort. A key driver for product selection was support for design-time governance, encompassing service discovery, and cataloging for reuse, dependency management, service lifecycle, and support for the enforcement of policies and standards. The product was also required to support effective runtime governance with support for a common UDDI registry, runtime policy enforcement (including security), exception handling, service deployment, and system monitoring. After an exhaustive product selection exercise, the Oracle SOA Governance Solution was chosen as the preferred software toolset to support the implementation of SOA Governance at Weir & Bell Telecom. As the preceding diagram suggests, the decision was made mainly for two reasons. Firstly, the product offering from Oracle was extremely strong at supporting design-time and runtime Governance, and therefore was highly aligned with the team's objectives for its governance effort. Secondly, as Oracle already had a considerable footprint in the technology landscape at Weir & Bell telecom, it was concluded that implementing the Oracle SOA Governance Solution would have considerable benefits as these products were certified to integrate with other Oracle Fusion Middleware products already implemented in the company, such as the Oracle SOA Suite 11g. The business case The enterprise architecture team, under the guidance of the new CIO, created a compelling business case that was presented in the board of directors at Weir & Bell Telecom. The outcome of this presentation was pivotal to transforming the image of the IT department, which was perceived as being inefficient and slow to react to business changes, and also to secure any needed funding. SOA was at the heart of the business case. The CIO recognized SOA to be the primary vehicle for introducing the agility required to meet the aggressive demands placed on the IT function by the retail arm of Weir & Bell Telecom. The maturity assessment was an essential part of the business case and it gave the CIO a real understanding of the strengths and weaknesses of his department and allowed him to focus on areas that needed to be addressed immediately. The alignment of IT strategy to business strategy was fundamental to realizing maximum return on investment (ROI) in terms of services and infrastructure. The business case, together with supporting evidence gathered during the SOA Maturity Assessment, highlighted how real synergy between IT and the business could result in automation and improvement of vital processes in the supply chain. Rationalization of the underlying infrastructure and a move to virtualization, together with the elimination of redundancy in terms of software applications and services across business units, could lead to significant cost savings across the organization. The following diagram depicts how a governed SOA implementation could significantly reduce the number of integration points and hand-offs between business units, paving the way to increased process efficiencies and cost savings: The business case discussed how the introduction of Oracle Enterprise Repository to support design-time Governance was essential to ensuring that services were rationalized across the enterprise. Better insight into existing services together with the introduction of design patterns as part of the governance effort would give rise to services that were more granular and more consumable for new processes. This in turn would allow IT to introduce new sales channels by utilizing existing services that communicated with back end order and logistics systems. It described how the reuse of service assets led to the decreased cost of development while the introduction of a new Software Development Lifecycle (SDLC) and continuous integration would lead to better quality software being produced by developers. This in turn would have a huge impact on the test department who were traditionally viewed in a bad light by management. Previously, quality was almost invariably introduced at the testing stage as testers battled with poor quality code leading to a huge testing cycle that in turn led to late delivery of software and increased costs. The Oracle Enterprise Repository combined with the Oracle Service Registry to enhance runtime Governance allowed Weir & Bell Telecom to share its services with third parties and preferred suppliers, leading to further efficiency savings in the supply chain processes. It illustrated how a large number of requisitions could be automatically raised when supplies fell below a threshold value; depending on business rules, some could even be automatically approved and sent to suppliers without human intervention. This would lead to much smoother delivery of important components and reduced component lead time while allowing the manufacturing department to have a far better control of inventory; all key factors in allowing the business to react to successful marketing and sales campaigns. Subsequent sections of the business case focused on justifying the cost of implementing governance and illustrated how the benefits would outweigh any spending in a relatively short period of time. This provided the board with key metrics for return on investment. It documented how savings could be realized in different areas. For example, on the current SOA implementation there was little or no reuse of existing services. Huge savings could be realized in the software development costs (CAPEX), by promoting code reuse by strict design-time Governance and by enforcing policies using Oracle Enterprise Repository. The business case also showed how reuse and policy enforcement would have a positive impact on operating support costs (OPEX), since with proper governance the number of services would increase gradually and not exponentially. Therefore, fewer people are required to support the services and also tasks that are usually complex, such as troubleshooting, would become more predictable since all services would implement known patterns and utilize a common exception handling framework. It is further explained that, by employing Oracle Web Services Manager (OWSM) to secure services in the production environment, the code itself could be separated from the security implementation, bringing with it the ability to declaratively define security for all service implementations, and delegate definition of security policies to the security department. OWSM also provided the ability to monitor whether SLAs were being met. Another key justification presented in the business case centered around how critical business logic embedded in old and extremely complex legacy systems could be extracted and presented as web services, and could be utilized in key value chain processes. The SOA layer would be used to provide an abstraction layer between legacy systems and the business processes themselves. This would allow the legacy applications to be replaced at a later date without having to change the process itself. The impact of change would be absorbed by the service implementation and transformation logic, the process being unaffected as the service interface would not change. This would, in turn, facilitate legacy modernization that was attractive to the board of directors and it served to illustrate how SOA could be used in different ways to deliver business benefits. Moving forward, all service implementations would be written to exacting standards to ease their consumption in the future processes and Oracle Enterprise Repository would act as a single source of truth for the SOA portfolio. Architects and analysts would be able to use the repository for version control, dependency tracking, impact analysis, and service discovery and cataloging. In short, the entire service lifecycle would be properly governed both at design-time and at runtime. The business case concluded by explaining how SOA Governance would lead to better management of the heterogeneous SOA landscape and would deliver greater visibility of end-to-end service networks and better tracking of usage, allowing the IT department to report on return on investment for SOA assets across the enterprise. All of this would ultimately transform into benefits to the business in the form of costs savings and greater flexibility. In short, a win-win situation for the business. The business case was presented to the board of directors and the feedback was very positive. The budget to begin the roadmap activities was approved. However, the board demanded that all benefits, as documented on the business case, should be supported with critical success factors that could be measured at different checkpoints throughout the SOA program. Critical success factors Based on the feedback from the board of directors, the new CIO of Weir & Bell Telecom placed the monitoring and measurement of critical success factors and business metrics at the heart of the architectural effort. By aligning the IT strategy with the business strategy and being able to report back to the board on the return on investment for IT-related projects, he would be able to demonstrate the effectiveness of IT across the enterprise. The key here was to prove that the use of SOA would result in significant cost savings while also increasing development productivity and reducing software maintenance costs. The resulting performance improvements were expected to have a significant impact on business agility and enable the organization to respond quickly to competitive challenges. The critical success factors for the architectural effort were heavily reliant on the fact that development and support costs would be reduced by enforcing governance, most notably in the reuse of existing SOA assets. The reasoning behind this proposition is explained in the following paragraphs. A SOA asset's estimated value is based on the development costs avoided by reusing rather than recreating. By reusing existing assets, an organization avoids both the costs of repeatedly developing the same functionality and the cost of maintaining different implementations of the same functionality. Thus, re-use consolidates functionality and reduces redundancy. The more that re-use occurs, the greater the savings as the diagram suggests. The difference between the time a developer would spend building an asset for single use and the time needed to use a reusable version of that asset is referred to as Predicted Net Hours Saved. This represents the development cost avoided through re-use of the asset. The following formula was used to make these calculations: Establishing the expected ROI of assets within the portfolio requires consideration of the annual usage of each asset, and the conversion of the total Predicted Net Hours Saved to a monetary value. An enterprise registry/repository with the appropriate capabilities is essential in tracking progress towards organizational SOA and re-use goals. It should be utilized to track each Asset's value, state (defined, designed, implemented, upgraded, and retired); then, as Assets are used, the registry/repository should gather the necessary data to generate reports that details an Asset's value. This provides a way for organizations to compare the estimated ROI to the actual returns in an SOA initiative. While reusable assets may vary in their size, scope, and purpose, their estimated valuation and the accurate reporting of their actual value are essential in guiding SOA Governance efforts to ensure sustainable alignment with business goals. SOA Governance is essential for increasing the value of assets by insuring their alignment with and support of policies and standards as established at the architectural, IT, and corporate levels. An asset's value is determined not solely by its reusability or re-use, but also by the contribution it makes in moving the organization toward its business goals. To that end, the importance of effective SOA lifecycle governance in guiding the design, development, support, and retirement of assets cannot be overstated. A new reference architecture The enterprise architecture team began by creating a new reference architecture that would be strongly aligned to the business strategy. This is presented in the following diagram: The IT strategy was developed with the target operating model in mind including the requirement for new sales channels. The underlying infrastructure was ported to a private cloud-based model to allow for the rapid growth predicted by the business and also in light of expected acquisitions that were due to be made over the coming years. SOA was endorsed as the preferred architectural style for software development with existing and new commercially available and off-the-shelf (COTS) products having little or no customizations moving forward. Proposed IT developments were assessed and aligned to the IT strategy with a number of projects scrapped and new initiatives introduced. The new governance process A new governance process was introduced across the enterprise to reduce redundancy of all software development efforts between business units. A clearly defined SOA Roadmap was created indicating how the current state of the IT organization could move as efficiently as possible to the new To-Be model. A whole suite of governance standards was created, documented, and distributed to the development teams. These standards clearly specified the development, programming, and deployment standards that were to be adopted for all SOA initiatives enterprise wide. Design-time governance was introduced, covering all aspects of service discovery, design, versioning, and decommissioning. The Oracle Enterprise Repository was introduced to support design-time Governance by ensuring that all service metadata and XML artifacts were accessible across the enterprise in an effort to promote re-use, and therefore to ensure that key benefits were realized from existing investment in services. Oracle Enterprise Repository was also used to harvest assets enterprise-wide, ensuring that existing investment was not lost as part of the governance initiative. Oracle JDeveloper was already used by the development teams for creating services. Integration of the JDeveloper SOA development environment with the Repository and Service Registry was seen as a key requirement to enable developers to access and synchronize between the repository and the development environment to automatically harvest assets and enhance productivity. Runtime governance standards were defined and documented. A new deployment framework was created to standardize system set up and service deployment. This would enable IT infrastructure to grow efficiently as the business demands increased. Oracle Service Registry was introduced to facilitate rapid discovery and consumption of existing services. Improved visibility Another key recommendation highlighted as part of the SOA Maturity Assessment was the requirement to improve runtime visibility of SOA assets. As a result, the runtime engines (such as the Oracle Service Registry and Oracle Enterprise Manager) were integrated with Oracle Enterprise Repository to ensure that key metrics around service utilization and service re-use were feedback into the asset's details page. Having design-time and runtime systems integrated, empowered project managers, SOA architects, and designers to enforce policies around service re-use, which ultimately accounted in savings to the company. Standardized error reporting and system monitoring was introduced with Oracle Enterprise Manager (OEM) being adopted across the enterprise. Business Activity Monitoring (BAM) was also introduced on selected business processes, following a review of business metrics and key performance indicators with the business. Oracle BAM and OEM gave the support team increased visibility into application performance and availability, allowing them to more proactively monitor problems and suggest improvements to a number of key business processes. In some cases, process lifecycles were decreased by as much as 60 percent and cost savings in one area ran into thousands of pounds. This exercise rapidly improved the availability of essential services and processes, which in turn provided a genuine real good factor to the business. The latter was important as the board started to fund further process improvement projects. Furthermore, continuous integration was implemented so that software was tested regularly ensuring that the quality of software was much higher than had previously been the case. The SOA Roadmap highlighted further areas of business automation and senior business executives were able to understand and digest how IT could now work hand in hand with the business to introduce real cost savings and to streamline the supply chain further. The exercise conducted for identifying key business metrics and KPIs also helped the business to focus on important processes that were key to the success of Weir & Bell Telecom. Understanding the business metrics helped the stakeholders to stop and realize a number of business improvements particularly in areas where previously hand-offs between business units were creating unnecessary delays and inefficiencies. Very importantly, IT and the business now had the ability to measure, qualitatively and quantitatively, key business areas, allowing the board for the first time to have a real understanding of the impact of IT investment moving forward. This could range from bottom line costs savings to simply improving the lives of some of its staff by freeing them from tasks that could otherwise be automated, thus freeing them up for more value-added activities. Summary In this article we described a case study based on a real live SOA Governance implementation as experienced by the authors. The article illustrated how the governance concepts such as maturity assessments, reference architectures, SOA assets, and design-time and runtime Governance, were put into practice in order to successfully implement SOA Governance into an organization. This article emphasized the importance of having the right level of sponsorship and buy-in from the business before embarking on a governance implementation. This was achieved up front, since without business sponsorship the benefits that SOA can bring to the table would have not been visible or achievable. Additionally, the article described how by presenting a business case with clear business benefits and objectives it was possible to qualitatively and quantitatively measure success. This in turn allowed the board of directors to understand the positive impact that IT and SOA can bring to business. The article concluded by describing how the Oracle SOA Governance toolset was implemented alongside the company's existing Oracle SOA and technological landscape in order to deliver design-time and runtime Governance. Resources for Article: Further resources on this subject: Oracle Integration and Consolidation Products [Article] SOA Application Design [Article] Process Driven SOA Development [Article]

Software architecture acts as a blueprint for the system, using abstraction to control the system's complexity and establish inter-component communication. In the ever-evolving landscape of software architecture, a groundbreaking innovation has emerged, reshaping the way developers design and optimize their systems. Enter ChatGPT, an advanced language model that has revolutionized the field with its remarkable capabilities. With its deep understanding of natural language, ChatGPT is unlocking new horizons in software architecture. From streamlining development processes to enhancing user interactions, this article delves into the transformative potential of ChatGPT, exploring how it is reshaping the very foundations of software architecture as we know it.In this piece, we'll examine the value of software architecture and how chatGPT may help us build it.  Architecture Design with ChatGPTThe entire design of the software, its elements, and its behavior are represented by the software system architect. In a nutshell, it is a visual representation of how software applications are made by connecting their many components. The following are examples of software architecture activities: Implementation Details: It could be architectural artifacts, source code, documentation, repository, etc.Implementation Design decisions: It includes options for technology (cloud or on-premises), architectural style (monolithic or distributed), storage (AWS S3, Azure Blob, ADLS Gen2, etc.), ingestion pattern (batch or real-time streaming pattern), and more.Infrastructure Considerations: Deployment Choices, Component Configurations, etc.Let’s understand in detail about the process of software architecture.Requirement Gathering: Any project should begin with functional and non-functional requirements since they will determine how to create the software architecture and how to prepare the finished product in accordance with the needs of the stakeholders.Create a Helicopter view of the solution: Using the mind map, provide a high-level overview of the system's constituent parts. It is a useful method for capturing your requirements as a diagram.Refine functional and non-functional requirements: Examine the non-functional needs in terms of performance, security, cost-effectiveness, etc. once you have thoroughly refined the functional requirements to comprehend the overall functioning of the program.Design each component in detail: Start with creating the platform's infrastructure and creating the application components while taking into account the functional and non-functional needs.Create a phase-wise approach for the implementation: Once the infrastructure and application implementation design considerations are clear, begin preparing deliverables for a phased rollout. It should include the state of architecture as it is today, architecture in transition, and architecture in the future. Make a visual representation of the application and infrastructure, taking into account networking, security, and interconnection issues.Below are the best practices that can be followed while designing the software architecture for any application:Design your application considering the best and worst-case scenarioDesign your application which should be able to scale up and scale downCreate loosely coupled architecture design for the smooth functioning of the system Create a multi-thread processing to speed up the processing for batch and real-time streaming implementationDesign your application with 7 layers of security implementationMake a choice to store your dataNow, Let’s use ChatGPT to create a software architecture for an application. Our goal is to create a data mesh architecture for the retail industry using the azure technology. The major requirement is also to define and capture data architecture requirements.Image 1 - Part 1, Data mesh responseImage 2 - Part 2, Data mesh responseImage 3- Part 3, Data mesh responseImage 4- Part 4, Data mesh response ChatGPT first offers suggestions for a group of Azure services to be used to implement the batch and real-time streaming patterns after receiving input from the user. The principles for implementing a data mesh architecture are then provided, including domain-driven data ownership, data-driven products, self-service data infrastructure, and data governance, including data discovery and monitoring.Let’s check with chatGPT on how the networking setup should be done for these Azure services:Image 5: Part 1, Networking responseImage 6: Part 2 Networking response Image 7: Part 3 Networking response  The configuration of a VNET, subnet, NSG rules, Azure firewall, VNET peering, VPN gateway, and the private link is advised by chatGPT. These Azure networking components can be used to manage the services' interconnection. Another major requirement to implement data mesh architecture is also to check how domain-specific data will be managed and operationalized for the data mesh architecture. Image 8: Part 1,  Domain-specific Data ManagementImage 9: Part 2,  Domain-specific Data ManagementImage 10: Part 3,  Domain-specific Data ManagementThe goal of chatGPT's proposals is to produce domain-specific cleansed data that end users can utilize directly to extract value from the data. Since these data domain stores are being built by domain experts, they are also in charge of managing, supporting, and operationalizing the data as needed.ConclusionIn this post, we looked in detail at the overall process of software architecture design as well as its sequential process. The best techniques for creating the software architecture for any application were also demonstrated. Later, we used chatGPT to develop a data mesh architecture implementation, complete with networking setup and operationalization, for a retail domain.Author BioSagar Lad is a Cloud Data Solution Architect with a leading organization and has deep expertise in designing and building Enterprise-grade Intelligent Azure Data and Analytics Solutions. He is a published author, content writer, Microsoft Certified Trainer, and C# Corner MVP.Link - Medium , Amazon , LinkedIn  

In this article written by Tom Ryder, author of the book Nagios Core Administration Cookbook, Second Edition, we will cover the following topics: Installing a plugin Removing a plugin Creating a new command Customizing an existing command (For more resources related to this topic, see here.) Introduction Nagios Core is perhaps best thought of as a monitoring framework and less as a monitoring tool.Its modular design allows any kind of program that returns appropriate values based on some kind of check as a check_command option for a host or service. This is where the concepts of commands and pluginscome into play. For Nagios Core, a plugin is any program that can be used to gather information about a host or service. To ensure that a host is responding to ping requests, we'd use a plugin, such as check_ping,which when run against a hostname or address—whether by Nagios Core or not—returns a status code to whatever called it, based on whether a response was received to the pingrequest within a certain period of time. This status code and any accompanying message is what Nagios Core uses to establish the state that a host or service is in. Plugins are generally just like any other program on a Unix-like system; they can be run from the command line, are subject to permissions and owner restrictions, can be written in any language, can read variables from their environment, and can take parameters and options to modify how they work. Most importantly, they are entirely separate from Nagios Core itself (even if programmed by the same people), and the way that they're used by the application can be changed. To allow for additional flexibility in how plugins are used, Nagios Core uses these programs according to the terms of a command definition. A command for a specific plugin defines the way in which that plugin is used, including its location in the filesystem, any parameters that should be passed to it, and any other options. In particular, parameters and options often include thresholds for the WARNINGand CRITICAL states. Nagios Core is usually downloaded and installed alongside a set of plugins called Nagios Plugins, available at https://nagios-plugins.org/, which this article assumes you have installed. These plugins were chosen because they cover the most common needs for a monitoring infrastructure quite well as a set, including checks for common services, such as web services, mail services, DNS services, and others as well as more generic checks, such as whether a TCP or UDP port is accessible and open on a server. It's possible that for most, if not all, of our monitoring needs, we won't need any other plugins—but if we do, Nagios Core makes it possible to use existing plugins in novel ways using custom command definitions, adding third-party plugins written by contributors on the Nagios Exchange website or even writing custom plugins ourselves from scratch in some special cases. Installing a plugin In this recipe, we'll install a custom plugin that we retrieved from Nagios Exchange onto a Nagios Core server so that we can use it in a Nagios Core command, and hence check a service with it. Getting ready You should have a Nagios Core 4.0 or newer server running with a few hosts and services configured already, and you should have found an appropriate plugin to install to solve some particular monitoring needs. Your Nagios Core server should have Internet connectivity to allow you to download the plugin directly from the website. In this example, we'll use check_rsync,which is available on the Web at https://exchange.nagios.org/directory/Plugins/Network-Protocols/Rsync/check_rsync/details. This particular plugin is quite simple,consisting of a single Perlscript with only very basic dependencies. If you want to install this script as an example,the server will also need to have a Perl interpreter installed, for example, in /usr/bin/perl. This example will also include directly testing a server running an rsync(1)daemon called troy.example.net. How to do it... We can download and install a new plugin using the following steps: Copy the URL for the download link for the most recent version of the check_rsync plugin. Navigate to the plugins directory for the Nagios Core server. The default location is /usr/local/nagios/libexec: # cd /usr/local/nagios/libexec Download the plugin using the wget command into a file called check_rsync. It's important to enclose the URL in quotes: # wget 'https://exchange.nagios.org/components/com_mtree/attachment. php?link_id=307&cf_id=29' -O check_rsync Make the plugin executable using the chmod(1) and chown(1) commands: # chown root.nagios check_rsync # chmod 0770 check_rsync Run the plugin directly with no arguments to check that it runs and to get usage instructions. It's a good idea to test it as the nagios user using the su(8) or sudo(8) commands:# sudo -s -u nagios$ ./check_rsync Usage: check_rsync -H <host> [-p <port>] [-m <module>[,<user>,<password>] [-m <module>[,<user>,<password>]...]] Try running the plugin directly against a host running rsync(1) to check whether it works and reports a status: $ ./check_rsync -H troy.example.net The output normally starts with the status determined, with any extra information after a colon: OK: Rsync is up If all of this works, then the plugin is now installed and working correctly. How it works... Because Nagios Core plugins are programs in themselves, all that installing a plugin really amounts to is saving a program or script into an appropriate directory, in this case, /usr/local/nagios/libexec, where all the other plugins live. It's then available to be used the same way as any other plugin. The next step once the plugin is working is defining a command for it in the Nagios Core configuration so that it can be used to monitor hosts and/or services. This can be done with the Creating a new commandrecipe in this article. There's more... If we inspect the Perl script, we can see a little bit of how it works. It works like any other Perl script except perhaps for the fact that its return valuesare defined in a hash called %ERRORS,and the return values it chooses depend on what happens when it tries to check the rsync(1)process. This is the most important part of implementing a plugin for Nagios Core. Installation procedures for different plugins vary. In particular, many plugins are written in languages like C, and hence, they need to be compiled. One such plugin is the popular check_nrpe plugin.Rather than simply being saved into a directory and made executable, these sorts of plugins often follow the usual pattern of configuration, compilation, and installation: $ ./configure $ make # make install For many plugins that are built in this style, the final step of make installwill often install the compiled plugin into the appropriate directory for us. In general, if instructions are included with the plugin, it pays to read them to see how best to install it. See also The Removing a plugin recipe The Creating a new command recipe Removing a plugin In this recipe, we'll remove a plugin that we no longer need as part of our Nagios Core installation. Perhaps it's not working correctly, the service it monitors is no longer available, or there are security or licensing concerns with its usage. Getting ready You should have a Nagios Core 4.0 or newer server running with a few hosts and services configured already and have a plugin that you would like to remove from the server. In this instance, we'll remove the now unneeded check_rsync plugin from our Nagios Core server. How to do it... We can remove a plugin from our Nagios Core instance using the following steps: Remove any part of the configuration that uses the plugin, including the hosts or services that use it for check_command and command definitions that refer to the program. As an example, the following definition for a command would no longer work after we remove the check_rsync plugin: define command { command_name check_rsync command_line $USER1$/check_rsync -H $HOSTADDRESS$ } Using a tool, such as grep(1), can be a good way to find mentions of the command and plugin: # grep -R check_rsync /usr/local/nagios/etc Change the directory on the Nagios Core server to wherever the plugins are kept. The default location is /usr/local/nagios/libexec: # cd /usr/local/nagios/libexec Delete the plugin with the rm(1) command: # rm check_rsync Validate the configuration and restart the Nagios Core server: # /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg # /etc/init.d/nagios reload How it works... Nagios Core plugins are simply external programs that the server uses to perform checks of hosts and services. If a plugin is no longer needed, all that we need to do is remove references to it in our configuration, if any, and delete it from /usr/local/nagios/libexec. There's more... There's not usually any harm in leaving the plugin's program on the server even if Nagios Core isn't using it. It doesn't slow anything down or cause any other problems, and it may be needed later. Nagios Core plugins are generally quite small programs and should not really cause disk space concerns on a modern server. See also The Installing a plugin recipe The Creating a new command recipe Creating a new command In this recipe, we'll create a new command for a plugin that was just installed into the /usr/local/nagios/libexecdirectory in the Nagios Core server. This will define the way in which Nagios Core should use the plugin, and thereby allow it to be used as part of a service definition. Getting ready You should have a Nagios Core 4.0 or newer server running with a few hosts and services configured already and have a plugin installed for which you'd like to define a new command so that you can use it as part of a service definition. In this instance, we'll define a command for an installed check_rsyncplugin. How to do it... We can define a new command in our configuration as follows: Change to the directory containing the objects configuration for Nagios Core. The default location is /usr/local/nagios/etc/objects: # cd /usr/local/nagios/etc/objects Edit the commands.cfg file: # vi commands.cfg At the bottom of the file, add the following command definition: define command {     command_name  check_rsync     command_line  $USER1$/check_rsync -H $HOSTADDRESS$ } Validate the configuration and restart the Nagios Core server: # /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg # /etc/init.d/nagios reload If the validation passed and the server restarted successfully, we should now be able to use the check_rsync command in a service definition. How it works... The configuration we added to the commands.cfgfile in the preceding steps defines a new command called check_rsync,which specifies a method for using the plugin of the same name to monitor a service. This enables us to use check_rsyncas a value for the check_commanddirective in a service declaration, which might look like this: define service {     use                  generic-service     host_name            troy.example.net     service_description  RSYNC     check_command        check_rsync } Only two directives are required for command definitions, and we've defined both: command_name: This defines the unique name with which we can reference the command when we use it in host or service definitions command_line: This defines the command line that should be executed by Nagios Core to make the appropriate check This particular command line also uses the following two macros: $USER1$: This expands to /usr/local/nagios/libexec, the location of the plugin binaries, including check_rsync. This is defined in the sample configuration in the /usr/local/nagios/etc/resource.cfg file. $HOSTADDRESS$: This expands to the address of any host for which this command is used as a host or service definition. So, if we used the command in a service, checking the rsync(1) server on troy.example.net, the completed command might look like this: $ /usr/local/nagios/libexec/check_rsync -H troy.example.net We can run this straight from the command line ourselves as the nagios userto see what kind of results it returns: $ /usr/local/nagios/libexec/check_rsync -H troy.example.net OK: Rsync is up There's more... A plugin can be used for more than one command. If we had a particular rsync(1) module, which we wanted to check named backup, we could write another command called check_rsync_backupas follows: define command {     command_name  check_rsync_backup     command_line  $USER1$/check_rsync -H $HOSTADDRESS$ -m backup } Alternatively, if one or more of our rsync(1) servers were running on an alternate port, say, port 5873, we could define a separate command check_rsync_altport for that: define command {     command_name  check_rsync_altport     command_line  $USER1$/check_rsync -H $HOSTADDRESS$ -p 5873 } Commands can thus be defined as precisely as we need them to be. We explore this in more detail in the Customizing an existing commandrecipe in this article. See also The Installing a plugin recipe The Customizing an existing command recipe Customizing an existing command In this recipe, we'll customize an existing command definition. There are a number of reasons why you might want to do this, but a common one is if a check is overzealous, sending notifications for the WARNING orCRITICALstates, which aren't actually terribly worrisome, or on the other hand, if a check is too "forgiving" and doesn't flag hosts or services as having problems when it would actually be appropriate to do so. Another reason is to account for peculiarities in your own network. For example, if you run HTTPdaemons on a large number of hosts in your hosts on the alternative port 8080 that you need to check, it would be convenient to have a check_http_altportcommand available. We can do this by copying and altering the definition for the vanilla check_httpcommand. Getting ready You should have a Nagios Core 4.0 or newer server running with a few hosts and services configured already. You should also already be familiar with the relationship between services, commands, and plugins. How to do it... We can customize an existing command definition as follows: Change to the directory containing the objects configuration for Nagios Core. The default location is /usr/local/nagios/etc/objects: # cd /usr/local/nagios/etc/objects Edit the commands.cfg or whichever file is an appropriate location for the check_http command: # vi commands.cfg Find the definition for the check_http command. In a default Nagios Core configuration, it should look something like this: # 'check_http' command_definition define command {     command_name  check_http     command_line  $USER1$/check_http -I $HOSTADDRESS$ $ARG1$ } Copy this definition into a new definition directly under it and alter it to look like the following, renaming the command and adding a new option to its command line: # 'check_http_altport' command_definition define command {     command_name  check_http_altport     command_line  $USER1$/check_http -I $HOSTADDRESS$ -p 8080 $ARG1$ } Validate the configuration and restart the Nagios Core server: # /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg # /etc/init.d/nagios reload If the validation passed and the server restarted successfully, we should now be able to use the check_http_altportcommand, which is based on the original check_httpcommand, in a service definition. How it works... The configuration we added to the commands.cfgfile in the preceding steps reproduces the command definition for check_http,but changes it in two ways: It renames the command from check_http to check_http_alt, which is necessary to distinguish the commands from one another. Command names in Nagios Core, like host names, must be unique. It adds the -p 8080 option to the command line call, specifying that when the call to check_http is made, the check will be made using TCP port 8080 rather than the default value for TCP port 80. The check_http_altcommand can now be used as a check command in the same way a check_httpcommand can be used. For example, a service definition that checks whether the sparta.example.nethost is running an HTTP daemon on port 8080 might look something like this: define service {     use                  generic-service     host_name            sparta.example.net     service_description  HTTP_8080     check_command        check_http_alt } There's more... This recipe's title implies that we should customize the existing commands by editing them in-place, and indeed, this works fine if we really do want to do things this way. Instead of copying the command definition, we can just add -p 8080 or any other customization to the command line and change the original command. However, this is bad practice in most cases, mostly because it can break existing monitoring and can be potentially confusing to other administrators of the Nagios Core server. If we have a special case for monitoring, in this case, checking a nonstandard port for HTTP, then it's wise to create a whole new command based on the existing one with the customisations we need. Particularly if you share monitoring configuration duties with someone else on your team, changing the command can break the monitoring for anyone who had set up the services using the check_http command beforeyou changed it, meaning that their checks would all start failing because port 8080 would be checked instead. There is no limit to the number of commands you can define, so you can be very liberal in defining as many alternative commands as you need. It's a good idea to give them instructive names that say something about what they do as well as to add explanatory comments to the configuration file. You can add a comment to the file by prefixing it with a # character: # # 'check_http_altport' command_definition. This is to keep track of the # servers that have administrative panels running on an alternative port # to confer special privileges to a separate instance of Apache HTTPD # that we don't want to confer to the one for running public-facing # websites. # define command {     command_name  check_http_altport     command_line  $USER1$/check_http -H $HOSTADDRESS$ -p 8080 $ARG1$ } See also The Creating a new command recipe Writing a new plugin from scratch Even given the very useful standard plugins in the Nagios Plugins set and the large number of custom plugins available on Nagios Exchange, occasionally, as our monitoring setup becomes more refined, we may well find that there is some service or property of a host that we would like to check, but for which there doesn't seem to be any suitable plugin available. Every network is different, and sometimes, the plugins that others have generously donated their time to make for the community don't quite cover all your bases. Generally, the more specific your monitoring requirements get, the less likely it is for there to be a plugin available that does exactly what you need. In this example, we'll deal with a very particular problem that we'll assume can't be dealt with effectively by any known Nagios Core plugins, and we'll write one ourselves using Perl. Here's the example problem. Our Linux security team wants to be able to automatically check whether any of our servers are running kernels that have known exploits. However, they're not worried about every vulnerable kernel, only certain ones. They have provided us with the version numbers of three kernels that have small vulnerabilities that they're not particularly worried about but that do need patching, and one they're extremely worried about. Let's say the minor vulnerabilities are in the kernels with version numbers 2.6.19, 2.6.24, and 3.0.1. The serious vulnerability is in the kernel with version number 2.6.39. Note that these version numbers in this case are arbitrary and don't necessarily reflect any real kernel vulnerabilities! The team could log in to all of the servers individually to check them, but the servers are of varying ages and access methods, and they are managed by different people. They would also have to check manually more than once because it's possible that a naive administrator could upgrade to a kernel that's known to be vulnerable in an older release, and they also might want to add other vulnerable kernel numbers for checking later on. So, the team have asked us to solve the problem with Nagios Core monitoring, and we've decided that the best way to do it is to write our own plugin, check_vuln_kernel, thatchecks the output of uname(1)for a kernel version string, and then does the following: If it's one of the slightly vulnerable kernels, it will return a WARNING state so that we can let the security team know that they should address it when they're next able to. If it's the highly vulnerable kernel version, it will return a CRITICAL state so that the security team knows that a patched kernel needs to be installed immediately. If uname(1) gives an error or output we don't understand, it will return an UNKNOWN state, alerting the team to a bug in the plugin or possibly more serious problems with the server. Otherwise, it returns an OK state, confirming that the kernel is not known to be a vulnerable one. Finally, in the Nagios Core monitoring, they want to be able to see at a glance what the kernel version is and whether it's vulnerable or not. For the purposes of this example, we'll only monitor the Nagios Core server; however, via NRPE, we'd be able to install this plugin on the other servers that require this monitoring, they'll work just fine here as well. While this problem is very specific, we'll approach it in a very general way, which you'll be able to adapt to any solution where it's required for a Nagios plugin to: Run a command and pull its output into a variable. Check the output for the presence or absence of certain patterns. Return an appropriate status based on those tests. All that this means is that if you're able to do this, you'll be able to monitor anything effectively from Nagios Core! Getting ready You should have a Nagios Core 4.0 or newer server running with a few hosts and services configured already. You should also already be familiar with the relationship between services, commands, and plugins. You should have Perl installed, at least version 5.10. This will include the required POSIX module. You should also have the Perl modules Nagios::Plugin(or Monitoring::Plugin) andReadonly installed. On Debian-like systems, you can install this with the following: # apt-get install libnagios-plugin-perl libreadonly-perl On RPM-based systems, such as CentOS or Fedora Core, the following command should work: # yum install perl-Nagios-Plugin perl-Readonly This will be a rather long recipe that ties in a lot of Nagios Core concepts. You should be familiar with all the following concepts: Defining new hosts and services and how they relate to one another Defining new commands and how they relate to the plugins they call Installing, testing, and using Nagios Core plugins Some familiarity with Perl would also be helpful, but it is not required. We'll include comments to explain what each block of code is doing in the plugin. How to do it... We can write, test, and implement our example plugin as follows: Change to the directory containing the plugin binaries for Nagios Core. The default location is /usr/local/nagios/libexec: # cd /usr/local/nagios/libexec Start editing a new file called check_vuln_kernel: # vi check_vuln_kernel Include the following code in it. Take note of the comments, which explain what each block of code is doing. #!/usr/bin/env perl   # Use strict Perl style use strict; use warnings; use utf8;   # Require at least Perl v5.10 use 5.010;   # Require a few modules, including Nagios::Plugin use Nagios::Plugin; use POSIX; use Readonly;   # Declare some constants with patterns that match bad kernels Readonly::Scalar my $CRITICAL_REGEX => qr/^2[.]6[.]39[^d]/msx; Readonly::Scalar my $WARNING_REGEX =>   qr/^(?:2[.]6[.](?:19|24)|3[.]0[.]1)[^d]/msx;   # Run POSIX::uname() to get the kernel version string my @uname   = uname(); my $version = $uname[2];   # Create a new Nagios::Plugin object my $np = Nagios::Plugin->new();   # If we couldn't get the version, bail out with UNKNOWN if ( !$version ) {     $np->nagios_die('Could not read kernel version     string'); }   # Exit with CRITICAL if the version string matches the critical pattern if ( $version =~ $CRITICAL_REGEX ) {     $np->nagios_exit( CRITICAL, $version ); }   # Exit with WARNING if the version string matches the warning pattern if ( $version =~ $WARNING_REGEX ) {     $np->nagios_exit( WARNING, $version ); }   # Exit with OK if neither of the patterns matched $np->nagios_exit( OK, $version ); Make the plugin owned by the nagios group and executable with chmod(1): # chown root.nagios check_vuln_kernel # chmod 0770 check_vuln_kernel Run the plugin directly to test it: # sudo -s -u nagios $ ./check_vuln_kernel VULN_KERNEL OK: 3.16.0-4-amd64 We should now be able to use the plugin in a command, and hence in a service check just like any other command. How it works... The code we added in the new plugin file, check_vuln_kernel,earlier is actually quite simple: It runs Perl's POSIX uname implementation to get the version number of the kernel If that didn't work, it exits with the UNKNOWN status If the version number matches anything in a pattern containing critical version numbers, it exits with the CRITICAL status If the version number matches anything in a pattern containing warning version numbers, it exits with the WARNING status Otherwise, it exits with the OK status It also prints the status as a string along with the kernel version number, if it was able to retrieve one. We might set up a command definition for this plugin, as follows: define command {     command_name  check_vuln_kernel     command_line  $USER1$/check_vuln_kernel } In turn, we might set up a service definition for that command, as follows: define service {     use                  local-service     host_name            localhost     service_description  VULN_KERNEL     check_command        check_vuln_kernel } If the kernel was not vulnerable, the service's appearance in the web interface might be something like this: However, if the monitoring server itself happened to be running a vulnerable kernel, it might look more like this (and send consequent notifications, if configured to do so): There's more... This may be a simple plugin, but its structure can be generalised to all sorts of monitoring tasks. If we can figure out the correct logic to return the status we want in an appropriate programming language, then we can write a plugin to do basically anything. A plugin like this can just as effectively be written in C or for improved performance, but we'll assume for simplicity's sake that high performance for the plugin is not required, we can instead use a language that's better suited for quick ad hoc scripts like this one, in this case, Perl. The utils.shfile,also in /usr/local/nagios/libexec, allows us to write in shell script if we'd prefer that. If you prefer Python, the nagiosplugin library should meet your needs for both Python 2 and Python 3. Ruby users may like the nagiosplugin gem. If you write a plugin that you think could be generally useful for the Nagios community at large, consider putting it under a free software license and submitting it to the Nagios Exchange so that others can benefit from your work. Community contribution and support is what has made Nagios Core such a great monitoring platform in such wide use. Any plugin you publish in this way should confirm to the Nagios Plugin Development Guidelines. At the time of writing, these are available at https://nagios-plugins.org/doc/guidelines.html. You may find older Nagios Core plugins written in Perl using the utils.pm file instead of Nagios::Plugin or Monitoring::Plugin. This will work fine, but Nagios::Plugin is recommended, as it includes more functionality out of the box and tends to be easier to use. See also The Creating a new command recipe The Customizing an existing command recipe Summary In this article, we learned about how to install a custom plugin that we retrieved from Nagios Exchange onto a Nagios Core server so that we can use it in a Nagios Core command, removing a plugin that we no longer need as part of our Nagios Core installation, creating new command, writing and customizing commands. Resources for Article: Further resources on this subject: An Introduction To NODE.JS Design Patterns [article] Developing A Basic Site With NODE.JS And EXPRESS [article] Creating Our First App With IONIC [article]

In this article series, we will develop a secure Java EE application based on Java EE and GlassFish capabilities. In course of the article, we will cover following topics: Analyzing Java EE application security requirements Including security requirements in Java EE application design Developing secure Business layer using EJBs Developing secure Presentation layer using JSP and Servlets Configuring deployment descriptors of Java EE applications Specifying security realm for enterprise applications Developing secure application client module Configuring Application Client Container Read Designing Secure Java EE Applications in GlassFish here. Developing the Presentation layer The Presentation layer is the closest layer to end users when we are developing applications that are meant to be used by humans instead of other applications. In our application, the Presentation layer is a Java EE web application consisting of the elements listed in the following table. In the table you can see that different JSP files are categorized into different directories to make the security description easier. Element Name Element Description Index.jsp Application entry point. It has some links to functional JSP pages like toMilli.jsp and so on. auth/login.html This file presents a custom login page to user when they try to access a restricted resource. This file is placed inside auth directory of the web application. auth/logout.jsp Logging users out of the system after their work is finished. auth/loginError.html Unsuccessful login attempt redirect users to this page. This file is placed inside auth directory of the web application jsp/toInch.jsp Converts given length to inch, it is only available for managers. jsp/toMilli.jsp Converts given length to millimeter, this page is available to any employee. jsp/toCenti.jsp Converts given length to centimeter, this functionality is available for everyone. Converter Servlet Receives the request and invoke the session bean to perform the conversion and returns back the value to user. auth/accessRestricted.html An error page for error 401 which happens when authorization fails. Deployment Descriptors The deployment descriptors which we describe the security constraints over resources we want to protect. Now that our application building blocks are identified we can start implementing them to complete the application. Before anything else let's implement JSP files that provides the conversion GUI. The directory layout and content of the Web module is shown in the following figure: Implementing the Conversion GUI In our application we have an index.jsp file that acts as a gateway to the entire system and is shown in the following listing: <html> <head><title>Select A conversion</title></head> <body><h1>Select A conversion</h1> <a href="auth/login.html">Login</a> <br/> <a href="jsp/toCenti.jsp">Convert Meter to Centimeter</a> <br/> <a href="jsp/toInch.jsp">Convert Meter to Inch</a> <br/> <a href="jsp/toMilli.jsp">Convert to Millimeter</a><br/> <a href="auth/logout.jsp">Logout</a> </body> </html> Implementing the Converter servlet The Converter servlet receives the conversion value and method from JSP files and calls the corresponding method of a session bean to perform the actual conversion. The following listing shows the Converter servlet content: @WebServlet(name="Converter", urlPatterns={"/Converter"}) public class Converter extends HttpServlet { @EJB private ConversionLocal conversionBean; protected void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { } @Override protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { System.out.println("POST"); response.setContentType("text/html;charset=UTF-8"); PrintWriter out = response.getWriter(); try{ int valueToconvert = Integer.parseInt(request.getParameter("meterValue")); String method = request.getParameter("method"); out.print("<hr/> <center><h2>Conversion Result is: "); if (method.equalsIgnoreCase("toMilli")) { out.print(conversionBean.toMilimeter(valueToconvert)); } else if (method.equalsIgnoreCase("toCenti")) { out.print(conversionBean.toCentimeter(valueToconvert)); } else if (method.equalsIgnoreCase("toInch")) { out.print(conversionBean.toInch(valueToconvert)); } out.print("</h2></center>"); }catch (AccessLocalException ale) { response.sendError(401); }finally { out.close(); } } } Starting from the beginning we are using annotation to configure the servlet mapping and servlet name instead of using the deployment descriptor for it. Then we use dependency injection to inject an instance of Conversion session bean into the servlet and decide which one of its methods we should invoke based on the conversion type that the caller JSP sends as a parameter. Finally, we catch javax.ejb.AccessLocalException and send an HTTP 401 error back to inform the client that it does not have the required privileges to perform the requested action. The following figure shows what the result of invocation could look like: Each servlet needs some description elements in the deployment descriptor or included as deployment descriptor elements. Implementing the conversion JSP files is the last step in implementing the functional pieces. In the following listing you can see content of the toMilli.jsp file. <html> <head><title>Convert To Millimeter</title></head> <body><h1>Convert To Millimeter</h1> <form method=POST action="../Converter">Enter Value to Convert: <input name=meterValue> <input type="hidden" name="method" value="toMilli"> <input type="submit" value="Submit" /> </form> </body> </html> The toCenti.jsp and toInch.jsp files look the same except for the descriptive content and the value of the hidden parameter which will be toCenti and toInch respectively for toCenti.jsp and toInch.jsp. Now we are finished with the functional parts of the Web layer; we just need to implement the required GUI for security measures. Implementing the authentication frontend For the authentication, we should use a custom login page to have a unified look and feel in the entire web frontend of our application. We can use a custom login page with the FORM authentication method. To implement the FORM authentication method we need to implement a login page and an error page to redirect the users to that page in case authentication fails. Implementing authentication requires us to go through the following steps: Implementing login.html and loginError.html Including security description in the web.xml and sun-web.xml or sun-application.xml Implementing a login page In FORM authentication we implement our own login form to collect username and password and we then pass them to the container for authentication. We should let the container know which field is username and which field is password by using standard names for these fields. The username field is j_username and the password field is j_password. To pass these fields to container for authentication we should use j_security_check as the form action. When we are posting to j_security_check the servlet container takes action and authenticates the included j_username and j_password against the configured realm. The listing below shows login.html content. <form method="POST" action="j_security_check"> Username: <input type="text" name="j_username"><br /> Password: <input type="password" name="j_password"><br /> <br /> <input type="submit" value="Login"> <input type="reset" value="Reset"> </form> The following figure shows the login page which is shown when an unauthenticated user tries to access a restricted resource: Implementing a logout page A user may need to log out of our system after they're finished using it. So we need to implement a logout page. The following listing shows the logout.jsp file: <% session.invalidate(); %> <body> <center> <h1>Logout</h1> You have successfully logged out. </center> </body> Implementing a login error page And now we should implement LoginError.html, an authentication error page to inform user about its authentication failure. <html> <body> <h2>A Login Error Occurred</h2> Please click <a href="login.html">here</a> for another try. </body> </html> Implementing an access restricted page When an authenticated user with no required privileges tries to invoke a session bean method, the EJB container throws a javax.ejb.AccessLocalException. To show a meaningful error page to our users we should either map this exception to an error page or we should catch the exception, log the event for audition purposes, and then use the sendError() method of the HttpServletResponse object to send out an error code. We will map the HTTP error code to our custom web pages with meaningful descriptions using the web.xml deployment descriptor. You will see which configuration elements we will use to do the mapping. The following snippet shows AccessRestricted.html file: <body> <center> <p>You need to login to access the requested resource. To login go to <a href="auth/login.html">Login Page</a></p></center> </body> Configuring deployment descriptors So far we have implemented required files for the FORM-based authentication and we only need to include required descriptions in the web.xml file. Looking back at the application requirement definitions, we see that anyone can use meter to centimeter conversion functionality and any other functionality that requires the user to login. We use three different HTML pages for different types of conversion. We do not need any constraint on toCentimeter.html therefore we do not need to include any definition for it. Per application description, any employee can access the toMilli.jsp page. Defining security constraint for this page is shown in the following listing: <security-constraint> <display-name>You should be an employee</display-name> <web-resource-collection> <web-resource-name>all</web-resource-name> <description/> <url-pattern>/jsp/toMillimeter.html</url-pattern> <http-method>GET</http-method> <http-method>POST</http-method> <http-method>DELETE</http-method> </web-resource-collection> <auth-constraint> <description/> <role-name>employee_role</role-name> </auth-constraint> </security-constraint> We should put enough constraints on the toInch.jsp page so that only managers can access the page. The listing included below shows the security constraint definition for this page. <security-constraint> <display-name>You should be a manager</display-name> <web-resource-collection> <web-resource-name>Inch</web-resource-name> <description/> <url-pattern>/jsp/toInch.html</url-pattern> <http-method>GET</http-method> <http-method>POST</http-method> </web-resource-collection> <auth-constraint> <description/> <role-name>manager_role</role-name> </auth-constraint> </security-constraint> Finally we need to define any role we used in the deployment descriptor. The following snippet shows how we define these roles in the web.xml page. <security-role> <description/> <role-name>manager_role</role-name> </security-role> <security-role> <description/> <role-name>employee_role</role-name> </security-role> Looking back at the application requirements, we need to define data constraint and ensure that username and passwords provided by our users are safe during transmission. The following listing shows how we can define the data constraint on the login.html page. <security-constraint> <display-name>Login page Protection</display-name> <web-resource-collection> <web-resource-name>Authentication</web-resource-name> <description/> <url-pattern>/auth/login.html</url-pattern> <http-method>GET</http-method> <http-method>POST</http-method> </web-resource-collection> <user-data-constraint> <description/> <transport-guarantee>CONFIDENTIAL</transport-guarantee> </user-data-constraint> </security-constraint> One more step and our web.xml file will be complete. In this step we define an error page for HTML 401 error code. This error code means that application server is unable to perform the requested action due to negative authorization result. The following snippet shows the required elements to define this error page. <error-page> <error-code>401</error-code> <location>AccessRestricted.html</location> </error-page> Now that we are finished with declaring the security we can create the conversion pages and after creating these pages we can start with Business layer and its security requirements. Specifying the security realm Up to this point we have defined all the constraints that our application requires but we still need to follow one more step to complete the application's security configuration. The last step is specifying the security realm and authentication. We should specify the FORM authentication and per-application description; authentication must happen against the company-wide LDAP server. Here we are going to use the LDAP security realm LDAPRealm. We need to import a new LDIF file into our LDAP server, which contains groups and users definition required for this article. To import the file we can use the following command, assuming that you downloaded the source code bundle from https://www.packtpub.com//sites/default/files/downloads/9386_Code.zip and you have it extracted. import-ldif --ldifFile path/to/chapter03/users.ldif --backendID userRoot --clearBackend --hostname 127.0.0.1 --port 4444 --bindDN cn=gf cn=admin --bindPassword admin --trustAll --noPropertiesFile The following table show users and groups that are defined inside the users.ldif file. Username and password Group membership james/james manager, employee meera/meera employee We used OpenDS for the realm data storage and it had two users, one in the employee group and the other one in the manager group. To configure the authentication realm we need to include the following snippet in the web.xml file. <login-config> <auth-method>FORM</auth-method> <realm-name>LDAPRealm</realm-name> <form-login-config> <form-login-page>/auth/login.html</form-login-page> <form-error-page>/auth/loginError.html</form-error-page> </form-login-config> </login-config> If we look at our Web and EJB modules as separate modules we must specify the role mappings for each module separately using the GlassFish deployment descriptors, which are sun-web.xml and sun-ejb.xml. But we are going to bundle our modules as an Enterprise Application Archive (EAR) file so we can use the GlassFish deployment descriptor for enterprise applications to define the role mapping in one place and let all modules use that definitions. The following listing shows roles and groups mapping in the sun-application.xml file. <sun-application> <security-role-mapping> <role-name>manager_role</role-name> <group-name>manager</group-name> </security-role-mapping> <security-role-mapping> <role-name>employee_role</role-name> <group-name>employee</group-name> </security-role-mapping> <realm>LDAPRealm</realm> </sun-application> The security-role-mapping element we used in sun-application.xml has the same schema as the security-role-mapping element of the sun-web.xml and sun-ejb-jar.xml files. You should have noticed that we have a realm element in addition to role mapping elements. We can use the realm element of the sun-application.xml to specify the default authentication realm for the entire application instead of specifying it for each module separately. Summary In this article series, we covered the following topics: Analyzing Java EE application security requirements Including security requirements in Java EE application design Developing secure Business layer using EJBs Developing secure Presentation layer using JSP and Servlets Configuring deployment descriptors of Java EE applications Specifying security realm for enterprise applications Developing secure application client module Configuring Application Client Container We learnt how to develop a secure Java EE application with all standard modules including Web, EJB, and application client modules.

 In this article by, Dharmesh Kakadia, author of the book Apache Mesos Essentials, explains how Mesos works internally in detail. We will start off with cluster scheduling and fairness concepts, understanding the Mesos architecture, and we will move on towards resource isolation and fault tolerance implementation in Mesos. In this article, we will cover the following topics: The Mesos architecture Resource allocation (For more resources related to this topic, see here.) The Mesos architecture Modern organizations have a lot of different kinds of applications for different business needs. Modern applications are distributed and they are deployed across commodity hardware. Organizations today run different applications in siloed environments, where separate clusters are created for different applications. This static partitioning of cluster leads to low utilization, and all the applications will duplicate the effort of dealing with distributed infrastructures. Not only is this a wasted effort, but it also undermines the fact that distributed systems are hard to build and maintain. This is challenging for both developers and operators. For developers, it is a challenge to build applications that scale elastically and can handle faults that are inevitable in large-scale environment. Operators, on the other hand, have to manage and scale all of these applications individually in siloed environments. The preceding situation is like trying to develop applications without having an operating system and managing all the devices in a computer. Mesos solves the problems mentioned earlier by providing a data center kernel. Mesos provides a higher-level abstraction to develop applications that treat distributed infrastructure just like a large computer. Mesos abstracts the hardware infrastructure away from the applications from the physical infrastructure. Mesos makes developers more productive by providing an SDK to easily write data center scale applications. Now, developers can focus on their application logic and do not have to worry about the infrastructure that runs it. Mesos SDK provides primitives to build large-scale distributed systems, such as resource allocation, deployment, and monitoring isolation. They only need to know and implement what resources are needed, and not how they get the resources. Mesos allows you to treat the data center just as a computer. Mesos makes the infrastructure operations easier by providing elastic infrastructure. Mesos aggregates all the resources in a single shared pool of resources and avoids static partitioning. This makes it easy to manage and increases the utilization. The data center kernel has to provide resource allocation, isolation, and fault tolerance in a scalable, robust, and extensible way. We will discuss how Mesos fulfills these requirements, as well as some other important considerations of modern data center kernel: Scalability: The kernel should be scalable in terms of the number of machines and number of applications. As the number of machines and applications increase, the response time of the scheduler should remain acceptable. Flexibility: The kernel should support a wide range of applications. It should also support diverse frameworks currently running on the cluster and future frameworks as well. The framework should also be able to cope up with the heterogeneity in the hardware, as most clusters are built over time and have a variety of hardware running. Maintainability: The kernel would be one of the very important pieces of modern infrastructure. As the requirements evolve, the scheduler should be able to accommodate new requirements. Utilization and dynamism: The kernel should adapt to the changes in resource requirements and available hardware resources and utilize resources in an optimal manner. Fairness: The kernel should be fair in allocating resources to the different users and/or frameworks. We will see what it means to be fair in detail in the next section. The design philosophy behind Mesos was to define a minimal interface to enable efficient resource sharing across frameworks and defer the task scheduling and execution to the frameworks. This allows the frameworks to implement diverse approaches toward scheduling and fault tolerance. It also makes the Mesos core simple, and the frameworks and core can evolve independently. The preceding figure shows the overall architecture (http://mesos.apache.org/documentation/latest/mesos-architecture) of a Mesos cluster. It has the following entities: The Mesos masters The Mesos slaves Frameworks Communication Auxiliary services We will describe each of these entities and their role, followed by how Mesos implements different requirements of the data center kernel. The Mesos slave The Mesos slaves are responsible for executing tasks from frameworks using the resources they have. The slave has to provide proper isolation while running multiple tasks. The isolation mechanism should also make sure that the tasks get resources that they are promised, and not more or less. The resources on slaves that are managed by Mesos can be described using slave resources and slave attributes. Resources are elements of slaves that can be consumed by a task, while we use attributes to tag slaves with some information. Slave resources are managed by the Mesos master and are allocated to different frameworks. Attributes identify something about the node, such as the slave having a specific OS or software version, it's part of a particular network, or it has a particular hardware, and so on. The attributes are simple key-value pairs of strings that are passed along with the offers to frameworks. Since attributes cannot be consumed by a running task, they will always be offered for that slave. Mesos doesn't understand the slave attribute, and interpretation of the attributes is left to the frameworks. More information about resource and attributes in Mesos can be found at https://mesos.apache.org/documentation/attributes-resources. A Mesos resource or attribute can be described as one of the following types: Scalar values are floating numbers Range values are a range of scalar values; they are represented as [minValue-maxValue] Set values are arbitrary strings Text values are arbitrary strings; they are applicable only to attributes Names of the resources can be an arbitrary string consisting of alphabets, numbers, "-", "/", ".", "-". The Mesos master handles the cpus, mem, disk, and ports resources in a special way. A slave without the cpus and mem resources will not be advertised to the frameworks. The mem and disk scalars are interpreted in MB. The ports resource is represented as ranges. The list of resources a slave has to offer to various frameworks can be specified as the resources flag. Resources and attributes are separated by a semicolon. For example: --resources='cpus:30;mem:122880;disk:921600;ports:[21000-29000];bugs:{a,b,c}' --attributes='rack:rack-2;datacenter:europe;os:ubuntuv14.4' This slave offers 30 cpus, 102 GB mem, 900 GB disk, ports from 21000 to 29000, and have bugs a, b, and c. The slave has three attributes: rack with value rack-2, datacenter with value europe, and os with value ubuntu14.4. Mesos does not yet provide direct support for GPUs, but does support custom resource types. This means that if we specify gpu(*):8 as part of --resources, then it will be part of the resource that offers to frameworks. Frameworks can use it just like other resources. Once some of the GPU resources are in use by a task, only the remaining resources will be offered. Mesos does not yet have support for GPU isolation, but it can be extended by implementing a custom isolator. Alternately, we can also specify which slaves have GPUs using attributes, such as --attributes="hasGpu:true". The Mesos master The Mesos master is primarily responsible for allocating resources to different frameworks and managing the task life cycle for them. The Mesos master implements fine-grained resource sharing using resource offers. The Mesos master acts as a resource broker for frameworks using pluggable policies. The master decides to offer cluster resources to frameworks in the form of resource offers based on them. Resources offer represents a unit of allocation in the Mesos world. It's a vector of resource available on a node. An offer represents some resources available on a slave being offered to a particular framework. Frameworks Distributed applications that run on top of Mesos are called frameworks. Frameworks implement the domain requirements using the general resource allocation API of Mesos. A typical framework wants to run a number of tasks. Tasks are the consumers of resources and they do not have to be the same. A framework in Mesos consists of two components: a framework scheduler and executors. Framework schedulers are responsible for coordinating the execution. An executor provides the ability to control the task execution. Executors can realize a task execution in many ways. An executor can choose to run multiple tasks, by spawning multiple threads, in an executor, or it can run one task in each executor. Apart from the life cycle and task management-related functions, the Mesos framework API also provides functions to communicate with framework schedulers and executors. Communication Mesos currently uses an HTTP-like wire protocol to communicate with the Mesos components. Mesos uses the libprocess library to implement the communication that is located in 3rdparty/libprocess. The libprocess library provides asynchronous communication with processes. The communication primitives have an actor message passing, such as semantics. The libprocess messages are immutable, which makes parallelizing the libprocess internals easier. Mesos communication happens along with the following APIs: Scheduler API: This is used to communicate with the framework scheduler and master. The internal communication is intended to be used only by the SchedulerDriver API. Executor API: This is used to communicate with an executor and the Mesos slave. Internal API: This is used to communicate with the Mesos master and slave. Operator API: This is the API exposed by Mesos for operators and is used by web UI, among other things. Unlike most Mesos API, the operator API is a synchronous API. To send a message, the actor does an HTTP POST request. The path is composed by the name of the actor followed by the name of the message. The User-Agent field is set to "libprocess/…" to distinguish from the normal HTTP requests. The message data is passed as the body of the HTTP request. Mesos uses protocol buffers to serialize all the messages (defined in src/messages/messages.proto). The parsing and interpretation of the message is left to the receiving actor. Here is an example header of a message sent to master to register the framework by scheduler(1) running at 10.0.1.7:53523 address: POST /master/mesos.internal.RegisterFrameworkMessage HTTP/1.1 User-Agent: libprocess/scheduler(1)@10.0.1.7:53523 The reply message header from the master that acknowledges the framework registration might look like this: POST /scheduler(1)/mesos.internal.FrameworkRegisteredMessage HTTP/1.1 User-Agent: libprocess/master@10.0.1.7:5050 At the time of writing, there is a very early discussion about rewiring the Mesos Scheduler API and Executor API as a pure HTTP API (https://issues.apache.org/jira/browse/MESOS-2288). This will make the API standard and integration with Mesos for various tools much easier without the need to be dependent on native libmesos. Also, there is an ongoing effort to convert all the internal messages into a standardized JSON or protocol buffer format (https://issues.apache.org/jira/browse/MESOS-1127). Auxiliary services Apart from the preceding main components, a Mesos cluster also needs some auxiliary services. These services are not part of Mesos itself, and are not strictly required, but they form a basis for operating the Mesos cluster in production environments. These services include, but are not limited to, the following: Shared filesystem: Mesos provides a view of the data center as a single computer and allows developers to develop for the data center scale application. With this unified view of resources, clusters need a shared filesystem to truly make the data center a computer. HDFS, NFS (Network File System), and Cloud-based storage options, such as S3, are popular among various Mesos deployments. Consensus service: Mesos uses a consensus service to be resilient in face of failure. Consensus services, such as ZooKeeper or etcd, provide a reliable leader election in a distributed environment. Service fabric: Mesos enables users to run a number of frameworks on unified computing resources. With a large number of applications and services running, it's important for users to be able to connect to them in a seamless manner. For example, how do users connect to Hive running on Mesos? How does the Ruby on Rails application discover and connect to the MongoDB database instances when one or both of them are running on Mesos? How is the website traffic routed to web servers running on Mesos? Answering these questions mainly requires service discovery and load balancing mechanisms, but also things such as IP/port management and security infrastructure. We are collectively referring to these services that connect frameworks to other frameworks and users as service fabric. Operational services: Operational services are essential in managing operational aspects of Mesos. Mesos deployments and upgrades, monitoring cluster health and alerting when human intervention is required, logging, and security are all part of the operational services that play a very important role in a Mesos cluster. Resource allocation As a data center kernel, Mesos serves a large variety of workloads and no single scheduler will be able to satisfy the needs of all different frameworks. For example, the way in which a real-time processing framework schedules its tasks will be very different from how a long running service will schedule its task, which, in turn, will be very different from how a batch processing framework would like to use its resources. This observation leads to a very important design decision in Mesos: separation of resource allocation and task scheduling. Resource allocation is all about deciding who gets what resources, and it is the responsibility of the Mesos master. Task scheduling, on the other hand, is all about how to use the resources. This is decided by various framework schedulers according to their own needs. Another way to understand this would be that Mesos handles coarse-grain resource allocation across frameworks, and then each framework does fine-grain job scheduling via appropriate job ordering to achieve its needs. The Mesos master gets information on the available resources from the Mesos slaves, and based on resource policies, the Mesos master offers these resources to different frameworks. Different frameworks can choose to accept or reject the offer. If the framework accepts a resource offer, the framework allocates the corresponding resources to the framework, and then the framework is free to use them to launch tasks. The following image shows the high-level flow of Mesos resource allocation: Mesos two level scheduler Here is the typical flow of events for one framework in Mesos: The framework scheduler registers itself with the Mesos master. The Mesos master receives the resource offers from slaves. It invokes the allocation module and decides which frameworks should receive the resource offers. The framework scheduler receives the resource offers from the Mesos master. On receiving the resource offers, the framework scheduler inspects the offer to decide whether it's suitable. If it finds it satisfactory, the framework scheduler accepts the offer and replies to the master with the list of executors that should be run on the slave, utilizing the accepted resource offers. Alternatively, the framework can reject the offer and wait for a better offer. The slave allocates the requested resources and launches the task executors. The executor is launched on slave nodes and runs the framework's tasks. It is up to the framework scheduler to accept or reject the resource offers. Here is an example of events that can happen when allocating resources. The framework scheduler gets notified about the task's completion or failure. The framework scheduler will continue receiving the resource offers and task reports and launch tasks as it sees fit. The framework unregisters with the Mesos master and will not receive any further resource offers. Note that this is optional and a long running service, and meta-framework will not unregister during the normal operation. Because of this design, Mesos is also known as a two-level scheduler. Mesos' two-level scheduler design makes it simpler and more scalable, as the resource allocation process does not need to know how scheduling happens. This makes the Mesos core more stable and scalable. Frameworks and Mesos are not tied to each other and each can iterate independently. Also, this makes porting frameworks easier. The choice of a two-level scheduler means that the scheduler does not have a global knowledge about resource utilization and the resource allocation decisions can be nonoptimal. One potential concern could be about the preferences that the frameworks have about the kind of resources needed for execution. Data locality, special hardware, and security constraints can be a few of the constraints on which tasks can run. In the Mesos realm, these preferences are not explicitly specified by a framework to the Mesos master, instead the framework rejects all the offers that do not meet its constraints. The Mesos scheduler Mesos was the first cluster scheduler to allow the sharing of resources to multiple frameworks. Mesos resource allocation is based on online Dominant Resource Fairness (DRF) called HierarchicalDRF. In a world of single resource static partitioning, fairness is easy to define. DRF extends this concept of fairness to multi-resource settings without the need for static partitioning. Resource utilization and fairness are equally important, and often conflicting, goals for a cluster scheduler. The fairness of resource allocation is important in a shared environment, such as data centers, to ensure that all the users/processes of the cluster get nearly an equal amount of resources. Min-max fairness provides a well-known mechanism to share a single resource among multiple users. Min-max fairness algorithm maximizes the minimum resources allocated to a user. In its simplest form, it allocates 1/Nth of the resource to each of the users. The weighted min-max fairness algorithm can also support priorities and reservations. Min-max resource fairness has been a basis for many well-known schedulers in operating systems and distributed frameworks, such as Hadoop's fair scheduler (http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html), capacity scheduler (https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html), Quincy scheduler (http://dl.acm.org/citation.cfm?id=1629601), and so on. However, it falls short when the cluster has multiple types of resources, such as the CPU, memory, disk, and network. When jobs in a distributed environment use different combinations of these resources to achieve the outcome, the fairness has to be redefined. For example, the two requests <1 CPU, 3 GB> and <3 CPU, 1 GB> come to the scheduler. How do they compare and what is the fair allocation? DRF generalizes the min-max algorithm for multiple resources. A user's dominant resource is the resource for which the user has a biggest share. For example, if the total resources are <8 CPU, 5 GB>, then for the user allocation of <2 CPU, 1 GB>, the user's dominant share is maximumOf(2/8,1/5) means CPU. A user's dominant share is the fraction of the dominant resource that's allocated to the user. In our example, it would be 25 percent (2/8). DRF applies the min-max algorithm to the dominant share of each user. It has many provable properties: Strategy proofness: A user cannot gain any advantage by lying about the demands. Sharing incentive: DRF has a minimum allocation guarantee for each user, and no user will prefer exclusive partitioned cluster of size 1/N over DRF allocation. Single resource fairness: In case of only one resource, DRF is equivalent to the min-max algorithm. Envy free: Every user prefers his allocation over any other allocation of other users. This also means that the users with the same requests get equivalent allocations. Bottleneck fairness: When one resource becomes a bottleneck, and each user has a dominant demand for it, DRF is equivalent to min-max. Monotonicity: Adding resources or removing users can only increase the allocation of the remaining users. Pareto efficiency: The allocation achieved by DRF will be pareto efficient, so it would be impossible to make a better allocation for any user without making allocation for some other user worse. We will not further discuss DRF but will encourage you to refer to the DRF paper for more details at http://static.usenix.org/event/nsdi11/tech/full_papers/Ghodsi.pdf. Mesos uses role specified in FrameworkInfo for resource allocation decision. A role can be per user or per framework or can be shared by multiple users and frameworks. If it's not set, Mesos will set it to the current user that runs the framework scheduler. An optimization is to use deny resource offers from particular slaves for a specified time period. Mesos can revoke tasks allocation killing those tasks. Before killing a task, Mesos gives the framework a grace period to clean up. Mesos asks the executor to kill the task, but if it does not oblige the request, it will kill the executor and all of its tasks. Weighted DRF DRF calculates each role's dominant share and allocates the available resources to the user with the smallest dominant share. In practice, an organization rarely wants to assign resources in a complete fair manner. Most organizations want to allocate resources in a weighted manner, such as 50 percent resources to ads team, 30 percent to QA, and 20 percent to R&D teams. To satisfy this command functionality, Mesos implements weighted DRF, where masters can be configured with weights for different roles. When weights are specified, a client's DRF share will be divided by the weight. For example, a role that has a weight of two will be offered twice as many resources as a role with weight of one. Mesos can be configured to use weighted DRF using the --weights and --roles flags on the master startup. The --weights flag expects a list of role/weight pairs in the form of role1=weight1 and role2=weight2. Weights do not need to be integers. We must provide weights for each role that appear in --roles on the master startup. Reservation One of the other most asked questions for requirement is the ability to reserve resources. For example, persistent or stateful services, such as memcache, or a database running on Mesos, would need a reservation mechanism to avoid being negatively affected on restart. Without reservation, memcache is not guaranteed to get a resource offer from the slave, which has all the data and would incur significant time in initialization and downtime for the service. Reservation can also be used to limit the resource per role. Reservation provides guaranteed resources for roles, but improper usage might lead to resource fragmentation and lower utilization of resources. Note that all the reservation requests go through a Mesos authorization mechanism to ensure that the operator or framework requesting the operation has the proper privileges. Reservation privileges are specified to the Mesos master through ACL along with the rest of the ACL configuration. Mesos supports the following two kinds of reservation: Static reservation Dynamic reservation Static reservation In static reservation, resources are reserved for a particular role. The restart of the slave after removing the checkpointed state is required to change static reservation. Static reservation is thus typically managed by operators using the --resources flag on the slave. The flag expects a list of name(role):value for different resources. If a resource is assigned to role A, then only frameworks with role A are eligible to get an offer for that resource. Any resources that do not include a role or resources that are not included in the --resources flag will be included in the default role (default *). For example, --resources="cpus:4;mem:2048;cpus(ads):8;mem(ads):4096" specifies that the slave has 8 CPUs and 4096 MB memory reserved for "ads" role and has 4 CPUs and 2048 MB memory unreserved. Nonuniform static reservation across slaves can quickly become difficult to manage. Dynamic reservation Dynamic reservation allows operators and frameworks to manage reservation more dynamically. Frameworks can use dynamic reservations to reserve offered resources, allowing those resources to only be reoffered to the same framework. At the time of writing, dynamic reservation is still being actively developed and is targeted toward the next release of Mesos (https://issues.apache.org/jira/browse/MESOS-2018). When asked for a reservation, Mesos will try to convert the unreserved resources to reserved resources. On the other hand, during the unreserve operation, the previously reserved resources are returned to the unreserved pool of resources. To support dynamic reservation, Mesos allows a sequence of Offer::Operations to be performed as a response to accepting resource offers. A framework manages reservation by sending Offer::Operations::Reserve and Offer::Operations::Unreserve as part of these operations, when receiving resource offers. For example, consider the framework that receives the following resource offer with 32 CPUs and 65536 MB memory: {   "id" : <offer_id>,   "framework_id" : <framework_id>,   "slave_id" : <slave_id>,   "hostname" : <hostname>,   "resources" : [     {       "name" : "cpus",       "type" : "SCALAR",       "scalar" : { "value" : 32 },       "role" : "*",     },     {       "name" : "mem",       "type" : "SCALAR",       "scalar" : { "value" : 65536 },       "role" : "*",     }   ] } The framework can decide to reserve 8 CPUs and 4096 MB memory by sending the Operation::Reserve message with resources field with the desired resources state: [   {     "type" : Offer::Operation::RESERVE,     "resources" : [       {         "name" : "cpus",         "type" : "SCALAR",         "scalar" : { "value" : 8 },         "role" : <framework_role>,         "reservation" : {           "framework_id" : <framework_id>,           "principal" : <framework_principal>         }       }       {         "name" : "mem",         "type" : "SCALAR",         "scalar" : { "value" : 4096 },         "role" : <framework_role>,         "reservation" : {           "framework_id" : <framework_id>,           "principal" : <framework_principal>         }       }     ]   } ] After a successful execution, the framework will receive resource offers with reservation. The next offer from the slave might look as follows: {   "id" : <offer_id>,   "framework_id" : <framework_id>,   "slave_id" : <slave_id>,   "hostname" : <hostname>,   "resources" : [     {       "name" : "cpus",       "type" : "SCALAR",       "scalar" : { "value" : 8 },       "role" : <framework_role>,       "reservation" : {         "framework_id" : <framework_id>,         "principal" : <framework_principal>       }     },     {       "name" : "mem",       "type" : "SCALAR",       "scalar" : { "value" : 4096 },       "role" : <framework_role>,       "reservation" : {         "framework_id" : <framework_id>,         "principal" : <framework_principal>       }     },     {       "name" : "cpus",       "type" : "SCALAR",       "scalar" : { "value" : 24 },       "role" : "*",     },     {       "name" : "mem",       "type" : "SCALAR",       "scalar" : { "value" : 61440 },       "role" : "*",     }   ] } As shown, the framework has 8 CPUs and 4096 MB memory reserved resources and 24 CPUs and 61440 MB memory underserved in the resource offer. The unreserve operation is similar. The framework on receiving the resource offer can send the unreserve operation message, and subsequent offers will not have reserved resources. The operators can use/reserve and/unreserve HTTP endpoints of the operator API to manage the reservation. The operator API allows operators to change the reservation specified when the slave starts. For example, the following command will reserve 4 CPUs and 4096 MB memory on slave1 for role1 with the operator authentication principal ops: ubuntu@master:~ $ curl -d slaveId=slave1 -d resources="{          {            "name" : "cpus",            "type" : "SCALAR",            "scalar" : { "value" : 4 },            "role" : "role1",            "reservation" : {              "principal" : "ops"            }          },          {            "name" : "mem",            "type" : "SCALAR",            "scalar" : { "value" : 4096 },            "role" : "role1",            "reservation" : {              "principal" : "ops"            }          },        }"        -X POST http://master:5050/master/reserve Before we end this discussion on resource allocation, it would be important to note that the Mesos community continues to innovate on the resource allocation front by incorporating interesting ideas, such as oversubscription (https://issues.apache.org/jira/browse/MESOS-354), from academic literature and other systems. Summary In this article, we looked at the Mesos architecture in detail and learned how Mesos deals with resource allocation, resource isolation, and fault tolerance. We also saw the various ways in which we can extend Mesos. Resources for Article: Further resources on this subject: Recommender systems dissected Tuning Solr JVM and Container [article] Transformation [article] Getting Started [article]

  Oracle WebCenter 11g PS3 Administration Cookbook Over 100 advanced recipes to secure, support, manage, and administer Oracle WebCenter         Introduction An enterprise portal is a framework that allows users to interact with different applications in a secure way. There is a single point of entry and the security to the composite applications is transparent for the user. Each user should be able to create their own view on the portal. A portal is highly customizable, which means that most of the work will be done at runtime. An administrator should be able to create and manage pages, users, roles, and so on. Users can choose whatever content they want to see on their pages so they can personalize the portal to their needs. In this article, you will learn some basics about the WebCenter Portal application. Later chapters will go into further details on most of the subjects covered in this chapter. It is intended as an introduction to the WebCenter Portal. Preparing JDeveloper for WebCenter When you want to build WebCenter portals, JDeveloper is the preferred IDE. JDeveloper has a lot of built-in features that will help us to build rich enterprise applications. It has a lot of wizards that can help in building the complex configuration files. Getting ready You will need to install JDeveloper before you can start with this recipe. JDeveloper is the IDE from Oracle and can be downloaded from the following link: http://www.oracle.com/technetwork/developer-tools/jdev/downloads/index.html. You will need to download JDeveloper 11.1.1.5 Studio Edition and not JDeveloper 11.1.2 because that version is not compatible with WebCenter yet. This edition is the full-blown edition with all the bells and whistles. It has all the libraries for building an ADF application, which is the basis for a WebCenter application. How to do it... Open JDeveloper that was installed. Choose Default Role. From JDeveloper, open the Help menu and select Check for updates. Click Next on the welcome screen. Make sure all the Update Centers are selected and press Next. In the available Updates, enter WebCenter and select all the found updates. Press Next to start the download. After the download is finished, you will need to restart JDeveloper. You can check if the updates have been installed by opening the About window from the Help menu. Select the Extensions tab and scroll down to the WebCenter extensions. You should be able to see them: How it works... When you first open JDeveloper, you first need to select a role. The role determines the functionality you have in JDeveloper. When you select the default role, all the functionality will be available. By installing the WebCenter extensions, you are installing all the necessary jar files containing the libraries for the WebCenter framework. JDeveloper will have three additional application templates: Portlet Producer Application: This template allows you to create a producer based upon the new JSR286 standard. WebCenter Portal Application: Template that will create a preconfigured portal with ADF and WebCenter technology. WebCenter Spaces Taskflow Customizations: This application is configured for customizing the applications and services taskflows used with the WebCenter Spaces Application. The extensions also include the taskflows and data controls for each of the WebCenter services that we will be integrating in our portal. Creating a WebCenter portal In this release of WebCenter, we can easily build enterprise portals by using the WebCenter Portal application template in JDeveloper. This template contains a preconfigured portal that we can modify to our needs. It has basic administration pages and security. Getting ready For this recipe, you need the latest version of JDeveloper with the WebCenter extensions installed, which is described in the previous recipe. How to do it... Select New from the File menu. Select Application in the General section on the left-hand side. Select WebCenter Portal Application from the list on the right. Press OK. The Create WebCenter Portal Application dialog will open. In the dialog, you will need to complete a few steps in order to create the portal application: Application Name: Specify the application name, directory, and application package prefix. Project Name: Specify the name and directory of the portal project. At this stage, you can also add additional libraries to the project. Project Java Settings: Specify the default package, java source, and output directory. Project WebCenter settings: With this step, you can request to build a default portal environment. When you disable the Configure the application with standard Portal features checkbox, you will have an empty project with only the reference to the WebCenter libraries, but no default portal will be configured. You can also let JDeveloper create a special test-role, so you can test your application. Press the Finish button to create the application. You can test the portal without needing to develop anything. Just start the integrated WebLogic server, right-click the portal project, and select Run from the context menu. When you start the WebLogic server for the first time, it can take a few minutes. This is because JDeveloper will create the WebLogic domain for the integrated WebLogic server. Because we have installed the WebCenter extensions, JDeveloper will also extend the domain with the WebCenter libraries. How it works... When the portal has been started, you will see a single page, which is the Home page that contains a login form at the top right corner: When you log in with the default WebLogic user, you should have complete administration rights. The default user of the integrated WebLogic server is weblogic with password weblogic1. When logged in, you should see an Administration link. This links to the Administration Console where you can manage the resources of your portal like pages, resource catalogs, navigations, and so on. In the Administration Console you have five tabs: Resources: In this tab, you manage all the resources of your portal. The resources are divided into three parts: Structure: In the structure, you manage the resources about the structure of your portal, such as pages, templates, navigations, and resource catalogs. Look and Layout: In the look and layout part, you manage things like skins, styles, templates for the content presenter, and mashup styles. Mashups: Mashups are taskflows created during runtime. You can also manage data controls in the mashup section. Services: In the services tab, you can manage the services that are configured for your portal. Security: In the security tab, you can add users or roles and define their access to the portal application. Configuration: In this tab, you can configure default settings for the portal like the default page template, default navigation, default resource catalog, and default skin. Propagation: This tab is only visible when you create a specific URL connection. From this tab, you can propagate changes from your staging environment to your production environment. There's more... The WebCenter Portal application will create a preconfigured portal for us. It has a basic structure and page navigation to build complex portals. JDeveloper has created a lot of files for us. Here is an overview of the most important files created for us by JDeveloper: Templates The default portal has two page templates. They can be found in the Web Content/oracle/Webcenter/portalapp/pagetemplates folder: pageTemplate_globe.jspx: This is the default template used for a page pageTemplate_swooshy.jspx: This is the same template as the globe template, but with another header image You can of course create your own templates. Pages JDeveloper will create four pages for us. These can be found in the Web Content/oracle/Webcenter/portalapp/pages folder: error.jspx: This page looks like the login page and is designed to show error messages upon login. home.jspx: This is an empty page that uses the globe template. login.jspx: This is the login page. It is also based upon the globe template. Resource catalogs By default, JDeveloper will create a default resource catalog. This can be found in the Web Content/oracle/Webcenter/portalapp/catalogs folder. In this folder, you will find the default-catalog.xml file which represents the resource catalog. When you open this file, you will notice that JDeveloper has a design view for this file. This way it is easier to manage and edit the catalog without knowing the underlying XML. Another file in the catalogs folder is the catalog-registry.xml. This is the set of components that the user can use when creating a resource catalog at runtime. Navigations By using navigations, you can allow users to find content on different pages, taskflow, or even external pages. By defining different navigation, you allow users to have a personalized navigation that fits their needs. By default, you will find one navigation model in the Web content/oracle/Webcenter/portalapp/navigations folder: default-navigation-model.xml. It contains the page hierarchy and a link to the administration page. This model is not used in the template, but it is there as an example. You can of course use this model and modify it, or you can create your own models. You will also find the navigation-registry.xml. This file contains the items that can be used to create a navigation model at runtime. Page hierarchy With the page hierarchy, you can create parent-child relationships between pages. It allows you to create multi-level navigation of existing pages. Within the page hierarchy, you can set the security of each node. You are able to define if a child node inherits the security from its parent or it has its own security. By default, JDeveloper will create the pages.xml page hierarchy in the Web Content/oracle/Webcenter/portalapp/pagehierarchy folder. This hierarchy has only one node, being the Home page.

  Alice 3 Cookbook 79 recipes to harness the power of Alice 3 for teaching students to build attractive and interactive 3D scenes and videos         Read more about this book       (For more resources related to this subject, see here.) Introduction You need to organize the statements that request the different actors to perform actions. Alice 3 provides blocks that allow us to configure the order in which many statements should be executed. This article provides many tasks that will allow us to start controlling the behavior of animations with many actors performing different actions. We will execute many actions with a specific order. We will use counters to run one or more statements many times. We will execute actions for many actors of the same class. We will run code for different actors at the same time to render complex animations. Performing many statements in order In this recipe, we will execute many statements for an actor with a specific order. We will add eight statements to control a sequence of movements for a bee. Getting ready We have to be working on a project with at least one actor. Therefore, we will create a new project and set a simple scene with a few actors. Select File | New... in the main menu to start a new project. A dialog box will display the six predefined templates with their thumbnail previews in the Templates tab. Select GrassyProject.a3p as the desired template for the new project and click on OK. Alice will display a grassy ground with a light blue sky. Click on Edit Scene, at the lower right corner of the scene preview. Alice will show a bigger preview of the scene and will display the Model Gallery at the bottom. Add an instance of the Bee class to the scene, and enter bee for the name of this new instance. First, Alice will create the MyBee class to extend Bee. Then, Alice will create an instance of MyBee named bee. Follow the steps explained in the Creating a new instance from a class in a gallery recipe, in the article, Alice 3: Making Simple Animations with Actors. Add an instance of the PurpleFlower class, and enter purpleFlower for the name of this new instance. Add another instance of the PurpleFlower class, and enter purpleFlower2 for the name of this new instance. The additional flower may be placed on top of the previously added flower. Add an instance of the ForestSky class to the scene. Place the bee and the two flowers as shown in the next screenshot: How to do it... Follow these steps to execute many statements for the bee with a specific order: Open an existing project with one actor added to the scene. Click on Edit Code, at the lower-right corner of the big scene preview. Alice will show a smaller preview of the scene and will display the Code Editor on a panel located at the right-hand side of the main window. Click on the class: MyScene drop-down list and the list of classes that are part of the scene will appear. Select MyScene | Edit run. Select the desired actor in the instance drop-down list located at the left-hand side of the main window, below the small scene preview. For example, you can select bee. Make sure that part: none is selected in the drop-down list located at the right-hand side of the chosen instance. Activate the Procedures tab. Alice will display the procedures for the previously selected actor. Drag the pointAt procedure and drop it in the drop statement here area located below the do in order label, inside the run tab. Because the instance name is bee, the pointAt statement contains the this.bee and pointAt labels followed by the target parameter and its question marks ???. A list with all the possible instances to pass to the first parameter will appear. Click on this.purpleFlower. The following code will be displayed, as shown in the next screenshot: this.bee.pointAt(this.purpleFlower) Drag the moveTo procedure and drop it below the previously dropped procedure call. A list with all the possible instances to pass to the first parameter will appear. Select this.purpleFlower getPart ??? and then IStemMiddle_IStemTop_IHPistil_IHPetal01, as shown in the following screenshot: Click on the more... drop-down menu button that appears at the right-hand side of the recently dropped statement. Click on duration and then on 1.0 in the cascade menu that appears. Click on the new more... drop-down menu that appears. Click on style and then on BEGIN_AND_END_ABRUPTLY. The following code will be displayed as the second statement: this.bee.moveTo(this.purpleFlower.getPart(IStemMiddle_IStemTop_IHPistil_IHPetal01), duration: 1.0, style: BEGIN_AND_END_ABRUPTLY) Drag the delay procedure and drop it below the previously dropped procedure call. A list with all the predefined direction values to pass to the first parameter will appear. Select 2.0 and the following code will be displayed as the third statement: this.bee.delay(2.0) Drag the moveAwayFrom procedure and drop it below the previously dropped procedure call. Select 0.25 for the first parameter. Click on the more... drop-down menu button that appears and select this.purpleFlower getPart ??? and then IStemMiddle_IStemTop_IHPistil_IHPetal01. Click on the additional more... drop-down menu button, on duration and then on 1.0 in the cascade menu that appears. Click on the new more... drop-down menu that appears, on style and then on BEGIN_ABRUPTLY_AND_END_GENTLY. The following code will be displayed as the fourth statement: this.bee.moveAwayFrom(0.25, this.purpleFlower.getPart(IStemMiddle_IStemTop_IHPistil_IHPetal01), duration: 1.0, style: BEGIN_ABRUPTLY_AND_END_GENTLY) Drag the turnToFace procedure and drop it below the previously dropped procedure call. Select this.purpleFlower2 getPart ??? and then IStemMiddle_IStemTop_IHPistil_IHPetal05. Click on the additional more... drop-down menu button, on duration and then on 1.0 in the cascade menu that appears. Click on the new more... drop-down menu that appears, on style and then on BEGIN_ABRUPTLY_AND_END_GENTLY. The following code will be displayed as the fifth statement: this.bee.turnToFace(this.purpleFlower2.getPart(IStemMiddle_IStemTop_IHPistil_IHPetal05), duration: 1.0, style: BEGIN_ABRUPTLY_AND_END_GENTLY) Drag the moveTo procedure and drop it below the previously dropped procedure call. Select this.purpleFlower2 getPart ??? and then IStemMiddle_IStemTop_IHPistil_IHPetal05. Click on the additional more... drop-down menu button, on duration and then on 1.0 in the cascade menu that appears. Click on the new more... drop-down menu that appears, on style and then on BEGIN_AND_END_ABRUPTLY. The following code will be displayed as the sixth statement: this.bee.moveTo(this.purpleFlower2.getPart(IStemMiddle_IStemTop_IHPistil_IHPetal05), duration: 1.0, style: BEGIN_AND_END_GENTLY) Drag the delay procedure and drop it below the previously dropped procedure call. A list with all the predefined direction values to pass to the first parameter will appear. Select 2.0 and the following code will be displayed as the seventh statement: this.bee.delay(2.0) Drag the move procedure and drop it below the previously dropped procedure call. Select FORWARD and then 10.0. Click on the more... drop-down menu button, on duration and then on 10.0 in the cascade menu that appears. Click on the additional more... drop-down menu that appears, on asSeenBy and then on this.bee. Click on the new more... drop-down menu that appears, on style and then on BEGIN_AND_END_ABRUPTLY. The following code will be displayed as the eighth and final statement. The following screenshot shows the eight statements that compose the run procedure: this.bee.move(FORWARD, duration: 10.0, asSeenBy: this.bee, style: BEGIN_ABRUPTLY_AND_END_GENTLY) (Move the mouse over the image to enlarge it.) Select File | Save as... from Alice's main menu and give a new name to the project. Then you can make changes to the project according to your needs. How it works... When we run a project, Alice creates the scene instance, creates and initializes all the instances that compose the scene, and finally executes the run method defined in the MyScene class. By default, the statements we add to a procedure are included within the do in order block. We added eight statements to the do in order block, and therefore Alice will begin with the first statement: this.bee.pointAt(this.purpleFlower) Once the bee finishes executing the pointAt procedure, the execution flow goes on with the next statement specified in the do in order block. Thus, Alice will execute the following second statement after the first one finishes: this.bee.moveTo(this.purpleFlower.getPart(IStemMiddle_IStemTop_IHPistil_IHPetal01), duration: 1.0, style: BEGIN_AND_END_ABRUPTLY) The do in order statement encapsulates a group of statements with a synchronous execution. Thus, when we add many statements within a do in order block, these statements will run one after the other. Each statement requires its previous statement to finish before starting its execution, and therefore we can use the do in order block to group statements that must run with a specific order. The moveTo procedure moves the 3D model that represents the actor until it reaches the position of the other actor. The value for the target parameter is the instance of the other actor. We want the bee to move to one of the petals of the first flower, purpleFlower, and therefore we passed this value to the target parameter: this.purpleFlower.getPart(IStemMiddle_IStemTop_IHPistil_IHPetal01) We called the getPart function for purpleFlower with IStemMiddle_IStemTop_IHPistil_IHPetal01 as the name of the part to return. This function allows us to retrieve one petal from the flower as an instance. We used the resulting instance as the target parameter for the moveTo procedure and we could make the bee move to the specific petal of the flower. Once the bee finishes executing the moveTo procedure, the execution flow goes on with the next statement specified in the do in order block. Thus, Alice will execute the following third statement after the second one finishes: this.bee.delay(2.0) The delay procedure puts the actor to sleep in its current position for the specified number of seconds. The next statement specified in the do in order block will run after waiting for two seconds. The statements added to the run procedure will perform the following visible actions in the specified order: Point the bee at purpleFlower. Begin and end abruptly a movement for the bee from its position to the petal named IStemMiddle_IStemTop_IHPistil_IHPetal01 of purpleFlower. The total duration for the animation must be 1 second. Make the bee stay in its position for 2 seconds. Move the bee away 0.25 units from the position of the petal named IStemMiddle_IStemTop_IHPistil_IHPetal01 of purpleFlower. Begin the movement abruptly but end it gently. The total duration for the animation must be 1 second. Turn the bee to the face of the petal named IStemMiddle_IStemTop_IHPistil_IHPetal05 of purpleFlower2. Begin the movement abruptly but end it gently. The total duration for the animation must be 1 second. Begin and end abruptly a movement for the bee from its position to the petal named IStemMiddle_IStemTop_IHPistil_IHPetal05 of purpleFlower2. The total duration for the animation must be 1 second. Make the bee stay in its position for 2 seconds. Move the bee forward 10 units. Begin the movement abruptly but end it gently. The total duration for the animation must be 10 seconds. The bee will disappear from the scene. The following screenshot shows six screenshots of the rendered frames: (Move the mouse over the image to enlarge it.) There's more... When you work with the Alice code editor, you can temporarily disable statements. Alice doesn't execute the disabled statements. However, you can enable them again later. It is useful to disable one or more statements when you want to test the results of running the project without these statements, but you might want to enable them back to compare the results. To disable a statement, right-click on it and deactivate the IsEnabled option, as shown in the following screenshot: The disabled statements will appear with diagonal lines, as shown in the next screenshot, and won't be considered at run-time: To enable a disabled statement, right-click on it and activate the IsEnabled option.

Remove everything that has no relevance to the story. If you say in the first chapter that there is a rifle hanging on the wall, in the second or third chapter it absolutely must go off. If it's not going to be fired, it shouldn't be hanging there.                                                                                              —Anton Chekhov King Julian: I don't know why the sacrifice didn't work. The science seemed so solid.                                                                                             —Madagascar: Escape 2 Africa (2008) Despite their strange design and mysterious engineering, Q's gadgets always prove useful and reliable. Bond has such faith in the technology that he never even asks how to charge the batteries. One of the inventive ideas in the Bond franchise is that even a lightly equipped spy should be able to see and photograph concealed objects, anyplace, anytime. Let's consider a timeline of a few relevant gadgets in the movies: 1967 (You Only Live Twice): An X-ray desk scans guests for hidden firearms. 1979 (Moonraker): A cigarette case contains an X-ray imaging system that is used to reveal the tumblers of a safe's combination lock. 1989 (License to Kill): A Polaroid camera takes X-ray photos. Oddly enough, its flash is a visible, red laser. 1995 (GoldenEye): A tea tray contains an X-ray scanner that can photograph documents beneath the tray. 1999 (The World is Not Enough): Bond wears a stylish pair of blue-lensed glasses that can see through one layer of clothing to reveal concealed weapons. According to the James Bond Encyclopedia (2007), which is an official guide to the movies, the glasses display infrared video after applying special processing to it. Despite using infrared, they are commonly called X-ray specs, a misnomer. These gadgets deal with unseen wavelengths of light (or radiation) and are broadly comparable to real-world devices such as airport security scanners and night vision goggles. However, it remains difficult to explain how Bond's equipment is so compact and how it takes such clear pictures in diverse lighting conditions and through diverse materials. Moreover, if Bond's devices are active scanners (meaning they emit X-ray radiation or infrared light), they will be clearly visible to other spies using similar hardware. To take another approach, what if we avoid unseen wavelengths of light but instead focus on unseen frequencies of motion? Many things move in a pattern that is too fast or too slow for us to easily notice. Suppose that a man is standing in one place. If he shifts one leg more than the other, perhaps he is concealing a heavy object, such as a gun, on the side that he shifts more. We also might fail to notice deviations from a pattern. Suppose the same man has been looking straight ahead but suddenly, when he believes no one is looking, his eyes dart to one side. Is he watching someone? We can make motions of a certain frequency more visible by repeating them, like a delayed afterimage or a ghost, with each repetition being more faint (less opaque) than the last. The effect is analogous to an echo or a ripple, and it is achieved using an algorithm called Eulerian video magnification. By applying this technique in this article by Joseph Howse, author of the book OpenCV for Secret Agents, we will build a desktop app that allows us to simultaneously see the present and selected slices of the past. The idea of experiencing multiple images simultaneously is, to me, quite natural because for the first 26 years of my life, I had strabismus—commonly called a lazy eye—that caused double vision. A surgeon corrected my eyesight and gave me depth perception but, in memory of strabismus, I would like to name this application Lazy Eyes. (For more resources related to this topic, see here.) Let's take a closer look—or two or more closer looks—at the fast-paced, moving world that we share with all the other secret agents. Planning the Lazy Eyes app Of all our apps, Lazy Eyes has the simplest user interface. It just shows a live video feed with a special effect that highlights motion. The parameters of the effect are quite complex and, moreover, modifying them at runtime would have a big effect on performance. Thus, we do not provide a user interface to reconfigure the effect, but we do provide many parameters in code to allow a programmer to create many variants of the effect and the app. The following is a screenshot illustrating one configuration of the app. This image shows me eating cake. My hands and face are moving often and we see an effect that looks like light and dark waves rippling around the places where moving edges have been. (The effect is more graceful in a live video than in a screenshot.) For more screenshots and an in-depth discussion of the parameters, refer to the section Configuring and testing the app for various motions, later in this article. Regardless of how it is configured, the app loops through the following actions: Capturing an image. Copying and downsampling the image while applying a blur filter and optionally an edge finding filter. We will downsample using so-called image pyramids, which will be discussed in Compositing two images using image pyramids, later in this article. The purpose of downsampling is to achieve a higher frame rate by reducing the amount of image data used in subsequent operations. The purpose of applying a blur filter and optionally an edge finding filter is to create haloes that are useful in amplifying motion. Storing the downsampled copy in a history of frames, with a timestamp. The history has a fixed capacity and once it is full, the oldest frame is overwritten to make room for the new one. If the history is not yet full, we continue to the next iteration of the loop. Decomposing the history into a list of frequencies describing fluctuations (motion) at each pixel. The decomposition function is called a Fast Fourier Transform. We will discuss this in the Extracting repeating signals from video using the Fast Fourier Transform section, later in this article. Setting all frequencies to zero except a certain chosen range that interests us. In other words, filter out the data on motions that are faster or slower than certain thresholds. Recomposing the filtered frequencies into a series of images that are motion maps. Areas that are still (with respect to our chosen range of frequencies) become dark, and areas that are moving might remain bright. The recomposition function is called an Inverse Fast Fourier Transform (IFFT), and we will discuss it later alongside the FFT. Upsampling the latest motion map (again using image pyramids), intensifying it, and overlaying it additively atop the original camera image. Showing the resulting composite image. There it is—a simple plan that requires a rather nuanced implementation and configuration. Let's prepare ourselves by doing a little background research. Understanding what Eulerian video magnification can do Eulerian video magnification is inspired by a model in fluid mechanics called Eulerian specification of the flow field. Let's consider a moving, fluid body, such as a river. The Eulerian specification describes the river's velocity at a given position and time. The velocity would be fast in the mountains in springtime and slow at the river's mouth in winter. Also, the velocity would be slower at a silt-saturated point at the river's bottom, compared to a point where the river's surface hits a rock and sprays. An alternative to the Eulerian specification is the Lagrangian specification, which describes the position of a given particle at a given time. A given bit of silt might make its way down from the mountains to the river's mouth over a period of many years and then spend eons drifting around a tidal basin. For a more formal description of the Eulerian specification, the Lagrangian specification, and their relationship, refer to this Wikipedia article http://en.wikipedia.org/wiki/Lagrangian_and_Eulerian_specification_of_the_flow_field. The Lagrangian specification is analogous to many computer vision tasks, in which we model the motion of a particular object or feature over time. However, the Eulerian specification is analogous to our current task, in which we model any motion occurring in a particular position and a particular window of time. Having modeled a motion from an Eulerian perspective, we can visually exaggerate the motion by overlaying the model's results for a blend of positions and times. Let's set a baseline for our expectations of Eulerian video magnification by studying other people's projects: Michael Rubenstein's webpage at MIT (http://people.csail.mit.edu/mrub/vidmag/) gives an abstract and demo videos of his team's pioneering work on Eulerian video magnification. Bryce Drennan's Eulerian-magnification library (https://github.com/brycedrennan/eulerian-magnification) implements the algorithm using NumPy, SciPy, and OpenCV. This implementation is good inspiration for us, but it is designed to process prerecorded videos and is not sufficiently optimized for real-time input. Now, let's discuss the functions that are building blocks of these projects and ours. Extracting repeating signals from video using the Fast Fourier Transform (FFT) An audio signal is typically visualized as a bar chart or wave. The bar chart or wave is high when the sound is loud and low when it is soft. We recognize that a repetitive sound, such as a metronome's beat, makes repetitive peaks and valleys in the visualization. When audio has multiple channels (being a stereo or surround sound recording), each channel can be considered as a separate signal and can be visualized as a separate bar chart or wave. Similarly, in a video, every channel of every pixel can be considered as a separate signal, rising and falling (becoming brighter and dimmer) over time. Imagine that we use a stationary camera to capture a video of a metronome. Then, certain pixel values rise and fall at a regular interval as they capture the passage of the metronome's needle. If the camera has an attached microphone, its signal values rise and fall at the same interval. Based on either the audio or the video, we can measure the metronome's frequency—its beats per minute (bpm) or its beats per second (Hertz or Hz). Conversely, if we change the metronome's bpm setting, the effect on both the audio and the video is predictable. From this thought experiment, we can learn that a signal—be it audio, video, or any other kind—can be expressed as a function of time and, equivalently, a function of frequency. Consider the following pair of graphs. They express the same signal, first as a function of time and then as a function of frequency. Within the time domain, we see one wide peak and valley (in other words, a tapering effect) spanning many narrow peaks and valleys. Within the frequency domain, we see a low-frequency peak and a high-frequency peak. The transformation from the time domain to the frequency domain is called the Fourier transform. Conversely, the transformation from the frequency domain to the time domain is called the inverse Fourier transform. Within the digital world, signals are discrete, not continuous, and we use the terms discrete Fourier transform (DFT) and inverse discrete Fourier transform (IDFT). There is a variety of efficient algorithms to compute the DFT or IDFT and such an algorithm might be described as a Fast Fourier Transform or an Inverse Fast Fourier Transform. For algorithmic descriptions, refer to the following Wikipedia article: http://en.wikipedia.org/wiki/Fast_Fourier_transform. The result of the Fourier transform (including its discrete variants) is a function that maps a frequency to an amplitude and phase. The amplitude represents the magnitude of the frequency's contribution to the signal. The phase represents a temporal shift; it determines whether the frequency's contribution starts on a high or a low. Typically, amplitude and phase are encoded in a complex number, a+bi, where amplitude=sqrt(a^2+b^2) and phase=atan2(a,b). For an explanation of complex numbers, refer to the following Wikipedia article: http://en.wikipedia.org/wiki/Complex_number. The FFT and IFFT are fundamental to a field of computer science called digital signal processing. Many signal processing applications, including Lazy Eyes, involve taking the signal's FFT, modifying or removing certain frequencies in the FFT result, and then reconstructing the filtered signal in the time domain using the IFFT. For example, this approach enables us to amplify certain frequencies while leaving others unchanged. Now, where do we find this functionality? Choosing and setting up an FFT library Several Python libraries provide FFT and IFFT implementations that can process NumPy arrays (and thus OpenCV images). Here are the five major contenders: NumPy: This provides FFT and IFFT implementations in a module called numpy.fft (for more information, refer to http://docs.scipy.org/doc/numpy/reference/routines.fft.html). The module also offers other signal processing functions to work with the output of the FFT. SciPy: This provides FFT and IFFT implementations in a module called scipy.fftpack (for more information refer to http://docs.scipy.org/doc/scipy/reference/fftpack.html). This SciPy module is closely based on the numpy.fft module, but adds some optional arguments and dynamic optimizations based on the input format. The SciPy module also adds more signal processing functions to work with the output of the FFT. OpenCV: This has implementations of FFT (cv2.dft) and IFT (cv2.idft). An official tutorial provides examples and a comparison to NumPy's FFT implementation at http://docs.opencv.org/doc/tutorials/core/discrete_fourier_transform/discrete_fourier_transform.html. OpenCV's FFT and IFT interfaces are not directly interoperable with the numpy.fft and scipy.fftpack modules that offer a broader range of signal processing functionality. (The data is formatted very differently.) PyFFTW: This is a Python wrapper (https://hgomersall.github.io/pyFFTW/) around a C library called the Fastest Fourier Transform in the West (FFTW) (for more information, refer to http://www.fftw.org/). FFTW provides multiple implementations of FFT and IFFT. At runtime, it dynamically selects implementations that are well optimized for given input formats, output formats, and system capabilities. Optionally, it takes advantage of multithreading (and its threads might run on multiple CPU cores, as the implementation releases Python's Global Interpreter Lock or GIL). PyFFTW provides optional interfaces matching NumPy's and SciPy's FFT and IFFT functions. These interfaces have a low overhead cost (thanks to good caching options that are provided by PyFFTW) and they help to ensure that PyFFTW is interoperable with a broader range of signal processing functionality as implemented in numpy.fft and scipy.fftpack. Reinka: This is a Python library for GPU-accelerated computations using either PyCUDA (http://mathema.tician.de/software/pycuda/) or PyOpenCL (http://mathema.tician.de/software/pyopencl/) as a backend. Reinka (http://reikna.publicfields.net/en/latest/) provides FFT and IFFT implementations in a module called reikna.fft. Reinka internally uses PyCUDA or PyOpenCL arrays (not NumPy arrays), but it provides interfaces for conversion from NumPy arrays to these GPU arrays and back. The converted NumPy output is compatible with other signal processing functionality as implemented in numpy.fft and scipy.fftpack. However, this compatibility comes at a high overhead cost due to locking, reading, and converting the contents of the GPU memory. NumPy, SciPy, OpenCV, and PyFFTW are open-source libraries under the BSD license. Reinka is an open-source library under the MIT license. I recommend PyFFTW because of its optimizations and its interoperability (at a low overhead cost) with all the other functionality that interests us in NumPy, SciPy, and OpenCV. For a tour of PyFFTW's features, including its NumPy- and SciPy-compatible interfaces, refer to the official tutorial at https://hgomersall.github.io/pyFFTW/sphinx/tutorial.html. Depending on our platform, we can set up PyFFTW in one of the following ways: In Windows, download and run a binary installer from https://pypi.python.org/pypi/pyFFTW. Choose the installer for either a 32-bit Python 2.7 or 64-bit Python 2.7 (depending on whether your Python installation, not necessarily your system, is 64-bit). In Mac with MacPorts, run the following command in Terminal: $ sudo port install py27-pyfftw In Ubuntu 14.10 (Utopic) and its derivatives, including Linux Mint 14.10, run the following command in Terminal: $ sudo apt-get install python-fftw3 In Ubuntu 14.04 and earlier versions (and derivatives thereof), do not use this package, as its version is too old. Instead, use the PyFFTW source bundle, as described in the last bullet of this list. In Debian Jessie, Debian Sid, and their derivatives, run the following command in Terminal: $ sudo apt-get install python-pyfftw In Debian Wheezy and its derivatives, including Raspbian, this package does not exist. Instead, use the PyFFTW source bundle, as described in the next bullet. For any other system, download the PyFFTW source bundle from https://pypi.python.org/pypi/pyFFTW. Decompress it and run the setup.py script inside the decompressed folder. Some old versions of the library are called PyFFTW3. We do not want PyFFTW3. However, on Ubuntu 14.10 and its derivatives, the packages are misnamed such that python-fftw3 is really the most recent packaged version (whereas python-fftw is an older PyFFTW3 version). We have our FFT and IFFT needs covered by FFTW (and if we were cowboys instead of secret agents, we could say, "Cover me!"). For additional signal processing functionality, we will use SciPy. Signal processing is not the only new material that we must learn for Lazy Eyes, so let's now look at other functionality that is provided by OpenCV. Compositing two images using image pyramids Running an FFT on a full-resolution video feed would be slow. Also, the resulting frequencies would reflect localized phenomena at each captured pixel, such that the motion map (the result of filtering the frequencies and then applying the IFFT) might appear noisy and overly sharpened. To address these problems, we want a cheap, blurry downsampling technique. However, we also want the option to enhance edges, which are important to our perception of motion. Our need for a blurry downsampling technique is fulfilled by a Gaussian image pyramid. A Gaussian filter blurs an image by making each output pixel a weighted average of many input pixels in the neighborhood. An image pyramid is a series in which each image is half the width and height of the previous image. The halving of image dimensions is achieved by decimation, meaning that every other pixel is simply omitted. A Gaussian image pyramid is constructed by applying a Gaussian filter before each decimation operation. Our need to enhance edges in downsampled images is fulfilled by a Laplacian image pyramid, which is constructed in the following manner. Suppose we have already constructed a Gaussian image pyramid. We take the image at level i+1 in the Gaussian pyramid, upsample it by duplicating the pixels, and apply a Gaussian filter to it again. We then subtract the result from the image at level i in the Gaussian pyramid to produce the corresponding image at level i of the Laplacian pyramid. Thus, the Laplacian image is the difference between a blurry, downsampled image and an even blurrier image that was downsampled, downsampled again, and upsampled. You might wonder how such an algorithm is a form of edge finding. Consider that edges are areas of local contrast, while non-edges are areas of local uniformity. If we blur a uniform area, it is still uniform—zero difference. If we blur a contrasting area, it becomes more uniform—nonzero difference. Thus, the difference can be used to find edges. The Gaussian and Laplacian image pyramids are described in detail in the journal article downloadable from http://web.mit.edu/persci/people/adelson/pub_pdfs/RCA84.pdf. This article is written by E. H. Adelson, C. H. Anderson, J. R. Bergen, P. J. Burt, and J. M. Ogden on Pyramid methods in image processing, RCA Engineer, vol. 29, no. 6, November/December 1984. Besides using image pyramids to downsample the FFT's input, we also use them to upsample the most recent frame of the IFFT's output. This upsampling step is necessary to create an overlay that matches the size of the original camera image so that we can composite the two. Like in the construction of the Laplacian pyramid, upsampling consists of duplicating pixels and applying a Gaussian filter. OpenCV implements the relevant downsizing and upsizing functions as cv2.pyrDown and cv2.pyrUp. These functions are useful in compositing two images in general (whether or not signal processing is involved) because they enable us to soften differences while preserving edges. The OpenCV documentation includes a good tutorial on this topic at http://docs.opencv.org/trunk/doc/py_tutorials/py_imgproc/py_pyramids/py_pyramids.html. Now, we are armed with the knowledge to implement Lazy Eyes! Implementing the Lazy Eyes app Let's create a new folder for Lazy Eyes and, in this folder, create copies of or links to the ResizeUtils.py and WxUtils.py files from any of our previous Python. Alongside the copies or links, let's create a new file, LazyEyes.py. Edit it and enter the following import statements: import collectionsimport numpyimport cv2import threadingimport timeitimport wx import pyfftw.interfaces.cachefrom pyfftw.interfaces.scipy_fftpack import fftfrom pyfftw.interfaces.scipy_fftpack import ifftfrom scipy.fftpack import fftfreq import ResizeUtilsimport WxUtils Besides the modules that we have used in previous projects, we are now using the standard library's collections module for efficient collections and timeit module for precise timing. Also for the first time, we are using signal processing functionality from PyFFTW and SciPy. Like our other Python applications, Lazy Eyes is implemented as a class that extends wx.Frame. Here are the declarations of the class and its initializer: class LazyEyes(wx.Frame):  def __init__(self, maxHistoryLength=360,   minHz=5.0/6.0, maxHz=1.0,   amplification=32.0, numPyramidLevels=2,   useLaplacianPyramid=True,   useGrayOverlay=True,   numFFTThreads = 4, numIFFTThreads=4,   cameraDeviceID=0, imageSize=(480, 360),   title='Lazy Eyes'): The initializer's arguments affect the app's frame rate and the manner in which motion is amplified. These effects are discussed in detail in the section Configuring and testing the app for various motions, later in this article. The following is just a brief description of the arguments: maxHistoryLength is the number of frames (including the current frame and preceding frames) that are analyzed for motion. minHz and maxHz, respectively, define the slowest and fastest motions that are amplified. amplification is the scale of the visual effect. A higher value means motion is highlighted more brightly. numPyramidLevels is the number of pyramid levels by which frames are downsampled before signal processing is done. Remember that each level corresponds to downsampling by a factor of 2. Our implementation assumes numPyramidLevels>0. If useLaplacianPyramid is True, frames are downsampled using a Laplacian pyramid before the signal processing is done. The implication is that only edge motion is highlighted. Alternatively, if useLaplacianPyramid is False, a Gaussian pyramid is used, and the motion in all areas is highlighted. If useGrayOverlay is True, frames are converted to grayscale before the signal processing is done. The implication is that motion is only highlighted in areas of grayscale contrast. Alternatively, if useGrayOverlay is False, motion is highlighted in areas that have contrast in any color channel. numFFTThreads and numIFFTThreads, respectively, are the numbers of threads used in FFT and IFFT computations. cameraDeviceID and imageSize are our usual capture parameters. The initializer's implementation begins in the same way as our other Python apps. It sets flags to indicate that the app is running and (by default) should be mirrored. It creates the capture object and configures its resolution to match the requested width and height if possible. Failing that, the device's default capture resolution is used. The relevant code is as follows:    self.mirrored = True    self._running = True    self._capture = cv2.VideoCapture(cameraDeviceID)   size = ResizeUtils.cvResizeCapture(       self._capture, imageSize)   w, h = size   self._imageWidth = w   self._imageHeight = h Next, we will determine the shape of the history of frames. It has at least 3 dimensions: a number of frames, a width, and height for each frame. The width and height are downsampled from the capture width and height based on the number of pyramid levels. If we are concerned about the color motion and not just the grayscale motion, the history also has a fourth dimension, consisting of 3 color channels. Here is the code to calculate the history's shape:    self._useGrayOverlay = useGrayOverlay   if useGrayOverlay:     historyShape = (maxHistoryLength,             h >> numPyramidLevels,             w >> numPyramidLevels)   else:     historyShape = (maxHistoryLength,             h >> numPyramidLevels,             w >> numPyramidLevels, 3) Note the use of >>, the right bit shift operator, to reduce the dimensions by a power of 2. The power is equal to the number of pyramid levels. We will store the specified maximum history length. For the frames in the history, we will create a NumPy array of the shape that we just determined. For timestamps of the frames, we will create a deque (double-ended queue), a type of collection that allows us to cheaply add or remove elements from either end:    self._maxHistoryLength = maxHistoryLength   self._history = numpy.empty(historyShape,                 numpy.float32)   self._historyTimestamps = collections.deque() We will store the remaining arguments because later, in each frame, we will pass them to the pyramid functions and signal the processing functions:    self._numPyramidLevels = numPyramidLevels   self._useLaplacianPyramid = useLaplacianPyramid    self._minHz = minHz   self._maxHz = maxHz   self._amplification = amplification    self._numFFTThreads = numFFTThreads   self._numIFFTThreads = numIFFTThreads To ensure meaningful error messages and early termination in the case of invalid arguments, we could add code such as the following for each argument: assert numPyramidLevels > 0,      'numPyramidLevels must be positive.' For brevity, such assertions are omitted from our code samples. We call the following two functions to tell PyFFTW to cache its data structures (notably, its NumPy arrays) for a period of at least 1.0 second from their last use. (The default is 0.1 seconds.) Caching is a critical optimization for the PyFFTW interfaces that we are using, and we will choose a period that is more than long enough to keep the cache alive from frame to frame:    pyfftw.interfaces.cache.enable()   pyfftw.interfaces.cache.set_keepalive_time(1.0) The initializer's implementation ends with code to set up a window, event bindings, a bitmap, layout, and background thread—all familiar tasks from our previous Python projects:    style = wx.CLOSE_BOX | wx.MINIMIZE_BOX |        wx.CAPTION | wx.SYSTEM_MENU |        wx.CLIP_CHILDREN   wx.Frame.__init__(self, None, title=title,             style=style, size=size)    self.Bind(wx.EVT_CLOSE, self._onCloseWindow)    quitCommandID = wx.NewId()   self.Bind(wx.EVT_MENU, self._onQuitCommand,         id=quitCommandID)   acceleratorTable = wx.AcceleratorTable([     (wx.ACCEL_NORMAL, wx.WXK_ESCAPE,       quitCommandID)   ])   self.SetAcceleratorTable(acceleratorTable)    self._staticBitmap = wx.StaticBitmap(self,                       size=size)   self._showImage(None)    rootSizer = wx.BoxSizer(wx.VERTICAL)   rootSizer.Add(self._staticBitmap)   self.SetSizerAndFit(rootSizer)    self._captureThread = threading.Thread(       target=self._runCaptureLoop)   self._captureThread.start() We must modify our usual _onCloseWindow callback to disable PyFFTW's cache. Disabling the cache ensures that resources are freed and that PyFFTW's threads terminate normally. The callback's implementation is given in the following code:  def _onCloseWindow(self, event):   self._running = False   self._captureThread.join()   pyfftw.interfaces.cache.disable()   self.Destroy() The Esc key is bound to our usual _onQuitCommand callback, which just closes the app:  def _onQuitCommand(self, event):   self.Close() The loop running on our background thread is similar to the one in our other Python apps. In each frame, it calls a helper function, _applyEulerianVideoMagnification. Here is the loop's implementation.  def _runCaptureLoop(self):   while self._running:     success, image = self._capture.read()     if image is not None:        self._applyEulerianVideoMagnification(           image)       if (self.mirrored):         image[:] = numpy.fliplr(image)     wx.CallAfter(self._showImage, image) The _applyEulerianVideoMagnification helper function is quite long so we will consider its implementation in several chunks. First, we will create a timestamp for the frame and copy the frame to a format that is more suitable for processing. Specifically, we will use a floating point array with either one gray channel or 3 color channels, depending on the configuration:  def _applyEulerianVideoMagnification(self, image):    timestamp = timeit.default_timer()    if self._useGrayOverlay:     smallImage = cv2.cvtColor(         image, cv2.COLOR_BGR2GRAY).astype(             numpy.float32)   else:     smallImage = image.astype(numpy.float32) Using this copy, we will calculate the appropriate level in the Gaussian or Laplacian pyramid:    # Downsample the image using a pyramid technique.   i = 0   while i < self._numPyramidLevels:     smallImage = cv2.pyrDown(smallImage)     i += 1   if self._useLaplacianPyramid:     smallImage[:] -=        cv2.pyrUp(cv2.pyrDown(smallImage)) For the purposes of the history and signal processing functions, we will refer to this pyramid level as "the image" or "the frame". Next, we will check the number of history frames that have been filled so far. If the history has more than one unfilled frame (meaning the history will still not be full after adding this frame), we will append the new image and timestamp and then return early, such that no signal processing is done until a later frame:    historyLength = len(self._historyTimestamps)    if historyLength < self._maxHistoryLength - 1:      # Append the new image and timestamp to the     # history.     self._history[historyLength] = smallImage     self._historyTimestamps.append(timestamp)      # The history is still not full, so wait.     return If the history is just one frame short of being full (meaning the history will be full after adding this frame), we will append the new image and timestamp using the code given as follows:    if historyLength == self._maxHistoryLength - 1:     # Append the new image and timestamp to the     # history.     self._history[historyLength] = smallImage     self._historyTimestamps.append(timestamp) If the history is already full, we will drop the oldest image and timestamp and append the new image and timestamp using the code given as follows:    else:     # Drop the oldest image and timestamp from the     # history and append the new ones.     self._history[:-1] = self._history[1:]     self._historyTimestamps.popleft()     self._history[-1] = smallImage     self._historyTimestamps.append(timestamp)    # The history is full, so process it. The history of image data is a NumPy array and, as such, we are using the terms "append" and "drop" loosely. NumPy arrays are immutable, meaning they cannot grow or shrink. Moreover, we are not recreating this array because it is large and reallocating it each frame would be expensive. We are just overwriting data within the array by moving the old data leftward and copying the new data in. Based on the timestamps, we will calculate the average time per frame in the history, as seen in the following code:    # Find the average length of time per frame.   startTime = self._historyTimestamps[0]   endTime = self._historyTimestamps[-1]   timeElapsed = endTime - startTime   timePerFrame =        timeElapsed / self._maxHistoryLength   #print 'FPS:', 1.0 / timePerFrame We will proceed with a combination of signal processing functions, collectively called a temporal bandpass filter. This filter blocks (zeros out) some frequencies and allows others to pass (remain unchanged). Our first step in implementing this filter is to run the pyfftw.interfaces.scipy_fftpack.fft function using the history and a number of threads as arguments. Also, with the argument axis=0, we will specify that the history's first axis is its time axis:    # Apply the temporal bandpass filter.   fftResult = fft(self._history, axis=0,           threads=self._numFFTThreads) We will pass the FFT result and the time per frame to the scipy.fftpack.fftfreq function. This function returns an array of midpoint frequencies (Hz in our case) corresponding to the indices in the FFT result. (This array answers the question, "Which frequency is the midpoint of the bin of frequencies represented by index i in the FFT?".) We will find the indices whose midpoint frequencies lie closest (minimum absolute value difference) to our initializer's minHz and maxHz parameters. Then, we will modify the FFT result by setting the data to zero in all ranges that do not represent the frequencies of interest:    frequencies = fftfreq(        self._maxHistoryLength, d=timePerFrame)   lowBound = (numpy.abs(       frequencies - self._minHz)).argmin()   highBound = (numpy.abs(       frequencies - self._maxHz)).argmin()   fftResult[:lowBound] = 0j   fftResult[highBound:-highBound] = 0j   fftResult[-lowBound:] = 0j The FFT result is symmetrical: fftResult[i] and fftResult[-i] pertain to the same bin of frequencies. Thus, we will modify the FFT result symmetrically. Remember, the Fourier transform maps a frequency to a complex number that encodes an amplitude and phase. Thus, while the indices of the FFT result correspond to frequencies, the values contained at those indices are complex numbers. Zero as a complex number is written in Python as 0+0j or 0j. Having thus filtered out the frequencies that do not interest us, we will finish applying the temporal bandpass filter by passing the data to the pyfftw.interfaces.scipy_fftpack.ifft function:    ifftResult = ifft(fftResult, axis=0,           threads=self._numIFFTThreads) From the IFFT result, we will take the most recent frame. It should somewhat resemble the current camera frame, but should be black in areas that do not exhibit recent motion matching our parameters. We will multiply this filtered frame so that the non-black areas become bright. Then, we will upsample it (using a pyramid technique) and add the result to the current camera frame so that areas of motion are lit up. Here is the relevant code, which concludes the _applyEulerianVideoMagnification method:    # Amplify the result and overlay it on the   # original image.   overlay = numpy.real(ifftResult[-1]) *            self._amplification   i = 0   while i < self._numPyramidLevels:     overlay = cv2.pyrUp(overlay)     i += 1   if self._useGrayOverlay:    overlay = cv2.cvtColor(overlay,                 cv2.COLOR_GRAY2BGR   cv2.convertScaleAbs(image + overlay, image) To finish the implementation of the LazyEyes class, we will display the image in the same manner as we have done in our other Python apps. Here is the relevant method:  def _showImage(self, image):   if image is None:     # Provide a black bitmap.     bitmap = wx.EmptyBitmap(self._imageWidth,                 self._imageHeight)   else:     # Convert the image to bitmap format.     bitmap = WXUtils.wxBitmapFromCvImage(image)   # Show the bitmap.   self._staticBitmap.SetBitmap(bitmap) Our module's main function just instantiates and runs the app, as seen in the following code: def main(): app = wx.App() lazyEyes = LazyEyes() lazyEyes.Show() app.MainLoop() if __name__ == '__main__': main() That's all! Run the app and stay quite still while it builds up its history of frames. Until the history is full, the video feed will not show any special effect. At the history's default length of 360 frames, it fills in about 20 seconds on my machine. Once it is full, you should see ripples moving through the video feed in areas of recent motion—or perhaps all areas if the camera is moved or the lighting or exposure is changed. The ripples will gradually settle and disappear in areas of the scene that become still, while new ripples will appear in new areas of motion. Feel free to experiment on your own. Now, let's discuss a few recipes for configuring and testing the parameters of the LazyEyes class. Configuring and testing the app for various motions Currently, our main function initializes the LazyEyes object with the default parameters. By filling in the same parameter values explicitly, we would have this statement:  lazyEyes = LazyEyes(maxHistoryLength=360,           minHz=5.0/6.0, maxHz=1.0,           amplification=32.0,           numPyramidLevels=2,           useLaplacianPyramid=True,           useGrayOverlay=True,           numFFTThreads = 4,           numIFFTThreads=4,           imageSize=(480, 360)) This recipe calls for a capture resolution of 480 x 360 and a signal processing resolution of 120 x 90 (as we are downsampling by 2 pyramid levels or a factor of 4). We are amplifying the motion only at frequencies of 0.833 Hz to 1.0 Hz, only at edges (as we are using the Laplacian pyramid), only in grayscale, and only over a history of 360 frames (about 20 to 40 seconds, depending on the frame rate). Motion is exaggerated by a factor of 32. These settings are suitable for many subtle upper-body movements such as a person's head swaying side to side, shoulders heaving while breathing, nostrils flaring, eyebrows rising and falling, and eyes scanning to and fro. For performance, the FFT and IFFT are each using 4 threads. Here is a screenshot of the app that runs with these default parameters. Moments before taking the screenshot, I smiled like a comic theater mask and then I recomposed my normal expression. Note that my eyebrows and moustache are visible in multiple positions, including their current, low positions and their previous, high positions. For the sake of capturing the motion amplification effect in a still image, this gesture is quite exaggerated. However, in a moving video, we can see the amplification of more subtle movements too. Here is a less extreme example where my eyebrows appear very tall because I raised and lowered them: The parameters interact with each other in complex ways. Consider the following relationships between these parameters: Frame rate is greatly affected by the size of the input data for the FFT and IFFT functions. The size of the input data is determined by maxHistoryLength (where a shorter length provides less input and thus a faster frame rate), numPyramidLevels (where a greater level implies less input), useGrayOverlay (where True means less input), and imageSize (where a smaller size implies less input). Frame rate is also greatly affected by the level of multithreading of the FFT and IFFT functions, as determined by numFFTThreads and numIFFTThreads (a greater number of threads is faster up to some point). Frame rate is slightly affected by useLaplacianPyramid (False implies a faster frame rate), as the Laplacian algorithm requires extra steps beyond the Gaussian image. Frame rate determines the amount of time that maxHistoryLength represents. Frame rate and maxHistoryLength determine how many repetitions of motion (if any) can be captured in the minHz to maxHz range. The number of captured repetitions, together with amplification, determines how greatly a motion or a deviation from the motion will be amplified. The inclusion or exclusion of noise is affected by minHz and maxHz (depending on which frequencies of noise are characteristic of the camera), numPyramidLevels (where more implies a less noisy image), useLaplacianPyramid (where True is less noisy), useGrayOverlay (where True is less noisy), and imageSize (where a smaller size implies a less noisy image). The inclusion or exclusion of motion is affected by numPyramidLevels (where fewer means the amplification is more inclusive of small motions), useLaplacianPyramid (False is more inclusive of motion in non-edge areas), useGrayOverlay (False is more inclusive of motion in areas of color contrast), minHz (where a lower value is more inclusive of slow motion), maxHz (where a higher value is more inclusive of fast motion), and imageSize (where bigger size is more inclusive of small motions). Subjectively, the visual effect is always best when the frame rate is high, noise is excluded, and small motions are included. Again subjectively, other conditions for including or excluding motion (edge versus non-edge, grayscale contrast versus color contrast, and fast versus slow) are application-dependent. Let's try our hand at reconfiguring Lazy Eyes, starting with the numFFTThreads and numIFFTThreads parameters. We want to determine the numbers of threads that maximize Lazy Eyes' frame rate on your machine. The more CPU cores you have, the more threads you can gainfully use. However, experimentation is the best guide to pick a number. To get a log of the frame rate in Lazy Eyes, uncomment the following line in the _applyEulerianVideoMagnification method:    print 'FPS:', 1.0 / timePerFrame Run LazyEyes.py from the command line. Once the history fills up, the history's average FPS will be printed to the command line in every frame. Wait until this average FPS value stabilizes. It might take a minute for the average to adjust to the effect of the FFT and IFFT functions that begin running once the history is full. Take note of the FPS value, close the app, adjust the thread count parameters, and test again. Repeat until you feel that you have enough data to pick good numbers of threads to use on your hardware. By activating additional CPU cores, multithreading can cause your system's temperature to rise. As you experiment, monitor your machine's temperature, fans, and CPU usage statistics. If you become concerned, reduce the number of FFT and IFFT threads. Having a suboptimal frame rate is better than overheating of your machine. Now, experiment with other parameters to see how they affect FPS. The numPyramidLevels, useGrayOverlay, and imageSize parameters should all have a large effect. For subjectively good visual results, try to maintain a frame rate above 10 FPS. A frame rate above 15 FPS is excellent. When you are satisfied that you understand the parameters' effects on frame rate, comment out the following line again because the print statement is itself an expensive operation that can reduce frame rate:    #print 'FPS:', 1.0 / timePerFrame Let's try another recipe. Although our default recipe accentuates motion at edges that have high grayscale contrast, this next recipe accentuates motion in all areas (edge or non-edge) that have high contrast (color or grayscale). By considering 3 color channels instead of one grayscale channel, we are tripling the amount of data being processed by the FFT and IFFT. To offset this change, we will cut each dimension of the capture resolution to two third times its default value, thus reducing the amount of data to 2/3 * 2/3 = 4/9 times the default amount. As a net change, the FFT and IFFT process 3 * 4/9 = 4/3 times the default amount of data, a relatively small change. The following initialization statement shows our new recipe's parameters:  lazyEyes = LazyEyes(useLaplacianPyramid=False,           useGrayOverlay=False,           imageSize=(320, 240)) Note that we are still using the default values for most parameters. If you have found non-default values that work well for numFFTThreads and numIFFTThreads on your machine, enter them as well. Here are the screenshots to show our new recipe's effect. This time, let's look at a non-extreme example first. I was typing on my laptop when this was taken. Note the haloes around my arms, which move a lot when I type, and a slight distortion and discoloration of my left cheek (viewer's left in this mirrored image). My left cheek twitches a little when I think. Apparently, it is a tic—already known to my friends and family but rediscovered by me with the help of computer vision! If you are viewing the color version of this image in the e-book, you should see that the haloes around my arms take a green hue from my shirt and a red hue from the sofa. Similarly, the haloes on my cheek take a magenta hue from my skin and a brown hue from my hair. Now, let's consider a more fanciful example. If we were Jedi instead of secret agents, we might wave a steel ruler in the air and pretend it was a lightsaber. While testing the theory that Lazy Eyes could make the ruler look like a real lightsaber, I took the following screenshot. The image shows two pairs of light and dark lines in two places where I was waving the lightsaber ruler. One of the pairs of lines passes through each of my shoulders. The Light Side (light line) and the Dark Side (dark line) show opposite ends of the ruler's path as I waved it. The lines are especially clear in the color version in the e-book. Finally, the moment for which we have all been waiting is … a recipe for amplifying a heartbeat! If you have a heart rate monitor, start by measuring your heart rate. Mine is approximately 87 bpm as I type these words and listen to inspiring ballads by the Canadian folk singer Stan Rogers. To convert bpm to Hz, divide the bpm value by 60 (the number of seconds per minute), giving (87 / 60) Hz = 1.45 Hz in my case. The most visible effect of a heartbeat is that a person's skin changes color, becoming more red or purple when blood is pumped through an area. Thus, let's modify our second recipe, which is able to amplify color motions in non-edge areas. Choosing a frequency range centered on 1.45 Hz, we have the following initializer:  lazyEyes = LazyEyes(minHz=1.4, maxHz=1.5,           useLaplacianPyramid=False,          useGrayOverlay=False,           imageSize=(320, 240)) Customize minHz and maxHz based on your own heart rate. Also, specify numFFTThreads and numIFFTThreads if non-default values work best for you on your machine. Even amplified, a heartbeat is difficult to show in still images; it is much clearer in the live video while running the app. However, take a look at the following pair of screenshots. My skin in the left-hand side screenshot is more yellow (and lighter) while in the right-hand side screenshot it is more purple (and darker). For comparison, note that there is no change in the cream-colored curtains in the background. Three recipes are a good start—certainly enough to fill an episode of a cooking show on TV. Go observe some other motions in your environment, try to estimate their frequencies, and then configure Lazy Eyes to amplify them. How do they look with grayscale amplification versus color amplification? Edge (Laplacian) versus area (Gaussian)? Different history lengths, pyramid levels, and amplification multipliers? Check the book's support page, http://www.nummist.com/opencv, for additional recipes and feel free to share your own by mailing me at josephhowse@nummist.com. Seeing things in another light Although we began this article by presenting Eulerian video magnification as a useful technique for visible light, it is also applicable to other kinds of light or radiation. For example, a person's blood (in veins and bruises) is more visible when imaged in ultraviolet (UV) or in near infrared (NIR) than in visible light. (Skin is more transparent to UV and NIR). Thus, a UV or NIR video is likely to be a better input if we are trying to magnify a person's pulse. Here are some examples of NIR and UV cameras that might provide useful results, though I have not tested them: The Pi NoIR camera (http://www.raspberrypi.org/products/pi-noir-camera/) is a consumer grade NIR camera with a MIPI interface. Here is a time lapse video showing the Pi NoIR renders outdoor scenes at https://www.youtube.com/watch?v=LLA9KHNvUK8. The camera is designed for Raspberry Pi, and on Raspbian it has V4L-compatible drivers that are directly compatible with OpenCV's VideoCapture class. Some Raspberry Pi clones might have drivers for it too. Unfortunately, Raspberry Pi is too slow to run Eulerian video magnification in real time. However, streaming the Pi NoIR input from Raspberry Pi to a desktop, via Ethernet, might allow for a real-time solution. The Agama V-1325R (http://www.agamazone.com/products_v1325r.html) is a consumer grade NIR camera with a USB interface. It is officially supported on Windows and Mac. Users report that it also works on Linux. It includes four NIR LEDs, which can be turned on and off via the vendor's proprietary software on Windows. Artray offers a series of industrial grade NIR cameras called InGaAs (http://www.artray.us/ingaas.html), as well as a series of industrial grade UV cameras (http://www.artray.us/usb2_uv.html). The cameras have USB interfaces. Windows drivers and an SDK are available from the vendor. A third-party project called OpenCV ARTRAY SDK (for more information refer to https://github.com/eiichiromomma/CVMLAB/wiki/OpenCV-ARTRAY-SDK) aims to provide interoperability with at least OpenCV's C API. Good luck and good hunting in the invisible light! Summary This article has introduced you to the relationship between computer vision and digital signal processing. We have considered a video feed as a collection of many signals—one for each channel value of each pixel—and we have understood that repetitive motions create wave patterns in some of these signals. We have used the Fast Fourier Transform and its inverse to create an alternative video stream that only sees certain frequencies of motion. Finally, we have superimposed this filtered video atop the original to amplify the selected frequencies of motion. There, we summarized Eulerian video magnification in 100 words! Our implementation adapts Eulerian video magnification to real time by running the FFT repeatedly on a sliding window of recently captured frames rather than running it once on an entire prerecorded video. We have considered optimizations such as limiting our signal processing to grayscale, recycling large data structures rather than recreating them, and using several threads. Resources for Article: Further resources on this subject: New functionality in OpenCV 3.0 [Article] Wrapping OpenCV [Article] Camera Calibration [Article]

This article will help you build your first HelloWorld WebSphere Enterprise Service Bus (WESB) mediation application and WESB mediation flow-based application. This article will then give an overview of Service Message Object (SMO) standards and how they relate to building applications using an SOA approach. At the completion of this article, you will have an understanding of how to begin building, testing, and deploying WESB mediations including: Overview of WESB-based programming fundamentals including WS-*? standards and Service Message Objects (SMOs) Building the first mediation module project in WID Using mediation flows Deploying the module on a server and testing the project Logging, debugging, and troubleshooting basics Exporting projects from WID WS standards Before we get down to discussing mediation flows, it is essential to take a moment and acquaint ourselves with some of the Web Service (WS) standards that WebSphere Integration Developers (WIDs) comply with. By using WIDs, user-friendly visual interfaces and drag-and-drop paradigms, developers are automatically building process flows that are globally standardized and compliant. This becomes critical in an ever-changing business environment that demands flexible integration with business partners. Here are some of the key specifications that you should be aware of as defined by the Worldwide Web Consortium (W3C): WS-Security: This is one of the secure standards used to secure Web Service at the message level, independent of the transport protocol. To learn more about WID and WS-Security refer to http://publib.boulder.ibm.com/infocenter/dmndhelp/v7r0mx/topic/com.ibm.wbit.help.runtime.doc/deploy/topics/cwssecurity.html. WS-Policy: A framework that helps define characteristics about Web Services through policies. To learn more about WS-Policy refer to: http://www.w3.org/TR/ws-policy/. WS-Addressing: This specification aids interoperability between Web Services, by standardizing ways to address Web Services and by providing addressing information in messages. In version 7.0, enhancements were made to WID to provide Web Services support for WS-Addressing and Attachments. For more details about WS-Addressing refer to http://www.w3.org/Submission/ws-addressing/. What are mediation flows? SOA, service consumers and service providers use an ESB as a communication vehicle. When services are loosely coupled through an ESB, the overall system has more flexibility and can be easily modified and enhanced to meet changing business requirements. We also saw that an ESB by itself is an enabler of many patterns and enables protocol transformations and provides mediation services, which can inspect, modify, augment and transform a message as it flows from requestor to provider. In WebSphere Enterprise Service, mediation modules provide the ESB functionality. The heart of a mediation module is the mediation flow component, which provides the mediation logic applied to a message as it flows from a service consumer to a provider. The mediation flow component is a type of SCA component that is typically used, but not limited to a mediation module. A mediation flow component contains a source interface and target references similar to other SCA components. The source interface is described using WSDL interface and must match the WSDL definition of the export to which it is wired. The target references are described using WSDL and must match the WSDL definitions of the imports or Java components to which they are wired. The mediation flow component handles most of the ESB functions including: Message filtering which is the capability to filter messages based on the content of the incoming message. Dynamic routing and selection of service provider, which is the capability to route incoming messages to the appropriate target at runtime based on predefined policies and rules. Message transformation, which is the capability to transform messages between source and target interfaces. This transformation can be defined using XSL stylesheets or business object maps. Message manipulation/enrichment, which is the capability to manipulate or enrich incoming message fields before they are sent to the target. This capability also allows you to do database lookups as needed. If the previous functionalities do not fit your requirements, you have the capability of defining a custom mediation behavior in JAVA. The following diagram describes the architectural layout of a mediation flow:   In the diagram, on the left-hand side you see the single service requester or source interface and on the right-hand side are the multiple service providers or target references. The mediation flow is the set of logical processing steps that efficiently route messages from the service requestor to the most appropriate service provider and back to the service requestor for that particular transaction and business environment. A mediation flow can be a request flow or a request and response flow. In a request flow message the sequence of processing steps is defined only from the source to the target. No message is returned to the source. However, in a request and response flow message the sequence of processing steps are defined from the single source to the multiple targets and back from the multiple targets to the single source. In the next section we take a deeper look into message objects and how they are handled in the mediation flows. Mediation primitives What are the various mediation primitives available in WESB? Mediation primitives are the core building blocks used to process the request and response messages in a mediation flow. Built-in primitives, which perform some predefined function that is configurable through the use of properties. Custom mediation primitives, which allow you to implement the function in Java. Mediation primitives have input and output terminals, through which the message flows. Almost all primitives have only one input terminal, but multiple input terminals are possible for custom mediation primitives. Primitives can have zero, one, or more output terminals. There is also a special terminal called the fail terminal through which the message is propagated when the processing of a primitive results in an exception. The different types of mediation primitives that are available in a mediation flow, along with their purpose, are summarized in the following table: Service invocation   Service Invoke Invoke external service, message modified with result Routing primitives   Message Filter Selectively forward messages based on element values Type Filter Selectively forward messages based on element types Routing primitives   Endpoint Lookup Find potential endpoint from a registry query Fan Out Starts iterative or split flow for aggregation Fan In Check completion of a split or aggregated flow Policy Resolution Set policy constrains from a registry query Flow Order Executes or fires the output terminal in a defined order Gateway Endpoint Lookup Finds potential endpoint in special cases from a registry SLA Check Verifies if the message complies with the SLA UDDI Endpoint Lookup Finds potential endpoints from a UDDI registry query Transformation primitives   XSL Transformation Update and modify messages using XSLT Business Object Map Update and modify messages using business object maps Message element setter Set, update, copy, and delete message elements Set message type Set elements to a more specific type Database Lookup Set elements from contents within a database Data Handler Update and modify messages using a data handler Custom Mediation Read, update, and modify messages using Java code SOAP header setter Read, update, copy, and delete SOAP header elements HTTP header setter Read, update, copy, and delete HTTP header elements JMS header setter Read, update, copy, and delete JMS header elements MQ header setter Read, update, copy, and delete MQ header elements Tracing primitives   Message logger Write a log message to a database or a custom destination Event emitter Raise a common base event to CEI Trace Send a trace message to a file or system out for debugging Error handling primitives   Stop Stop a single path in flow without an exception Fail Stop the entire flow and raise an exception Message Validator Validate a message against a schema and assertions Mediation subflow primitive   Subflow Represents a user-defined subflow