Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-writing-3d-space-rail-shooter-threejs-part-1
Martin Naumann
01 Oct 2015
7 min read
Save for later

Writing a 3D space rail shooter in Three.js, Part 1

Martin Naumann
01 Oct 2015
7 min read
In the course of this 3 part article series, you will learn how to write a simple 3D space shooter game with Three.js. The game will look like this: It will introduce the basic concepts of a Three.js application, how to write modular code and the core principles of a game, such as camera, player motion and collision detection. But now it's time to get ready! Prerequisites I will introduce you to a small set of tools that have proven to work reliably and help build things in a quick and robust fashion. The tools we will use are: npm - the node.js package manager browserify / watchify for using npm modules in the browser Three.js - a wrapper around WebGL and other 3D rendering APIs three-world - a highlevel wrapper around Three.js to reduce boilerplate. uglify-js - a tool to minify and strip down Javascript code To set up our project, create a directory and bring up a terminal inside this directory, e.g. mkdir space-blaster cd space-blaster Initializing our package As we'll use npm, it's a good idea to setup our project as an npm package. This way we'll get dependency management and some handy meta information - and it's easy: npm init Answer the questions and you're ready. Getting the dependencies Next up is getting the dependencies installed. This can be done conveniently by having npm install the dependencies and save them to our package.json as well. This way others can quickly grab our source code along with the package.json and just run npm install to get all required dependencies installed. So we now do npm install --save three three-world npm install --save-dev browserify watchify uglify-js The first line installs the two dependencies that our application needs to run, the second installs dependencies that we'll use throughout development. Creating the scaffold Now we'll need an HTML file to host our game and the main entry point Javascript file for our game. Let's start with the index.html: <!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <title>Space Blaster</title> <style> html, body { padding: 0; margin: 0; height: 100%; height: 100vh; overflow: hidden; } </style> </head> <body> <script src="app.js"></script> <noscript>Yikes, you'll need Javascript to play the game :/</noscript> </body> </html> That is enough to get us a nice and level field to put our game on. Note that the app.js will be the result of running watchify while we're developing and browserify when making a release build, respectively. Now we just create a js folder where our Javascript files will be placed in and we're nearly done with our setup, just one more thing: Build scripts In order to not have to clutter our system with globally-installed npm packages for our development tools, we can use npm's superpowers by extending the package.json a bit. Replace the scripts entry in your package.json with: "scripts": { "build": "browserify js/main.js | uglifyjs -mc > app.js", "dev": "watchify js/main.js -v -o app.js" }, and you'll have two handy new commands available: build runs browserify and minifies the output into app.js, starting from js/main.js which will be our game code's entry point dev runs watchify which will look for changes in our Javascript code and update app.js, but without minifying, which decreases the cycle time but produces a larger output file. To kick off development, we now create an empty js/main.js file and run npm run dev which should output something like this: > space-blaster@1.0.0 dev /code/space-blast > watchify js/main.js -v -o app.js 498 bytes written to app.js (0.04 seconds) With this running, open the js/main.js file in your favourite editor - it's time to start coding! Example repository and assets In the course of this article series, we will be using resources such as images and 3D models that are available for free on the internet. The models come from cgtrader, the space background from mysitemyway. All sources can be found at the Github repository for this article. Light, Camera, Action! Let's start creating - first of all we need a world and a function that is updating this world on each frame: var World = require('three-world') function render() { // here we'll update the world on each frame } World.init({ renderCallback: render }) // Things will be put into the world here World.start() Behind the curtain, this code creates a camera, a scene (think of this as our stage) and an ambient light. With World.start() the rendering loop is started and the world starts to "run". But so far our world has been empty and, thus, just a black void. Let's start to fill this void - we'll start with making a vortex in which we'll fly through space. Into the vortex To add things to our stage, we put more code in between World.init and World.start. var tunnel = new THREE.Mesh( new THREE.CylinderGeometry(100, 100, 5000, 24, 24, true), new THREE.MeshBasicMaterial({ map: THREE.ImageUtils.loadTexture('images/space.jpg'), side: THREE.BackSide }) ) tunnel.rotation.x = -Math.PI/2 World.add(tunnel) With this code we're creating a new mesh with a cylindrical geometry that is open at the ends and our space background, lay it on the side and add it to the world. The THREE.BackSide means that the texture shall be drawn on the inside of the cylinder, which is what we need as we'll be inside the cylinder. Now our world looks like this: There's a few things that we should tweak at this point. First of all, there's an ugly black hole at the end - let's fix that. First of all, we'll change the color that the World uses to clear out parts that are too far for the camera to see by adjusting the parameters to World.init: World.init({ renderCallback: render, clearColor: 0x000022}) With this we'll get a very dark blue at the end of our vortex. But it still looks odd as it is very sharp on the edges. But fortunately we can fix that by adding fog to our scene with a single line: World.getScene().fog = newTHREE.FogExp2(0x0000022, 0.00125) Note that the fog has the same color as the clearColor of the World and is added to the Scene. And voilà: It looks much nicer! Now let's do something about this stretched look of the space texture on the vortex by repeating it a few times to make it look better: var tunnel = new THREE.Mesh( new THREE.CylinderGeometry(100, 100, 5000, 24, 24, true), new THREE.MeshBasicMaterial({ map: THREE.ImageUtils.loadTexture('images/space.jpg', null, function(tex) { tex.wrapS = tex.wrapT = THREE.RepeatWrapping tex.repeat.set(5, 10) tex.needsUpdate = true }), side: THREE.BackSide }) ) Alright, that looks good. In Part 2 of this series we will add the spaceship and add asteroids to our game. About the author Martin Naumann is an open source contributor and web evangelist by heart from Zurich with a decade of experience from the trenches of software engineering in multiple fields. He works as a software engineer at Archilogic in front and backend. He devotes his time to moving the web forward, fixing problems, building applications and systems and breaking things for fun & profit. Martin believes in the web platform and is working with bleeding edge technologies that will allow the web to prosper.
Read more
  • 0
  • 0
  • 5673

article-image-ogre3d-scene-graph
Packt
20 Dec 2010
13 min read
Save for later

The Ogre 3D scene graph

Packt
20 Dec 2010
13 min read
Creating a scene node We will learn how to create a new scene node and attach our 3D model to it. How to create a scene node with Ogre 3D We will follow these steps: In the old version of our code, we had the following two lines in the createScene() function: Ogre::Entity* ent = mSceneMgr->createEntity("MyEntity","Sinbad.mesh"); mSceneMgr->getRootSceneNode()->attachObject(ent); Replace the last line with the following: Ogre::SceneNode* node = mSceneMgr->createSceneNode("Node1"); Then add the following two lines; the order of those two lines is irrelevant forthe resulting scene: mSceneMgr->getRootSceneNode()->addChild(node); node->attachObject(ent); Compile and start the application. What just happened? We created a new scene node named Node 1. Then we added the scene node to the root scene node. After this, we attached our previously created 3D model to the newly created scene node so it would be visible. How to work with the RootSceneNode The call mSceneMgr->getRootSceneNode() returns the root scene node. This scene node is a member variable of the scene manager. When we want something to be visible, we need to attach it to the root scene node or a node which is a child or a descendent in any way. In short, there needs to be a chain of child relations from the root node to the node; otherwise it won't be rendered. As the name suggests, the root scene node is the root of the scene. So the entire scene will be, in some way, attached to the root scene node. Ogre 3D uses a so-called scene graph to organize the scene. This graph is like a tree, it has one root, the root scene node, and each node can have children. We already have used this characteristic when we called mSceneMgr->getRootSceneNode()->addChild(node);. There we added the created scene node as a child to the root. Directly afterwards, we added another kind of child to the scene node with node->attachObject(ent);. Here, we added an entity to the scene node. We have two different kinds of objects we can add to a scene node. Firstly, we have other scene nodes, which can be added as children and have children themselves. Secondly, we have entities that we want rendered. Entities aren't children and can't have children themselves. They are data objects which are associated with the node and can be thought of as leaves of the tree. There are a lot of other things we can add to a scene, like lights, particle systems, and so on. We will later learn what these things are and how to use them. Right now, we only need entities. Our current scene graph looks like this: The first thing we need to understand is what a scene graph is and what it does. A scene graph is used to represent how different parts of a scene are related to each other in 3D space. 3D space Ogre 3D is a 3D rendering engine, so we need to understand some basic 3D concepts. The most basic construct in 3D is a vector, which is represented by an ordered triple (x,y,z). Each position in a 3D space can be represented by such a triple using the Euclidean coordination system for three dimensions. It is important to know that there are different kinds of coordinate systems in 3D space. The only difference between the systems is the orientation of the axis and the positive rotation direction. There are two systems that are widely used, namely, the left-handed and the right-handed versions. In the following image, we see both systems—on the left side, we see the left-handed version; and on the right side, we see the right-handed one. Source:http://en.wikipedia.org/wiki/File:Cartesian_coordinate_system_handedness.svg The names left-and right-handed are based on the fact that the orientation of the axis can be reconstructed using the left and right hand. The thumb is the x-axis, the index finger the y-axis, and the middle finger the z-axis. We need to hold our hands so that we have a ninety-degree angle between thumb and index finger and also between middle and index finger. When using the right hand, we get a right-handed coordination system. When using the left hand, we get the left-handed version. Ogre uses the right-handed system, but rotates it so that the positive part of the x-axis is pointing right and the negative part of the x-axis points to the left. The y-axis is pointing up and the z-axis is pointing out of the screen and it is known as the y-up convention. This sounds irritating at first, but we will soon learn to think in this coordinate system. The website http://viz.aset.psu.edu/gho/sem_notes/3d_fundamentals/html/3d_coordinates.html contains a rather good picture-based explanation of the different coordination systems and how they relate to each other. Scene graph A scene graph is one of the most used concepts in graphics programming. Simply put, it's a way to store information about a scene. We already discussed that a scene graph has a root and is organized like a tree. But we didn't touch on the most important function of a scene graph. Each node of a scene graph has a list of its children as well as a transformation in the 3D space. The transformation is composed of three aspects, namely, the position, the rotation, and the scale. The position is a triple (x,y,z), which obviously describes the position of the node in the scene. The rotation is stored using a quaternion, a mathematical concept for storing rotations in 3D space, but we can think of rotations as a single floating point value for each axis, describing how the node is rotated using radians as units. Scaling is quite easy; again, it uses a triple (x,y,z), and each part of the triple is simply the factor to scale the axis with. The important thing about a scene graph is that the transformation is relative to the parent of the node. If we modify the orientation of the parent, the children will also be affected by this change. When we move the parent 10 units along the x-axis, all children will also be moved by 10 units along the x-axis. The final orientation of each child is computed using the orientation of all parents. This fact will become clearer with the next diagram. The position of MyEntity in this scene will be (10,0,0) and MyEntity2 will be at (10,10,20). Let's try this in Ogre 3D. Pop quiz – finding the position of scene nodes Look at the following tree and determine the end positions of MyEntity and MyEntity2: MyEntity(60,60,60) and MyEntity2(0,0,0) MyEntity(70,50,60) and MyEntity2(10,-10,0) MyEntity(60,60,60) and MyEntity2(10,10,10) Setting the position of a scene node Now, we will try to create the setup of the scene from the diagram before the previous image. Time for action – setting the position of a scene node Add this new line after the creation of the scene node: node->setPosition(10,0,0); To create a second entity, add this line at the end of the createScene() function: Ogre::Entity* ent2 = mSceneMgr->createEntity("MyEntity2","Sinbad. mesh"); Then create a second scene node: Ogre::SceneNode* node2 = mSceneMgr->createSceneNode("Node2"); Add the second node to the first one: node->addChild(node2); Set the position of the second node: node2->setPosition(0,10,20); Attach the second entity to the second node: node2->attachObject(ent2); Compile the program and you should see two instances of Sinbad: What just happened? We created a scene which matches the preceding diagram. The first new function we used was at step 1. Easily guessed, the function setPosition(x,y,z) sets the position of the node to the given triple. Keep in mind that this position is relative to the parent. We wanted MyEntity2 to be at (10,10,20), because we added node2, which holds MyEntity2, to a scene node which already was at the position (10,0,0). We only needed to set the position of node2 to (0,10,20). When both positions combine, MyEntity2 will be at (10,10,20). Pop quiz – playing with scene nodes We have the scene node node1 at (0,20,0) and we have a child scene node node2, which has an entity attached to it. If we want the entity to be rendered at (10,10,10), at which position would we need to set node2? (10,10,10) (10,-10,10) (-10,10,-10) Have a go hero – adding a Sinbad Add a third instance of Sinbad and let it be rendered at the position (10,10,30). Rotating a scene node We already know how to set the position of a scene node. Now, we will learn how to rotate a scene node and another way to modify the position of a scene node. Time for action – rotating a scene node We will use the previous code, but create completely new code for the createScene() function. Remove all code from the createScene() function. First create an instance of Sinbad.mesh and then create a new scene node. Set the position of the scene node to (10,10,0), at the end attach the entity to the node, and add the node to the root scene node as a child: Ogre::Entity* ent = mSceneMgr->createEntity("MyEntity","Sinbad. mesh"); Ogre::SceneNode* node = mSceneMgr->createSceneNode("Node1"); node->setPosition(10,10,0); mSceneMgr->getRootSceneNode()->addChild(node); node->attachObject(ent); Again, create a new instance of the model, also a new scene node, and set the position to (10,0,0): Ogre::Entity* ent2 = mSceneMgr->createEntity("MyEntity2","Sinbad. mesh"); Ogre::SceneNode* node2 = mSceneMgr->createSceneNode("Node2"); node->addChild(node2); node2->setPosition(10,0,0); Now add the following two lines to rotate the model and attach the entity to the scene node: node2->pitch(Ogre::Radian(Ogre::Math::HALF_PI)); node2->attachObject(ent2); Do the same again, but this time use the function yaw instead of the function pitch and the translate function instead of the setPosition function: Ogre::Entity* ent3 = mSceneMgr->createEntity("MyEntity3","Sinbad. mesh"); Ogre::SceneNode* node3 = mSceneMgr->createSceneNode("Node3",); node->addChild(node3); node3->translate(20,0,0); node3->yaw(Ogre::Degree(90.0f)); node3->attachObject(ent3); And the same again with roll instead of yaw or pitch: Ogre::Entity* ent4 = mSceneMgr->createEntity("MyEntity4","Sinbad. mesh"); Ogre::SceneNode* node4 = mSceneMgr->createSceneNode("Node4"); node->addChild(node4); node4->setPosition(30,0,0); node4->roll(Ogre::Radian(Ogre::Math::HALF_PI)); node4->attachObject(ent4); Compile and run the program, and you should see the following screenshot: What just happened? We repeated the code we had before four times and always changed some small details. The first repeat is nothing special. It is just the code we had before and this instance of the model will be our reference model to see what happens to the other three instances we made afterwards. In step 4, we added one following additional line: node2->pitch(Ogre::Radian(Ogre::Math::HALF_PI)); The function pitch(Ogre::Radian(Ogre::Math::HALF_PI)) rotates a scene node around the x-axis. As said before, this function expects a radian as parameter and we used half of pi, which means a rotation of ninety degrees. In step 5, we replaced the function call setPosition(x,y,z) with translate(x,y,z). The difference between setPosition(x,y,z) and translate(x,y,z) is that setPosition sets the position—no surprises here. translate adds the given values to the position of the scene node, so it moves the node relatively to its current position. If a scene node has the position (10,20,30) and we call setPosition(30,20,10), the node will then have the position (30,20,10). On the other hand, if we call translate(30,20,10), the node will have the position (40,40,40). It's a small, but important, difference. Both functions can be useful if used in the correct circumstances, like when we want to position in a scene, we would use the setPosition(x,y,z) function. However, when we want to move a node already positioned in the scene, we would use translate(x,y,z). Also, we replaced pitch(Ogre::Radian(Ogre::Math::HALF_PI))with yaw(Ogre::Degree(90.0f)). The yaw() function rotates the scene node around the y-axis. Instead of Ogre::Radian(), we used Ogre::Degree(). Of course, Pitch and yaw still need a radian to be used. However, Ogre 3D offers the class Degree(), which has a cast operator so the compiler can automatically cast into a Radian(). Therefore, the programmer is free to use a radian or degree to rotate scene nodes. The mandatory use of the classes makes sure that it's always clear which is used, to prevent confusion and possible error sources. Step 6 introduces the last of the three different rotate function a scene node has, namely, roll(). This function rotates the scene node around the z-axis. Again, we could use roll(Ogre::Degree(90.0f)) instead of roll(Ogre::Radian(Ogre::Math::HALF_PI)). The program when run shows a non-rotated model and all three possible rotations. The left model isn't rotated, the model to the right of the left model is rotated around the x-axis, the model to the left of the right model is rotated around the y-axis, and the right model is rotated around the z-axis. Each of these instances shows the effect of a different rotate function. In short, pitch() rotates around the x-axis, yaw() around the y-axis, and roll() around the z-axis. We can either use Ogre::Degree(degree) or Ogre::Radian(radian) to specify how much we want to rotate. Pop quiz – rotating a scene node Which are the three functions to rotate a scene node? pitch, yawn, roll pitch, yaw, roll pitching, yaw, roll Have a go hero – using Ogre::Degree Remodel the code we wrote for the previous section in such a way that each occurrence of Ogre::Radian is replaced with an Ogre::Degree and vice versa, and the rotation is still the same. Scaling a scene node We already have covered two of the three basic operations we can use to manipulate our scene graph. Now it's time for the last one, namely, scaling. Time for action – scaling a scene node Once again, we start with the same code block we used before. Remove all code from the createScene() function and insert the following code block: Ogre::Entity* ent = mSceneMgr->createEntity("MyEntity","Sinbad. mesh"); Ogre::SceneNode* node = mSceneMgr->createSceneNode("Node1"); node->setPosition(10,10,0); mSceneMgr->getRootSceneNode()->addChild(node); node->attachObject(ent); Again, create a new entity: Ogre::Entity* ent2 = mSceneMgr->createEntity("MyEntity2","Sinbad. mesh"); Now we use a function that creates the scene node and adds it automatically as a child. Then we do the same thing we did before: Ogre::SceneNode* node2 = node->createChildSceneNode("node2"); node2->setPosition(10,0,0); node2->attachObject(ent2); Now, after the setPosition() function, call the following line to scale the model: node2->scale(2.0f,2.0f,2.0f); Create a new entity: Ogre::Entity* ent3 = mSceneMgr->createEntity("MyEntity3","Sinbad. mesh"); Now we call the same function as in step 3, but with an additional parameter: Ogre::SceneNode* node3 = node->createChildSceneNode("node3",Ogre:: Vector3(20,0,0)); After the function call, insert this line to scale the model: node3->scale(0.2f,0.2f,0.2f); Compile the program and run it, and you should see the following image:
Read more
  • 0
  • 0
  • 5671

article-image-creating-shopping-cart-using-zend-framework-part-1
Packt
27 Oct 2009
13 min read
Save for later

Creating a Shopping Cart using Zend Framework: Part 1

Packt
27 Oct 2009
13 min read
Our next task in creating the storefront is to create the shopping cart. This will allow users to select the products they wish to purchase. Users will be able to select, edit, and delete items from their shopping cart. Lets get started. Creating the Cart Model and Resources We will start by creating our model and model resources. The Cart Model differs from our previous model in the fact that it will use the session to store its data instead of the database. Cart Model The Cart Model will store the products that they wish to purchase. Therefore, the Cart Model will contain Cart Items that will be stored in the session. Let's create this class now. application/modules/storefront/models/Cart.php class Storefront_Model_Cart extends SF_Model_Abstract implements SeekableIterator, Countable, ArrayAccess { protected $_items = array(); protected $_subTotal = 0; protected $_total = 0; protected $_shipping = 0; protected $_sessionNamespace; public function init() { $this->loadSession(); } public function addItem( Storefront_Resource_Product_Item_Interface $product,$qty) { if (0 > $qty) { return false; } if (0 == $qty) { $this->removeItem($product); return false; } $item = new Storefront_Resource_Cart_Item( $product, $qty ); $this->_items[$item->productId] = $item; $this->persist(); return $item; } public function removeItem($product) { if (is_int($product)) { unset($this->_items[$product]); } if ($product instanceof Storefront_Resource_Product_Item_Interface) { unset($this->_items[$product->productId]); } $this->persist(); } public function setSessionNs(Zend_Session_Namespace $ns) { $this->_sessionNamespace = $ns; } public function getSessionNs() { if (null === $this->_sessionNamespace) { $this->setSessionNs(new Zend_Session_Namespace(__CLASS__)); } return $this->_sessionNamespace; } public function persist() { $this->getSessionNs()->items = $this->_items; $this->getSessionNs()->shipping = $this->getShippingCost(); } public function loadSession() { if (isset($this->getSessionNs()->items)) { $this->_items = $this->getSessionNs()->items; } if (isset($this->getSessionNs()->shipping)) { $this->setShippingCost($this->getSessionNs()->shipping); } } public function CalculateTotals() { $sub = 0; foreach ($this as $item) { $sub = $sub + $item->getLineCost(); } $this->_subTotal = $sub; $this->_total = $this->_subTotal + (float) $this->_shipping; } public function setShippingCost($cost) { $this->_shipping = $cost; $this->CalculateTotals(); $this->persist(); } public function getShippingCost() { $this->CalculateTotals(); return $this->_shipping; } public function getSubTotal() { $this->CalculateTotals(); return $this->_subTotal; } public function getTotal() { $this->CalculateTotals(); return $this->_total; } /*...*/ } We can see that the Cart Model class is fairly weighty and in fact, we have not included the full class here. The reason we have slightly truncated the class is that we are implementing the SeekableIterator, Countable, and ArrayAccess interfaces. These interfaces are defined by PHP's SPL Library and we use them to provide a better way to interact with the cart data. For the complete code, copy the methods below getTotal() from the example files for this article. We will look at what each method does shortly in the Cart Model implementation section, but first, let's look at what functionality the SPL interfaces allow us to add. Cart Model interfaces The SeekableIterator interface allows us to access our cart data in these ways: // iterate over the cartforeach($cart as $item) {}// seek an item at a position$cart->seek(1);// standard iterator access$cart->rewind();$cart->next();$cart->current(); The Countable interface allows us to count the items in our cart: count($cart); The ArrayAccess interface allows us to access our cart like an array: $cart[0]; Obviously, the interfaces provide no concrete implementation for the functionality, so we have to provide it on our own. The methods not listed in the previous code listing are: offsetExists($key) offsetGet($key) offsetSet($key, $value) offsetUnset($key) current() key() next() rewind() valid() seek($index) count() We will not cover the actual implementation of these interfaces, as they are standard to PHP. However, you will need to copy all these methods from the example files to get the Cart Model working. Documentation for the SPL library can be found athttp://www.php.net/~helly/php/ext/spl/     Cart Model implementation Going back to our code listing, let's now look at how the Cart Model is implemented. Let's start by looking at the properties and methods of the class. The Cart Model has the following class properties: $_items:An array of cart items $_subTotal: Monetary total of cart items $_total: Monetary total of cart items plus shipping $_shipping: The shipping cost $_sessionNamespace: The session store The Cart Model has the following methods: init(): Called during construct and loads the session data addItem(Storefront_Resource_Product_Item_Interface $product, $qty): Adds or updates items in the cart removeItem($product): Removes a cart item setSessionNs(Zend_Session_Namespace $ns): Sets the session instance to use for storage getSessionNs(): Gets the current session instance persist(): Saves the cart data to the session loadSession(): Loads the stored session values calculateTotals(): Calculates the cart totals setShippingCost($cost): Sets the shipping cost getShippingCost(): Gets the shipping cost getSubTotal(): Gets the total cost for items in the cart (not including the shipping) getTotal(): Gets the subtotal plus the shipping cost When we instantiate a new Cart Model instance, the init() method is called. This is defined in the SF_Model_Abstract class and is called by the __construct() method. This enables us to easily extend the class's instantiation process without having to override the constructor. The init() method simply calls the loadSession() method. This method populates the model with the cart items and shipping information stored in the session. The Cart Model uses Zend_Session_Namespace to store this data, which provides an easy-to-use interface to the $_SESSION variable. If we look at the loadSession() method, we see that it tests whether the items and shipping properties are set in the session namespace. If they are, then we set these values on the Cart Model. To get the session namespace, we use the getSessionNs() method. This method checks if the $_sessionNs property is set and returns it. Otherwise it will lazy load a new Zend_Session_Namespace instance for us. When using Zend_Session_ Namespace, we must provide a string to its constructor that defines the name of the namespace to store our data in. This will then create a clean place to add variables to, without worrying about variable name clashes. For the Cart Model, the default namespace will be Storefront_Model_Cart. The Zend_Session_Namespace component provides a range of functionality that we can use to control the session. For example, we can set the expiration time as follows: $ns = new Zend_Session_Namespace('test');$ns->setExpirationSeconds(60, 'items');$ns->setExpirationHops(10);$ns->setExpirationSeconds(120); This code would set the item's property expiration to 60 seconds and the namespaces expiration to 10 hops (requests) or 120 seconds, whichever is reached first. The useful thing about this is that the expiration is not global. Therefore, we can have specialized expiration per session namespace. There is a full list of Zend_Session_Namespace functionalities in the reference manual. Testing with Zend_Session and Zend_Session_NamespaceTesting with the session components can be fairly diffi cult. For the Cart Model, we use the setSessionNs() method to allow us to inject a mock object for testing, which you can see in the Cart Model unit tests. There are plans to rewrite the session components to make testing easier in the future, so keep an eye out for those updates. To add an item to the cart, we use the addItem() method. This method accepts two parameters,$product and $qty. The $product parameter must be an instance of the Storefront_Resource_Product_Item class, and the $qty parameter must be an integer that defines the quantity that the customer wants to order. If the addItem() method receives a valid $qty, then it will create a new Storefront_Resource_Cart_Item and add it to the $_items array using the productId as the array key. We then call the persist() method. This method simply stores all the relevant cart data in the session namespace for us. You will notice that we are not using a Model Resource in the Cart Model and instead we are directly instantiating a Model Resource Item. This is because the Model Resources represent store items and the Cart Model is already doing this for us so it is not needed. To remove an item, we use the removeItem() method. This accepts a single parameter $product which can be either an integer or a Storefront_Resource_Product_Item instance. The matching cart item will be removed from the $_items array and the data will be saved to the session. Also, addItem() will call removeItem() if the quantity is set to zero. The other methods in the Cart Model are used to calculate the monetary totals for the cart and to set the shipping. We will not cover these in detail here as they are fairly simple mathematical calculations. Cart Model Resources Now that we have our Model created, let's create the Resource Interface and concrete Resource class for our Model to use. application/modules/storefront/models/resources/Cart/Item/Interface.php interface Storefront_Resource_Cart_Item_Interface { public function getLineCost(); } The Cart Resource Item has a very simple interface that has one method, getLineCost(). This method is used when calculating the cart totals in the Cart Model. application/modules/storefront/models/resources/Cart/Item.php class Storefront_Resource_Cart_Item implements Storefront_Resource_Cart_Item_Interface { public $productId; public $name; public $price; public $taxable; public $discountPercent; public $qty; public function __construct(Storefront_Resource_Product_Item_ Interface $product, $qty) { $this->productId = (int) $product->productId; $this->name = $product->name; $this->price = (float) $product->getPrice(false,false); $this->taxable = $product->taxable; $this->discountPercent = (int) $product->discountPercent; $this->qty = (int) $qty; } public function getLineCost() { $price = $this->price; if (0 !== $this->discountPercent) { $discounted = ($price*$this->discountPercent)/100; $price = round($price - $discounted, 2); } if ('Yes' === $this->taxable) { $taxService = new Storefront_Service_Taxation(); $price = $taxService->addTax($price); } return $price * $this->qty; } } The concrete Cart Resource Item has two methods __construct() and getLineCost(). The constructor accepts two parameters $product and $qty that must be a Storefront_Resource_Product_Item_Interface instance and integer respectively. This method will then simply copy the values from the product instance and store them in the matching public properties. We do this because we do not want to simply store the product instance because it has all the database connection data contained within. This object will be serialized and stored in the session. The getLineCost() method simply calculates the cost of the product adding tax and discounts and then multiplies it by the given quantity. Shipping Model We also need to create a Shipping Model so that the user can select what type of shipping they would like. This Model will simply act as a data store for some predefined shipping values. application/modules/storefront/models/Shipping.php class Storefront_Model_Shipping extends SF_Model_Abstract { protected $_shippingData = array( 'Standard' => 1.99, 'Special' => 5.99, ); public function getShippingOptions() { return $this->_shippingData; } } The shipping Model is very simple and only contains the shipping options and a single method to retrieve them. In a normal application, shipping would usually be stored in the database and most likely have its own set of business rules. For the Storefront, we are not creating a complete ordering process so we do not need these complications. Creating the Cart Controller With our Model and Model Resources created, we can now start wiring the application layer together. The Cart will have a single Controller, CartController that will be used to add, view, and update cart items stored in the Cart Model. application/modules/storefront/controllers/CartController.php class Storefront_CartController extends Zend_Controller_Action { protected $_cartModel; protected $_catalogModel; public function init() { $this->_cartModel = new Storefront_Model_Cart(); $this->_catalogModel = new Storefront_Model_Catalog(); } public function addAction() { $product = $this->_catalogModel->getProductById( $this->_getParam('productId') ); if(null === $product) { throw new SF_Exception('Product could not be added to cart as it does not exist' ); } $this->_cartModel->addItem( $product, $this->_getParam('qty') ); $return = rtrim( $this->getRequest()->getBaseUrl(), '/' ) . $this->_getParam('returnto'); $redirector = $this->getHelper('redirector'); return $redirector->gotoUrl($return); } public function viewAction() { $this->view->cartModel = $this->_cartModel; } public function updateAction() { foreach($this->_getParam('quantity') as $id => $value) { $product = $this->_catalogModel->getProductById($id); if (null !== $product) { $this->_cartModel->addItem($product, $value); } } $this->_cartModel->setShippingCost( $this->_getParam('shipping') ); return $this->_helper->redirector('view'); } } The Cart Controller has three actions that provide a way to: add: add cart items view: view the cart contents update: update cart items The addAction() first tries to find the product to be added to the cart. This is done by searching for the product by its productId field, which is passed either in the URL or by post using the Catalog Model. If the product is not found, then we throw an SF_Exception stating so. Next, we add the product to the cart using the addItem() method. When adding the product, we also pass in the qty. The qty can again be either in the URL or post. Once the product has been successfully added to the cart, we then need to redirect back to the page where the product was added. As we can have multiple locations, we send a returnto variable with the add request. This will contain the URL to redirect back to, once the item has been added to the cart. To stop people from being able to redirect away from the storefront, we prepend the baseurl to the redirect string. To perform the actual redirect, we use the redirector Action Helper's gotoUrl() method. This will create an HTTP redirect for us. The viewAction() simply assigns the Cart Model to the cartModel View property. Most of the cart viewing functionality has been pushed to the Cart View Helper and Forms, which we will create shortly. The updateAction() is used to update the Cart Items already stored in the cart. The first part of this updates the quantities. The quantities will be posted to the Action as an array in the quantity parameter. The array will contain the productId as the array key, and the quantity as the value. Therefore, we iterate over the array fi nding the product by its ID and adding it to the cart. The addItem() method will then update the quantity for us if the item exists and remove any with a zero quantity. Once we have updated the cart quantities, we set the shipping and redirect back to the viewAction. >> Continue Reading Creating a Shopping Cart using Zend Framework: Part 2
Read more
  • 0
  • 0
  • 5671

article-image-oracle-environmental-variables-and-scripting
Packt
22 Jun 2010
10 min read
Save for later

Oracle: Environmental Variables and Scripting

Packt
22 Jun 2010
10 min read
(For more resources on Oracle, see here.) Unix scripting will be one of the most powerful tools in your arsenal and only with constant use will you become proficient. Your standards for scripting need to produce code that is robust, modular, consistent, and scalable. This article won't cover everything you need to know about scripting. There are many excellent lessons available from your operating system vendor. See the following documents for more information specific to how Oracle products behave in a Unix environment: Note: 131207.1 Subject: How to Set Unix Environment Variables Note: 1011994.6 Subject: How do Applications Act in Different Unix Shells. How to Integrate the Shell, SQL*Plus Scripts and PL/SQL in any Permutation? [ID 400195.1] (Excellent document) It is suggested that the reader should read the Oracle Database Concepts 11g Release 2 (11.2) of the documentation, the book every DBA should start with. Usually cron is used for scheduling in Unix and the AT command with Windows. For Oracle utilities that run outside the database (or which must only run when the database is down), Unix shell scripting is best used. A well-written script would parse /etc/oratab for a list of ORACLE_SIDs on a single server for tasks such as parsing log files, consistent backups, exports, dbverify, and RMAN. If you have a solitary database, then DBMS_SCHEDULER can be used with a combination of SQL and PL/SQL integrated with OS commands. Occasionally, DBAs rename the oraenv located in $ORACLE_HOME/bin when they have added custom code to the /usr/local/bin/oraenv (default location on Linux), so that they can make sure they know exactly which one is executed at run time. If you have any problems related to running a modified oraenv file, check which one is being executed and adjust the $PATH variable as needed. The following is a list of some of the Oracle-provided Unix commands with a quick synopsis of their purpose. Most of them are located in $ORACLE_HOME/bin: wrap—encrypts stored procedures for advanced use oerr—displays oracle errors. Syntax: oerr ora 12154 Sysresv—instance and shared memory segments Tkprof—formats output trace file into readable format Dbshut—shell script to shut down all instances dbstart—shell script to start up all instances at boot Dbhome—sets ORACLE_HOME Oraenv—sets environmental variables for ORACLE_SID trcasst—trace assistant Guidelines for scripting These are general suggestions for someone with some experience with a Unix operating system. You will need more background information to understand this section. The best sources of information will be the software vendor of your operating system, because there are small differences between the different versions and flavors of Unix. As with any code, you must test on a non-production system first, as inexperience may lead to unexpected results. Separating the configuration file Use the OS-provided default profile for environmental variables, but use a separate configuration file to set the $ORACLE_SID variable. There are several key environmental variables that are commonly used with Oracle software, all of which are found in the documentation specific to the operating system. Optimal Flexible Architecture (OFA) mentions setting the $ORACLE_SID in the profile, but if you have more than one database or Oracle product installed on a system, it is best done interactively. A default profile is loaded when you first log in to Unix. So if the $ORACLE_SID is loaded when you log on, what happens when you want to change ORACLE_SID(s)? This is when the environment becomes mixed. It just keeps appending the $PATH variable each time you execute the oraenv script. Which set of executables will you find first? It will be those executables for which you ran oraenv the first time. At this point I wanted to execute SQL*Plus out of the 11g directory, but was able to determine that the 10gR2 client was first in the $PATH. How did I know that? Use the which Unix command to find out. It is also a good practice to use a separate terminal window for each ORACLE_HOME. Normally, you will operate with multiple windows open, one for each ORACLE_HOME or ORACLE_SID in different colors to visually remind you which one is production. The example profile is provided in the code: example_profile.txt. The profile sets the entire user environment at first logon. This one is specific to the ksh or korn shell on the Linux operating system and will also work for bash. Differences in bash include that the line history is scrollable with the up and down arrows instead of vi commands. To set the ORACLE_SID and activate all of the variables located in profile, source the file oraenv (bash, Bourne, or korn shell) or coraenv (C shell). Source means the variables will be in effect for the entire session and not just the current line in the command window. You indicate source by putting a '.' (dot) in front of the file. As the oraenv file is located in /usr/local/bin (on Linux) and this location is in the $PATH, typing it at the command line works. Putting key Oracle files, such as oraenv, oratab, and oraInst.loc, in locations that ORACLE_HOMEwill not be affected by standard Oracle installations is also part of the OFA. The oraenv script is installed into /usr/local/ bin/ automatically when running .runInstaller for the first time. Notice the prompt that you will see if you use the command as in the profile listed above: A note about prompts: Every person will want to customize their own prompt so; look around for various versions that tickle your fancy. This one is better than most examples to compare to. Notice the small difference in the prompt before and after? ORACLE_SID is now defined because oraenv is executed, which also runs dbhome (also located in /usr/local/bin), but these scripts require a valid entry in /etc/oratab. If you type the ORACLE_SID incorrectly on Unix, this will be case sensitive. It will ask where the ORACLE_HOME is if there is no entry in oratab. Making an ORACLE_SID lowercase conforms to the OFA standard (see the install guide for your OS for more information). Some DBAs use an uppercase ORACLE_SID because it makes it more prominent for any type of naming convention and is meant to reduce human error. You can use an ORACLE_SID entry in the oratab file to reference other Oracle products such as the Grid Control Intelligent Agent. The ea, which is an ORACLE_SID in the following oratab example, is what I use to indicate the Enterprise Manager Agent. The third letter after the ORACLE_HOME (N) indicates when Oracle-supplied utilities (like db_shut and db_start) are not to execute against this ORACLE_HOME. I personally use the N for my own scripting to indicate which utilities shouldn't run against this ORACLE_HOME. What this does is take advantage of Oracle-provided files—oratab and oraenv—to accomplish other types of tasks. This is only a suggested use. There are other ways of setting environmental variables for non-database products. You will need to create a test database to work through all of the examples and practice scenarios. How should you create the test database? Use the Oracle-provided Database Configuration Assistant (DBCA) utility to create a test database. There are default templates provided that will work for most of the tasks outlined in this article. If you are interested in duplicating some of the advanced tasks (like Data Guard), then it will require the installation of the Enterprise Edition of Oracle Database. All tasks in this article were done with 11.1.0.7 version of Oracle Database with some references to 11.2.0.1, which had just been released. Host commands relative location This will be important as you begin scripting. Host commands are relative to the location of the executable. As a general rule, you should execute database-specific utilities (imp, exp, datapump, RMAN, and so forth) on the server where the database is located in the correct ORACLE_HOME. This reduces the amount of issues such as core dumps and version compatibilities. This is different from what is usually thought of as a client utilities such as SQL*Plus. There are exceptions to this rule, for it is recommended to run a compiled code (C, C++, Cobol) on a separate server rather than a database. See the following document for setting the TWO_TASK variable when using a separate node for compiled programs. TWO_TASK is an environmental variable. Subject: How to Customize Pro*C and Pro*Cobol Makefiles demo_proc.mk And demo_procob.mk On Linux/Unix [Doc ID: 602936.1]. Notice the WARNING! message that is set using the new 11g sqlnet.ora parameter SEC_USER_UNAUTHORIZED_ACCESS_BANNER. The sqlnet.ora file is part of the SQL*Net components of Oracle RDBMS, which handle the communication between clients and the database. oracle@nodename:/u01/app/oracle/admin/newdb[newdb]> sqlplus /nologSQL*Plus: Release 11.1.0.7.0 - Production on Thu Nov 5 19:00:292009Copyright (c) 1982, 2008, Oracle. All rights reserved.@> connect / as sysdba####################################################################WARNING! This computer system is the property of YOURORGANIZATIONand may be accessed only by authorized users.Unauthorized use of this system is strictly prohibited and may besubject to criminal prosecution.Connected.SYS@newdb> If you wanted to execute something that is available on the operating system level, then you would use a host command (either Windows or Unix), or on Unix the ! symbol. The output below shows that I am logged into the newdb as sys and lists (ls command) the files located in the $ORACLE_HOME/sqlplus/admin directory: Notice how the SQL prompt is populated with the ORACLE_SID and the username that is connected to the database. This is done by adding a line to the glogin.sql file, which can be done within SQL*Plus as shown below (I used the text editor vi): Host commands work based on the location of SQL*Plus. If you want to execute these same commands from a Windows desktop connecting to a remote Unix database, then it would require a Windows equivalent command like Notepad or another editor. If you have Cygwin installed and configured on the Windows desktop, then it would allow you to run Unix-equivalent commands like vi. Separating the variable part of the script into its own configuration file There are scripts that will need some variables set, but you don't necessarily want to use the profile to set a variable at every login. The variables may need to contain commands specific to applications such as RMAN, SQL*Plus or specific to certain tasks. This is where a configuration file comes in handy, which is a personal preference as to what you call them. Be sure not to use reserved keywords for variables, as that leads to unexpected results. In the example below, we use emal instead of the word email. To call this configuration file, you source it starting with the prerequisite . (dot) within a script. This file can also be used in the custom code section of the ioraenv file.
Read more
  • 0
  • 0
  • 5671

article-image-first-steps
Packt
24 Mar 2014
7 min read
Save for later

First Steps

Packt
24 Mar 2014
7 min read
(For more resources related to this topic, see here.) Installing matplotlib Before experimenting with matplotlib, you need to install it. Here we introduce some tips to get matplotlib up and running without too much trouble. How to do it... We have three likely scenarios: you might be using Linux, OS X, or Windows. Linux Most Linux distributions have Python installed by default, and provide matplotlib in their standard package list. So all you have to do is use the package manager of your distribution to install matplotlib automatically. In addition to matplotlib, we highly recommend that you install NumPy, SciPy, and SymPy, as they are supposed to work together. The following list consists of commands to enable the default packages available in different versions of Linux: Ubuntu: The default Python packages are compiled for Python 2.7. In a command terminal, enter the following command: sudo apt-get install python-matplotlib python-numpy python-scipy python-sympy ArchLinux: The default Python packages are compiled for Python 3. In a command terminal, enter the following command: sudo pacman -S python-matplotlib python-numpy python-scipy python-sympy If you prefer using Python 2.7, replace python by python2 in the package names Fedora: The default Python packages are compiled for Python 2.7. In a command terminal, enter the following command: sudo yum install python-matplotlib numpy scipy sympy There are other ways to install these packages; in this article, we propose the most simple and seamless ways to do it. Windows and OS X Windows and OS X do not have a standard package system for software installation. We have two options—using a ready-made self-installing package or compiling matplotlib from the code source. The second option involves much more work; it is worth the effort to have the latest, bleeding edge version of matplotlib installed. Therefore, in most cases, using a ready-made package is a more pragmatic choice. You have several choices for ready-made packages: Anaconda, Enthought Canopy, Algorete Loopy, and more! All these packages provide Python, SciPy, NumPy, matplotlib, and more (a text editor and fancy interactive shells) in one go. Indeed, all these systems install their own package manager and from there you install/uninstall additional packages as you would do on a typical Linux distribution. For the sake of brevity, we will provide instructions only for Enthought Canopy. All the other systems have extensive documentation online, so installing them should not be too much of a problem. So, let's install Enthought Canopy by performing the following steps: Download the Enthought Canopy installer from https://www.enthought.com/products/canopy. You can choose the free Express edition. The website can guess your operating system and propose the right installer for you. Run the Enthought Canopy installer. You do not need to be an administrator to install the package if you do not want to share the installed software with other users. When installing, just click on Next to keep the defaults. You can find additional information about the installation process at http://docs.enthought.com/canopy/quick-start.html. That's it! You will have Python 2.7, NumPy, SciPy, and matplotlib installed and ready to run. Plotting one curve The initial example of Hello World! for a plotting software is often about showing a simple curve. We will keep up with that tradition. It will also give you a rough idea about how matplotlib works. Getting ready You need to have Python (either v2.7 or v3) and matplotlib installed. You also need to have a text editor (any text editor will do) and a command terminal to type and run commands. How to do it... Let's get started with one of the most common and basic graph that any plotting software offers—curves. In a text file saved as plot.py, we have the following code: import matplotlib.pyplot as plt X = range(100) Y = [value ** 2 for value in X] plt.plot(X, Y) plt.show() Assuming that you installed Python and matplotlib, you can now use Python to interpret this script. If you are not familiar with Python, this is indeed a Python script we have there! In a command terminal, run the script in the directory where you saved plot.py with the following command: python plot.py Doing so will open a window as shown in the following screenshot: The window shows the curve Y = X ** 2 with X in the [0, 99] range. As you might have noticed, the window has several icons, some of which are as follows: : This icon opens a dialog, allowing you to save the graph as a picture file. You can save it as a bitmap picture or a vector picture. : This icon allows you to translate and scale the graphics. Click on it and then move the mouse over the graph. Clicking on the left button of the mouse will translate the graph according to the mouse movements. Clicking on the right button of the mouse will modify the scale of the graphics. : This icon will restore the graph to its initial state, canceling any translation or scaling you might have applied before. How it works... Assuming that you are not very familiar with Python yet, let's analyze the script demonstrated earlier. The first line tells Python that we are using the matplotlib.pyplot module. To save on a bit of typing, we make the name plt equivalent to matplotlib.pyplot. This is a very common practice that you will see in matplotlib code. The second line creates a list named X, with all the integer values from 0 to 99. The range function is used to generate consecutive numbers. You can run the interactive Python interpreter and type the command range(100) if you use Python 2, or the command list(range(100)) if you use Python 3. This will display the list of all the integer values from 0 to 99. In both versions, sum(range(100)) will compute the sum of the integers from 0 to 99. The third line creates a list named Y, with all the values from the list X squared. Building a new list by applying a function to each member of another list is a Python idiom, named list comprehension. The list Y will contain the squared values of the list X in the same order. So Y will contain 0, 1, 4, 9, 16, 25, and so on. The fourth line plots a curve, where the x coordinates of the curve's points are given in the list X, and the y coordinates of the curve's points are given in the list Y. Note that the names of the lists can be anything you like. The last line shows a result, which you will see on the window while running the script. There's more... So what we have learned so far? Unlike plotting packages like gnuplot, matplotlib is not a command interpreter specialized for the purpose of plotting. Unlike Matlab, matplotlib is not an integrated environment for plotting either. matplotlib is a Python module for plotting. Figures are described with Python scripts, relying on a (fairly large) set of functions provided by matplotlib. Thus, the philosophy behind matplotlib is to take advantage of an existing language, Python. The rationale is that Python is a complete, well-designed, general purpose programming language. Combining matplotlib with other packages does not involve tricks and hacks, just Python code. This is because there are numerous packages for Python for pretty much any task. For instance, to plot data stored in a database, you would use a database package to read the data and feed it to matplotlib. To generate a large batch of statistical graphics, you would use a scientific computing package such as SciPy and Python's I/O modules. Thus, unlike many plotting packages, matplotlib is very orthogonal—it does plotting and only plotting. If you want to read inputs from a file or do some simple intermediary calculations, you will have to use Python modules and some glue code to make it happen. Fortunately, Python is a very popular language, easy to master and with a large user base. Little by little, we will demonstrate the power of this approach.
Read more
  • 0
  • 0
  • 5668

article-image-hitting-the-right-notes-in-2017-ai-song-for-data-scientists
Aarthi Kumaraswamy
26 Dec 2017
3 min read
Save for later

Hitting the right notes in 2017: AI in a song for Data Scientists

Aarthi Kumaraswamy
26 Dec 2017
3 min read
A lot, I mean lots and lots of great articles have been written already about AI’s epic journey in 2017. They all generally agree that 2017 sets the stage for AI in very real terms.  We saw immense progress in academia, research and industry in terms of an explosion of new ideas (like capsNets), questioning of established ideas (like backprop, AI black boxes), new methods (Alpha Zero’s self-learning), tools (PyTorch, Gluon, AWS SageMaker), and hardware (quantum computers, AI chips). New and existing players gearing up to tap into this phenomena even as they struggled to tap into the limited talent pool at various conferences and other community hangouts. While we have accelerated the pace of testing and deploying some of those ideas in the real world with self-driving cars, in media & entertainment, among others, progress in building a supportive and sustainable ecosystem has been slow. We also saw conversations on AI ethics, transparency, interpretability, fairness, go mainstream alongside broader contexts such as national policies, corporate cultural reformation setting the tone of those conversations. While anxiety over losing jobs to robots keeps reaching new heights proportional to the cryptocurrency hype, we saw humanoids gain citizenship, residency and even talk about contesting in an election! It has been nothing short of the stuff, legendary tales are made of: struggle, confusion, magic, awe, love, fear, disgust, inspiring heroes, powerful villains, misunderstood monsters, inner demons and guardian angels. And stories worth telling must have songs written about them! Here’s our ode to AI Highlights in 2017 while paying homage to an all-time favorite: ‘A few of my favorite things’ from Sound of Music. Next year, our AI friends will probably join us behind the scenes in the making of another homage to the extraordinary advances in data science, machine learning, and AI. [box type="shadow" align="" class="" width=""] Stripes on horses and horsetails on zebras Bright funny faces in bowls full of rameN Brown furry bears rolled into pandAs These are a few of my favorite thinGs   TensorFlow projects and crisp algo models Libratus’ poker faces, AlphaGo Zero’s gaming caboodles Cars that drive and drones that fly with the moon on their wings These are a few of my favorite things   Interpreting AI black boxes, using Python hashes Kaggle frenemies and the ones from ML MOOC classes R white spaces that melt into strings These are a few of my favorite things   When models don’t converge, and networks just forget When I am sad I simply remember my favorite things And then I don’t feel so bad[/box]   PS: We had to leave out many other significant developments in the above cover as we are limited in our creative repertoire. We invite you to join in and help us write an extended version together! The idea is to make learning about data science easy, accessible, fun and memorable!    
Read more
  • 0
  • 0
  • 5666
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-photo-compositing-gimp-part-1
Packt
07 Dec 2009
7 min read
Save for later

Photo Compositing with The GIMP: Part 1

Packt
07 Dec 2009
7 min read
Basing from my previous GIMP article titled Creating Pseudo-3D Imagery with GIMP, you learned how to do some basic selection manipulation, gradient application, faking Depth of Field, etc.  In line with that, I’m following it with a new article very much related to the concepts discussed therein but we’ll raise the bar a bit by having a glimpse on compositing, where we’ll use an existing image or photograph and later add in our 2-dimensional element seamlessly with the said picture. So if you haven’t read yet “Creating Pseudo-3D Imagery with GIMP”, I highly suggest you do so since almost all major concepts we’ll tackle here are based off of that article.  But if you have an idea on how to do the implied concepts here, then you’re good to go. If you have been following my advices lately, this might feel cliché to you, but you can’t blame me if I say “Always plan what you have to do!”, right? There you go, another useful and tad overused advice. Just to give you an overview, this article you are about to spend some time on will teach you basically how to: 1) add 2-dimensional elements to photos or just any other image you wish to, 2) apply effects to better enhance the composition, 3) plan out your scenes well However, this guide doesn’t teach you how to pick the right color combination nor does it help you how to shoot great photographs, but hopefully though, at the end of your reading, you’ll soon be able to apply the concepts with no hassle and get comfortable with it each time you do. Some of you might be a bit daunted by the title alone of this article, especially those of you most inclined with specialized compositing software, but as much as I would want to make use of those applications, I’m much more comfortable exploring what GIMP is capable of, not only as a simple drawing application but as a minor compositing app as well.  The concepts that I present here though are just basic representations of what compositing actually is.  And in this context, we’ll only be focusing on still images as reference and output all throughout this article.  If you wanted however to do compositing on series of images, animation, or movie, I highly suggest GIMP’s 3D partner – Blender. Ok, promotion set aside, let’s head back to the topic at hand. To give you an idea (because I believe [and I’m positive you do too] that pictures speak louder than words), here’s what we should be having by the end of this article, probably not exactly matching it but fairly close enough and I’ll try my best to be as guiding as possible. So let’s hop on! Heart and Sphere Composited with GIMP Compose, Compose, Compose! Yup, you read it thrice, I did too, don’t worry.  So what’s the fuss about composing anyway? The answer is pretty straightforward, though. Just like how a song is written through a composition, a photo/image is almost the same thing.  Without the proper composition, your image would never give life.  By composition, I mean a proper mix of colors, framing, lighting, etc.  This is one of the hardest obstacles any artist or photographer might face.  It will either ruin a majestic idea or it will turn your doodle into a wonderful creation you could almost hear the melody of your lines rhythm through your senses (wow, that was almost a mouthful!). Whichever tool you’re comfortable using, it really doesn’t differ a lot as compared to how you could easily interpret your ideas into something much more fruitful than worrying how to work your way around. That’s probably one reason I stuck into using GIMP, not only am I confident it can deliver anything I could 2-dimensionally think of but more importantly I am comfortable using it, which is a very important thing regarding design in my opinion. Just like how I wrote this article, composition comes into play (or you might already have doubted me already?).  Without the drafts and planning I made, I don’t believe I could even finish writing a paragraph of this one. To start off the process, we’ll use one photograph I shot just for this article (in an attempt to recreate the first image I showed you). Or if you don’t want to follow this article thoroughly, you can grab a sample photo from Google Images or from Stock Exchange (www.sxc.hu), just be sure to credit the owner though or whatever conditions or licenses the image has. Photo to work on Photo Enhancement Honestly, the photo we have is already decent enough to work with, but let’s just try making it better so we won’t have to go and adjust it later on. First, let’s open our image and do some primary color correction to it, just in case you’re the type who thinks “something has got to be better, always”.  Go ahead and fire up our tool of choice (GIMP in this case) and open the image (as you can see below). Opening the image in GIMP   With our photo active in our canvas and the layer it is on (which is the only layer that you see in the Layer Window by default), right click on the image, select Color, then choose Levels. Adjusting the image’s color levels is one good way to fix some color cast problems and to edit the color range of your colors non-destructively (extreme cases excluded), another great tool is using the Curves Tool to manipulate your image the same way that you do with Levels. But again, for the sake of this tutorial, we’ll use the levels tool since it’s much easier and faster to edit. You can see a screenshot below of the Levels Tool that we’ll be using in awhile. Levels Tool One nifty tool we can use under our Levels Tool is the Auto function which (you guessed it right again!), automates the color adjustment on our image based on the histogram reading and graph analysis of GIMP. Oftentimes, it makes the task easier for you but it might also ruin your image.  Nothing beats your visual judgment still so if you’re not contented with what the Auto Leveling gives you, go on and move the sliders that you see in the window.  Normally, I only adjust the Value data of the image to correct it’s overall brightness and contrast without altering the overall color mood of the photo.  But if in case you weren’t lucky enough to set your color balance settings on your camera the moment you shot the photo or if you felt the image you’re seeing infront of you is color casted too much, you can freely choose the other color channels (Red, Green, and Blue respectively) from the drop-down menu. You can see a screenshot below on how I adjusted the photo we currently loaded. Value Level Adjustment   RGB Color Level Choices That’s basically all that we need to do to enhance our photo (or you could go ahead and repeat the process a few more times to get the appropriate feel you wanted). If you wanted a safer way of editing (just in case you might run out of undo steps), duplicate your base layer that holds your image and work on the duplicate layer instead of the original one, then you can just switch the visibility on and off to see the changes you’ve made so far.
Read more
  • 0
  • 0
  • 5663

article-image-windows-azure-service-bus-key-features
Packt
06 Dec 2012
13 min read
Save for later

Windows Azure Service Bus: Key Features

Packt
06 Dec 2012
13 min read
(For more resources related to this topic, see here.) Service Bus The Windows Azure Service Bus provides a hosted, secure, and widely available infrastructure for widespread communication, large-scale event distribution, naming, and service publishing. Service Bus provides connectivity options for Windows Communication Foundation (WCF) and other service endpoints, including REST endpoints, that would otherwise be difficult or impossible to reach. Endpoints can be located behind Network Address Translation (NAT) boundaries, or bound to frequently changing, dynamically assigned IP addresses, or both. Getting started To get started and use the features of Services Bus, you need to make sure you have the Windows Azure SDK installed. Queues Queues in the AppFabric feature (different from Table Storage queues) offer a FIFO message delivery capability. This can be an outcome for those applications that expect messages in a certain order. Just like with ordinary Azure Queues, Service Bus Queues enable the decoupling of your application components and can still function, even if some parts of the application are offline. Some differences between the two types of queues are (for example) that the Service Bus Queues can hold larger messages and can be used in conjunction with Access Control Service. Working with queues To create a queue, go to the Windows Azure portal and select the Service Bus, Access Control & Caching tab. Next, select Service Bus, select the namespace, and click on New Queue. The following screen will appear. If you did not set up a namespace earlier you need to create a namespace before you can create a queue: There are some properties that can be configured during the setup process of a queue. Obviously, the name uniquely identifies the queue in the namespace. Default Message Time To Live configures messages having this default TTL. This can also be set in code and is a TimeSpan value. Duplicate Detection History Time Window implicates how long the message ID (unique) of the received messages will be retained to check for duplicate messages. This property will be ignored if the Required Duplicate Detection option is not set. Keep in mind that a long detection history results in the persistency of message IDs during that period. If you process many messages, the queue size will grow and so does your bill. When a message expires or when the limit of the queue size is reached, it will be deadlettered . This means that they will end up in a different queue named $DeadLetterQueue. Imagine a scenario where a lot of traffic in your queue results in messages in the dead letter queue. Your application should be robust and process these messages as well. The lock duration property defines the duration of the lock when the PeekLock() method is called. The PeekLock() method hides a specific message from other consumers/processors until the lock duration expires. Typically, this value needs to be sufficient to process and delete the message. A sample scenario Remember the differences between the two queue types that Windows Azure offers, where the Service Bus queues are able to guarantee first-in first-out and to support transactions. The scenario is when a user posts a geotopic on the canvas containing text and also uploads a video by using the parallel upload functionality. What should happen next is for the WCF service CreateGeotopic() to post a message in the queue to enter the geotopic, but when the file finishes uploading, there is also a message sent to the queue. These two together should be in a single transaction. Geotopia.Processor processes this message but only if the media file is finished uploading. In this example, you can see how a transaction is handled and how a message can be abandoned and made available on the queue again. If the geotopic is validated as a whole (file is uploaded properly), the worker role will reroute the message to a designated audit trail queue to keep track of actions made by the system and also send to a topic (see next section) dedicated to keeping messages that need to be pushed to possible mobile devices. The messages in this topic will again be processed by a worker role. The reason for choosing a separate worker role is that it creates a role, a loosely-coupled solution, and possible to be fine-grained by only scaling the back-end worker role. See the following diagram for an overview of this scenario: In the previous section, we already created a queue named geotopicaqueue. In order to work with queues, you need the service identity (in this case we use a service identity with a symmetric issuer and the key credentials) of the service namespace. Preparing the project In order to make use of the Service Bus capabilities, you need to add a reference to Microsoft.ServiceBus.dll, located in <drive>:Program FilesMicrosoft SDKsWindows Azure.NET SDK2012-06ref. Next, add the following using statements to your file: using Microsoft.ServiceBus; using Microsoft.ServiceBus.Messaging; Your project is now ready to make use of Service Bus queues. In the configuration settings of the web role project hosting the WCF services, add a new configuration setting named ServiceBusQueue with the following value: "Endpoint=sb://<servicenamespace>.servicebus.windows. net/;SharedSecretIssuer=<issuerName>;SharedSecretValue=<yoursecret>" The properties of the queue you configured in the Windows Azure portal can also be set programmatically. Sending messages Messages that are sent to a Service Bus queue are instances of BrokeredMessage. This class contains standard properties such as TimeToLive and MessageId. An important property is Properties, which is of type IDictionary<string, object>, where you can add additional data. The body of the message can be set in the constructor of BrokerMessage, where the parameter must be of a type decorated with the [Serializable] attribute. The following code snippet shows how to send a message of type BrokerMessage: MessagingFactory factory = MessagingFactory.CreateFromConnectionString (connectionString); MessageSender sender = factory.CreateMessageSender("geotopiaqueue"); sender.Send(new BrokeredMessage( new Geotopic { id = id, subject = subject, text = text, PostToFacebook = PostToFacebook, accessToken = accessToken, MediaFile = MediaFile //Uri of uploaded mediafile })); As the scenario depicts a situation where two messages are expected to be sent in a certain order and to be treated as a single transaction, we need to add some more logic to the code snippet. Right before this message is sent, the media file is uploaded by using the BlobUtil class. Consider sending the media file together with BrokeredMessage if it is small enough. This might be a long-running operation, depending on the size of the file. The asynchronous upload process returns Uri, which is passed to BrokeredMessage. The situation is: A multimedia file is uploaded from the client to Windows Azure Blob storage using a parallel upload (or passed on in the message). A Parallel upload is breaking up the media file in several chunks and uploading them separately by using multithreading. A message is sent to geotopiaqueue, and Geotopia.Processor processes the messages in the queues in a single transaction. Receiving messages On the other side of the Service Bus queue resides our worker role, Geotopia. Processor, which performs the following tasks: It grabs the messages from the queue Sends the message straight to a table in Windows Azure Storage for auditing purposes Creates a geotopic that can be subscribed to The following code snippet shows how to perform these three tasks: MessagingFactory factory = MessagingFactory.CreateFromConnectionString (connectionString); MessageReceiver receiver = factory.CreateMessageReceiver("geotopiaqueue "); BrokeredMessage receivedMessage = receiver.Receive(); try { ProcessMessage(receivedMessage); receivedMessage.Complete(); } catch (Exception e) { receivedMessage.Abandon(); } Cross-domain communication We created a new web role in our Geotopia solution, hosting the WCF services we want to expose. As the client is a Silverlight one (and runs in the browser), we face cross-domain communication. To protect against security vulnerabilities and to prevent cross-site requests from a Silverlight client to some services (without the notice of the user), Silverlight by default allows only site-of-origin communication. A possible exploitation of a web application is cross-site forgery, exploits that can occur when cross-domain communication is allowed; for example, a Silverlight application sending commands to some service running on the Internet somewhere. As we want the Geotopia Silverlight client to access the WCF service running in another domain, we need to explicitly allow cross-domain operations. This can be achieved by adding a file named clientaccesspolicy.xml at the root of the domain where the WCF service is hosted and allowing this cross-domain access. Another option is to add a crossdomain.xml file at the root where the service is hosted. Please go to http://msdn.microsoft.com/en-us/library/cc197955(v=vs.95).aspx to find more details on the cross-domain communication issues. Comparison The following table shows the similarities and differences between Windows Azure and Service Bus queues: Criteria Windows Azure queue Service Bus queue Ordering guarantee No, but based on best effort first-in, first out First-in, first-out Delivery guarantee At least once At most once; use the PeekLock() method to ensure that no messages are missed. PeekLock() together with the Complete() method enable a two-stage receive operation. Transaction support No Yes, by using TransactionScope Receive Mode Peek & Lease Peek & Lock Receive & Delete Lease/Lock duration Between 30 seconds and 7 days Between 60 seconds and 5 minutes Lease/Lock granularity Message level Queue level Batched Receive Yes, by using GetMessages(count) Yes, by using the prefetch property or the use of transactions Scheduled Delivery Yes Yes Automatic dead lettering No Yes In-place update Yes No Duplicate detection No Yes WCF integration No Yes, through WCF bindings WF integration Not standard; needs a customized activity Yes, out-of-the-box activities Message Size Maximum 64 KB Maximum 256 KB Maximum queue size 100 TB, the limits of a storage account 1, 2, 3, 4, or 5 GB; configurable Message TTL Maximum 7 days Unlimited Number of queues Unlimited 10,000 per service namespace Mgmt protocol REST over HTTP(S) REST over HTTP(S) Runtime protocol REST over HTTP(S) REST over HTTP(S) Queue naming rules Maximum of 63 characters Maximum of 260 characters Queue length function Yes, value is approximate Yes, exact value Throughput Maximum of 2,000 messages/second Maximum of 2,000 messages/second Authentication Symmetric key ACS claims Role-based access control No Yes through ACS roles Identity provider federation No Yes Costs $0.01 per 10,000 transactions $ 0.01 per 10,000 transactions Billable operations Every call that touches "storage"' Only Send and Receive operations Storage costs $0.14 per GB per month None ACS transaction costs None, since ACS is not supported $1.99 per 100,000 token requests Background information There are some additional characteristics of Service Bus queues that need your attention: In order to guarantee the FIFO mechanism, you need to use messaging sessions. Using Receive & Delete in Service Bus queues reduces transaction costs, since it is counted as one. The maximum size of a Base64-encoded message on the Window Azure queue is 48 KB and for standard encoding it is 64 KB. Sending messages to a Service Bus queue that has reached its limit will throw an exception that needs to be caught. When the throughput has reached its limit, the HTTP 503 error response is returned from the Windows Azure queue service. Implement retrying logic to tackle this issue. Throttled requests (thus being rejected) are not billable. ACS transactions are based on instances of the message factory class. The received token will expire after 20 minutes, meaning that you will only need three tokens per hour of execution. Topics and subscriptions Topics and subscriptions can be useful in a scenario where (instead of a single consumer, in the case of queues) multiple consumers are part of the pattern. Imagine in our scenario where users want to be subscribed to topics posted by friends. In such a scenario, a subscription is created on a topic and the worker role processes it; for example, mobile clients can be push notified by the worker role. Sending messages to a topic works in a similar way as sending messages to a Service Bus queue. Preparing the project In the Windows Azure portal, go to the Service Bus, Access Control & Caching tab. Select Topics and create a new topic, as shown in the following screenshot: Next, click on OK and a new topic is created for you. The next thing you need to do is to create a subscription on this topic. To do this, select New Subscription and create a new subscription, as shown in the following screenshot: Using filters Topics and subscriptions, by default, it is a push/subscribe mechanism where messages are made available to registered subscriptions. To actively influence the subscription (and subscribe only to those messages that are of your interest), you can create subscription filters. SqlFilter can be passed as a parameter to the CreateSubscription method of the NamespaceManager class. SqlFilter operates on the properties of the messages so we need to extend the method. In our scenario, we are only interested in messages that are concerning a certain subject. The way to achieve this is shown in the following code snippet: BrokeredMessage message = new BrokeredMessage(new Geotopic { id = id, subject = subject, text = text, PostToFacebook = PostToFacebook, accessToken = accessToken, mediaFile = fileContent }); //used for topics & subscriptions message.Properties["subject"] = subject; The preceding piece of code extends BrokeredMessage with a subject property that can be used in SqlFilter. A filter can only be applied in code on the subscription and not in the Windows Azure portal. This is fine, because in Geotopia, users must be able to subscribe to interesting topics, and for every topic that does not exist yet, a new subscription is made and processed by the worker role, the processor. The worker role contains the following code snippet in one of its threads: Uri uri = ServiceBusEnvironment.CreateServiceUri ("sb", "<yournamespace>", string.Empty); string name = "owner"; string key = "<yourkey>"; //get some credentials TokenProvider tokenProvider = TokenProvider.CreateSharedSecretTokenProvider(name, key); // Create namespace client NamespaceManager namespaceClient = new NamespaceManager(ServiceBusEnvironment.CreateServiceUri ("sb", "geotopiaservicebus", string.Empty), tokenProvider); MessagingFactory factory = MessagingFactory.Create(uri, tokenProvider); BrokeredMessage message = new BrokeredMessage(); message.Properties["subject"] = "interestingsubject"; MessageSender sender = factory.CreateMessageSender("dataqueue"); sender.Send(message); //message is send to topic SubscriptionDescription subDesc = namespaceClient.CreateSubscription("geotopiatopic", "SubscriptionOnMe", new SqlFilter("subject='interestingsubject'")); //the processing loop while(true) { MessageReceiver receiver = factory.CreateMessageReceiver ("geotopiatopic/subscriptions/SubscriptionOnMe"); //it now only gets messages containing the property 'subject' //with the value 'interestingsubject' BrokeredMessage receivedMessage = receiver.Receive(); try { ProcessMessage(receivedMessage); receivedMessage.Complete(); } catch (Exception e) { receivedMessage.Abandon(); } } Windows Azure Caching Windows Azure offers caching capabilities out of the box. Caching is fast, because it is built as an in-memory (fast), distributed (running on different servers) technology. Windows Azure Caching offers two types of cache: Caching deployed on a role Shared caching When you decide to host caching on your Windows Azure roles, you need to pick from two deployment alternatives. The first is dedicated caching, where a worker role is fully dedicated to run as a caching store and its memory is used for caching. The second option is to create a co-located topology, meaning that a certain percentage of available memory in your roles is assigned and reserved to be used for in-memory caching purposes. Keep in mind that the second option is the most costeffective one, as you don't have a role running just for its memory. Shared caching is the central caching repository managed by the platform which is accessible for your hosted services. You need to register the shared caching mechanism on the portal in the Service Bus, Access Control & Caching section of the portal. You need to configure a namespace and the size of the cache (remember, there is money involved). This caching facility is a shared one and runs inside a multitenant environment.
Read more
  • 0
  • 0
  • 5661

article-image-advanced-system-management
Packt
25 Oct 2013
11 min read
Save for later

Advanced System Management

Packt
25 Oct 2013
11 min read
(For more resources related to this topic, see here.) Beyond backups Of course, backups are not the only issue with managing multiple, remote systems. In particular, managing such multiple configurations using a centralized application is often desirable. Configuration management One of the issues frequently faced by administrators is that of having multiple, remote systems all with similar software for the most part, but with minor differences in what is installed or running. Debian provides several packages that can help manage such an environment in a unified manner. Two of the more popular packages, both available in Debian, are FAI and Puppet. While we don't have the space to go into details, both applications are described briefly here. Fully Automated Installation Fully Automated Installation (FAI) focuses on managing Linux installations, and is developed using Debian, although it works with many different distributions, not just Debian. FAI uses a class concept for categorizing similar systems, and provides a good deal of flexibility and customization via hooks. FAI provides for unattended, automatic installation as well as tools for monitoring and updating groups of systems. FAI is frequently used for creating and maintaining clusters. More information is available at http://fai-project.org/. Puppet Probably the best known application for distributed management is Puppet, developed by Puppet Labs. Unlike FAI, only the Open Source edition is free, the Enterprise edition, which has many additional features, is not. Puppet does include support for environments other than Linux. The desired configuration is described in a custom, high-level definition language, and distributed to systems with installed clients. Unlike FAI, Puppet does not provide its own bare metal remote installation method, but does use existing methods (such as kickstart) to provide this function. A number of companies that make heavy use of distributed and clustered systems use Puppet to manage their environments. More information is available at http://puppetlabs.com/. Other packages There are other packages that can be used to manage a distributed environment, such as Chef and BCFG2. While simpler than Puppet or FAI, they support similar functions and have been used in some distributed and clustered environments. The use of FAI, Puppet, and others in cluster management warrants a brief look at clustering next, and what packages in Debian support clustering. Clusters A cluster is a group of systems that work together in such a way that the whole functions as a single unit. Such clusters can be loosely coupled or tightly coupled. A loosely coupled environment, each system is complete in itself, and can handle all of the tasks any of the other systems can handle. The environment provides mechanisms for redundancy, load sharing, and fail-over between systems, and is often called a High Availability (HA) cluster. In a tightly coupled environment, the systems involved are highly dependent on one another, often sharing memory and disk storage, and all work on the same task together. The environment provides mechanisms for data sharing, avoiding storage conflicts, keeping the systems in synchronization, and splitting up tasks appropriately. This design is often used in super-computing environments. Clustering is an advanced technique that involves more than just installing and configuring software. It also involves hardware integration, and systems and network design, and implementation. Along with the URLs mentioned below, a good text on the subject is Building Clustered Linux Systems, by Robert W. Lucke, Prentice Hall. Here we will only touch the very basics, along with what tools Debian provides. Let's take a brief look at each environment, and some of the tools used to create them. High Availability clusters Two primary functions are required to implement a high availability cluster: A way to handle load balancing and individual host fail-over. A way to synchronize storage so that all servers provide the same view of the data they serve. Debian includes meta packages that bring together software from the Linux High Availability project, including cluster-agents and resource-agents, two of the higher-level meta packages. These packages install various agents that are useful in coordinating and managing load balancing and fail-over. In some cases, a master server is designated to distribute the processing load among other servers. Data synchronization is handled by using shared storage and any of the filesystems that provide for multiple accesses and shared files, such as NFS or AFS. High Availability clusters generally use standard software, along with software that is readily available to manage the dynamics of such environments. Beowulf clusters In addition to the considerations for High Availability clusters, more tightly coupled environments such as Beowulf clusters also require an infrastructure to manage and distribute computing tasks. There are several web pages devoted to creating a Beowulf cluster using Debian as well as packages that aid in creating such a cluster. One such page is https://wiki.debian.org/StartaBeowulf, a Debian Wiki page on Beowulf basics. The manual for FAI also has a section on creating a Beowulf cluster. Books are available as well. Debian provides several packages that are helpful in building such a cluster, such as the OpenMPI libraries for message passing, and various utilities that run commands on multiple systems, such as those in the kadif package. There are even projects that have released scripts and live CDs that allow you to set up a cluster quickly (one such project is the PelicanHPC project, developed for Debian Lenny, hosted at http://www.pelicanhpc.org/. This type of cluster is not something that you can set up and go. Beowulf and other tightly coupled clusters are intended for highly parallel computing, and the programs that do the actual computing must be designed specifically for such an environment. That said, some packages for specific parallel computations do exist in Debian, such as nwchem, which provides several applications for computational chemistry that take advantage of parallelism. Common tools Some common components of clusters have already been mentioned, such as the OpenMPI libraries. Aside from the meta-packages already mentioned, the redhat-cluster suite of tools is available in Debian, as well as many useful libraries, scheduling tools, and failover tools such as booth. All of these can be found using apt-cache or Synaptic by searching for "cluster". Webmin Many administrators will never have to administer a cluster, and many won't be responsible for a large number of systems requiring central backup solutions. However, even administering a single system using command line tools and text editors can be a chore. Even clusters sometimes require administrative tasks on individual systems. Fortunately, there is an application that can ease many administrative tasks, is easy to use, and can handle many aspects of Linux administration. It is called Webmin. Up until Debian Sarge, Webmin was a part of Debian distributions. However, the Debian developer in charge of packaging it had difficulty keeping up with the frequent releases, and it was eventually dropped from Debian. However, the upstream Webmin developers maintain current packages that install cleanly. Some users have reported issues because Webmin does not always handle configuration files exactly as Debian intends, but it most certainly attempts to handle them in a compatible manner, and while some users have experienced problems with upgrades, many administrators are quite happy with Webmin. As long as you are willing to deal with conflicts during upgrades, or restrict use of modules that have major configuration impacts, you will find Webmin quite useful. Installing Webmin Webmin may be installed by adding the following lines to your apt sources file: deb http://download.webmin.com/download/repository sarge contrib deb http://webmin.mirror.somersettechsolutions.co.uk/repository sarge contrib Usually, this is added to a separate webmin.list file in /etc/apt/sources.list.d. The use of 'sarge' for the release name in the configuration is not a mistake. Since Webmin was dropped after the Sarge release (Debian 3.1), the developers update the repository as it is and haven't bothered changing it to keep up with the Debian code names. However, the versions available in the repository are compatible with any Debian release since 3.1. After updating your cache file, Webmin can be installed and maintained using apt-get, aptitude, or Synaptic. Also, if you request a Webmin upgrade from within Webmin itself on a Debian system, it will use the proper Debian package to upgrade. Using Webmin Webmin runs in the background, and provides an HTTP or HTTPS server on localhost port 10,000. You can use any web browser to connect to http://localhost:10000/ to access Webmin. Upon first installation, only the root user or those in a group allowed to use sudo to access the root account, may log in but Webmin users can be managed separately or in conjunction with local users. Webmin provides extensive and easy to understand menus and icons for various configuration tasks. Webmin is also highly modular and extensible, and an extensive list of standard modules is included with the base package. It is not possible to cover Webmin as fully here as it deserves, but a short list of some of its capabilities includes: Configuration of Webmin itself (the server, users, modules, and security) Local system user and password management Filesystem management Bootup and service management CRON job management Software updates Basic filesystem backups Authentication and security configuration APACHE, DNS, SSH, and FTP (if you're using ProFTP) configuration User mail management Qmail or sendmail configuration Network and Firewall configuration and management Bandwidth monitoring Printer management There are even modules that apply to clusters. Also, Webmin can search and allow access to other Webmin servers on the local network or you can define remote servers manually. This allows a central Webmin server, installed on a particular system, to be the gateway to all of the other servers in your environment, essentially providing a single point of access to manage all Webmin enabled servers. Webmin and Debian Webmin understands the configuration file layout of many distributions. The main problem is when a particular module does not handle certain types of configuration in the way the Debian developers prefer, which can make package upgrades somewhat difficult. This can be handled in a couple of ways. Most modules provide a means to edit configuration files directly, so if you have read the Debian documentation you can modify the configuration appropriately to use Debian specific configuration techniques. Or, you may choose to allow Webmin to modify files as it sees fit, and handle any conflicts manually when you upgrade the software involved. Finally, you can avoid those modules involved with specific software that are more likely to cause problems. One such module is Apache, which doesn't use links from sites-enabled to sites-available. Rather, it configures directly in the sites-enabled directory. Some administrators create the configuration in Webmin, and then move and link the files. Others prefer to manually configure Apache outside of Webmin. Webmin modules are constantly changing, and some actually recognize the Debian file layouts well, so it is not possible to give a comprehensive list of modules to avoid at this time. Best practice when using Webmin is to read the documentation and check the configuration files for specific software prior to using Webmin. Then, after configuring with Webmin, check the files again to determine whether changes may be required to work within the particular package's Debian configuration framework. Based upon this, you can decide whether to continue to configure using Webmin or switch back to manual configuration of that particular software. Webmin security Security is always a concern when remote access to a system is involved. Webmin handles this by requiring authentication and providing for detailed access restrictions that provide a layer of control beyond the firewall. Webmin users can be defined separately, or certain local users can be designated. Access to the various modules in Webmin can be restricted to certain users or groups of users, and detailed logs of Webmin actions are kept. Usermin In addition to Webmin, there is a server called Usermin which may be installed from the same repository as Webmin. It allows individual users to perform a number of functions more easily, such as changing their password, accessing their files, read and manage their email, and managing some aspects of their user profile. It is also modular and has the same security features as Webmin. Summary Several powerful and flexible central backup solutions exist that help manage backups for multiple remote servers and sites. Debian provides packages that assist in building High Availability and Beowulf style multiprocessing clusters as well. And, whether you are involved in managing clusters or not, or even a single system, Webmin can ease an administrator's tasks. Resources for Article: Further resources on this subject: Customizing a Linux kernel [Article] Microsoft SharePoint 2010 Administration: Farm Governance [Article] Testing Workflows for Microsoft Dynamics AX 2009 Administration [Article]
Read more
  • 0
  • 0
  • 5659

article-image-introduction-object-oriented-programming-using-python-javascript-and-c
Packt
17 Feb 2016
5 min read
Save for later

Introduction to Object-Oriented Programming using Python, JavaScript, and C#

Packt
17 Feb 2016
5 min read
In this extract from Learning Object-Oriented Programming by Gaston Hillar, we will show you the benefits of thinking in an object-oriented way. From encapsulation and inheritance to polymorphism or object-overloading, we will hammer home the thought process that must be mastered in order to truly make the most of the object-oriented approach. In this article, we will see how to generate blueprints for objects and we will design classes which include the attributes or fields that provide data for each instance. We will explore the different object-oriented approaches in different languages, such as Python, JavaScript, and C#. Generating blueprints for objects Imagine that you want to draw and calculate the areas of four different rectangles. You will end up with four rectangles each with different widths, heights, and areas. Now imagine having a blueprint to simplify the process of drawing each different rectangle. In object-oriented programming, a class is a blueprint or a template definition from which the objects are created. Classes are models that define the state and behavior of an object. After defining a class that determines the state and behavior of a rectangle, we can use it to generate objects that represent the state and behavior of each real-world rectangle. Objects are also known as instances. For example, we can say each rectangle object is an instance of the rectangle class. The following image shows four rectangle instances, with their widths and heights specified: Rectangle #1, Rectangle #2, Rectangle #3, and Rectangle #4. We can use a rectangle class as a blueprint to generate the four different rectangle instances. It is very important to understand the difference between a class and the objects or instances generated through its usage. Object-oriented programming allows us to discover the blueprint we used to generate a specific object. Thus, we are able to infer that each object is an instance of the rectangle class. Recognizing attributes/fields Now, we need to design the classes to include the attributes that provide the required data to each instance. In other words, we have to make sure that each class has the necessary variables that encapsulate all the data required by the objects to perform all the tasks. Let's start with the Square class. It is necessary to know the length of the sides for each instance of this class - for each square object. Therefore, we need an encapsulated variable that allows each instance of this class to specify the value of the length of a side. The variables defined in a class to encapsulate data for each instance of the class are known as attributes or fields. Each instance has its own independent value for the attributes or fields defined in the class. The Square class defines a floating point attribute named LengthOfSide whose initial value is equal to 0 for any new instance of the class. After you create an instance of the Square class, it is possible to change the value of the LengthOfSide attribute. For example, imagine that you create two instances of the Square class. One of the instances is named square1, and the other is square2. The instance names allow you to access the encapsulated data for each object, and therefore, you can use them to change the values of the exposed attributes. Imagine that our object-oriented programming language uses a dot (.) to allow us to access the attributes of the instances. So, square1.LengthOfSide provides access to the length of side for the Square instance named square1, and square2.LengthOfSide does the same for the Square instance named square2. You can assign the value 10 to square1.LengthOfSide and 20 to square2.LengthOfSide. This way, each Square instance is going to have a different value for their LengthOfSide attribute. Now, let's move to the Rectangle class. We can define two floating-point attributes for this class: Width and Height. Their initial values are also going to be 0. Then, you can create two instances of the Rectangle class: rectangle1 and rectangle2. You can assign the value 10 to rectangle1.Width and 20 to rectangle1.Height. This way, rectangle1 represents a 10 x 20 rectangle. You can assign the value 30 to rectangle2.Width and 50 to rectangle2.Height to make the second Rectangle instance representing a 30 x 50 rectangle. Object-oriented approaches in Python, JavaScript, and C# Python, JavaScript, and C# support object-oriented programming, also known as OOP. However, each programming language takes a different approach. Both Python and C# support classes and inheritance. Therefore, you can use the different syntax provided by each of these programming languages to declare the Shape class and its four subclasses. Then, you can create instances of each of the subclasses and call the different methods. On the other hand, JavaScript uses an object-oriented model that doesn't use classes. This object-oriented model is known as prototype-based programming. However, don't worry. Everything you have learned so far in your simple object-oriented design journey can be coded in JavaScript. Instead of using inheritance to achieve behavior reuse, we can expand upon existing objects. Thus, we can say that objects serve as prototypes in JavaScript. Instead of focusing on classes, we work with instances and decorate them to emulate inheritance in class-based languages. The object-oriented model named prototype-based programing is also known with other names such as classless programming, instance-based programming, or prototype-oriented programming. There are other important differences between Python, JavaScript, and C#. They have a great impact on the way you can code object-oriented designs. Carry on reading Learning Object-Oriented Programming to learn different ways to code the same object-oriented design in three programming languages. So what are you waiting for? Take a look at what else the book offers now!
Read more
  • 0
  • 0
  • 5658
article-image-blocking-common-attacks-using-modsecurity-25-part-1
Packt
01 Dec 2009
11 min read
Save for later

Blocking Common Attacks using ModSecurity 2.5: Part 1

Packt
01 Dec 2009
11 min read
Web applications can be attacked from a number of different angles, which is what makes defending against them so difficult. Here are just a few examples of where things can go wrong to allow a vulnerability to be exploited: The web server process serving requests can be vulnerable to exploits. Even servers such as Apache, that have a good security track record, can still suffer from security problems - it's just a part of the game that has to be accepted. The web application itself is of course a major source of problems. Originally, HTML documents were meant to be just that - documents. Over time, and especially in the last few years, they have evolved to also include code, such as client-side JavaScript. This can lead to security problems. A parallel can be drawn to Microsoft Office, which in earlier versions was plagued by security problems in its macro programming language. This, too, was caused by documents and executable code being combined in the same file. Supporting modules, such as mod_php which is used to run PHP scripts, can be subject to their own security vulnerabilities. Backend database servers, and the way that the web application interacts with them, can be a source of problems ranging from disclosure of confidential information to loss of data. HTTP fingerprinting Only amateur attackers blindly try different exploits against a server without having any idea beforehand whether they will work or not. More sophisticated adversaries will map out your network and system to find out as much information as possible about the architecture of your network and what software is running on your machines. An attacker looking to break in via a web server will try to find one he knows he can exploit, and this is where a method known as HTTP fingerprinting comes into play. We are all familiar with fingerprinting in everyday life - the practice of taking a print of the unique pattern of a person's finger to be able to identify him or her - for purposes such as identifying a criminal or opening the access door to a biosafety laboratory. HTTP fingerprinting works in a similar manner by examining the unique characteristics of how a web server responds when probed and constructing a fingerprint from the gathered information. This fingerprint is then compared to a database of fingerprints for known web servers to determine what server name and version is running on the target system. More specifically, HTTP fingerprinting works by identifying subtle differences in the way web servers handle requests - a differently formatted error page here, a slightly unusual response header there - to build a unique profile of a server that allows its name and version number to be identified. Depending on which viewpoint you take, this can be useful to a network administrator to identify which web servers are running on a network (and which might be vulnerable to attack and need to be upgraded), or it can be useful to an attacker since it will allow him to pinpoint vulnerable servers. We will be focusing on two fingerprinting tools: httprint One of the original tools - the current version is 0.321 from 2005, so it hasn't been updated with new signatures in a while. Runs on Linux, Windows, Mac OS X, and FreeBSD. httprecon This is a newer tool which was first released in 2007. It is still in active development. Runs on Windows. Let's first run httprecon against a standard Apache 2.2 server: And now let's run httprint against the same server and see what happens: As we can see, both tools correctly guess that the server is running Apache. They get the minor version number wrong, but both tell us that the major version is Apache 2.x. Try it against your own server! You can download httprint at http://www.net-square.com/httprint/ and httprecon at http://www.computec.ch/projekte/httprecon/. Tip If you get the error message Fingerprinting Error: Host/URL not found when running httprint, then try specifying the IP address of the server instead of the hostname. The fact that both tools are able to identify the server should come as no surprise as this was a standard Apache server with no attempts made to disguise it. In the following sections, we will be looking at how fingerprinting tools distinguish different web servers and see if we are able to fool them into thinking the server is running a different brand of web server software. How HTTP fingerprinting works There are many ways a fingerprinting tool can deduce which type and version of web server is running on a system. Let's take a look at some of the most common ones. Server banner The server banner is the string returned by the server in the Server response header (for example: Apache/1.3.3 (Unix) (Red Hat/Linux)). This banner can be changed by using the ModSecurity directive SecServerSignature. Here is what to do to change the banner: # Change the server banner to MyServer 1.0ServerTokens FullSecServerSignature "MyServer 1.0" Response header The HTTP response header contains a number of fields that are shared by most web servers, such as Server, Date, Accept-Ranges, Content-Length, and Content-Type. The order in which these fields appear can give a clue as to which web server type and version is serving the response. There can also be other subtle differences - the Netscape Enterprise Server, for example, prints its headers as Last-modified and Accept-ranges, with a lowercase letter in the second word, whereas Apache and Internet Information Server print the same headers as Last-Modified and Accept-Ranges. HTTP protocol responses An other way to gain information on a web server is to issue a non-standard or unusual HTTP request and observe the response that is sent back by the server. Issuing an HTTP DELETE request The HTTP DELETE command is meant to be used to delete a document from a server. Of course, all servers require that a user is authenticated before this happens, so a DELETE command from an unauthorized user will result in an error message - the question is just which error message exactly, and what HTTP error number will the server be using for the response page? Here is a DELETE request issued to our Apache server: $ nc bytelayer.com 80DELETE / HTTP/1.0HTTP/1.1 405 Method Not AllowedDate: Mon, 27 Apr 2009 09:10:49 GMTServer: Apache/2.2.8 (Fedora) mod_jk/1.2.27 DAV/2Allow: GET,HEAD,POST,OPTIONS,TRACEContent-Length: 303Connection: closeContent-Type: text/html; charset=iso-8859-1<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>405 Method Not Allowed</title></head><body><h1>Method Not Allowed</h1><p>The requested method DELETE is not allowed for the URL /index.html.</p><hr><address>Apache/2.2.8 (Fedora) mod_jk/1.2.27 DAV/2 Server at www.bytelayer.com Port 80</address></body></html> As we can see, the server returned a 405 - Method Not Allowed error. The error message accompanying this response in the response body is given as The requested method DELETE is not allowed for the URL/index.html. Now compare this with the following response, obtained by issuing the same request to a server at www.iis.net: $ nc www.iis.net 80DELETE / HTTP/1.0HTTP/1.1 405 Method Not AllowedAllow: GET, HEAD, OPTIONS, TRACEContent-Type: text/htmlServer: Microsoft-IIS/7.0Set-Cookie: CSAnonymous=LmrCfhzHyQEkAAAANWY0NWY1NzgtMjE2NC00NDJjLWJlYzYtNTc4ODg0OWY5OGQz0; domain=iis.net; expires=Mon, 27-Apr-2009 09:42:35GMT; path=/; HttpOnlyX-Powered-By: ASP.NETDate: Mon, 27 Apr 2009 09:22:34 GMTConnection: closeContent-Length: 1293<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><html ><head><meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"/><title>405 - HTTP verb used to access this page is not allowed.</title><style type="text/css"><!--body{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica,sans-serif;background:#EEEEEE;}fieldset{padding:0 15px 10px 15px;}h1{font-size:2.4em;margin:0;color:#FFF;}h2{font-size:1.7em;margin:0;color:#CC0000;}h3{font-size:1.2em;margin:10px 0 0 0;color:#000000;}#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;fontfamily:"trebuchet MS", Verdana, sans-serif;color:#FFF;background-color:#555555;}#content{margin:0 0 0 2%;position:relative;}.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}--></style>< /head><body><div id="header"><h1>Server Error</h1></div><div id="content"><div class="content-container"><fieldset> <h2>405 - HTTP verb used to access this page is not allowed.</h2> <h3>The page you are looking for cannot be displayed because aninvalid method (HTTP verb) was used to attempt access.</h3> </fieldset></div></div></body></html> The site www.iis.net is Microsoft's official site for its web server platform Internet Information Services, and the Server response header indicates that it is indeed running IIS-7.0. (We have of course already seen that it is a trivial operation in most cases to fake this header, but given the fact that it's Microsoft's official IIS site we can be pretty sure that they are indeed running their own web server software.) The response generated from IIS carries the same HTTP error code, 405; however there are many subtle differences in the way the response is generated. Here are just a few: IIS uses spaces in between method names in the comma separated list for the Allow field, whereas Apache does not The response header field order differs - for example, Apache has the Date field first, whereas IIS starts out with the Allow field IIS uses the error message The page you are looking for cannot be displayed because an invalid method (HTTP verb) was used to attempt access in the response body Bad HTTP version numbers A similar experiment can be performed by specifying a non-existent HTTP protocol version number in a request. Here is what happens on the Apache server when the request GET / HTTP/5.0 is issued: $ nc bytelayer.com 80GET / HTTP/5.0HTTP/1.1 400 Bad RequestDate: Mon, 27 Apr 2009 09:36:10 GMTServer: Apache/2.2.8 (Fedora) mod_jk/1.2.27 DAV/2Content-Length: 295Connection: closeContent-Type: text/html; charset=iso-8859-1<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>400 Bad Request</title></head><body><h1>Bad Request</h1><p>Your browser sent a request that this server could notunderstand.<br /></p><hr><address>Apache/2.2.8 (Fedora) mod_jk/1.2.27 DAV/2 Server at www.bytelayer.com Port 80</address></body></html> There is no HTTP version 5.0, and there probably won't be for a long time, as the latest revision of the protocol carries version number 1.1. The Apache server responds with a 400 - Bad Request Error, and the accompanying error message in the response body is Your browser sent a request that this server could not understand. Now let's see what IIS does: $ nc www.iis.net 80GET / HTTP/5.0HTTP/1.1 400 Bad RequestContent-Type: text/html; charset=us-asciiServer: Microsoft-HTTPAPI/2.0Date: Mon, 27 Apr 2009 09:38:37 GMTConnection: closeContent-Length: 334<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd"><HTML><HEAD><TITLE>Bad Request</TITLE><META HTTP-EQUIV="Content-Type" Content="text/html; charset=usascii"></HEAD><BODY><h2>Bad Request - Invalid Hostname</h2><hr><p>HTTP Error 400. The request hostname is invalid.</p></BODY></HTML> We get the same error number, but the error message in the response body differs - this time it's HTTP Error 400. The request hostname is invalid. As HTTP 1.1 requires a Host header to be sent with requests, it is obvious that IIS assumes that any later protocol would also require this header to be sent, and the error message reflects this fact.
Read more
  • 0
  • 0
  • 5656

article-image-getting-and-running-cassandra
Packt
27 Feb 2015
20 min read
Save for later

Getting Up and Running with Cassandra

Packt
27 Feb 2015
20 min read
As an application developer, you have almost certainly worked with databases extensively. You must have built products using relational databases like MySQL and PostgreSQL, and perhaps experimented with a document store like MongoDB or a key-value database like Redis. While each of these tools has its strengths, you will now consider whether a distributed database like Cassandra might be the best choice for the task at hand. In this article by Mat Brown, author of the book Learning Apache Cassandra, we'll talk about the major reasons to choose Cassandra from among the many database options available to you. Having established that Cassandra is a great choice, we'll go through the nuts and bolts of getting a local Cassandra installation up and running. By the end of this article, you'll know: When and why Cassandra is a good choice for your application How to install Cassandra on your development machine How to interact with Cassandra using cqlsh How to create a keyspace (For more resources related to this topic, see here.) What Cassandra offers, and what it doesn't Cassandra is a fully distributed, masterless database, offering superior scalability and fault tolerance to traditional single master databases. Compared with other popular distributed databases like Riak, HBase, and Voldemort, Cassandra offers a uniquely robust and expressive interface for modeling and querying data. What follows is an overview of several desirable database capabilities, with accompanying discussions of what Cassandra has to offer in each category. Horizontal scalability Horizontal scalability refers to the ability to expand the storage and processing capacity of a database by adding more servers to a database cluster. A traditional single-master database's storage capacity is limited by the capacity of the server that hosts the master instance. If the data set outgrows this capacity, and a more powerful server isn't available, the data set must be sharded among multiple independent database instances that know nothing of each other. Your application bears responsibility for knowing to which instance a given piece of data belongs. Cassandra, on the other hand, is deployed as a cluster of instances that are all aware of each other. From the client application's standpoint, the cluster is a single entity; the application need not know, nor care, which machine a piece of data belongs to. Instead, data can be read or written to any instance in the cluster, referred to as a node; this node will forward the request to the instance where the data actually belongs. The result is that Cassandra deployments have an almost limitless capacity to store and process data; when additional capacity is required, more machines can simply be added to the cluster. When new machines join the cluster, Cassandra takes care of rebalancing the existing data so that each node in the expanded cluster has a roughly equal share. Cassandra is one of the several popular distributed databases inspired by the Dynamo architecture, originally published in a paper by Amazon. Other widely used implementations of Dynamo include Riak and Voldemort. You can read the original paper at http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf. High availability The simplest database deployments are run as a single instance on a single server. This sort of configuration is highly vulnerable to interruption: if the server is affected by a hardware failure or network connection outage, the application's ability to read and write data is completely lost until the server is restored. If the failure is catastrophic, the data on that server might be lost completely. A master-follower architecture improves this picture a bit. The master instance receives all write operations, and then these operations are replicated to follower instances. The application can read data from the master or any of the follower instances, so a single host becoming unavailable will not prevent the application from continuing to read data. A failure of the master, however, will still prevent the application from performing any write operations, so while this configuration provides high read availability, it doesn't completely provide high availability. Cassandra, on the other hand, has no single point of failure for reading or writing data. Each piece of data is replicated to multiple nodes, but none of these nodes holds the authoritative master copy. If a machine becomes unavailable, Cassandra will continue writing data to the other nodes that share data with that machine, and will queue the operations and update the failed node when it rejoins the cluster. This means in a typical configuration, two nodes must fail simultaneously for there to be any application-visible interruption in Cassandra's availability. How many copies?When you create a keyspace - Cassandra's version of a database - you specify how many copies of each piece of data should be stored; this is called the replication factor. A replication factor of 3 is a common and good choice for many use cases. Write optimization Traditional relational and document databases are optimized for read performance. Writing data to a relational database will typically involve making in-place updates to complicated data structures on disk, in order to maintain a data structure that can be read efficiently and flexibly. Updating these data structures is a very expensive operation from a standpoint of disk I/O, which is often the limiting factor for database performance. Since writes are more expensive than reads, you'll typically avoid any unnecessary updates to a relational database, even at the expense of extra read operations. Cassandra, on the other hand, is highly optimized for write throughput, and in fact never modifies data on disk; it only appends to existing files or creates new ones. This is much easier on disk I/O and means that Cassandra can provide astonishingly high write throughput. Since both writing data to Cassandra, and storing data in Cassandra, are inexpensive, denormalization carries little cost and is a good way to ensure that data can be efficiently read in various access scenarios. Because Cassandra is optimized for write volume, you shouldn't shy away from writing data to the database. In fact, it's most efficient to write without reading whenever possible, even if doing so might result in redundant updates. Just because Cassandra is optimized for writes doesn't make it bad at reads; in fact, a well-designed Cassandra database can handle very heavy read loads with no problem. Structured records The first three database features we looked at are commonly found in distributed data stores. However, databases like Riak and Voldemort are purely key-value stores; these databases have no knowledge of the internal structure of a record that's stored at a particular key. This means useful functions like updating only part of a record, reading only certain fields from a record, or retrieving records that contain a particular value in a given field are not possible. Relational databases like PostgreSQL, document stores like MongoDB, and, to a limited extent, newer key-value stores like Redis do have a concept of the internal structure of their records, and most application developers are accustomed to taking advantage of the possibilities this allows. None of these databases, however, offer the advantages of a masterless distributed architecture. In Cassandra, records are structured much in the same way as they are in a relational database—using tables, rows, and columns. Thus, applications using Cassandra can enjoy all the benefits of masterless distributed storage while also getting all the advanced data modeling and access features associated with structured records. Secondary indexes A secondary index, commonly referred to as an index in the context of a relational database, is a structure allowing efficient lookup of records by some attribute other than their primary key. This is a widely useful capability: for instance, when developing a blog application, you would want to be able to easily retrieve all of the posts written by a particular author. Cassandra supports secondary indexes; while Cassandra's version is not as versatile as indexes in a typical relational database, it's a powerful feature in the right circumstances. Efficient result ordering It's quite common to want to retrieve a record set ordered by a particular field; for instance, a photo sharing service will want to retrieve the most recent photographs in descending order of creation. Since sorting data on the fly is a fundamentally expensive operation, databases must keep information about record ordering persisted on disk in order to efficiently return results in order. In a relational database, this is one of the jobs of a secondary index. In Cassandra, secondary indexes can't be used for result ordering, but tables can be structured such that rows are always kept sorted by a given column or columns, called clustering columns. Sorting by arbitrary columns at read time is not possible, but the capacity to efficiently order records in any way, and to retrieve ranges of records based on this ordering, is an unusually powerful capability for a distributed database. Immediate consistency When we write a piece of data to a database, it is our hope that that data is immediately available to any other process that may wish to read it. From another point of view, when we read some data from a database, we would like to be guaranteed that the data we retrieve is the most recently updated version. This guarantee is called immediate consistency, and it's a property of most common single-master databases like MySQL and PostgreSQL. Distributed systems like Cassandra typically do not provide an immediate consistency guarantee. Instead, developers must be willing to accept eventual consistency, which means when data is updated, the system will reflect that update at some point in the future. Developers are willing to give up immediate consistency precisely because it is a direct tradeoff with high availability. In the case of Cassandra, that tradeoff is made explicit through tunable consistency. Each time you design a write or read path for data, you have the option of immediate consistency with less resilient availability, or eventual consistency with extremely resilient availability. Discretely writable collections While it's useful for records to be internally structured into discrete fields, a given property of a record isn't always a single value like a string or an integer. One simple way to handle fields that contain collections of values is to serialize them using a format like JSON, and then save the serialized collection into a text field. However, in order to update collections stored in this way, the serialized data must be read from the database, decoded, modified, and then written back to the database in its entirety. If two clients try to perform this kind of modification to the same record concurrently, one of the updates will be overwritten by the other. For this reason, many databases offer built-in collection structures that can be discretely updated: values can be added to, and removed from collections, without reading and rewriting the entire collection. Cassandra is no exception, offering list, set, and map collections, and supporting operations like "append the number 3 to the end of this list". Neither the client nor Cassandra itself needs to read the current state of the collection in order to update it, meaning collection updates are also blazingly efficient. Relational joins In real-world applications, different pieces of data relate to each other in a variety of ways. Relational databases allow us to perform queries that make these relationships explicit, for instance, to retrieve a set of events whose location is in the state of New York (this is assuming events and locations are different record types). Cassandra, however, is not a relational database, and does not support anything like joins. Instead, applications using Cassandra typically denormalize data and make clever use of clustering in order to perform the sorts of data access that would use a join in a relational database. For data sets that aren't already denormalized, applications can also perform client-side joins, which mimic the behavior of a relational database by performing multiple queries and joining the results at the application level. Client-side joins are less efficient than reading data that has been denormalized in advance, but offer more flexibility. MapReduce MapReduce is a technique for performing aggregate processing on large amounts of data in parallel; it's a particularly common technique in data analytics applications. Cassandra does not offer built-in MapReduce capabilities, but it can be integrated with Hadoop in order to perform MapReduce operations across Cassandra data sets, or Spark for real-time data analysis. The DataStax Enterprise product provides integration with both of these tools out-of-the-box. Comparing Cassandra to the alternatives Now that you've got an in-depth understanding of the feature set that Cassandra offers, it's time to figure out which features are most important to you, and which database is the best fit. The following table lists a handful of commonly used databases, and key features that they do or don't have: Feature Cassandra PostgreSQL MongoDB Redis Riak Structured records Yes Yes Yes Limited No Secondary indexes Yes Yes Yes No Yes Discretely writable collections Yes Yes Yes Yes No Relational joins No Yes No No No Built-in MapReduce No No Yes No Yes Fast result ordering Yes Yes Yes Yes No Immediate consistency Configurable at query level Yes Yes Yes Configurable at cluster level Transparent sharding Yes No Yes No Yes No single point of failure Yes No No No Yes High throughput writes Yes No No Yes Yes As you can see, Cassandra offers a unique combination of scalability, availability, and a rich set of features for modeling and accessing data. Installing Cassandra Now that you're acquainted with Cassandra's substantial powers, you're no doubt chomping at the bit to try it out. Happily, Cassandra is free, open source, and very easy to get running. Installing on Mac OS X First, we need to make sure that we have an up-to-date installation of the Java Runtime Environment. Open the Terminal application, and type the following into the command prompt: $ java -version You will see an output that looks something like the following: java version "1.8.0_25"Java(TM) SE Runtime Environment (build 1.8.0_25-b17)Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode) Pay particular attention to the java version listed: if it's lower than 1.7.0_25, you'll need to install a new version. If you have an older version of Java or if Java isn't installed at all, head to https://www.java.com/en/download/mac_download.jsp and follow the download instructions on the page. You'll need to set up your environment so that Cassandra knows where to find the latest version of Java. To do this, set up your JAVA_HOME environment variable to the install location, and your PATH to include the executable in your new Java installation as follows: $ export JAVA_HOME="/Library/Internet Plug- Ins/JavaAppletPlugin.plugin/Contents/Home"$ export PATH="$JAVA_HOME/bin":$PATH You should put these two lines at the bottom of your .bashrc file to ensure that things still work when you open a new terminal. The installation instructions given earlier assume that you're using the latest version of Mac OS X (at the time of writing this, 10.10 Yosemite). If you're running a different version of OS X, installing Java might require different steps. Check out https://www.java.com/en/download/faq/java_mac.xml for detailed installation information. Once you've got the right version of Java, you're ready to install Cassandra. It's very easy to install Cassandra using Homebrew; simply type the following: $ brew install cassandra$ pip install cassandra-driver cql$ cassandra Here's what we just did: Installed Cassandra using the Homebrew package manager Installed the CQL shell and its dependency, the Python Cassandra driver Started the Cassandra server Installing on Ubuntu First, we need to make sure that we have an up-to-date installation of the Java Runtime Environment. Open the Terminal application, and type the following into the command prompt: $ java -version You will see an output that looks similar to the following: java version "1.7.0_65"OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3- 0ubuntu0.14.04.1)OpenJDK 64-bit Server VM (build 24.65-b04, mixed mode) Pay particular attention to the java version listed: it should start with 1.7. If you have an older version of Java, or if Java isn't installed at all, you can install the correct version using the following command: $ sudo apt-get install openjdk-7-jre-headless Once you've got the right version of Java, you're ready to install Cassandra. First, you need to add Apache's Debian repositories to your sources list. Add the following lines to the file /etc/apt/sources.list: deb http://www.apache.org/dist/cassandra/debian 21x maindeb-src http://www.apache.org/dist/cassandra/debian 21x main In the Terminal application, type the following into the command prompt: $ gpg --keyserver pgp.mit.edu --recv-keys F758CE318D77295D$ gpg --export --armor F758CE318D77295D | sudo apt-key add -$ gpg --keyserver pgp.mit.edu --recv-keys 2B5C1B00$ gpg --export --armor 2B5C1B00 | sudo apt-key add -$ gpg --keyserver pgp.mit.edu --recv-keys 0353B12C$ gpg --export --armor 0353B12C | sudo apt-key add -$ sudo apt-get update$ sudo apt-get install cassandra$ cassandra Here's what we just did: Added the Apache repositories for Cassandra 2.1 to our sources list Added the public keys for the Apache repo to our system and updated our repository cache Installed Cassandra Started the Cassandra server Installing on Windows The easiest way to install Cassandra on Windows is to use the DataStax Community Edition. DataStax is a company that provides enterprise-level support for Cassandra; they also release Cassandra packages at both free and paid tiers. DataStax Community Edition is free, and does not differ from the Apache package in any meaningful way. DataStax offers a graphical installer for Cassandra on Windows, which is available for download at planetcassandra.org/cassandra. On this page, locate Windows Server 2008/Windows 7 or Later (32-Bit) from the Operating System menu (you might also want to look for 64-bit if you run a 64-bit version of Windows), and choose MSI Installer (2.x) from the version columns. Download and run the MSI file, and follow the instructions, accepting the defaults: Once the installer completes this task, you should have an installation of Cassandra running on your machine. Bootstrapping the project We will build an application called MyStatus, which allows users to post status updates for their friends to read. CQL – the Cassandra Query Language Since this is about Cassandra and not targeted to users of any particular programming language or application framework, we will focus entirely on the database interactions that MyStatus will require. Code examples will be in Cassandra Query Language (CQL). Specifically, we'll use version 3.1.1 of CQL, which is available in Cassandra 2.0.6 and later versions. As the name implies, CQL is heavily inspired by SQL; in fact, many CQL statements are equally valid SQL statements. However, CQL and SQL are not interchangeable. CQL lacks a grammar for relational features such as JOIN statements, which are not possible in Cassandra. Conversely, CQL is not a subset of SQL; constructs for retrieving the update time of a given column, or performing an update in a lightweight transaction, which are available in CQL, do not have an SQL equivalent. Interacting with Cassandra Most common programming languages have drivers for interacting with Cassandra. When selecting a driver, you should look for libraries that support the CQL binary protocol, which is the latest and most efficient way to communicate with Cassandra. The CQL binary protocol is a relatively new introduction; older versions of Cassandra used the Thrift protocol as a transport layer. Although Cassandra continues to support Thrift, avoid Thrift-based drivers, as they are less performant than the binary protocol. Here are CQL binary drivers available for some popular programming languages: Language Driver Available at Java DataStax Java Driver github.com/datastax/java-driver Python DataStax Python Driver github.com/datastax/python-driver Ruby DataStax Ruby Driver github.com/datastax/ruby-driver C++ DataStax C++ Driver github.com/datastax/cpp-driver C# DataStax C# Driver github.com/datastax/csharp-driver JavaScript (Node.js) node-cassandra-cql github.com/jorgebay/node-cassandra-cql PHP phpbinarycql github.com/rmcfrazier/phpbinarycql While you will likely use one of these drivers in your applications, you can simply use the cqlsh tool, which is a command-line interface for executing CQL queries and viewing the results. To start cqlsh on OS X or Linux, simply type cqlsh into your command line; you should see something like this: $ cqlshConnected to Test Cluster at localhost:9160.[cqlsh 4.1.1 | Cassandra 2.0.7 | CQL spec 3.1.1 | Thrift protocol 19.39.0]Use HELP for help.cqlsh> On Windows, you can start cqlsh by finding the Cassandra CQL Shell application in the DataStax Community Edition group in your applications. Once you open it, you should see the same output we just saw. Creating a keyspace A keyspace is a collection of related tables, equivalent to a database in a relational system. To create the keyspace for our MyStatus application, issue the following statement in the CQL shell:    CREATE KEYSPACE "my_status"   WITH REPLICATION = {      'class': 'SimpleStrategy', 'replication_factor': 1    }; Here we created a keyspace called my_status. When we create a keyspace, we have to specify replication options. Cassandra provides several strategies for managing replication of data; SimpleStrategy is the best strategy as long as your Cassandra deployment does not span multiple data centers. The replication_factor value tells Cassandra how many copies of each piece of data are to be kept in the cluster; since we are only running a single instance of Cassandra, there is no point in keeping more than one copy of the data. In a production deployment, you would certainly want a higher replication factor; 3 is a good place to start. A few things at this point are worth noting about CQL's syntax: It's syntactically very similar to SQL; as we further explore CQL, the impression of similarity will not diminish. Double quotes are used for identifiers such as keyspace, table, and column names. As in SQL, quoting identifier names is usually optional, unless the identifier is a keyword or contains a space or another character that will trip up the parser. Single quotes are used for string literals; the key-value structure we use for replication is a map literal, which is syntactically similar to an object literal in JSON. As in SQL, CQL statements in the CQL shell must terminate with a semicolon. Selecting a keyspace Once you've created a keyspace, you would want to use it. In order to do this, employ the USE command: USE "my_status"; This tells Cassandra that all future commands will implicitly refer to tables inside the my_status keyspace. If you close the CQL shell and reopen it, you'll need to reissue this command. Summary In this article, you explored the reasons to choose Cassandra from among the many databases available, and having determined that Cassandra is a great choice, you installed it on your development machine. You had your first taste of the Cassandra Query Language when you issued your first command via the CQL shell in order to create a keyspace. You're now poised to begin working with Cassandra in earnest. Resources for Article: Further resources on this subject: Getting Started with Apache Cassandra [article] Basic Concepts and Architecture of Cassandra [article] About Cassandra [article]
Read more
  • 0
  • 0
  • 5655

article-image-introduction-color-theory-and-lighting-basics-blender
Packt
14 Apr 2011
7 min read
Save for later

Introduction to Color Theory and Lighting Basics in Blender

Packt
14 Apr 2011
7 min read
Basic color theory To fully understand how light works, we need to have a basic understanding of what color is and how different colors interact with each other. The study of this phenomenon is known as color theory. What is color? When light comes in contact with an object, the object absorbs a certain amount of that light. The rest is reflected into the eye of the viewer in the form of color. The easiest way to visualize colors and their relations is in the form of a color wheel. Primary colors There are millions of colors, but there are only three colors that cannot be created through color mixing—red, yellow, and blue. These colors are known as primary colors, which are used to create the other colors on the color wheel through a process known as color mixing. Through color mixing, we get other "sets" of colors, including secondary and tertiary colors. Secondary colors Secondary colors are created when two primary colors are mixed together. For example, mixing red and blue makes purple, red and yellow make orange, and blue and yellow make green. Tertiary colors It's natural to assume that, because mixing two primary colors creates a secondary color, mixing two secondary colors would create a tertiary color. Surprisingly, this isn't the case. A tertiary color is, in fact, the result of mixing a primary and secondary color together. This gives us the remainder of the color wheel: Red-orange Orange-yellow Chartreuse Turquoise Indigo Violet-red Color relationships There are other relationships between colors that we should know about before we start using Blender. The first is complimentary colors. Complimentary colors are colors that are across from each other on the color wheel. For example, red and green are compliments. Complimentary colors are especially useful for creating contrast in an image, because mixing them together darkens the hue. In a computer program, mixing perfect compliments together will result in black, but mixing compliments in a more traditional medium such as oil pastels results in more of a dark brown hue. In both situations, though, the compliments are used to create a darker value. Be wary of using complimentary colors in computer graphics—if complimentary colors mix accidentally, it will result in black artifacts in images or animations. The other color relationship that we should be aware of is analogous colors. Analogous colors are colors found next to each other on the color wheel. For example, red, red-orange, and orange are analogous. Here's the kicker—red, orange, and yellow can also be analogous as well. A good rule to follow is as long as you don't span more than one primary color on the color wheel, they're most likely considered analogous colors. Color temperature Understanding color temperature is an essential step in understanding how lights work—at the very least, it helps us understand why certain lights emit the colors they do. No light source emits a constant light wavelength. Even the sun, although considered a constant light source, is filtered by the atmosphere to various degrees based on the time of the day, changing its perceived color. Color temperature is typically measured in degrees Kelvin (°K), and has a color range from a red to blue hue, like in the image below: Real world, real lights So how is color applicable beyond a two-dimensional color wheel? In the real world, our eyes perceive color because light from the sun—which contains all colors in the visible color spectrum—is reflected off of objects in our field of vision. As light hits an object, some wavelengths are absorbed, while the rest are reflected. Those reflected rays are what determine the color we perceive that particular object to be. Of course, the sun isn't the only source of light we have. There are many different types of natural and artificial light sources, each with its own unique properties. The most common types of light sources we may try to simulate in Blender include: Candlelight Incandescent light Florescent light Sunlight Skylight Candlelight Candlelight is a source of light as old as time. It has been used for thousands of years and is still used today in many cases. The color temperature of a candle's light is about 1500 K, giving it a warm red-orange hue. Candlelight also has a tendency to create really high contrast between lit areas and unlit areas in a room, which creates a very successful dramatic effect. Incandescent light bulbs When most people hear the term "light bulb", the incandescent light bulb immediately comes to mind. It's also known as a tungsten-halogen light bulb. It's your typical household light bulb, burning at approximately 2800 K-3200 K. This color temperature value still allows it to fall within the orange-yellow part of the spectrum, but it is noticeably brighter than the light of a candle. Florescent light bulbs Florescent lights are an alternative to incandescent. Also known as mercury vapor lights, fluorescents burn at a color temperature range of 3500 K-5900 K, allowing them to emit a color anywhere between a yellow and a white hue. They're commonly used when lighting a large area effectively, such as a warehouse, school hallway, or even a conference room. The sun and the sky Now let's take a look at some natural sources of light! The most obvious example is the sun. The sun burns at a color temperature of approximately 5500 K, giving it its bright white color. We rarely use pure white as a light's color in 3D though—it makes your scene look too artificial. Instead, we may choose to use a color that best suits the scene at hand. For example, if we are lighting a desert scene, we may choose to use a beige color to simulate light bouncing off the sand. But even so, this still doesn't produce an entirely realistic effect. This is where the next source of light comes in—the sky. The sky can produce an entire array of colors from deep purple to orange to bright blue. It produces a color temperature range of 6000 K-20,000 K. That's a huge range! We can really use this to our advantage in our 3D scenes—the color of the sky can have the final say in what the mood of your scene ends up being. Chromatic adaptation What is chromatic adaptation? We're all more familiar with this process than you may realize. As light changes, the color we perceive from the world around us changes. To accommodate for those changes, our eyes adjust what we see to something we're more familiar with (or what our brains would consider normal). When working in 3D you have to keep this in mind, because even though your 3D scene may be physically lit correctly, it may not look natural because the computer renders the final image objectively, without the chromatic adaptation that we, as humans, are used to. Take this image for example. In the top image, the second card from the left appears to be a stronger shade of pink than the corresponding card in the bottom picture. Believe it or not, they are the exact same color, but because of the red hue of the second photo, our brains change how we perceive that image.
Read more
  • 0
  • 0
  • 5651
article-image-administering-swarm-cluster
Packt
02 Dec 2016
12 min read
Save for later

Administering a Swarm Cluster

Packt
02 Dec 2016
12 min read
In this article by Fabrizio Soppelsa and Chanwit Kaewkasi, the author of Native Docker Clustering with Swarm we're now going to see how to administer a running Swarm cluster. The topics include scaling the cluster size (adding and removing nodes), updating the cluster and nodes information; handling the node status (promotion and demotion), troubleshooting, and graphical interfaces (UI). (For more resources related to this topic, see here.) Docker Swarm standalone In standalone mode, cluster operations must be done directly inside the container 'swarm'. We're not going to cover every option in detail. Swarm standalone is not deprecated yet, and is used around, the reason for which we're discussing it here, but it will be probably declared deprecated soon. It is obsoleted by the Swarm mode. The commands to administer a Docker Swarm standalone cluster are: Create (c): Typically, in production people use Consul or Etcd, so this command has no relevance for production List (l): This shows the list of cluster nodes, basing on a iteration through Consul or Etcd, that is, the Consul or Etcd must be passed as an argument Join (j): This joins a node on which the swarm container is running to the cluster. Here, still, a discovery mechanism must be passed at the command line Manage (m): This is the core of the Standalone mode. By managing a cluster, here it's meant how to change some cluster properties, such as Filters, Schedulers, external CA URLs, and timeouts. Docker Swarm mode: Scale a cluster size Manually adding nodes You can choose to create Docker hosts either way you prefer. If you plan to use Docker Machine, you're probably going to hit Machine's limits very soon, and you will need to be very patient while even listing machines, having to wait several seconds for Machine to get and print all the information on the whole. My favorite method is to use Machine with the generic driver, thus delegate to something else (that is, Ansible) the host provisioning (Operating System installation, network and security groups configurations, and so on), and later exploit Machine to install Docker the proper way: Manually configure the cloud environment (security groups, networks, and so on) Provision Ubuntu hosts with a third-party tool Run Machine with the generic driver on these hosts with the only goal to properly install Docker Then handle hosts with the tool in part 2, or even others. If you use Machine's generic driver, it will select the latest stable Docker binaries. While we were writing this article, in order to use Docker 1.12, we had to overcome this by passing Machine a special option to get the latest, unstable, version of Docker: docker-machine create -d DRIVER--engine-install-url https://test.docker.com mymachine For a production Swarm (mode), at the time you'll be reading this article, 1.12 will be already stable, so this trick will not be necessary anymore, unless you need to use some of the very latest Docker features. Managers The theory of HA suggests us that the number of managers must be odd, and equal or more than 3. This is to grant a quorum in high availability, that is the majority of nodes agree on what part of nodes are leading the operations. If there were two managers, and one goes down and comes back, it's possible that both will think to be the leaders. That causes a logical crash in the cluster organization called split brain. The more managers you have, the higher is the resistance ratio to failures. Refer to the following table: Number of managers Quorum (majority) Maximum possible failures 3 2 1 5 3 2 7 4 3 9 5 4 Also, in Swarm Mode, an overlay network is created automatically and associated as ingress traffic to the nodes. Its purpose is to be used with containers: You will want that your containers be associated to an internal overlay (VxLAN meshed) network to communicate with each other, rather than using public or other networks. Thus, Swarm creates this already for you, ready to use. We recommend, further, to geographically distribute managers. If an earthquake hits the datacenter where all managers are serving, the cluster would go down, wouldn't it? So, consider to place each manager or groups of managers into different physical locations. With the advent of cloud computing, that's really easy, you can spawn up each manager in a different AWS region, or even better have a manager running each on different providers on different regions, that is on AWS, on Digital Ocean, on Azure and also on private cloud, such as OpenStack. IMAGE OF A WORLD WITH SCATTERED MANAGERS IN CONTINENTS? Workers You can add an arbitrary number of workers. This is the elastic part of the Swarm. It's totally fine to have 5, 15, 200, or 2,300 running workers. This is the easiest part to handle: You can add and remove workers with no burdens, at any time, at any size. Scripted nodes addition The very easiest way to add nodes, if you plan to not go over 100 nodes total, is to use basic scripting. At the time of docker swarm init, just copy and paste the lines printed in the output. Then, create a certain bunch of workers: #!/bin/bash for i in `seq 0 9`; do docker-machine create -d amazonec2 --engine-install-url https://test.docker.com --amazonec2-instance-type "t2.large" swarm-worker-$i done After that, it will be only necessary to go through the list of machines, ssh into them, and join the nodes: #!/bin/bash SWARMWORKER="swarm-worker-" for machine in `docker-machine ls --format {{.Name}} | grep $SWARMWORKER`; do docker-machine ssh $machine sudo docker swarm join --token SWMTKN-1-5c3mlb7rqytm0nk795th0z0eocmcmt7i743ybsffad5e04yvxt-9m54q8xx8m1wa1g68im8srcme 172.31.10.250:2377 done This script runs through the machines, and for each with a name starting with swarm-worker-, will ssh into, and join the node to the existing Swarm, to the leader manager, here 172.31.10.250. Refer to https://github.com/swarm2k/swarm2k/tree/master/amazonec2 for some further details or to download these one liners. Belt Belt is another tool for massively provisioning Docker Engines. It is basically a SSH wrapper on steroids and it requires you to prepare provider-specific images as well as provisioning templates before go massively. In this section, we'll learn to do so: You can compile Belt yourself by getting its source from Github. # Set $GOPATH here go get https://github.com/chanwit/belt Currently, Belt supports the DigitalOcean driver. We can prepare our template for provisioning such as the following inside config.yml: digitalocean: image: "docker-1.12-rc4" region: nyc3 ssh_key_fingerprint: "your SSH ID" ssh_user: root Then we can create a hundred nodes basically with a couple of commands. First we create three boxes of 16 GB, namely, mg0, mg1, and mg2. $ belt create 16gb mg[0:2] NAME IPv4 MEMORY REGION IMAGE STATUS mg2 104.236.231.136 16384 nyc3 Ubuntu docker-1.12-rc4 active mg1 45.55.136.207 16384 nyc3 Ubuntu docker-1.12-rc4 active mg0 45.55.145.205 16384 nyc3 Ubuntu docker-1.12-rc4 active Then we can use the status command to wait for all nodes to become active: $ belt status --wait active=3 STATUS #NODES NAMES active 3 mg2, mg1, mg0 We'll do this again for 10 worker nodes. $ belt create 512mb node[1:10] $ belt status --wait active=13 STATUS #NODES NAMES active 3 node10, node9, node8, node7 Use Ansible You can alternatively use Ansible (as you like, and it's becoming very popular) to make things more repeatable. I (Fabrizio) created some Ansible modules to work directly with Machine and Swarm (Mode), compatible with Docker 1.12 (https://github.com/fsoppelsa/ansible-swarm). They require Ansible 2.2+, the very first version of Ansible compatible with binary modules. You will need to compile the modules (written in Go), and then pass them to the ansible-playbook -M parameter. git clone https://github.com/fsoppelsa/ansible-swarm cd ansible-swarm/library go build docker_machine_ go build docker_swarm_ cd .. There are some examples of plays in playbooks/. Ansible's plays syntax is that easy to understand, that's even superfluous to explain in detail. I used this play to join 10 workers to the Swarm2k experiment: --- name: Join the Swarm2k project hosts: localhost connection: local gather_facts: False #mg0 104.236.18.183 #mg1 104.236.78.154 #mg2 104.236.87.10 tasks: name: Load shell variables shell: > eval $(docker-machine env "{{ machine_name }}") echo $DOCKER_TLS_VERIFY && echo $DOCKER_HOST && echo $DOCKER_CERT_PATH && echo $DOCKER_MACHINE_NAME register: worker name: Set facts set_fact: whost: "{{ worker.stdout_lines[0] }}" wcert: "{{ worker.stdout_lines[1] }}" name: Join a worker to Swarm2k docker_swarm: role: "worker" operation: "join" join_url: ["tcp://104.236.78.154:2377"] secret: "d0cker_swarm_2k" docker_url: "{{ whost }}" tls_path: "{{ wcert }}" register: swarm_result name: Print final msg debug: msg="{{ swarm_result.msg }}" Basically, after loading some host facts from Machine, it invokes the docker_swarm module: The operation is join. The role of the new node is worker. The new node joins "tcp://104.236.78.154:2377", that was the leader manager at the time of joining. This argument takes an array of managers, such as might be ["tcp://104.236.78.154:2377", "104.236.18.183:2377", "tcp://104.236.87.10:2377"]. It passes the password (secret). It specifies some basic Engine connection facts. The module will connect to the dockerurl using the certificates at tlspath. After having docker_swarm.go compiled in library/, adding workers to the swarm is as easy as: #!/bin/bash SWARMWORKER="swarm-worker-" for machine in `docker-machine ls --format {{.Name}} | grep $SWARMWORKER`; do ansible-playbook -M library --extra-vars "{machine_name: $machine}" playbook.yaml don Cluster management We now operate a little bit with this example, made of 3 managers and 10 workers. You can reference the nodes by calling them either by their hostname (manager1) or by their ID (ctv03nq6cjmbkc4v1tc644fsi). The other columns in this list statement describe the properties of the cluster nodes. STATUS: This is about the physical reachability of the node. If the node is up, it's Ready, otherwise Down AVAILABILITY: This is the node availability. A node can be either Active (so participating to the cluster operations), Pause (in standby, suspended, not accepting tasks), or Drain (waiting to evacuate its tasks). MANAGER STATUS: This is about the current status of manager. If a node is not a manager, this field will be empty. If a node is a manager, this field can be either Reachable (one of the managers presents to guarantee high availability) or Leader (the host leading all operations). Nodes operations The docker node command comes with some possible options. Demotion and promotion Promotion is possible for worker nodes (transforming them into managers), while demotion is possible for manager nodes (transforming them into workers). Always keep in mind the table to guarantee high availability when managing the number of managers and workers (odd number, more than or equal to 3). Use the following syntax to promote worker0 and worker1 to managers: docker node promote worker0 docker node promote worker1 There is nothing magic behind the curtain. It is just that Swarm attempts to change the node role, with an on-the-fly instruction. Demote is the same (docker node demote worker1). But be careful to not demote the node you're working from, otherwise you'll get locked out. What happens if you try to demote a Leader manager? In this case, the RAFT algorithm will start an election and a new leader will be selected among the Active managers. Tagging nodes You must have noticed, in the preceding screenshot, that worker9 is in Drain availability. This means that the node is in the process of evacuating its tasks (if any), which will be rescheduled somewhere else on the cluster. You can change the availability of a node by updating its status using the docker node update command: The --availability option can take either active, pause, or drain. Here we just restored worker9 to the active state. Active: This means that the node is running and ready to accept tasks pause: This means that the node is running, but not accepting tasks drain: This means that the node is running and not accepting tasks, it is currently draining its tasks, that are getting rescheduled somewhere else Another powerful update argument is about labels. There are --label-add and --label-rm that respectively allow us to add labels to Swarm nodes. Docker Swarm labels do not affect the Engine labels. It's possible to specify labels when starting the Docker Engine (dockerd [...] --label "staging" --label "dev" [...]). But Swarm has no power to edit/change them. The labels we see here only affect the Swarm behavior. Labels are useful to categorize nodes. When you start services, you can then filter and decide where to physically spawn containers, using labels. For instance, if you want to dedicate a bunch of nodes with SSD to host MySQL, you can actually do this: docker node update --label-add type=ssd --label-add type=mysql worker1 docker node update --label-add type=ssd --label-add type=mysql worker2 docker node update --label-add type=ssd --label-add type=mysql worker3 Later, when you will start a service with some replica factor, say 3, you'll be sure that it will start MySQL containers exactly on worker1, worker2, and worker3, if you filter by node.type: docker service create --replicas 3 --constraint 'node.type == mysql' --name mysql-service mysql:5.5. Summary In this article, we went through the typical Swarm administration procedures and options. After showing how to add managers and workers to the cluster, we explained in detail how to update cluster and node properties, how to check the Swarm health, and we encountered Shipyard as a UI. After this focus on infrastructure, now it's time to use our Swarms. Resources for Article: Further resources on this subject: Hands On with Docker Swarm [article] Setting up a Project Atomic host [article] Getting Started with Flocker [article]
Read more
  • 0
  • 0
  • 5649

article-image-module-facts-types-and-reporting-tools-puppet
Packt
11 Jul 2014
11 min read
Save for later

Module, Facts, Types and Reporting tools in Puppet

Packt
11 Jul 2014
11 min read
(For more resources related to this topic, see here.) Module Files and Templates Transferring files with puppet is something best done within modules, when you define a file resource, you can use content => "something" or you can push a file from the puppet master using source. As an example, using our judy database, we could have judy::config with the following file definition: class judy::config { file {'/etc/judy/judy.conf': source => 'puppet:///modules/judy/judy.conf' } } Now puppet will search for this file in the directory judy/files. It is also possible to add full paths and have your module mimic the filesystem, the previous source line would be changed to source => 'puppet:///modules/judy/etc/judy/judy.conf' and the file would be found in judy/files/etc/judy/judy.conf. The puppet:/// url given previously has three backslashes, optionally the name of a puppet server may appear between the second and third backslash. If left blank, the puppet server performing catalog compilation is used to retrieve the file. You can alternatively specify the server using source => 'puppet://puppetfile.example.com/modules/judy/judy.conf'. Having files come from specific puppet servers can make maintenance difficult. If you change the name of your puppet server, you have to change all references to that name as well. Templates are searched in a similar fashion, in this example in judy/templates, when specifying the template, you use content => template('judy/template.erb') to have puppet look for the template in your modules templates directory. As an example another config file for judy could be: file {'/etc/judy/judyadm.conf': content => template('judy/judyadm.conf.erb') } Puppet will look for the file 'judy/judyadm.conf.erb' in modulepath/judy/templates/judyadm.conf.erb. We haven't covered ruby templates up to this point, templates are files that are parsed according to erb syntax rules. If you need to distribute a file where you need to change some settings based on variables, then a template is the thing to use. ERB syntax is covered in detail at http://docs.puppetlabs.com/guides/templating.html. In the next section, we will discuss module implementations in a large organization before writing custom modules. Creating a Custom Fact for use in Hiera The most useful custom facts are those that return a calculated value that you can use to organize your nodes. Such facts allow you to group your nodes into smaller groups or create groups with functionality or locality. These facts allow you to separate the data component of your modules from the logic or code components. Such a fact can be used in your hiera.yaml file to add a level to the hierarchy. One aspect of the system that can be used to determine information about the node is the ipaddress. Assuming you do not reuse ipaddresses within your organization, the ipaddress can be used to determine on which part of the network a node resides, the zone. In this example, we will define three zone's in which machines reside, production, development, and sandbox. The ipaddresses in each zone are on different subnets. We'll start by building up a script to calculate the zone and then turn it into a fact like our last example. Our script will need to calculate IP ranges using netmasks, so we'll import the ipaddr library and use IPAddr objects to calculate ranges. require('ipaddr') require('facter') require('puppet') Next we'll define a function that takes an ipaddress as the argument and returns the zone to which that ipaddress belongs: def zone(ip) zones = { 'production' => [IPAddr.new('192.168.122.0/24'),IPAddr.new('192.168.124.0/23')], 'development' => [IPAddr.new('192.168.123.0/24'),IPAddr.new('192.168.126.0/23')], 'sandbox' => [IPAddr.new('192.168.128.0/22')] } for zone in zones.keys do for subnet in zones[zone] do if subnet.include?(ip) return zone end end end return 'undef' end This function will loop through the zones hash looking for a match on IP address. If no match is found, the value of 'undef' is returned. We then obtain the ipaddress for the machine using the ipaddress fact from facter. ip = IPAddr.new(Facter.value('ipaddress')) Then we call the zone function with this ipaddress to obtain the zone. print zone(ip),"n" Now we can make this script executable and test. node1# facteripaddress 192.168.122.132 node1# ./example_zone.rb production Now all we have to do is replace print zone(ip),"n" with the following to define the fact. Facter.add('example_zone') do setcode do zone(ip) end end Now when we insert this code into our example_facts module and run puppet on our nodes, the custom fact is available. # facter -p example_zone production Now that we can define a zone based on a custom fact, we can go back to our hiera.yaml file and add %{::example_zone} to the hierarchy. The hiera.yaml hierarchy will now contain the following --- :hierarchy: - "zones/%{::example_zone}" - "hosts/%{::hostname}" - "roles/%{::role}" - "%{::kernel}/%{::osfamily}/%{::lsbmajdistrelease}" - "is_virtual/%{::is_virtual}" - common After restarting httpd to have the hiera.yaml file reread, we create a zones directory in hieradata and add production.yaml with the following contents. --- welcome: "example_zone - production" Now when we run puppet on our node1, we see the motd updated with the new welcome message node1# cat /etc/motd PRODUCTION example_zone - production Managed Node: node1 Managed by Puppet version 3.4.2 Creating a few key facts that can be used to build up your hierarchy can greatly reduce the complexity of your modules. There are several workflows available, in addition to the custom fact we just described above, you can use the /etc/facter/facts.d directory with static files or scripts, or you can have tasks run from other tools dump files into that directory to create custom facts. When writing ruby scripts you can use any other fact by calling Facter.value('factname'). If you write your script in ruby you can access any ruby library using require. Your custom fact could query the system using lspci or lsusb to determine what hardware is specifically installed on that node. As an example, you could use lspci to determine the make and model of graphics card on the machine and return that as a fact, such as videocard. In the next section we'll write our own custom modules that will take such a fact and install the appropriate driver for the videocard based on the custom fact. Parameterized Classes Parameterized classes are classes where you have defined several parameters that can be overridden when you instantiate the class for your node. The use case for parameterized classes is when you have something that won't be repeated within a single node. You cannot define the same parameterized class more than once per node. As a simple example, we'll create a class which installs a database program and starts that databases service. We'll call this class example::db, the definition will live in modules/example/manifests/db.pp class example::db ($db) { case $db { 'mysql': { $dbpackage = 'mysql-server' $dbservice = 'mysqld' } 'postgresql': { $dbpackage = 'postgresql-server' $dbservice = 'postgresql' } } package { "$dbpackage": } service { "$dbservice": ensure => true, enable => true, require => Package["$dbpackage"] } } This class takes a single parameter ($db) that specifies the type of the database, in this case either postgresql or mysql. To use this class, we have to instantiate it. class { 'example::db': db => 'mysql' } Now when we apply this to a node we see that mysql-server is installed and mysqld is started and enabled at boot. This works great for something like a database, since we don't think we will have more than one type of database server on a single node. If we try to instantiate the example::db class with postgresql on our node, we'll get an error as follows: Types and Providers Puppet separates the implementation of a type into the type definition and any one of many providers for that type. For instance, the package type in puppet has multiple providers depending on the platform in use (apt, yum, rpm and others). Early on in puppet development there were only a few core types defined. Since then, the core types have expanded to the point where anything that I feel should be a type is already defined by core puppet. The lvm module created a type for defining logical volumes, the concat module created types for defining file fragments. The firewall module created a type for defining firewall rules. Each of these types represents something on the system with the following properties: unique searchable creatable destroyable atomic When creating a new type, you have to make sure your new type has these properties. The resource defined by the type has to be unique, this is why the file type uses the path to a file as the naming variable (namevar), a system may have files with the same name (not unique) but it cannot have more than one file with an identical path. As an example, the ldap configuration file for openldap is /etc/openldap/ldap.conf, the ldap configuration file for the name services library is /etc/ldap.conf, if you used filename, then they would both be the same resource. Resources must be unique. By atomic I mean that it is indivisible, it cannot be made of smaller components. For instance, the firewall module creates a type for single iptables rules. Creating a type for the tables (INPUT, OUTPUT, FORWARD) within iptables wouldn't be atomic, each table is made up of multiple smaller parts, the rules. Your type has to be searchable so that puppet can determine the state of the thing you are modifying. A mechanism has to exist to know what the current state is of the thing in question. The last two properties are equally important, puppet must be able to remove the thing, destroy it and likewise, puppet must be able to create the thing anew. Given these criteria, there are several modules that define new types, some examples include types which manage: git repositories apache virtual hosts ldap entries network routes gem modules perlcpan modules databases drupalmultisites Foreman Foreman is more than just a puppet reporting tool, it bills itself as a complete lifecycle management platform. Foreman can act as the ENC (external node classifier) for your entire installation and configure DHCP, DNS and PXE booting. It's a one stop shop. We'll configure foreman to be our report backend in this example. mcollective Mcollective is an orchestration tool created by puppetlabs that is not specific to puppet, plugins exist to work with other configuration management systems. Mcollective uses a message queue (MQ) with active connections from all active nodes to enable parallel job execution on large numbers of nodes. To understand how mcollective works, we'll consider the following high level diagram and work through the various components. The configuration of mcollective is still somewhat involved and prone to errors. Still, once mcollective is working properly, the power it provides can become addicting, it will be worth the effort. The default MQ install for marionette is using activemq, the activemq provided by the puppetlabs repo is known to work. Mcollective uses message queue and can use your existing message queue infrastructure. If using activemq, a single server can handle 800 nodes. After that you'll need to spread out. We'll cover the standard mcollective install using puppet's certificate authority to provide ssl security to mcollective. The theory here being that we trust puppet to configure the machines already, we can trust it a little more to run arbitrary commands. We'll also require that users of mcollective have proper ssl authentication as well. Summary In this article we learned how to deal with puppet. We also created a custom fact for use in Hiera and covered different topics like Foreman, mcollective, and much more. Resources for Article: Further resources on this subject: Puppet: Integrating External Tools [article] Quick start – Using the core Puppet resource types [article] External Tools and the Puppet Ecosystem [article]
Read more
  • 0
  • 0
  • 5645
Modal Close icon
Modal Close icon