How-To Tutorials

(For more resources related to this topic, see here.) The PlayStation Mobile (PSM) SDK represents an exciting opportunity for game developers of all stripes, from hobbyists to indie and professional developers. It contains everything you need to quickly develop a game using the C# programming language. Perhaps more importantly, it provides a market for those games. If you are currently using XNA, you will feel right at home with the PSM SDK. You may be wondering at this point, Why develop for PlayStation Mobile at all? Obviously, the easiest answer is, so you can develop for PlayStation Vita , which of itself will be enough for many people. Perhaps, though the most important reason is that it represents a group of dedicated gamers hungry for games. While there are a wealth of games available for Android, finding them on the App Store is a mess, while supporting the literally thousands of devices is a nightmare. With PlayStation Mobile, you have a common development environment, targeting powerful devices with a dedicated store catering to gamers. We are now going to jump right in and get those tools up and running. Of course, we will also write some code and show how easy it is to get it running on your device. PlayStation Mobile allows you to target a number of different devices and we will cover the three major targets (the Simulator, PlayStation Vita, and Android). You do not need to have a device to follow along, although certain functionality will not be available on the Simulator. One thing to keep in mind with the PlayStation Mobile SDK is that it is essentially two SDKs in one. There is a much lower level set of libraries for accessing graphics, audio, and input, as well as a higher-level layer build over the top of this layer, mostly with the complete source available. Of course, underneath this all there is the .NET framework. In this article, we are going to deal with the lower level graphics interface. If the code seems initially quite long or daunting for what seems like a simple task, don't worry! There is a much easier way that we will cover later in the book. Accessing the PlayStation Mobile portal This recipe looks at creating a PSM portal account. For this process it is mandatory to download and use the PSM SDK. Getting ready You need to have a Sony Entertainment Network (SEN) account to register with the PSM portal. This is the standard account you use to bring your PlayStation device online, so you may already have one. If not, create one at http://bit.ly/Yiglfk before continuing. How to do it... Open a web browser and log in to http://psm.playstation.net. Locate and click on the Register button. Sign in using the SEN account. Agree to the Terms and Conditions. You need to scroll to the bottom of the text before the Agree button is enabled. But, you always read the fine print anyways... don't you? Finally select the e-mail address and language you want for the PlayStation Mobile portal. You can use the same e-mail you used for your SEN account. Click on Register. An e-mail will be sent to the e-mail account you used to sign up. Locate the activation link and either click on it, or copy and paste into a browser window: Your account is now completed, and you can log in to the PSM developer portal now. How it works... A PlayStation Mobile account is mandatory to download the PSM tools. Many of the links to the portal require you to be logged in before they will work. It is very important that you create and activate your account and log in to the portal before continuing on with the book! All future recipes assume you are logged in to the portal. Installing the PlayStation Mobile SDK This recipe demonstrates how to install the PlayStation Mobile SDK. Getting ready First you need to download the PlayStation Mobile SDK; you can download it from http://bit.ly/W8rhhx. How to do it... Locate the installation file you downloaded earlier and double-click to launch the installer. Say yes to any security related questions. Take the default settings when prompting, making sure to install the runtimes and GTK# libraries. The installer for the Vita drivers will now launch. There is no harm in installing them even if you do not have a Vita: Installation is now complete; a browser window with the current release notes will open. How it works... The SDK is now installed on your machines. Assuming you used default directories, the SDK will be installed to C:Program Files (x86)SCEPSM if you are running 64 bit Windows, or to C:Program FilesSCEPSM if you are running 32 bit Windows. Additionally all of the documentation and samples have been installed under the Public account, located in C:UsersPublicDocumentsPSM. There's more... There are a number of samples available in the samples directory and you should certainly take a moment to check them out. They range in complexity from simple Hello World applications, up to a full blown 3rd person 3D role playing game (RPG). They are, however, often documented in Japanese and often rely on other samples, making learning from them a frustrating experience at times, at least, for those of us who do not understand Japanese! Creating a simple game loop We are now going to create our first PSM SDK application, which is the main loop of your application. Actually all the code in this sample is going to be generated by PSM Studio for us. Getting ready From the start menu, locate and launch PSM Studio in the PlayStation Mobile folder. How to do it... In PSM Studio, select the File | New | Solution... menu. In the resulting dialog box, in the left-hand panel expand C# and select PlayStation Suite, then in the right-hand panel, select PlayStation Suite Application. Fill in the Name field, which will automatically populate the Solution name field. Click on OK. Your workspace and boilerplate code will now be created; hit the F5 key or select the Run | Start Debugging menu to run your code in the Simulator. Not much to look at, but it's your first running PlayStation Mobile application! Now let's take a quick look at the code it generated: using System; using System.Collections.Generic; using Sce.PlayStation.Core; using Sce.PlayStation.Core.Environment; using Sce.PlayStation.Core.Graphics; using Sce.PlayStation.Core.Input; namespace Ch1_Example1 { public class AppMain{ private static GraphicsContext graphics; public static void Main (string[] args){ Initialize (); while (true) { SystemEvents.CheckEvents (); Update (); Render (); } } public static void Initialize (){ graphics = new GraphicsContext (); } public static void Update (){ var gamePadData = GamePad.GetData (0); } public static void Render () { graphics.SetClearColor (0.0f, 0.0f, 0.0f, 0.0f); graphics.Clear (); graphics.SwapBuffers (); } } } How it works... This recipe shows us the very basic skeleton of an application. Essentially it loops forever, displaying a black screen. private static GraphicsContext graphics; The GraphicsContext variable represents the underlying OpenGL context. It is used to perform almost every graphically related action. Additionally, it contains the capabilities (resolution, pixel depth, and so on) of the underlying graphics device. All C# based applications have a main function, and this one is no exception. Within Main() we call our Initialize() method, then loop forever, checking for events, updating, and finally rendering the frame. The Initialize() method simply creates a new GraphicsContext variable. The Update() method polls the first gamepad for updates. Finally Render() uses our GraphicsContext variable to first clear the screen to black using an RGBA color value, then clears the screen and swaps the buffers, making it visible. Graphic operations in PSM SDK generally are drawn to a back buffer. There's more... The same process is used to create PlayStation Suite library projects, which will generate a DLL file. You can use almost any C# library that doesn't rely on native code (pInvoke or Unsafe); however, they need to be recompiled into a PSM compatible DLL format. Color in the PSM SDK is normally represented as an RGBA value. The RGBA acronym stands for red, green, blue, and alpha. Each is an int variable type, with values ranging from 0 to 255 representing the strength of each primary color. Alpha represents the level of transparency, with 0 being completely transparent and 256 being opaque.

(For more resources related to this topic, see here.) Getting ready For this article, all you need is your browser and favorite text editor. How to do it... Perform the following steps: Let's begin with creating a blank document in your text editor. Then, add the following code and save it as a test localdemo.html: <!DOCTYPE html> <html> <head> <script src = "http://code.jquery.com/jquery-1.8.1.min.js"></script> <script type="text/javascript"> </script> </head> <body> </body> </html> In between the <script> tags, add the following function. This copes with storing the information within the browser. <script type="text/javascript"> function storeItem() { var item = $('#item').val(); var items = localStorage.getItem('myItems'); if (items != null) { items = JSON.parse(items); } else { items = new Array(); } items.push(item); localStorage.setItem('myItems', JSON.stringify(items)); refresh(); } </script> We need to add another function to retrieve information and refresh the content displayed on screen. So go ahead and update the script as highlighted: <script type="text/javascript"> function storeItem() { var item = $('#item').val(); var items = localStorage.getItem('myItems'); if (items != null) { items = JSON.parse(items); } else { items = new Array(); } items.push(item); localStorage.setItem('myItems', JSON.stringify(items)); refresh(); } function refresh() { var items = localStorage.getItem('myItems'); var ul = $('ul'); ul.html(''); if (items != null) { items = JSON.parse(items); $(items).each(function (index, data) { ul.append('<li>' + data + '</li>'); }); } } $(function () { refresh(); }); </script> We finish by adding a basic form—while the purists amongst you will notice that it doesn't have all of the proper forms of tag, it is enough to illustrate how this demo works. Add the following code snippet, just above the closing </body> tag: Enter item: <input type="text" id="item" /> <input type="button" value= "store" onclick="storeItem()" /> <br /> <ul></ul> Crack open your browser and preview the results. Here's a screenshot of what you should see, with some example values already entered: How it works... Now we've seen Local Storage in action, let's take a look at how it works in detail. HTML5 Local Storage works on the principle of named key/value pairs, where you store information using a named key and retrieve it by calling that named key. Everything is stored locally on the user's PC; it cuts down the need to retrieve information from the server, thereby acting as a form of caching. You may have noticed that we've used jQuery in this article—basic use of LocalStorage (and SessionStorage) doesn't necessarily need jQuery; you could use pure JavaScript if you prefer. It all depends on your requirements; if you are already using jQuery in your pages, for example, you may prefer to use this over JavaScript. (You will see I have used a mix of both throughout this book, to show you how you can use either jQuery or JavaScript). In this article, we've used jQuery to reference LocalStorage; if you take a look at the code, you will see two lines of particular importance: var items = localStorage.getItem('myItems'); localStorage.setItem('myItems', JSON.stringify(items)); These two handle the retrieval and setting of values respectively. In this demo, we begin with either fetching the contents of any existing stored information and inserting them into an array, or creating a new one, if nothing exists within the store. We then use JSON. stringify() to convert information from the form into a string, push this into the storage, and then refresh the page so that you can see the updated list. To get the information back, we simply repeat the same steps, but in the reverse way. The beauty of using JSON as part of storing information in this way is that you are not entirely limited to just plain text; you can store some other things in the LocalStorage area, as we will see later in this book. There's more... By now, you will start to see that using Local Storage works very much in the same way that cookies do—indeed some people often refer to Local Storage as "cookies on steroids". This said, there are still some limitations that you need to be aware of when using Local Storage, such as the following: Local Storage will only support text as a format and is set to a suggested arbitary limit of 5 MB, although this is inconsistent across browsers. If you exceed this, the QUOTA_EXCEEDED_ERR error is thrown. At the time of writing this book, there is no support built in for requesting more space. Some browsers such as Opera will allow the user to control each site's quota, but this is a purely userbased action. Web Storage is no more secure than cookies; although use of the HTTPS protocol can resolve a lot of security issues, it is still up to you as a developer to ensure that sensitive information (such as passwords) is not sent to or stored locally on the client using Web Storage. With careful use, we can take advantage of the ability to store relevant information on a user's PC, and avoid the need to push it back to the server. Once the information has been stored, there will be occasions when you will need to view the raw information from within your browser—this is easy enough to do, although the method varies from browser to browser, which we will see as part of the next article. Summary In this article we discussed basic use of HTML5 Local Storage. Resources for Article : Further resources on this subject: Blocking versus Non blocking scripts [Article] Building HTML5 Pages from Scratch [Article] HTML5 Games Development: Using Local Storage to Store Game Data [Article]

(For more resources related to this topic, see here.) Creating a gun that fires homing missiles UDK already has a homing rocket launcher packaged with the dev kit (UTWeap_ RocketLauncher). The problem however, is that it isn't documented well; it has a ton of excess code only necessary for multiplayer games played over a network, and can only lock on when you have loaded three rockets. We're going to change all of that, and allow our homing weapon to lock onto a pawn and fire any projectile of our choice. We also need to change a few functions, so that our weapon fires from the correct location and uses the pawn's rotation and not the camera's. Getting ready As I mentioned earlier, our main weapon for this article will extend from the UTWeap_ ShockRifle, as that gun offers a ton of great base functionality which we can build from. Let's start by opening your IDE and creating a new weapon called MyWeapon, and have it extend from UTWeap_ShockRifle as shown as follows: class MyWeapon extends UTWeap_ShockRifle; How to do it... We need to start by adding all of the variables that we'll be needing for our lock on feature. There are quite a few here, but they're all commented in pretty great detail. Much of this code is straight from UDK's rocket launcher, that is why it looks familiar. In this recipe, we'll be creating a base weapon which extends from one of the Unreal Tournament's most commonly used weapons, the shock rifle, and base all of our weapons from that. I've gone ahead and removed an unnecessary information, added comments, and altered functionality so that we can lock onto pawns with any weapon, and fire only one missile while doing so. /********************************************************* Weapon lock on support********************************************************//** Class of the rocket to use when seeking */var class<UTProjectile> SeekingRocketClass;/** The frequency with which we will check for a lock */var(Locking) float LockCheckTime;/** How far out should we be considering actors for a lock */var float LockRange;/** How long does the player need to target an actor to lock on toit*/var(Locking) float LockAcquireTime;/** Once locked, how long can the player go without painting theobject before they lose the lock */var(Locking) float LockTolerance;/** When true, this weapon is locked on target */var bool bLockedOnTarget;/** What "target" is this weapon locked on to */var Actor LockedTarget;var PlayerReplicationInfo LockedTargetPRI;/** What "target" is current pending to be locked on to */var Actor PendingLockedTarget;/** How long since the Lock Target has been valid */var float LastLockedOnTime;/** When did the pending Target become valid */var float PendingLockedTargetTime;/** When was the last time we had a valid target */var float LastValidTargetTime;/** angle for locking for lock targets */var float LockAim;/** angle for locking for lock targets when on Console */var float ConsoleLockAim;/** Sound Effects to play when Locking */var SoundCue LockAcquiredSound;var SoundCue LockLostSound;/** If true, weapon will try to lock onto targets */var bool bTargetLockingActive;/** Last time target lock was checked */var float LastTargetLockCheckTime; With our variables in place, we can now move onto the weapon's functionality. The InstantFireStartTrace() function is the same function we added in our weapon. It allows our weapon to start its trace from the correct location using the GetPhysicalFireStartLoc() function. function. As mentioned before, this simply grabs the rotation of the weapon's muzzle flash socket, and tells the weapon to fire projectiles from that location, using the socket's rotation. The same goes for GetEffectLocation(), which is where our muzzle flash will occur. The v in vector for the InstantFireStartTrace() function is not capitalized. The reason being that vector is actually of struct type, and not a function, and that is standard procedure in UDK. /********************************************************* Overriden to use GetPhysicalFireStartLoc() instead of* Instigator.GetWeaponStartTraceLocation()* @returns position of trace start for instantfire()********************************************************/simulated function vector InstantFireStartTrace(){return GetPhysicalFireStartLoc();}/********************************************************* Location that projectiles will spawn from. Works for secondaryfire on* third person mesh********************************************************/simulated function vector GetPhysicalFireStartLoc(optional vectorAimDir){Local SkeletalMeshComponent AttachedMesh;local vector SocketLocation;Local TutorialPawn TutPawn;TutPawn = TutorialPawn(Owner);AttachedMesh = TutPawn.CurrentWeaponAttachment.Mesh;/** Check to prevent log spam, and the odd situation winwhich a cast to type TutPawn can fail */if (TutPawn != none){AttachedMesh.GetSocketWorldLocationAndRotation(MuzzleFlashSocket, SocketLocation);}return SocketLocation;}/********************************************************* Overridden from UTWeapon.uc* @return the location + offset from which to spawn effects(primarily tracers)********************************************************/simulated function vector GetEffectLocation(){Local SkeletalMeshComponent AttachedMesh;local vector SocketLocation;Local TutorialPawn TutPawn;TutPawn = TutorialPawn(Owner);AttachedMesh = TutPawn.CurrentWeaponAttachment.Mesh;if (TutPawn != none){AttachedMesh.GetSocketWorldLocationAndRotation(MuzzleFlashSocket, SocketLocation);}MuzzleFlashSocket, SocketLocation);return SocketLocation;} Now we're ready to dive into the parts of code that are applicable to the actual homing of the weapon. Let's start by adding our debug info, which allows us to troubleshoot any issues we may have along the way. ********************************************************** Prints debug info for the weapon********************************************************/simulated function GetWeaponDebug( out Array<String> DebugInfo ){Super.GetWeaponDebug(DebugInfo);DebugInfo[DebugInfo.Length] = "Locked:"@bLockedOnTarget@LockedTarget@LastLockedontime@(WorldInfo.TimeSeconds-LastLockedOnTime);DebugInfo[DebugInfo.Length] ="Pending:"@PendingLockedTarget@PendingLockedTargetTime@WorldInfo.TimeSeconds;} Here we are simply stating which target our weapon is currently locked onto, in addition to the pending target. It does this by grabbing the variables we've listed before, after they've returned from their functions, which we'll add in the next part. We need to have a default state for our weapon to begin with, so we mark it as inactive. /********************************************************* Default state. Go back to prev state, and don't use our* current tick********************************************************/auto simulated state Inactive{ignores Tick;simulated function BeginState(name PreviousStateName){Super.BeginState(PreviousStateName);// not looking to lock onto a targetbTargetLockingActive = false;// Don't adjust our target lockAdjustLockTarget(None);} We ignore the tick which tells the weapon to stop updating any of its homing functions. Additionally, we tell it not to look for an active target or adjust its current target, if we did have one at the moment. While on the topic of states, if we finish our current one, then it's time to move onto the next: /********************************************************* Finish current state, & prepare for the next one********************************************************/simulated function EndState(Name NextStateName){Super.EndState(NextStateName);// If true, weapon will try to lock onto targetsbTargetLockingActive = true;}} If our weapon is destroyed or we are destroyed, then we want to prevent the weapon from continuing to lock onto a target. /********************************************************* If the weapon is destroyed, cancel any target lock********************************************************/simulated event Destroyed(){// Used to adjust the LockTarget.AdjustLockTarget(none);//Calls the previously defined Destroyed functionsuper.Destroyed();} Our next chunk of code is pretty large, but don't let it intimidate you. Take your time and read it through to have a thorough understanding of what is occurring. When it all boils down, the CheckTargetLock() function verifies that we've actually locked onto our target. We start by checking that we have a pawn, a player controller, and that we are using a weapon which can lock onto a target. We then check if we can lock onto the target, and if it is possible, we do it. At the moment we only have the ability to lock onto pawns. /****************************************************************** Have we locked onto our target?****************************************************************/function CheckTargetLock(){local Actor BestTarget, HitActor, TA;local UDKBot BotController;local vector StartTrace, EndTrace, Aim, HitLocation,HitNormal;local rotator AimRot;local float BestAim, BestDist;if((Instigator == None)||(Instigator.Controller ==None)||(self != Instigator.Weapon) ){return;}if ( Instigator.bNoWeaponFiring)// TRUE indicates that weapon firing is disabled for thispawn{// Used to adjust the LockTarget.AdjustLockTarget(None);// "target" is current pending to be locked on toPendingLockedTarget = None;return;}// We don't have a targetBestTarget = None;BotController = UDKBot(Instigator.Controller);// If there is BotController...if ( BotController != None ){// only try locking onto bot's targetif((BotController.Focus != None) &&CanLockOnTo(BotController.Focus) ){// make sure bot can hit itBotController.GetPlayerViewPoint( StartTrace, AimRot );Aim = vector(AimRot);if((Aim dot Normal(BotController.Focus.Location -StartTrace)) > LockAim ){HitActor = Trace(HitLocation, HitNormal,BotController.Focus.Location, StartTrace, true,,,TRACEFLAG_Bullet);if((HitActor == None)||(HitActor == BotController.Focus) ){// Actor being looked atBestTarget = BotController.Focus;}}}} Immediately after that, we do a trace to see if our missile can hit the target, and check for anything that may be in the way. If we determine that we can't hit our target then it's time to start looking for a new one. else{// Trace the shot to see if it hits anyoneInstigator.Controller.GetPlayerViewPoint( StartTrace, AimRot );Aim = vector(AimRot);// Where our trace stopsEndTrace = StartTrace + Aim * LockRange;HitActor = Trace(HitLocation, HitNormal, EndTrace, StartTrace,true,,, TRACEFLAG_Bullet);// Check for a hitif((HitActor == None)||!CanLockOnTo(HitActor) ){/** We didn't hit a valid target? Controllerattempts to pick a good target */BestAim = ((UDKPlayerController(Instigator.Controller)!=None)&&UDKPlayerController(Instigator.Controller).bConsolePlayer) ? ConsoleLockAim : LockAim;BestDist = 0.0;TA = Instigator.Controller.PickTarget(class'Pawn', BestAim, BestDist, Aim, StartTrace,LockRange);if ( TA != None && CanLockOnTo(TA) ){/** Best target is the target we've locked */BestTarget = TA;}}// We hit a valid targetelse{// Best Target is the one we've done a trace onBestTarget = HitActor;}} If we have a possible target, then we note its time mark for locking onto it. If we can lock onto it, then start the timer. The timer can be adjusted in the default properties and determines how long we need to track our target before we have a solid lock. // If we have a "possible" target, note its time markif ( BestTarget != None ){LastValidTargetTime = WorldInfo.TimeSeconds;// If we're locked onto our best targetif ( BestTarget == LockedTarget ){/** Set the LLOT to the time in seconds sincelevel began play */LastLockedOnTime = WorldInfo.TimeSeconds;} Once we have a good target, it should turn into our current one, and start our lock on it. If we've been tracking it for enough time with our crosshair (PendingLockedTargetTime), then lock onto it. else{if ( LockedTarget != None&&((WorldInfo.TimeSeconds - LastLockedOnTime >LockTolerance)||!CanLockOnTo(LockedTarget)) ){// Invalidate the current locked TargetAdjustLockTarget(None);}/** We have our best target, see if they shouldbecome our current target Check for a newpending lock */if (PendingLockedTarget != BestTarget){PendingLockedTarget = BestTarget;PendingLockedTargetTime =((Vehicle(PendingLockedTarget) != None)&&(UDKPlayerController(Instigator.Controller)!=None)&&UDKPlayerController(Instigator.Controller).bConsolePlayer)? WorldInfo.TimeSeconds + 0.5*LockAcquireTime: WorldInfo.TimeSeconds + LockAcquireTime;}/** Otherwise check to see if we have beentracking the pending lock long enough */else if (PendingLockedTarget == BestTarget&& WorldInfo.TimeSeconds = PendingLockedTargetTime ){AdjustLockTarget(PendingLockedTarget);LastLockedOnTime = WorldInfo.TimeSeconds;PendingLockedTarget = None;PendingLockedTargetTime = 0.0;}}} Otherwise, if we can't lock onto our current or our pending target, then cancel our current target, along with our pending target. else{if ( LockedTarget != None&&((WorldInfo.TimeSeconds -LastLockedOnTime > LockTolerance)||!CanLockOnTo(LockedTarget)) ){// Invalidate the current locked TargetAdjustLockTarget(None);}// Next attempt to invalidate the Pending Targetif ( PendingLockedTarget != None&&((WorldInfo.TimeSeconds - LastValidTargetTime >LockTolerance)||!CanLockOnTo(PendingLockedTarget)) ){// We are not pending another target to lock ontoPendingLockedTarget = None;}}} That was quite a bit to digest. Don't worry, because the functions from here on out are pretty simple and straightforward. As with most other classes, we need a Tick() function to check for something in each frame. Here, we'll be checking whether or not we have a target locked in each frame, as well as setting our LastTargetLockCheckTime to the number of seconds passed during game-time. /********************************************************* Check target locking with each update********************************************************/event Tick( Float DeltaTime ){if ( bTargetLockingActive && ( WorldInfo.TimeSeconds >LastTargetLockCheckTime + LockCheckTime ) ){LastTargetLockCheckTime = WorldInfo.TimeSeconds;// Time, in seconds, since level began playCheckTargetLock();// Checks to see if we are locked on a target}} As I mentioned earlier, we can only lock onto pawns. Therefore, we need a function to check whether or not our target is a pawn. /********************************************************* Given an potential target TA determine if we can lock on to it.By* default, we can only lock on to pawns.********************************************************/simulated function bool CanLockOnTo(Actor TA){if ( (TA == None) || !TA.bProjTarget || TA.bDeleteMe ||(Pawn(TA) == None) || (TA == Instigator) ||(Pawn(TA).Health <= 0) ){return false;}return ( (WorldInfo.Game == None) ||!WorldInfo.Game.bTeamGame || (WorldInfo.GRI == None) ||!WorldInfo.GRI.OnSameTeam(Instigator,TA) );} Once we have a locked target we need to trigger a sound, so that the player is aware of the lock. The whole first half of this function simply sets two variables to not have a target, and also plays a sound cue to notify the player that we've lost track of our target. /********************************************************* Used to adjust the LockTarget.********************************************************/function AdjustLockTarget(actor NewLockTarget){if ( LockedTarget == NewLockTarget ){// No need to updatereturn;}if (NewLockTarget == None){// Clear the lockif (bLockedOnTarget){// No targetLockedTarget = None;// Not locked onto a targetbLockedOnTarget = false;if (LockLostSound != None && Instigator != None &&Instigator.IsHumanControlled() ){// Play the LockLostSound if we lost track of thetargetPlayerController(Instigator.Controller).ClientPlaySound(LockLostSound);}}}else{// Set the lockbLockedOnTarget = true;LockedTarget = NewLockTarget;LockedTargetPRI = (Pawn(NewLockTarget) != None) ?Pawn(NewLockTarget).PlayerReplicationInfo : None;if ( LockAcquiredSound != None && Instigator != None &&Instigator.IsHumanControlled() ){PlayerController(Instigator.Controller).ClientPlaySound(LockAcquiredSound);}}} Once it looks like everything has checked out we can fire our ammo! We're just setting everything back to 0 at this point, as our projectile is seeking our target, so it's time to start over and see whether we will use the same target or find another one. /********************************************************* Everything looks good, so fire our ammo!********************************************************/simulated function FireAmmunition(){Super.FireAmmunition();AdjustLockTarget(None);LastValidTargetTime = 0;PendingLockedTarget = None;LastLockedOnTime = 0;PendingLockedTargetTime = 0;} With all of that out of the way, we can finally work on firing our projectile, or in our case, our missile. ProjectileFile() tells our missile to go after our currently locked target, by setting the SeekTarget variable to our currently locked target. /********************************************************* If locked on, we need to set the Seeking projectile's* LockedTarget.********************************************************/simulated function Projectile ProjectileFire(){local Projectile SpawnedProjectile;SpawnedProjectile = super.ProjectileFire();if (bLockedOnTarget &&UTProj_SeekingRocket(SpawnedProjectile) != None){/** Go after the target we are currently lockedonto */UTProj_SeekingRocket(SpawnedProjectile).SeekTarget =LockedTarget;}return SpawnedProjectile;} Really though, our projectile could be anything at this point. We need to tell our weapon to actually use our missile (or rocket, they are used interchangeably) which we will define in our defaultproperties block. /********************************************************* We override GetProjectileClass to swap in a Seeking Rocket if weare* locked on.********************************************************/function class<Projectile> GetProjectileClass(){// if we're locked on...if (bLockedOnTarget){// use our homing rocketreturn SeekingRocketClass;}// Otherwise...else{// Use our default projectilereturn WeaponProjectiles[CurrentFireMode];}} If we don't have a SeekingRocketClass class defined, then we just use the currently defined projectile from our CurrentFireMode array. The last part of this class involves the defaultproperties block. This is the same thing we saw in our Camera class. We're setting our muzzle flash socket, which is used for not only firing effects, but also weapon traces, to actually use our muzzle flash socket. defaultproperties{// Forces the secondary fire projectile to fire fromthe weapon attachment */MuzzleFlashSocket=MuzzleFlashSocket} Our MyWeapon class is complete. We don't want to clog our defaultproperties block and we have some great base functionality, so from here on out our weapon classes will generally be only changes to the defaultproperties block. Simplicity! Create a new class called MyWeapon_HomingRocket. Have it extend from MyWeapon. class MyWeapon_HomingRocket extends MyWeapon; In our defaultproperties block, let's add our skeletal and static meshes. We're just going to keep using the shock rifle mesh. Although it's not necessary to do this, as we're already a child class of (that is, inheriting from) UTWeap_ShockRifle, I still want you to see where you would change the mesh if you ever wanted to. defaultproperties{// Weapon SkeletalMeshBegin Object class=AnimNodeSequence Name=MeshSequenceAEnd Object// Weapon SkeletalMeshBegin Object Name=FirstPersonMeshSkeletalMesh=SkeletalMesh'WP_ShockRifle.Mesh.SK_WP_ShockRifle_1P'AnimSets(0)=AnimSet'WP_ShockRifle.Anim.K_WP_ShockRifle_1P_Base'Animations=MeshSequenceARotation=(Yaw=-16384)FOV=60.0End Object// PickupMeshBegin Object Name=PickupMeshSkeletalMesh=SkeletalMesh'WP_ShockRifle.Mesh.SK_WP_ShockRifle_3P'End Object// Attachment classAttachmentClass=class'UTGameContent.UTAttachment_ShockRifle' Next, we want to declare the type of projectile, the type of damage it does, and the frequency at which it can be fired. Moreover, we want to declare that each shot fired will only deplete one round from our inventory. We can declare how much ammo the weapon starts with too. // Defines the type of fire for each modeWeaponFireTypes(0)=EWFT_InstantHitWeaponFireTypes(1)=EWFT_ProjectileWeaponProjectiles(1)=class'UTProj_Rocket'// Damage typesInstantHitDamage(0)=45FireInterval(0)=+1.0FireInterval(1)=+1.3InstantHitDamageTypes(0)=class'UTDmgType_ShockPrimary'InstantHitDamageTypes(1)=None// Not an instant hit weapon, so set to "None"// How much ammo will each shot use?ShotCost(0)=1ShotCost(1)=1// # of ammo gun should start withAmmoCount=20// Initial ammo count if weapon is lockedLockerAmmoCount=20// Max ammo countMaxAmmoCount=40 Our weapon will use a number of sounds that we didn't previously need, such as locking onto a pawn, as well as losing lock. So let's add those now. // Sound effectsWeaponFireSnd[0] =SoundCue'A_Weapon_ShockRifle.Cue.A_Weapon_SR_FireCue'WeaponFireSnd[1]=SoundCue'A_Weapon_RocketLauncher.Cue.A_Weapon_RL_Fire_Cue'WeaponEquipSnd=SoundCue'A_Weapon_ShockRifle.Cue.A_Weapon_SR_RaiseCue'WeaponPutDownSnd=SoundCue'A_Weapon_ShockRifle.Cue.A_Weapon_SR_LowerCue'PickupSound=SoundCue'A_Pickups.Weapons.Cue.A_Pickup_Weapons_Shock_Cue'LockAcquiredSound=SoundCue'A_Weapon_RocketLauncher.Cue.A_Weapon_RL_SeekLock_Cue'LockLostSound=SoundCue'A_Weapon_RocketLauncher.Cue.A_Weapon_RL_SeekLost_Cue' We won't be the only one to use this weapon, as bots will be picking it up during Deathmatch style games as well. Therefore, we want to declare some logic for the bots, such as how strongly they will desire it, and whether or not they can use it for things like sniping. // AI logicMaxDesireability=0.65 // Max desireability for botsAIRating=0.65CurrentRating=0.65bInstantHit=false // Is it an instant hit weapon?bSplashJump=false// Can a bot use this for splash damage?bRecommendSplashDamage=true// Could a bot snipe with this?bSniping=false// Should it fire when the mouse is released?ShouldFireOnRelease(0)=0// Should it fire when the mouse is released?ShouldFireOnRelease(1)=0 We need to create an offset for the camera too, otherwise the weapon wouldn't display correctly as we switch between first and third person cameras. // Holds an offset for spawning projectile effectsFireOffset=(X=20,Y=5)// Offset from view center (first person)PlayerViewOffset=(X=17,Y=10.0,Z=-8.0) Our homing properties section is the bread and butter of our class. This is where you'll alter the default values for anything to do with locking onto pawns. // Homing properties/** angle for locking for locktargets when on Console */ConsoleLockAim=0.992/** How far out should we be before considering actors fora lock? */LockRange=9000// Angle for locking, for lockTargetLockAim=0.997// How often we check for lockLockChecktime=0.1// How long does player need to hover over actor to lock?LockAcquireTime=.3// How close does the trace need to be to the actual targetLockTolerance=0.8SeekingRocketClass=class'UTProj_SeekingRocket' Animations are an essential part of realism, so we want the camera to shake when firing a weapon, in addition to an animation for the weapon itself. // camera anim to play when firing (for camera shakes)FireCameraAnim(1)=CameraAnim'Camera_FX.ShockRifle.C_WP_ShockRifle_Alt_Fire_Shake'// Animation to play when the weapon is firedWeaponFireAnim(1)=WeaponAltFire While we're on the topic of visuals, we may as well add the flashes at the muzzle, as well as the crosshairs for the weapon. // Muzzle flashesMuzzleFlashPSCTemplate=WP_ShockRifle.Particles.P_ShockRifle_MF_AltMuzzleFlashAltPSCTemplate=WP_ShockRifle.Particles.P_ShockRifle_MF_AltMuzzleFlashColor=(R=200,G=120,B=255,A=255)MuzzleFlashDuration=0.33MuzzleFlashLightClass=class'UTGame.UTShockMuzzleFlashLight'CrossHairCoordinates=(U=256,V=0,UL=64,VL=64)LockerRotation=(Pitch=32768,Roll=16384)// CrosshairIconCoordinates=(U=728,V=382,UL=162,VL=45)IconX=400IconY=129IconWidth=22IconHeight=48/** The Color used when drawing the Weapon's Name on theHUD */WeaponColor=(R=160,G=0,B=255,A=255) Since weapons are part of a pawn's inventory, we need to declare which slot this weapon will fall into (from one to nine). // InventoryInventoryGroup=4 // The weapon/inventory set, 0-9GroupWeight=0.5 // position within inventory group.(used by prevweapon and nextweapon) Our final piece of code has to do with rumble feedback with the Xbox gamepad. This is not only used on consoles, but also it is generally reserved for it. /** Manages the waveform data for a forcefeedback device,specifically for the xbox gamepads. */Begin Object Class=ForceFeedbackWaveformName=ForceFeedbackWaveformShooting1Samples(0)=(LeftAmplitude=90,RightAmplitude=40,LeftFunction=WF_Constant,RightFunction=WF_LinearDecreasing,Duration=0.1200)End Object// controller rumble to play when firingWeaponFireWaveForm=ForceFeedbackWaveformShooting1} All that's left to do is to add the weapon to your pawn's default inventory. You can easily do this by adding the following line to your TutorialGame class's defaultproperties block: defaultproperties{DefaultInventory(0)=class'MyWeapon_HomingRocket'} Load up your map with a few bots on it, hold your aiming reticule over it for a brief moment and when you hear the lock sound, fire away! How it works... To keep things simple we extend from UTWeap_ShockRifle. This gave us a great bit of base functionality to work from. We created a MyWeapon class which offers not only everything that the shock rifle does, but also the ability to lock onto targets. When we aim our target reticule over an enemy bot, it checks for a number of things. First, it verifies that it is an enemy and also whether or not the target can be reached. It does this by drawing a trace and returns any actors which may fall in our weapon's path. If all of these things check out, then it begins to lock onto our target after we've held the reticule over the enemy for a set period of time. We then fire our projectile, which is either the weapon's firing mode, or in our case, a rocket. We didn't want to clutter the defaultproperties block for MyWeapon; so we create a child class called MyWeapon_HomingRocket that makes use of all the functionality and only changes the defaultproperties block, which will influence the weapon's aesthetics, sound effects, and even some functionality with the target lock.

(For more resources related to this topic, see here.) Tools to migrate data There are several ways to migrate data into Microsoft Dynamics NAV 2013. You will choose the method depending on what is to be migrated and whether any additional processes need to be carried out on the provided data to meet Dynamics NAV requirements. We'll go through the different tools available in Dynamics NAV to migrate data. We'll also explain how to write our own tools if the ones provided out of the box do not meet our requirement or expectation. RapidStart Services RapidStart Services is a new feature of Microsoft Dynamics NAV 2013. It allows you to configure your company using out of the box configurations. Microsoft Dynamics NAV 2013 also allows streamlined importing of opening balances into journals and active documents with dimensions. Typically, a Dynamics NAV implementation project can go on for a great length of time. With RapidStart Services, Dynamics NAV 2013 delivers a new way for partners and customers to speed up implementations. RapidStart Services is a tool designed not only to shorten deployment time but also to improve quality, to introduce a repeatable approach to implementations, and to automate and simplify recurring tasks. With RapidStart Services, you can set up the tables often involved in the configuration process of new companies. You can create a questionnaire to guide your customers through the collection of setup information. Your customers have the option of using the questionnaire to set up application areas, or they can open the setup page directly and complete the setup there. Most importantly, RapidStart Services helps you, as a customer, prepare the company with default setup data that you can fine-tune and customize. Lastly, when you use RapidStart Services, you can configure and migrate existing customer data, such as a list of customers or items, into the new company. The RapidStart Services tools can be found under the Department menu, Departments/Administration/Application Setup/RapidStart Services for Microsoft Dynamics NAV. The following components can be used to set up a new company: Configuration wizard Configuration packages Configuration worksheet Configuration templates Configuration questionnaire We will explain how these components work by following a step-by-step example of each of them. Before starting with the example, you need to create a new company, and we also recommend that you change your role to that of the RapidStart Services implementer. Follow these steps to do both these things: Open Microsoft Dynamics NAV Development Environment. Click on File | Company | New. A window opens; in it, you can enter the name of the new company. Click on the OK button and wait for a few seconds until the system finishes creating the company. Open the Windows client. Click on the the Dynamics icon found on the upper-right corner of the page. Then, click on the Select Company option. On the Select Company page, choose the company called New Company. Then click on the OK button. You have now entered in the New Company section. Follow the ensuing steps to change your role. Open the User Personalization page, found via Departments/ Administration/Application Setup/RoleTailored Client/ User personalization. Select your user ID and click on the Edit option. In the Profile ID field, select RAPIDSTART SERVICES. Close the Windows client and open it again. Your Role Center now looks like the following screenshot Now that we have a new company and we have selected the RapidStart Services implementer role, we are ready to use all the components of the RapidStart Services tool to set up our company. Configuration wizard The configuration wizard is used to quickly configure a new company. Click on the RapidStart Services Wizard option found on the ribbon bar. A new page will open where you will be able to enter basic information about the new company. You can go through all the tabs of the page entering the required information. The Select Package tab will be explained in the Configuration packages section, next. Configuration packages There are more than 200 tables that could be considered configuration tables. You will need to fill them in when you create a new company. First of all, you will find almost 50 tables with the word setup as their description, but there are many other tables that could also be considered as setup tables. Here you can see a list of some setup tables: Posting groups: There are 10 tables located here Journal batch tables and journal template tables: More than 20 tables are located here G/L accounts, account schedules and VAT statements: Almost 10 tables are present here. Payment terms, payment methods, currencies, languages, countries and regions, post codes, series, and so on are the other setup tables without the word setup in their description. Having to edit all those tables manually on each implementation could take forever. Many companies can use Dynamics NAV with the same data or almost the same data on those configuration tables. The best approach will be to create a configuration package for the data on the configuration tables and then apply it on each new implementation, like a template. You can create one configuration package per functional area, for example, you can create one package for the manufacturing functionality. Another approach would be to create one package for each type of data, for example, you could create one package with data related to all the posting groups found in the application. In this section we will see how to create a configuration package and also how to apply it to a new company. Creating a configuration package In this section we are going to create a new configuration package with all posting groups tables found on the application. Since posting groups refer to general ledger accounts, we are also going to include the chart of accounts in our package. Follow these steps to create the new configuration package: Select a company containing the data that you want to include in your configuration package, for example, we have selected the demonstration company CRONUS International Ltd. From the RapidStart Services implementer Role Center, click on the Packages option. Click on the New button on the ribbon bar. The Config.Package Cardpage opens. Fill in the fields in the General tab, as shown in the following screenshot: Add tables to the packages by creating new lines on the Tables tab, as shown in the following screenshot. You will only have to fill in the Table ID column. When you insert a table on the package, all table fields are included by default. In some cases, you may want to exclude certain fields from the package. Select the G/L Account table and click on Table | Fields. On the Config. Package Fields page, uncheck the Include Field column for the Global Dimension 1 Code field and the Global Dimension 2 Code field. Click on the Export Package option on the ribbon bar. This will create a RAPIDSTART file that you can save. Applying a configuration package In the previous section we created a new configuration package. In this section we are going to apply this package to the company New Company, that we created earlier in this article. Follow these steps to apply the configuration package: On the Windows client, open the company New Company. From the RapidStart Services implementer's role center, click on the RapidStart Services Wizard option. On the Select package tab, select the configuration package that you created in the previous section. Click on the Apply Package option found on the ribbon bar. Click on the Configuration Worksheet option if you want to edit the applied data check errors found during the importation. The configuration worksheet is explained in the next section. The data contained in the configuration data has been imported to the new company. You can also import packages from the packages page we saw while creating the configuration package. Configuration worksheet The configuration worksheet is the central location in which you can plan, track, and perform your configuration work. For those of you that have used previous versions of Dynamics NAV, the configuration worksheet is the old migration tool with some new features. You use the configuration worksheet to create the structure of tables that need to be imported with the company data. You will be able to export this structure to Microsoft Office Excel, fill in the data, and then import it b ack to Dynamics NAV. This makes it easy for companies to copy and paste informa tion from another ERP system. We'll explain how the configuration worksheet works by creating a migration structure for the sales area and then importing some demo data into the customer's table. Creating the migration structure To create a migration structure, you need to use a company that is already configured. We will use the demo company CRONUS International Ltd. Follow the steps described in this section: Open the configuration worksheet. Create a line for table 18, Customer. You only need to fill in the Line Type field and the Table ID field. Now, we have to put the tables into a configuration package. Click on the Assign Package option from the ribbon bar. The Config.Packages page opens. Create a new package and call it Customer. Then, click on the OK button. Now we have to define which fields will be included in the migration process.Click on the Fields option on the Actions tab of the ribbon bar. The Config. Package Fields page will open, showing all fields available in the table. In this list, we will have to place checkmarks on the Include Field column for all the fields we want to be part of the migration process. By default, all fields are checked. Click on the Clear Included option to uncheck all the fields. All primary key fields must be included in the migration process. The Include Fields checkmark cannot be unchecked for these fields. Select the following fields to be included. Then, click on the OK button. When importing the data into the table, fields will be validated according to their processing order. You can use the Move Up and Move Down options to change the default validation order. You can also uncheck the Validate Field column for a field if you don't want to run the OnValidate trigger of the field. If you do so, you will have to validate the data consistency on your own. We will now add a filter to determinate which records will be included in the template we are going to create later on. Click on the Filters option on the Actions tab of the ribbon bar. We will add a filter to the No. field to only include customers with a blank No. field. Since this is the primary key of the table, all customers should have filled up this field. No records will match the filter, so no records will be included while exporting the template. Add the filter as shown in the following screenshot: Back at the configuration worksheet; the Related Tables FactBox shows the tables that are related to the Customer table. The first related table is Payment Terms. We will not be able to fill in the payment terms code for a customer unless it exists on the Payment Terms table. Therefore, we need to include related tables in the configuration worksheet. Use the Get Related Tables option found on the Actions tab of the ribbon bar to add new tables to the worksheet. The system has included all the related tables. However for this example, delete all tables, except the ones shown in the following screenshot: We are now going to group the tables by adding areas and groups. Create a new line and select the Area option on the Line Type field. Write Sales Area in the Name field. Use the Move Up option from the Actions tab of the ribbon bar to move the new line to the first position. Create three new lines for the Master Data, Secondary Data, and General Settings groups. Use the Move Up and Move Down options to rearrange the tables as shown in the following screenshot: Now, we have to put the tables into a configuration package. Click on the Assign Package option from the ribbon bar. The Config. Packages page opens. Create a new package and call it Customer. Then, click on the OK button. Back at the configuration worksheet page; you can see that the Package Code field is now filled for all the rows. Click on Package Card, and then click on the Export Package option to save the package file for the migration structure we have defined. Now that you, as a partner, have defined the migration structure, it's time for the users to complete the migration by providing their data. Migrating your data The partners are responsible for creating the migration structure according to company's needs. We have done that in the previous section. Then, the rest of the work can be done by the end users. Therefore, you will need to train the end users on the steps explained in this section. To provide data for the new company, you can follow the ensuing steps: Open the company named New Company that we created earlier in this article. Open the package page and click on the Import Package option. Select the file created in the previous section. The data included in the package is stored in special tables. It will not be applied to the database yet. Open the Configuration Worksheet page. The system has created the configuration structure of the new company while importing the package file. Select the line for the table with the table-ID 3, Payment Terms. The Package Table FactBox shows us some information, as shown in the following screenshot: For instance, we can see that 6 records were included in the package. Click on the Package Data option on the ribbon bar. A page opens showing all the data included in the package. Before applying it to the database, you can delete or insert lines or modify the existing ones. Go back to the Config. Worksheet page and click on the Apply Data option. Now the payment terms are created on the database. You can repeat this action for the Payment Method table. Change the Status column for both tables and select the option Completed. On your Role Center page, you will be able to see the level of completion of the migration tasks. To migrate data for the Customer table, open the Config. Worksheet page and select the line for the Customer table. Click on the Export to Template option on the ribbon bar. The system will export the template, and will open the Excel file. We will use the information extracted from the old system to fil l in our Excel template. We are going to import the customers shown in the following screenshot: In the configuration worksheet, click on the Import from Template option and import your customers' file. Use the Package Data option to check the data and then click on the Apply Data option. Click on the Database Data option to see the records that have been created in the Customer table.

(For more resources related to this topic, see here.) Step 1 – Understanding vCloud resources This step will introduce you to how resources work in vCloud Director. The following diagram shows how resources are managed in vCloud and how they work together. The diagram is simplified and doesn't show all the vCloud properties; however, it is sufficient to explain the resource design. PvDC A Provider Virtual Data Center (PvDC) represents a portion of all the virtual resources of a vSphere environment. It will take all the CPU and memory resources from a given resource pool or cluster and present them to the vCloud as consumable resources. A typical cluster or resource pool contains multiple datastores, storage profiles, and networks as well as multiple hosts. A PvDC will automatically gain access to these resources. It is basically the link between vSphere and the vCloud world. Org An organization (Org) is a container that holds users and groups and regulates their access to the vCloud resources. Users can be either locally created or imported from Lightweight Directory Access Protocol (LDAP) or Active Directory (AD); however, groups can only be imported. It is possible to assign different LDAP, e-mail, and notification settings to each organization. This is one of the most powerful features of vCloud Director. Its usage becomes clear if you think about a public cloud model. You could link different organizations into the different customers' LDAP/ AD and e-mail systems (assuming a VPN tunnel between vCloud and the customer network), extending the customer's sphere of influence into the cloud. If a customer doesn't have or doesn't want to use his / her own LDAP/AD, he / she could make use of the local user function. OvDC An Organizational Virtual Data Center (OvDC) is a mixture of an Org with a PvDC. The Org defines who can do what and the PvDC defines where it is happening. Each OvDC is assigned one of the three allocation models as well as storage profiles. The three allocation models are designed to provide different methods of resource allocation. Let's first look at the difference between the models: Reservation pool: This allocates a fixed amount of resources (in GHz and GB) from the PvDC to the OvDC. This model is good if the users want to define a per-VM resource allocation. Only this model enables the Resource Allocation tab in VMs. Allocation pool: This is similar to reservation pool; however, you can also assign how many resources are guaranteed (reserved) for this OvDC. This model is good for overcommitting resources. Pay-as-you-go (PAYG): This is similar to the allocation pool; however, recourses are only consumed if vApps/VMs are running. All other models reserve resources even if the OvDC doesn't contain any running VMs. This model is useful if the number of resources is unknown or fluctuating. There are different settings that one can choose from for each model.   Allocation Pool PAYG Reservation Pool CPU allocation (GHz) Yes Yes and unlimited Yes CPU resources guaranteed (percentage) Yes Yes NA vCPU max speed (GHz) Yes Yes NA Memory allocation (GB) Yes Yes and unlimited Yes Memory resources guaranteed (percentage) Yes Yes NA Maximum number of VMs (number or unlimited) Yes Yes Yes vApp You might have encountered the name before in vCenter; however, the vApp of vCD and the vApp of vCenter are totally different beasts. vApps in vCenter are essentially resource pools with extras, such as a startup sequence. A vApp in vCD is a container that exists only in vCD. However, it can also contain isolated networks and allows the configuration of a start-and-stop sequence for its member VMs. In addition to all this, you can allow this vApp to be shared with other members of your organization. VM The most atomic part of a vCD is the VM. VMs live in vApps. Here you can configure all the settings you are familiar with from vSphere, and some more. You are able to add/update/delete vHardware as well as define guest customization. Step 2 – Connecting vCenter to vCD Let's start with the process of assigning resources to the vCloud. The first step is to assign a vCenter to this vCD installation. For future reference, one vCD installation can use multiple vCenters. As a starting point for steps 2 to 5, we will use the home screen, as shown in the following screenshot: On the Home screen (or if you like, the welcome screen), click on the first link, Attach a vCenter. A pop up will ask you for the following details for your vCenter: Host name or IP address: Enter the fully qualified domain name (FQDN) or IP address of your vCenter Port number: Port 443 is the correct default port User name and Password: Enter the username and password of an account that has administrator rights in vCenter vCenter name: Give your vCenter a name with which you would like to identify it in vCloud Description: A description isn't required; however, it doesn't hurt either vSphere Web Client URL: Now enter the URL for the web vCenter client https://(FDQN or IP)/vsphere-client After vCD has accepted the information and contacted vCenter, we now need to enter all the details for the vShield installation (from the Step 2 – downloading vCloud Director subsection in the Installation section). Enter the FQDN or IP address of the vShield VM And if you didn't change the default password, you can log in with admin as ID and default as the password vCD contacts vShield and that's that. You have now connected vCD to vCenter and are now able to use resources presented by this vCenter in your vCloud. Step 3 – Creating a PvDC Now we will create our first PvDC and assign resources to our vCloud. To create a new PvDC, you click on the second link, Create a Provider VDC (refer to the first image in the Step 2 – connecting vCenter to vCD subsection of the Quick start – creating your first VM section). Enter a name for the new PvCD. A good idea is to develop a naming standard for any item in vCenter and vCD. My PvDC will be called PvDC_myLab. Choose the highest supported virtual hardware version that your vCenter/ESXi supports. If you are running VMware 5.1, it is Version 9. In the next window, we choose the cluster or the resource pool that vCloud should use to create the PvDC. Please note that you need to create a resource pool before starting this wizard, or else it won't show up. For this example, I choose the cluster myCluster. In the next window, we are prompted to choose a storage profile. For the time being, just choose any and continue. Now vCD shows us all the ESXi hosts that belong to the cluster or the resource pool we selected. vCD will need to install some extra software on them and will need to connect directly to the ESXi hosts. That's why it is asking for the credentials of the ESXi hosts. Finish the wizard. At the end of this wizard, vCD will put the ESXi into maintenance mode to install the extra software package. If you only have one ESXi host and it is also running vCD and vCenter, you will have to manually install the vCD software package (not in the scope of this book). You have now successfully carved off a slice of resources to be used inside your vCloud. Storage profiles vSphere storage profiles are defined in vCenter. The idea is to group datastores together by their capabilities or by a user-defined label. For example, group datstores by their types (NFS, Fiber, SATA, or SSD), different RAID types, or by features that are provided, such as backup or replication. Enterprises use storage profiles such as gold, silver, and bronze, depending on the speed of the disks (SATA or SSD) and on whether a datastore is backed up or replicated for DR purposes. vCloud Director can assign different storage profiles to PvDCs and OvDCs. If an OvDC has multiple storage profiles assigned to it, you can choose a specific storage profile to be the default for this OvDC. Also, when you create a vApp in this OvDC, you can choose the storage profile with which you want to store the vApp. Step 4 – Creating an Org And now we will create an organization (Org). On the Home panel, click on the fifth link, Create a new organization (refer to the first image in the Step 2 – connecting vCenter to vCD subsection of the Quick start – creating your first VM section). Give the Org a name, for example, MyOrg and the organization's full name. In the next window, choose the first option, Do not use LDAP. Next, we could add a local user but we won't. So let's just click on Next. Our first Org should be able to share. So click on Allow publishing…, and then click on Next. We keep clicking on Next. The first Org will use the e-mail and notification settings of vCD. Now we need to configure the leases. You can just click on Next, or if you like, set all leases to unlimited. The last window shows us all the settings we have selected, and by clicking on Finish, our first organization will be created. System Org You have actually created a second Org as the first Org is called system and was created when we installed vCD. If you look at your home screen, you will see that there is a small tab that says System. The system Org is the mother of all Orgs. It's where other Orgs, PvDCs, OvDCs, and basically all settings are defined in vCloud Director. The system organization can only be accessed by vCloud system administrators. Step 5 – Creating an OvDC Now that we have our first Org, we can proceed with assigning resources to it for consumption. To do that, we need to create an Organization Virtual Data Center (OvDC). On the Home Screen, we click on the sixth link, Allocate resources to an organization. First we have to select the Org to which we want to assign the resources. As we only have one Org, the choice is easy. Next, we are asked which PvDC we want to take the resources from. Again, we only have one PvDC, so we choose that one. Note that the screen shows you what percentage of various resources of this PvDC are already committed and which networks are associated with this PvDC. Don't be alarmed that no networks are showing; we haven't configured any yet. Next we choose the allocation model. We have discussed the details of all the three models earlier: allocation pool, pay-as-you-go, and reservation pool. Choose Pay-as-you-go and click on Next. Have a look at the settings and click on Next. The next window lets you define which storage profile you would like to use for this OvDC. If you don't have a storage profile configured (as I do in my lab), just select any and click on the Add button. Enable Thin Provisioning to save on storage. This setting is the same as the normal thin provisioning in vSpere. Enable Fast Provisioning. This setting will use vCloud-linked clones (explained later). This window lets us configure the network resources for the organization. As we haven't configured any networking yet, just click on Next. We will discuss the network options in the next section about networks. We don't want to create an edge gateway so we leave the setting as it is and click Next. Again, more information about this is to follow in the next section. Finally, we will give this OvDC a name and finish the creation. I normally add a little descriptor in the name to say what allocation model I used, for example, res, payg, or allo. We have now successfully assigned memory, CPU, and storage to be consumed by our organization. Linked clones Linked clones save an enormous amount of storage space. When a VM is created from a template, a full clone of the template is created. When linked clones are used, only the changes to the VM are written to disk. As an example, we have a VM with 40 GB storage capacity (ignore thin provisioning for this example). A full clone would need another 40 GB of disk space. If linked clones are used, only a few MB will be used. As more changes are made to the cloned VM, it will demand more storage (up to the maximum of 40 GB). If this reminds you of the way snapshots work in vSphere, that's because that is what is actually used in the background. vCloud linked clones are not the same technology as VMware View linked clones; they are a more advanced version of the VMware Lab Manager linked clones. Step 6 – Creating a vApp Now that we have resources within our organization, we can create a vAPP and the VM inside it. vApps are created inside organizations, so we first need to access the organization that was created in the Step 4 – creating an Org subsection of the Quick start – creating your first VM section. Click on the Manage & Monitor tab and double-click on the Organizations menu item. Now double-click on the organization we created in the Step 4 – creating an Org subsection of the Quick start – creating your first VM section. You will see that a new tab is opened with the name of the new Org. You are now on the home screen of this Org. We will take the easy road here. Click on Build New vApp. Give your first vApp a name (for example, MyFirstVapp), a description, and if you like, explore the settings of the leases. After you click on Next, we are asked to choose a template. As we currently don't have one, we click on New Virtual Machine in the left-hand side menu of the screen. We will learn about templates in the Top features you need to know about section. A pop up will appear and we will then select all the settings we would expect when creating a new VM, such as name and hostname, CPU, memory, OS type and version, hard disk, and network. Note that if you are using virtual ESXi servers in your lab, you may be limited to 32-bit VMs only. After clicking on OK, we will find ourselves back at the previous screen. However, our VM should now show up in the lower table. Click on Next. We can now choose in which OvDC and in what storage profile we will deploy the vApp. The choices should be very limited at the moment, so just click on Next. Next, we are asked to choose a network. As we don't have one, we just click on Next. Another window will open; click on Next. Normally, we could define a fencing here. At last, we see a summary of all the settings, and clicking on Finish will create our first vApp. After the vApp is created, you can power it on and have a closer look. Click on the play button to power the vApp on. Wait for a few seconds, and then click on the black screen of the VM. A console pop up should come up and show you the BIOS of the booting VM. If that's not happening, check your browser security settings. That's it! You have installed vCD, and you've configured your resources and created your first vApp. Summary This article explained how we could create VM in vCloud technology. Resources for Article : Further resources on this subject: VMware View 5 Desktop Virtualization [Article] Supporting hypervisors by OpenNebula [Article] Tips and Tricks on BackTrack 4 [Article]

(For more resources related to this topic, see here.) Creating a new component package We are going to create a custom-logging component, and add it to the Misc tab of the component palette. To do this, we first need to create a new package and add out component to that package along with any other required resources, such as an icon for the component. To create a new package, do the following: Select package from the main menu. Select New Package.... from the submenu. Select a directory that appears in the Save dialog and create a new directory called MyComponents. Select the MyComponents directory. Enter MyComponents as the filename and press the Save button. Now, you have a new package that is ready to have components added to it. Follow these steps: On the Package dialog window, click on the add (+) button. Select the New Component tab. Select TComponent as Ancestor Type. Set New class name to TMessageLog. Set Palette Page to Misc. Leave all the other settings as they are. You should now have something similar to the following screenshot. If so, click on the Create New Component button: You should see messagelog.pas listed under the Files node in the Package dialog window. Let's open this file and see what the auto-generated code contains. Double-click on the file or choose Open file from More menu in the Package dialog. Do not name your component the same as the package. This will cause you problems when you compile the package later. If you were to do this, the .pas file would be over written, because the compile procedure creates a .pas file for the package automatically. The code in the Source Editor window is given as follows: unit TMessageLog;{$mode objfpc}{$H+}interfaceusesClasses, SysUtils, LResources, Forms, Controls, Graphics, Dialogs,StdCtrls;typeTMessageLog = class(TComponent)private{ Private declarations }protected{ Protected declarations }public{ Public declarations }published{ Published declarations }end;procedure Register;implementationprocedure Register;beginRegisterComponents('Misc',[TMessageLog]);end;end. What should stand out in the auto-generated code is the global procedure RegisterComponents. RegisterComponents is contained in the Classes unit. The procedure registers the component (or components if you create more than one in the unit) to the component page that is passed to it as the first parameter of the procedure. Since everything is in order, we can now compile the package and install the component. Click the Compile button on the toolbar. Once the compile procedure has been completed, select Install, which is located in the menu under the Use button. You will be presented with a dialog telling you that Lazarus needs to be rebuilt. Click on the Yes button, as shown in the following screenshot: The Lazarus rebuilding process will take some time. When it is complete, it will need to be restarted. If this does not happen automatically, then restart Lazarus yourself. On restarting Lazarus, select the Misc tab on the component palette. You should see the new component as the last component on the tab, as shown in the following screenshot: You have now successfully created and installed a new component. You can now create a new application and add this component to a Lazarus form. The component in its current state does not perform any action. Let us now look at adding properties and events to the component that will be accessible in the Object Inspector window at design time. Adding properties Properties of a component that you would like to have visible in the Object Inspector window must be declared as published. Properties are attributes that determine an object's status and behavior. A property is a name that is mapped to read and write methods or access data directly. This means, when you read or write a property, you are accessing a field or calling a method of the object. For example, let us add a FileName property to TMessageLog, which is the name of the file that messages will be written to. The actual field of the object that will store this data will be named fFileName. To the TMessageLog private declaration section, add: fFileName: String; To the TMessagLog published declaration section, add: property FileName: String read fFileName write fFileName; With these changes, when the packages are compiled and installed, the property FileName will be visible in the Object Inspector window when the TMessageLog declaration is added to a form in a project. You can do this now if you would like to verify this. Adding events Any interaction that a user has with a component, such as clicking it, generates an event. Events are also generated by the system in response to a method call or a change in a component's property, or if different component's property changes, such as the focus being set on one component causes the current component in focus to lose it, which triggers an event call. Event handlers are methods of the form containing the component; this technique is referred to as delegation. You will notice that when you double-click on a component's event in the object inspector it creates a new procedure of the form. Events are properties, and such methods are assigned to event properties, as we just saw with normal properties. Because events are the properties and use of delegation, multiple events can share the same event handler. The simplest way to create an event is to define a method of the type TNotifyEvent. For example, if we want to add an OnChange event to TMessageLog, we could add the following code: ...privateFonChange : TNotifyEvent;...publicproperty OnChange: TNotifyEvent read FOnChange write FOnChange;…end; When you double-click on the OnChange event in Object Inspector, the following method stub would be created in the form containing the TMessageLog component: procedure TForm.MessageLogChange(Sender: TObject);beginend; Some properties, such as OnChange or OnFocus, are sometimes called on the change of value of a component's property or the firing of another event. Traditionally, in this case, a method with the prefix of Do and with the suffix of the On event are called. So, in the case of our OnChange event, it would be called from the DoChange method (as called by some other method). Let us assume that, when a filename is set for the TMessageLog component, the procedure SetFileName is called, and that calls DoChange. The code would look as follows: procedure SetFileName(name : string);beginFFileName = name;//fire the eventDoChange;end;procedure DoChange;beginif Assigned(FOnChange) thenFOnChange(Self);end; The DoChange procedure checks to see if anything has been assigned to the FOnChange field. If it is assigned, then it executes what is assigned to it. What this means is that if you double-click on the OnChange event in Object Inspector, it assigns the method name you enter to FOnChange, and this is the method that is called by DoChange. Events with more parameters You probably noticed that the OnChange event only had one parameter, which was Sender and is of the type Object. Most of the time, this is adequate, but there may be times when we want to send other parameters into an event. In those cases, TNotifyEvent is not an adequate type, and we will need to define a new type. The new type will need to be a method pointer type, which is similar to a procedural type but has the keyword of object at the end of the declaration. In the case of TMessageLog, we may need to perform some action before or after a message is written to the file. To do this, we will need to declare two method pointers, TBeforeWriteMsgEvent and TAfterWriteMsgEvent, both of which will be triggered in another method named WriteMessage. The modification of our code will look as follows: typeTBeforeWriteMsgEvent = procedure(var Msg: String; var OKToWrite:Boolean) of Object;TAfterWriteMsgEvent = procedure(Msg: String) of Object;TmessageLog = class(TComponent)…publicfunction WriteMessage(Msg: String): Boolean;...publishedproperty OnBeforeWriteMsg: TBeforeWriteMsgEvent read fBeforeWriteMsgwrite fBeforeWriteMsg;property OnAfterWriteMsg: TAfterWriteMsgEvent read fAfterWriteMsgwrite fAfterWriteMsg;end;implementationfunction TMessageLog.WriteMessage(Msg: String): Boolean;varOKToWrite: Boolean;beginResult := FALSE;OKToWrite := TRUE;if Assigned(fBeforeWriteMsg) thenfBeforeWriteMsg(Msg, OKToWrite);if OKToWrite thenbegintryAssignFile(fLogFile, fFileName);if FileExists(fFileName) thenAppend(fLogFile)elseReWrite(fLogFile);WriteLn(fLogFile, DateTimeToStr(Now()) + ' - ' + Msg);if Assigned(fAfterWriteMsg) thenfAfterWriteMsg(Msg);Result := TRUE;CloseFile(fLogFile);exceptMessageDlg('Cannot write to log file, ' + fFileName + '!',mtError, [mbOK], 0);CloseFile(fLogFile);end; // try...exceptend; // ifend; // WriteMessage While examining the function WriteMessage, we see that, before the Msg parameter is written to the file, the FBeforeWriteMsg field is checked to see if anything is assigned to it, and, if so, the write method of that field is called with the parameters Msg and OKToWrite. The method pointer TBeforeWriteMsgEvent declares both of these parameters as var types. So if any changes are made to the method, the changes will be returned to WriteMessage function. If the Msg parameter is successfully written to the file, the FAfterWriteMsg parameter is checked for assigned and executed parameter (if it is). The file is then closed and the function's result is set to True. If the Msg parameter value is not able to be written to the file, then an error dialog is shown, the file is closed, and the function's result is set to False. With the changes that we have made to the TMessageLog unit, we now have a functional component. You can now save the changes, recompile, reinstall the package, and try out the new component by creating a small application using the TMessageLog component. Property editors Property editors are custom dialogs for editing special properties of a component. The standard property types, such as strings, images, or enumerated types, have default property editors, but special property types may require you to write custom property editors. Custom property editors must extend from the class TPropertyEditor or one of its descendant classes. Property editors must be registered in the Register procedure using the function RegisterPropertyEditor from the unit PropEdits. An example of property editor class declaration is given as follows: TPropertyEditor = classpublicfunction AutoFill: Boolean; Virtual;procedure Edit; Virtual; // double-clicking the property value toactivateprocedure ShowValue; Virtual; //control-clicking the propertyvalue to activatefunction GetAttributes: TPropertyAttributes; Virtual;function GetEditLimit: Integer; Virtual;function GetName: ShortString; Virtual;function GetHint(HintType: TPropEditHint; x, y: integer): String;Virtual;function GetDefaultValue: AnsiString; Virtual;function SubPropertiesNeedsUpdate: Boolean; Virtual;function IsDefaultValue: Boolean; Virtual;function IsNotDefaultValue: Boolean; Virtual;procedure GetProperties(Proc: TGetPropEditProc); Virtual;procedure GetValues(Proc: TGetStrProc); Virtual;procedure SetValue(const NewValue: AnsiString); Virtual;procedure UpdateSubProperties; Virtual;end; Having a class as a property of a component is a good example of a property that would need a custom property editor. Because a class has many fields with different formats, it is not possible for Lazarus to have the object inspector make these fields available for editing without a property editor created for a class property, as with standard type properties. For such properties, Lazarus shows the property name in parentheses followed by a button with an ellipsis (…) that activates the property editor. This functionality is handled by the standard property editor called TClassPropertyEditor, which can then be inherited to create a custom property editor, as given in the following code: TClassPropertyEditor = class(TPropertyEditor)publicconstructor Create(Hook: TPropertyEditorHook; APropCount: Integer);Override;function GetAttributes: TPropertyAttributes; Override;procedure GetProperties(Proc: TGetPropEditProc); Override;function GetValue: AnsiString; Override;property SubPropsTypeFilter: TTypeKinds Read FSubPropsTypeFilterWrite SetSubPropsTypeFilterDefault tkAny;end; Using the preceding class as a base class, all you need to do to complete a property editor is add a dialog in the Edit method as follows: TMyPropertyEditor = class(TClassPropertyEditor)publicprocedure Edit; Override;function GetAttributes: TPropertyAttributes; Override;end;procedure TMyPropertyEditor.Edit;varMyDialog: TCommonDialog;beginMyDialog := TCommonDialog.Create(NIL);try…//Here you can set attributes of the dialogMyDialog.Options := MyDialog.Options + [fdShowHelp];...finallyMyDialog.Free;end;end; Component editors Component editors control the behavior of a component when double-clicked or right-clicked in the form designer. Classes that define a component editor must descend from TComponentEditor or one of its descendent classes. The class should be registered in the Register procedure using the function RegisterComponentEditor. Most of the methods of TComponentEditor are inherited from it's ancestor TBaseComponentEditor, and, if you are going to write a component editor, you need to be aware of this class and its methods. Declaration of TBaseComponentEditor is as follows: TBaseComponentEditor = classprotectedpublicconstructor Create(AComponent: TComponent;ADesigner: TComponentEditorDesigner); Virtual;procedure Edit; Virtual; Abstract;procedure ExecuteVerb(Index: Integer); Virtual; Abstract;function GetVerb(Index: Integer): String; Virtual; Abstract;function GetVerbCount: Integer; Virtual; Abstract;procedure PrepareItem(Index: Integer; const AnItem: TMenuItem);Virtual; Abstract;procedure Copy; Virtual; Abstract;function IsInInlined: Boolean; Virtual; Abstract;function GetComponent: TComponent; Virtual; Abstract;function GetDesigner: TComponentEditorDesigner; Virtual;Abstract;function GetHook(out Hook: TPropertyEditorHook): Boolean;Virtual; Abstract;procedure Modified; Virtual; Abstract;end; Let us look at some of the more important methods of the class. The Edit method is called on the double-clicking of a component in the form designer. GetVerbCount and GetVerb are called to build the context menu that is invoked by right-clicking on the component. A verb is a menu item. GetVerb returns the name of the menu item. GetVerbCount gets the total number of items on the context menu. The PrepareItem method is called for each menu item after the menu is created, and it allows the menu item to be customized, such as adding a submenu or hiding the item by setting its visibility to False. ExecuteVerb executes the menu item. The Copy method is called when the component is copied to the clipboard. A good example of a component editor is the TCheckListBox component editor. It is a descendant from TComponentEditor so all the methods of the TBaseComponentEditor do not need to be implemented. TComponentEditor provides empty implementation for most methods and sets defaults for others. Using this, methods that are needed for the TCheckListBoxComponentEditor component are overwritten. An example of the TCheckListBoxComponentEditor code is given as follows: TCheckListBoxComponentEditor = class(TComponentEditor)protectedprocedure DoShowEditor;publicprocedure ExecuteVerb(Index: Integer); override;function GetVerb(Index: Integer): String; override;function GetVerbCount: Integer; override;end;procedure TCheckGroupComponentEditor.DoShowEditor;varDlg: TCheckGroupEditorDlg;beginDlg := TCheckGroupEditorDlg.Create(NIL);try// .. shortenedDlg.ShowModal;// .. shortenedfinallyDlg.Free;end;end;procedure TCheckGroupComponentEditor.ExecuteVerb(Index: Integer);begincase Index of0: DoShowEditor;end;end;function TCheckGroupComponentEditor.GetVerb(Index: Integer): String;beginResult := 'CheckBox Editor...';end;function TCheckGroupComponentEditor.GetVerbCount: Integer;beginResult := 1;end; Summary In this article, we learned how to create a new Lazarus package and add a new component to that using the New Package dialog window to create our own custom component, TMessageLog. We also learned about compiling and installing a new component into the IDE, which requires Lazarus to rebuild itself in order to do so. Moreover, we discussed component properties. Then, we became acquainted with the events, which are triggered by any interaction that a user has with a component, such as clicking it, or by a system response, which could be caused by the change in any component of a form that affects another component. We studied that Events are properties, and they are handled through a technique called delegation. We discovered the simplest way to create an event is to create a descendant of TNotifyEvent—if you needed to send more parameters to an event and a single parameter provided by TNotifyEvent, then you need to declare a method pointer. We learned that property editors are custom dialogs for editing special properties of a component that aren't of a standard type, such as string or integer, and that they must extend from TPropertyEditor. Then, we discussed the component editors, which control the behavior of a component when it is right-clicked or double- clicked in the form designer, and that a component editor must descend from TComponentEditor or a descendant class of it. Finally, we looked at an example of a component editor for the TCheckListBox. Resources for Article : Further resources on this subject: User Extensions and Add-ons in Selenium 1.0 Testing Tools [Article] 10 Minute Guide to the Enterprise Service Bus and the NetBeans SOA Pack [Article] Support for Developers of Spring Web Flow 2 [Article]

(For more resources related to this topic, see here.) Understanding the Oracle RDBMS The term RDBMS, or Relational Database Management System can be used in many ways and, more often than not, simply refers to a commercial database management product. However, the true meaning of the term RDBMS refers to the way the product operates internally to store and retrieve data. There are many commercial and open source database management systems available today. While they all attempt to interface with the user in a manner that is generally compliant with international standards, they all operate differently internally. We will explore the characteristics that make the Oracle RDBMS unique and, in doing so, understand how it works. Distinguishing between an instance and a database To define the Oracle database architecture, we must first define two terms and make a distinction between them. An instance is the set of background processes and memory structures that enable the Oracle kernel to operate. A database is the set of files that stores the data contained in the RDBMS. These two entities account for the three basic resources available to computers, namely the CPU, RAM, and disk. The instance represents the usage of the CPU and RAM, while the database represents the usage of the disk. These definitions are architecturally oriented, and the distinction between the two isn't always fully recognized. In fact, the two are often used interchangeably. In most database configurations, it is reasonable to do so since, on a single server, the instance and database operate together. However, it's important to draw a distinction at this point, for three reasons. First, the terms themselves are used differently with other RDBMS products, such as Microsoft SQL Server. Secondly, when investigating the Oracle architecture, it is simpler to see the relationship between the various architectural components if we divide them accordingly. Lastly, although an instance and a database generally operate together on a single server, other database configurations are possible. In Oracle's Real Application Clusters, or RAC, the instance and database are both logically and physically separated. We will examine the subject of RAC later in the article. The following diagram gives us a broad look of the Oracle architecture as a whole. We will refer back to portions of it throughout the article: Note that this is a logical diagram that groups similar functions together. We display this as a reverse tree diagram. The top of the tree is the RDBMS itself. From there, it branches into the instance and the database, the terms we have just defined. We will look at each branch of this diagram and examine the individual components, as well as how they operate and contribute to the inner working of the Oracle RDBMS. Examining the Oracle instance As we mentioned before, the Oracle instance is composed of the necessary processes and memory structures required to operate the Oracle kernel. The instance is the first thing invoked when the database is started. Recognizing the primary Oracle processes The first components of the Oracle instance that we will examine are the Oracle background processes. These processes run in the background of the operating system and are not interacted with directly. Each process is highly specialized and has a specific function in the overall operation of the Oracle kernel. While these processes accomplish the same functions regardless of the host operating system, their implementation is significantly different. On Unix-based systems, owing to Unix's multiprocess architecture, each Oracle process runs as a separate operating system process. Thus, we can actually see the processes themselves from within the operating system. For instance, we can use the ps command on Linux to see these processes, as shown in the following screenshot. We've highlighted a few of them that we will examine in depth. Note that our background processes are named in the format ora_ processtype_SID. Since the SID for our database is ORCL, that name forms a part of the full process name: On Windows, rather than implementing each Oracle process as a separate OS process, the Oracle processes are implemented as threads since Windows is a multithreaded operating system. As a result, the Oracle kernel runs under a single executable called Oracle.exe. The background processes then run as threads under that single process. So, if we attempt to see the Oracle background processes using Windows's Task Manager, we only see the Oracle.exe executable, as shown in the next screenshot. The threads representing the Oracle processes are masked to us, at least at the operating system level: We can also display this information from the Windows command line by running the tasklist command: Regardless of whether we can see each process directly from the operating system, once the instance starts, they are present. Each has a specific job and all run in concert to service the needs of the database. PMON – the Process Monitor The core process of the Oracle architecture is the PMON process—the Process Monitor. The PMON is tasked with monitoring and regulating all other Oracle-related processes. This includes not only background processes but server processes as well. Most databases run in a dedicated server mode. In this mode, any user that connects to the database is granted a server process with which to do work. In Linux systems, this process can actually be viewed at the server level with the ps -ef command. When the user connects over the network, the process will be labeled with LOCAL=NO in the process description. Privileged users such as database administrators can also make an internal connection to the database, provided that we are logging in from the server that hosts the database. When an internal connection is made, the process is labeled with LOCAL=YES. We see an example of each in the following screenshot of the ps –ef command on a Linux machine hosting Oracle: Under ordinary circumstances, when a user properly disconnects his or her session from the database by exiting the tool used to connect to it, the server process given to that user terminates cleanly. However, what if instead of disconnecting the connection properly, the machine that the user was connected to was rebooted? In situations like these, the server process on the database is left running since it hasn't received the proper instructions to terminate. When this occurs, it is the job of PMON to monitor sessions and clean up orphaned processes. The PMON normally "wakes up" every 3 seconds to check these processes and clean them up. In addition to this primary function, PMON is also responsible for registering databases with network listeners. The real-world DBA Since the instance cannot run unless PMON is running, DBAs sometimes check for it using the ps command as a way of determining whether the instance is down, because, on Unix-based systems, we can actually see the processes at the server level using the command ps –ef | grep pmon. If a process is not returned, we know the instance is down. SMON – the System Monitor The SMON, or System Monitor process, has several very important duties. Chiefly SMON is responsible for instance recovery. Under normal circumstances, databases are shut down using the proper commands to do so. When this occurs, all of the various components, mainly the datafiles, are properly recorded and synchronized so that the database is left in a consistent state. However, if the database crashes for some reason (the database's host machine loses power, for instance), this synchronization cannot occur. When the database is restarted, it will begin from an inconsistent state. Every time the instance is started, SMON will check for these marks of synchronization. In a situation where the database is in an inconsistent state, SMON will perform instance recovery to resynchronize these inconsistencies. Once this is complete, the instance and database can open correctly. Unlike database recovery, where some data loss has occurred, instance recovery occurs without intervention from the DBA. It is an automatic process that is handled by SMON. The SMON process is also responsible for various cleanup operations within the datafiles themselves. tempfiles are the files that hold the temporary data that is written when an overflow from certain memory caches occurs. This temporary data is written in the form of temporary segments within the tempfile. When this data is no longer needed, SMON is tasked with removing them. The SMON process can also coalesce data within datafiles, removing gaps, which allows the data to be stored more efficiently. DBWn – the Database Writer process For all of the overhead duties of processes such as PMON and SMON, we can probably intuit that there must be a process that actually reads and writes data from the datafiles. Until later versions, that process was named DBWR – the Database Writer process. The DBWR is responsible for reading and writing the data that services user operations, but it doesn't do it in the way that we might expect. In Oracle, almost no operation is executed directly on the disk. The Oracle processing paradigm is to read data into memory, complete a given operation while the data is still in memory, and write it back to the disk. We will cover the reason for this in greater depth when we discuss memory caches, but for now let's simply say it is for performance reasons. Thus, the DBWR process will read a unit of data from the disk, called a database block, and place it into a specialized memory cache. If data is changed using an UPDATE statement, for instance, it is changed in memory. After some time, it is written back to the disk in its new state. If we think about it, it should be obvious that the amount of reading and writing in a database would constitute a great deal of work for one single process. It is certainly possible that a single DBWR process would become overloaded and begin to affect performance. That's why, in more recent versions of Oracle, we have the ability to instantiate multiple database writer processes. So we can refer to DBWR as DBWn, where "n" is a given instantiation of a database writer process. If our instance is configured to spawn three database writers, they would be dbw0, dbw1, and dbw2. The number of the DBWn processes that are spawned is governed by one of our initialization parameters, namely, db_writer_processes. Let's take a closer look at how the value for db_writer_processes affects the database writer processes that we can see in the Linux operating system. We won't go into great depth with the commands that we'll be using at this point, but we can still see how the spawning of multiple DBWn processes works. We will become very familiar with commands such as these as we revisit them frequently throughout many of the examples in this book. First, let's examine the number of DBWn processes on our system using the ps command, with which we're familiar: From the Linux command line, we use the ps –ef command along with the grep command that searches through the processes in the system with the string dbw in their names. This restricts our output to only those processes that contain dbw, which will be the database writer processes. As we can see in the preceding screenshot, there is only one database writer process named ora_dbw0_orcl. As mentioned, the number of the database writer processes is determined by an initialization parameter. The name of that parameter is db_writer_processes.We can determine the value of this parameter by logging into the database using SQL*Plus (the command sqlplus / as sysdba) and showing its value using the show parameter command, as in the following screenshot: Since we've already determined that we only have a single dbw0 process, it should come as no surprise that the value for our parameter is 1. However, if we wish to add more database writers, it is simple to do so. From the SQL*Plus command line, we issue the following command, followed by the shutdown immediate and startup commands to shut down and start up the database: The alter system command instructs Oracle to set the db_writer_processes parameter to 4. The change is recognized when the database is restarted. From here, we type exit to leave SQL*Plus and return to the Linux command line. We then issue our ps command again and view the results: As we can see in the preceding screenshot, there are four database writer processes, called ora_dbw0_orcl, ora_dbw1_orcl, ora_dbw2_orcl, and ora_dbw3_orcl, that align with our value for db_writer_processes. We now have four database writer processes with which to read and write data. The real-world DBA What's the optimal number of database writers? The answer is that, as with many aspects of database administration, it depends. The parameter has a maximum value of 20, so does that mean more is better? Not necessarily. The simplest answer is that the default value, either 1 or the integer value resulting from the number of CPUs divided by 8 (whichever is greater), will generally provide the best performance. Most opinions regarding best practices vary greatly and are usually based on the number of CPUs in the host box. Generally, the default value will serve you well unless your server is very large or heavy tuning is needed. CKPT – the Checkpoint process We mentioned in the preceding section that the purpose of the DBWn process is to move data in and out of memory. Once a block of data is moved into memory, it is referred to as a buffer. When a buffer in memory is changed using an UPDATE statement, for instance, it is called a dirty buffer. Dirty buffers can remain in memory for a time and are not automatically flushed to disk. The event that signals the writing of dirty buffers to disk is known as a checkpoint. The checkpoint ensures that memory is kept available for other new buffers and establishes a point for recovery. In earlier versions of Oracle, the type of checkpoint that occurred was known as a full checkpoint. This checkpoint will flush all dirty buffers back to the datafiles on the disk. While full checkpoints represent a complete flush of the dirty buffers, they are expensive in terms of performance. Since Version 8i, the Oracle kernel makes use of an incremental checkpoint that intelligently flushes only part of the available dirty buffers when needed. Full checkpoints only occur now during a shutdown of the database or on demand, using a command. The process in the instance that orchestrates checkpointing is the CKPT process. The CKPT process uses incremental checkpoints at regular intervals to ensure that dirty buffers are written out and any changes recorded in the redo logs are kept consistent for recovery purposes. Unlike the DBWn process, there is only one CKPT process. Although the incremental checkpoint method is used by CKPT, we can also force a full checkpoint using the command shown in the following screenshot: LGWR – the Log Writer process Redo logs are files that serially store the changes that occur in the database. These changes can be anything from INSERT, DELETE, or UPDATE statements executed against the database to the creation of new tables. Note, however, that queries against the database using SELECT statements do not constitute changes and are not recorded in the redo logs. The primary function of redo logs is to act during database recovery, where database changes can be "rolled forward" from a backup. When a change occurs in the database, that change is first written into a memory buffer called the log buffer that is specifically tasked with handling database changes. It is then written into the redo logs. The process that writes changes from memory to disk is the LGWR process. There is only one LGWR process, so it is important that it moves data as efficiently as possible. There are three conditions that can occur to cause LGWR to move changes from memory into the redo logfiles: When a COMMIT command is issued When the log buffer fills to one-third of its capacity Every three seconds The last of these conditions, "every three seconds," is actually caused by the DBWn process. The database writer has a three-second timeout before it must write a limited number of dirty buffers to disk. When this occurs, the LGWR process also flushes its changes just before DBWn does its work. This ensures that uncommitted transactions can always be rolled back. ARCn – the Archiver process The ARCH, or the Archiver process, is an optional but very important process. Let's continue describing the life cycle of a change in Oracle. To review, as sessions change data, those changes are written into the log buffer. At periodic intervals, those changes are written out serially to the redo logs by the LGWR process. The number and size of the redo logs can vary, but there are always a limited number of them. When one redo log becomes full, LGWR switches to the next one. Since there are a finite number of redo logs, LGWR eventually fills them all. When that happens, LGWR switches back to the original redo log and overwrites the changes that exist in the log. This effectively destroys those changes and invalidates the purpose of storing the changes at all. When the database operates in this manner, overwriting changes that were stored in the redo logs, we say that we're operating in NOARCHIVELOG mode. In order to prevent changes from being overwritten, we must operate using a different mode—ARCHIVELOG mode. When the database is in ARCHIVELOG mode, the contents of the redo logs are written out to a different type of file called archive logs. Archive logs simply contain the data that was in the redo log and serve as its static copy. However, archive logs never overwrite each other. Thus, the history of changes stored in the archive logs and redo logs constitute all the changes that have occurred from a given point in time. The process that enables ARCHIVELOG mode is the Archiver process, or ARCH. The ARCH copies data from the redo logs to the archive logs and does so before the data in the redo logs can be overwritten. Like DBWn, the Archiver process can run as multiple processes, albeit with a different purpose. In previous versions of Oracle, the Archiver process ran as a single process— ARCH. In recent versions, it is more accurate to call the Archiver process by its proper name, ARCn, where "n" is the number for one of multiple Archiver processes. When ARCn runs as multiple processes, it does so in order to write to multiple locations, rather than using multiple processes to write to a single location. Since, along with the redo logs, the archive logs contain all of the changes that have occurred in the database, they are critical to database recovery. As such, we can configure our database to write out archive log copies to more than one location. For instance, we may want to configure the Archiver process to write archive logs to two different locations on disk or write a copy of them out to a tape drive to mitigate the risk of data loss.

(For more resources related to this topic, see here.) Counting distinct IPs in weblog data using MapReduce and Combiners This recipe will walk you through creating a MapReduce program to count distinct IPs in weblog data. We will demonstrate the application of a combiner to optimize data transfer overhead between the map and reduce stages. The code is implemented in a generic fashion and can be used to count distinct values in any tab-delimited dataset. Getting ready This recipe assumes that you have a basic familiarity with the Hadoop 0.20 MapReduce API. You will need access to the weblog_entries dataset supplied with this book and stored in an HDFS folder at the path /input/weblog. You will need access to a pseudo-distributed or fully-distributed cluster capable of running MapReduce jobs using the newer MapReduce API introduced in Hadoop 0.20. You will also need to package this code inside a JAR file to be executed by the Hadoop JAR launcher from the shell. Only the core Hadoop libraries are required to compile and run this example. How to do it... Perform the following steps to count distinct IPs using MapReduce: Open a text editor/IDE of your choice, preferably one with Java syntax highlighting. Create a class named DistinctCounterJob.java in your JAR file at whatever source package is appropriate. The following code will serve as the Tool implementation for job submission: import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; import java.io.IOException; import java.util.regex.Pattern; public class DistinctCounterJob implements Tool { private Configuration conf; public static final String NAME = "distinct_counter"; public static final String COL_POS = "col_pos"; public static void main(String[] args) throws Exception { ToolRunner.run(new Configuration(), new DistinctCounterJob(), args); } The run() method is where we set the input/output formats, mapper class configuration, combiner class, and key/value class configuration: public int run(String[] args) throws Exception { if(args.length != 3) { System.err.println("Usage: distinct_counter <input> <output> <element_position>"); System.exit(1); } conf.setInt(COL_POS, Integer.parseInt(args[2])); Job job = new Job(conf, "Count distinct elements at position"); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); job.setMapperClass(DistinctMapper.class); job.setReducerClass(DistinctReducer.class); job.setCombinerClass(DistinctReducer.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class); job.setJarByClass(DistinctCounterJob.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); return job.waitForCompletion(true) ? 1 : 0; } public void setConf(Configuration conf) { this.conf = conf; } public Configuration getConf() { return conf; } } The map() function is implemented in the following code by extending mapreduce.Mapper: public static class DistinctMapper extends Mapper<LongWritable, Text, Text, IntWritable> { private static int col_pos; private static final Pattern pattern = Pattern. compile("t"); private Text outKey = new Text(); private static final IntWritable outValue = new IntWritable(1); @Override protected void setup(Context context ) throws IOException, InterruptedException { col_pos = context.getConfiguration(). getInt(DistinctCounterJob.COL_POS, 0); } @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String field = pattern.split(value.toString())[col_ pos]; outKey.set(field); context.write(outKey, outValue); } } The reduce() function is implemented in the following code by extending mapreduce.Reducer: public static class DistinctReducer extends Reducer<Text, IntWritable, Text, IntWritable> { private IntWritable count = new IntWritable(); @Override protected void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int total = 0; for(IntWritable value: values) { total += value.get(); } count.set(total); context.write(key, count); } } The following command shows the sample usage against weblog data with column position number 4, which is the IP column: hadoop jar myJobs.jar distinct_counter /input/weblog/ /output/ weblog_distinct_counter 4 How it works... First we set up DistinctCounterJob to implement a Tool interface for remote submission. The static constant NAME is of potential use in the Hadoop Driver class, which supports the launching of different jobs from the same JAR file. The static constant COL_POS is initialized to the third required argument from the command line <element_position>. This value is set within the job configuration, and should match the position of the column you wish to count for each distinct entry. Supplying 4 will match the IP column for the weblog data. Since we are reading and writing text, we can use the supplied TextInputFormat and TextOutputFormat classes. We will set the Mapper and Reduce classes to match our DistinctMapper and DistinctReducer implemented classes respectively. We also supply DistinctReducer as a combiner class. This decision is explained in more detail as follows: It's also very important to call setJarByClass() so that the TaskTrackers can properly unpack and find the Mapper and Reducer classes. The job uses the static helper methods on FileInputFormat and FileOutputFormat to set the input and output directories respectively. Now we're set up and ready to submit the job. The Mapper class sets up a few member variables as follows: col_pos: This is initialized to a value supplied in the configuration. It allows users to change which column to parse and apply the count distinct operation on. pattern: This defines the column's split point for each row based on tabs. outKey: This is a class member that holds output values. This avoids having to create a new instance for each output that is written. outValue: This is an integer representing one occurrence of the given key. It is similar to the WordCount example. The map() function splits each incoming line's value and extracts the string located at col_ pos. We reset the internal value for outKey to the string found on that line's position. For our example, this will be the IP value for the row. We emit the value of the newly reset outKey variable along with the value of outValue to mark one occurrence of that given IP address. Without the assistance of the combiner, this would present the reducer with an iterable collection of 1s to be counted. The following is an example of a reducer {key, value:[]} without a combiner: {10.10.1.1, [1,1,1,1,1,1]} = six occurrences of the IP "10.10.1.1". The implementation of the reduce() method will sum the integers and arrive at the correct total, but there's nothing that requires the integer values to be limited to the number 1. We can use a combiner to process the intermediate key-value pairs as they are output from each mapper and help improve the data throughput in the shuffle phase. Since the combiner is applied against the local map output, we may see a performance improvement as the amount of data we need to transfer for an intermediate key/value can be reduced considerably. Instead of seeing {10.10.1.1, [1,1,1,1,1,1]}, the combiner can add the 1s and replace the value of the intermediate value for that key to {10.10.1.1, [6]}. The reducer can then sum the various combined values for the intermediate key and arrive at the same correct total. This is possible because addition is both a commutative and associative operation. In other words: Commutative: The order in which we process the addition operation against the values has no effect on the final result. For example, 1 + 2 + 3 = 3 + 1 + 2. Associative: The order in which we apply the addition operation has no effect on the final result. For example, (1 + 2) + 3 = 1 + (2 + 3). For counting the occurrences of distinct IPs, we can use the same code in our reducer as a combiner for output in the map phase. When applied to our problem, the normal output with no combiner from two separate independently running map tasks might look like the following where {key: value[]} is equal to the intermediate key-value collection: Map Task A = {10.10.1.1, [1,1,1]} = three occurrences Map Task B = {10.10.1.1, [1,1,1,1,1,1]} = six occurrences Without the aid of a combiner, this will be merged in the shuffle phase and presented to a single reducer as the following key-value collection: {10.10.1.1, [1,1,1,1,1,1,1,1,1]} = nine total occurrences Now let's revisit what would happen when using a Combiner against the exact same sample output: Map Task A = {10.10.1.1, [1,1,1]} = three occurrences Combiner = {10.10,1,1, [3] = still three occurrences, but reduced for this mapper. Map Task B = {10.10.1.1, [1,1,1,1,1,1] = six occurrences Combiner = {10.10.1.1, [6] = still six occurrences Now the reducer will see the following for that key-value collection: {10.10.1.1, [3,6]} = nine total occurrences We arrived at the same total count for that IP address, but we used a combiner to limit the amount of network I/O during the MapReduce shuffle phase by pre-reducing the intermediate key-value output from each mapper. There's more... The combiner can be confusing to newcomers. Here are some useful tips: The Combiner does not always have to be the same class as your Reducer The previous recipe and the default WordCount example show the Combiner class being initialized to the same implementation as the Reducer class. This is not enforced by the API, but ends up being common for many types of distributed aggregate operations such as sum(), min(), and max(). One basic example might be the min() operation of the Reducer class that specifically formats output in a certain way for readability. This will take a slightly different form from that of the min() operator of the Combiner class, which does not care about the specific output formatting. Combiners are not guaranteed to run Whether or not the framework invokes your combiner during execution depends on the intermediate spill file size from each map output, and is not guaranteed to run for every intermediate key. Your job should not depend on the combiner for correct results, it should be used only for optimization. You can control the spill file threshold when MapReduce tries to combine intermediate values with the configuration property min.num.spills.for.combine. Using Hive date UDFs to transform and sort event dates from geographic event data This recipe will illustrate the efficient use of the Hive date UDFs to list the 20 most recent events and the number of days between the event date and the current system date. Getting ready Make sure you have access to a pseudo-distributed or fully-distributed Hadoop cluster with Apache Hive 0.7.1 installed on your client machine and on the environment path for the active user account. This recipe depends on having the Nigera_ACLED_cleaned.tsv dataset loaded into a Hive table named acled_nigeria_cleaned with the fields mapped to the respective datatypes. Issue the following command to the Hive client to see the mentioned fields: describe acled_nigeria_cleaned You should see the following response: OK Loc string event_date string event_type string actor string latitude double longitude double source string fatalities int How to do it... Perform the following steps to utilize Hive UDFs for sorting and transformation: Open a text editor of your choice, ideally one with SQL syntax highlighting. Add the inline creation and transform syntax: SELECT event_type,event_date,days_since FROM ( SELECT event_type,event_date, datediff(to_date(from_unixtime(unix_timestamp())), to_date(from_unixtime( unix_timestamp(event_date, 'yyyy-MM-dd')))) AS days_since FROM acled_nigeria_cleaned) date_differences ORDER BY event_date DESC LIMIT 20; Save the file as top_20_recent_events.sql in the active folder. Run the script from the operating system shell by supplying the –f option to the Hive client. You should see the following five rows appear first in the output console: OK Battle-No change of territory 2011-12-31 190 Violence against civilians 2011-12-27 194 Violence against civilians 2011-12-25 196 Violence against civilians 2011-12-25 196 Violence against civilians 2011-12-25 196 How it works... Let's start with the nested SELECT subqueries. We select three fields from our Hive table acled_nigeria_cleaned: event_type, event_date, and the result of calling the UDF datediff(), which takes as arguments an end date and a start date. Both are expected in the form yyyy-MM-dd. The first argument to datediff() is the end date, with which we want to represent the current system date. Calling unix_timestamp() with no arguments will return the current system time in milliseconds. We send that return value to from_ unixtimestamp() to get a formatted timestamp representing the current system date in the default Java 1.6 format (yyyy-MM-dd HH:mm:ss). We only care about the date portion, so calling to_date() with the output of this function strips the HH:mm:ss. The result is the current date in the yyyy-MM-dd form. The second argument to datediff() is the start date, which for our query is the event_ date. The series of function calls operate in almost the exact same manner as our previous argument, except that when we call unix_timestamp(), we must tell the function that our argument is in the SimpleDateFormat format that is yyyy-MM-dd. Now we have both start_date and end_date arguments in the yyyy-MM-dd format and can perform the datediff() operation for the given row. We alias the output column of datediff() as days_since for each row. The outer SELECT statement takes these three columns per row and sorts the entire output by event_date in descending order to get reverse chronological ordering. We arbitrarily limit the output to only the first 20. The net result is the 20 most recent events with the number of days that have passed since that event occurred. There's more... The date UDFs can help tremendously in performing string date comparisons. Here are some additional pointers: Date format strings follow Java SimpleDateFormat guidelines Check out the Javadocs for SimpleDateFormat to learn how your custom date strings can be used with the date transform UDFs. Default date and time formats Many of the UDFs operate under a default format assumption. For UDFs requiring only date, your column values must be in the form yyyy-MM-dd. For UDFs that require date and time, your column values must be in the form yyyy- MM-dd HH:mm:ss. Using Hive to build a per-month report of fatalities over geographic event data This recipe will show a very simple analytic that uses Hive to count fatalities for every month appearing in the dataset and print the results to the console. Getting ready Make sure you have access to a pseudo-distributed or fully-distributed Hadoop cluster with Apache Hive 0.7.1 installed on your client machine and on the environment path for the active user account. This recipe depends on having the Nigera_ACLED_cleaned.tsv dataset loaded into a Hive table named acled_nigeria_cleaned with the following fields mapped to the respective datatypes. Issue the following command to the Hive client: describe acled_nigeria_cleaned You should see the following response: OK loc string event_date string event_type string actor string latitude double longitude double source string fatalities int How to do it... Follow the steps to use Hive for report generation: Open a text editor of your choice, ideally one with SQL syntax highlighting. Add the inline creation and transformation syntax: SELECT from_unixtime(unix_timestamp(event_date, 'yyyy-MM-dd'), 'yyyy-MMM'), COALESCE(CAST(sum(fatalities) AS STRING), 'Unknown') FROM acled_nigeria_cleaned GROUP BY from_unixtime(unix_timestamp(event_date, 'yyyy-MMdd'),' yyyy-MMM'); Save the file as monthly_violence_totals.sql in the active folder. Run the script from the operating system shell by supplying the –f option to the Hive client. You should see the following three rows appear first in the output console. Note that the output is sorted lexicographically, and not on the order of dates. OK 1997-Apr 115 1997-Aug 4 1997-Dec 26 How it works... The SELECT statement uses unix_timestamp() and from_unixtime() to reformat the event_date for each row as just a year-month concatenated field. This is also in the GROUP BY expression for totaling fatalities using sum(). The coalesce() method returns the first non-null argument passed to it. We pass as the first argument, the value of fatalities summed for that given year-month, cast as a string. If that value is NULL for any reason, return the constant Unknown. Otherwise return the string representing the total fatalities counted for that year-month combination. Print everything to the console over stdout. There's more... The following are some additional helpful tips related to the code in this recipe: The coalesce() method can take variable length arguments. As mentioned in the Hive documentation, coalesce() supports one or more arguments. The first non-null argument will be returned. This can be useful for evaluating several different expressions for a given column before deciding the right one to choose. The coalesce() will return NULL if no argument is non-null. It's not uncommon to provide a type literal to return if all other arguments are NULL. Date reformatting code template Having to reformat dates stored in your raw data is very common. Proper use of from_ unixtime() and unix_timestamp() can make your life much easier. Remember this general code template for concise date format transformation in Hive: from_unixtime(unix_timestamp(<col>,<in-format>),<out-format>);

(For more resources related to this topic, see here.) NGINX's architecture NGINX consists of a single master process and multiple worker processes. Each of these is single-threaded and designed to handle thousands of connections simultaneously. The worker process is where most of the action takes place, as this is the component that handles client requests. NGINX makes use of the operating system's event mechanism to respond quickly to these requests. The NGINX master process is responsible for reading the configuration, handling sockets, spawning workers, opening log files, and compiling embedded Perl scripts. The master process is the one that responds to administrative requests via signals. The NGINX worker process runs in a tight event loop to handle incoming connections. Each NGINX module is built into the worker, so that any request processing, filtering, handling of proxy connections, and much more is done within the worker process. Due to this worker model, the operating system can handle each process separately and schedule the processes to run optimally on each processor core. If there are any processes that would block a worker, such as disk I/O, more workers than cores can be configured to handle the load. There are also a small number of helper processes that the NGINX master process spawns to handle dedicated tasks. Among these are the cache loader and cache manager processes. The cache loader is responsible for preparing the metadata for worker processes to use the cache. The cache manager process is responsible for checking cache items and expiring invalid ones. NGINX is built in a modular fashion. The master process provides the foundation upon which each module may perform its function. Each protocol and handler is implemented as its own module. The individual modules are chained together into a pipeline to handle connections and process requests. After a request is handled, it is then passed on to a series of filters, in which the response is processed. One of these filters is responsible for processing subrequests, one of NGINX's most powerful features. Subrequests are how NGINX can return the results of a request that differs from the URI that the client sent. Depending on the configuration, they may be multiply nested and call other subrequests. Filters can collect the responses from multiple subrequests and combine them into one response to the client. The response is then finalized and sent to the client. Along the way, multiple modules come into play. See http://www.aosabook.org/en/nginx.html for a detailed explanation of NGINX internals. We will be exploring the http module and a few helper modules in the remainder of this article. The HTTP core module The http module is NGINX's central module, which handles all interactions with clients over HTTP. We will have a look at the directives in the rest of this section, again divided by type. The server The server directive starts a new context. We have already seen examples of its usage throughout the book so far. One aspect that has not yet been examined in-depth is the concept of a default server. A default server in NGINX means that it is the first server defined in a particular configuration with the same listen IP address and port as another server. A default server may also be denoted by the default_server parameter to the listen directive. The default server is useful to define a set of common directives that will then be reused for subsequent servers listening on the same IP address and port: server { listen 127.0.0.1:80; server_name default.example.com; server_name_in_redirect on; } server { listen 127.0.0.1:80; server_name www.example.com; } In this example, the www.example.com server will have the server_name_in_redirect directive set to on as well as the default.example.com server. Note that this would also work if both servers had no listen directive, since they would still both match the same IP address and port number (that of the default value for listen, which is *:80). Inheritance, though, is not guaranteed. There are only a few directives that are inherited, and which ones are changes over time. A better use for the default server is to handle any request that comes in on that IP address and port, and does not have a Host header. If you do not want the default server to handle requests without a Host header, it is possible to define an empty server_name directive. This server will then match those requests. server { server_name ""; } The following table summarizes the directives relating to server: Table: HTTP server directives Directive Explanation port_in_redirect Determines whether or not the port will be specified in a redirect issued by NGINX. server Creates a new configuration context, defining a virtual host. The listen directive specifies the IP address(es) and port(s); the server_name directive lists the Host header values that this context matches. server_name Configures the names that a virtual host may respond to. server_name_in_redirect Activates using the first value of the server_name directive in any redirect issued by NGINX within this context. server_tokens Disables sending the NGINX version string in error messages and the Server response header (default value is on). Logging NGINX has a very flexible logging model . Each level of configuration may have an access log. In addition, more than one access log may be specified per level, each with a different log_format. The log_format directive allows you to specify exactly what will be logged, and needs to be defined within the http section. The path to the log file itself may contain variables, so that you can build a dynamic configuration. The following example describes how this can be put into practice: http { log_format vhost '$host $remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"'; log_format downloads '$time_iso8601 $host $remote_addr ' '"$request" $status $body_bytes_sent $request_ time'; open_log_file_cache max=1000 inactive=60s; access_log logs/access.log; server { server_name ~^(www.)?(.+)$; access_log logs/combined.log vhost; access_log logs/$2/access.log; location /downloads { access_log logs/downloads.log downloads; } } } The following table describes the directives used in the preceding code: Table: HTTP logging directives Directive Explanation access_log Describes where and how access logs are to be written. The first parameter is a path to the file where the logs are to be stored. Variables may be used in constructing the path. The special value off disables the access log. An optional second parameter indicates log_format that will be used to write the logs. If no second parameter is configured, the predefined combined format is used. An optional third parameter indicates the size of the buffer if write buffering should be used to record the logs. If write buffering is used, this size cannot exceed the size of the atomic disk write for that filesystem. If this third parameter is gzip, then the buffered logs will be compressed on-the-fly, provided that the nginx binary was built with the zlib library. A final flush parameter indicates the maximum length of time buffered log data may remain in memory before being flushed to disk. log_format Specifies which fields should appear in the log file and what format they should take. See the next table for a description of the log-specific variables. log_not_found Disables reporting of 404 errors in the error log (default value is on). log_subrequest Enables logging of subrequests in the access log (default value is off ). open_log_file_cache Stores a cache of open file descriptors used in access_logs with a variable in the path. The parameters used are: max: The maximum number of file descriptors present in the cache inactive: NGINX will wait this amount of time for something to be written to this log before its file descriptor is closed min_uses: The file descriptor has to be used this amount of times within the inactive period in order to remain open valid: NGINX will check this often to see if the file descriptor still matches a file with the same name off: Disables the cache In the following example, log entries will be compressed at a gzip level of 4. The buffer size is the default of 64 KB and will be flushed to disk at least every minute. access_log /var/log/nginx/access.log.gz combined gzip=4 flush=1m; Note that when specifying gzip the log_format parameter is not optional.The default combined log_format is constructed like this: log_format combined '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent"'; As you can see, line breaks may be used to improve readability. They do not affect the log_format itself. Any variables may be used in the log_format directive. The variables in the following table which are marked with an asterisk ( *) are specific to logging and may only be used in the log_format directive. The others may be used elsewhere in the configuration, as well. Table: Log format variables Variable Name Value $body_bytes_sent The number of bytes sent to the client, excluding the response header. $bytes_sent The number of bytes sent to the client. $connection A serial number, used to identify unique connections. $connection_requests The number of requests made through a particular connection. $msec The time in seconds, with millisecond resolution. $pipe * Indicates if the request was pipelined (p) or not (.). $request_length * The length of the request, including the HTTP method, URI, HTTP protocol, header, and request body. $request_time The request processing time, with millisecond resolution, from the first byte received from the client to the last byte sent to the client. $status The response status. $time_iso8601 * Local time in ISO8601 format. $time_local * Local time in common log format (%d/%b/%Y:%H:%M:%S %z). In this section, we have focused solely on access_log and how that can be configured. You can also configure NGINX to log errors. Finding files In order for NGINX to respond to a request, it passes it to a content handler, determined by the configuration of the location directive. The unconditional content handlers are tried first: perl, proxy_pass, flv, mp4, and so on. If none of these is a match, the request is passed to one of the following, in order: random index, index, autoindex, gzip_static, static. Requests with a trailing slash are handled by one of the index handlers. If gzip is not activated, then the static module handles the request. How these modules find the appropriate file or directory on the filesystem is determined by a combination of certain directives. The root directive is best defined in a default server directive, or at least outside of a specific location directive, so that it will be valid for the whole server: server { root /home/customer/html; location / { index index.html index.htm; } location /downloads { autoindex on; } } In the preceding example any files to be served are found under the root /home/customer/html. If the client entered just the domain name, NGINX will try to serve index.html. If that file does not exist, then NGINX will serve index.htm. When a user enters the /downloads URI in their browser, they will be presented with a directory listing in HTML format. This makes it easy for users to access sites hosting software that they would like to download. NGINX will automatically rewrite the URI of a directory so that the trailing slash is present, and then issue an HTTP redirect. NGINX appends the URI to the root to find the file to deliver to the client. If this file does not exist, the client receives a 404 Not Found error message. If you don't want the error message to be returned to the client, one alternative is to try to deliver a file from different filesystem locations, falling back to a generic page, if none of those options are available. The try_files directive can be used as follows: location / { try_files $uri $uri/ backups/$uri /generic-not-found.html; } As a security precaution, NGINX can check the path to a file it's about to deliver, and if part of the path to the file contains a symbolic link, it returns an error message to the client: server { root /home/customer/html; disable_symlinks if_not_owner from=$document_root; } In the preceding example, NGINX will return a "Permission Denied" error if a symlink is found after /home/customer/html, and that symlink and the file it points to do not both belong to the same user ID. The following table summarizes these directives: Table: HTTP file-path directives Directive Explanation disable_symlinks Determines if NGINX should perform a symbolic link check on the path to a file before delivering it to the client. The following parameters are recognized: off : Disables checking for symlinks (default) on: If any part of a path is a symlink, access is denied if_not_owner: If any part of a path contains a symlink in which the link and the referent have different owners, access to the file is denied from=part: When specified, the path up to part is not checked for symlinks, everything afterward is according to either the on or if_not_owner parameter root Sets the path to the document root. Files are found by appending the URI to the value of this directive. try_files Tests the existence of files given as parameters. If none of the previous files are found, the last entry is used as a fallback, so ensure that this path or named location exists, or is set to return a status code indicated by  =<status code>. Name resolution If logical names instead of IP addresses are used in an upstream or *_pass directive, NGINX will by default use the operating system's resolver to get the IP address, which is what it really needs to connect to that server. This will happen only once, the first time upstream is requested, and won't work at all if a variable is used in the *_pass directive. It is possible, though, to configure a separate resolver for NGINX to use. By doing this, you can override the TTL returned by DNS, as well as use variables in the *_pass directives. server { resolver 192.168.100.2 valid=300s; } Table: Name resolution directives Directive Explanation resolver   Configures one or more name servers to be used to resolve upstream server names into IP addresses. An optional  valid parameter overrides the TTL of the domain name record. In order to get NGINX to resolve an IP address anew, place the logical name into a variable. When NGINX resolves that variable, it implicitly makes a DNS look-up to find the IP address. For this to work, a resolver directive must be configured: server { resolver 192.168.100.2; location / { set $backend upstream.example.com; proxy_pass http://$backend; } } Of course, by relying on DNS to find an upstream, you are dependent on the resolver always being available. When the resolver is not reachable, a gateway error occurs. In order to make the client wait time as short as possible, the resolver_timeout parameter should be set low. The gateway error can then be handled by an error_ page designed for that purpose. server { resolver 192.168.100.2; resolver_timeout 3s; error_page 504 /gateway-timeout.html; location / { proxy_pass http://upstream.example.com; } } Client interaction There are a number of ways in which NGINX can interact with clients. This can range from attributes of the connection itself (IP address, timeouts, keepalive, and so on) to content negotiation headers. The directives listed in the following table describe how to set various headers and response codes to get the clients to request the correct page or serve up that page from its own cache: Table: HTTP client interaction directives Directive Explanation default_type Sets the default MIME type of a response. This comes into play if the MIME type of the file cannot be matched to one of those specified by the types directive. error_page Defines a URI to be served when an error level response code is encountered. Adding an = parameter allows the response code to be changed. If the argument to this parameter is left empty, the response code will be taken from the URI, which must in this case be served by an upstream server of some sort. etag Disables automatically generating the ETag response header for static resources (default is on). if_modified_since Controls how the modification time of a response is compared to the value of the If-Modified-Since request header: off: The If-Modified-Since header is ignored exact: An exact match is made (default) before: The modification time of the response is less than or equal to the value of the If-Modified-Since header ignore_invalid_headers Disables ignoring headers with invalid names (default is on). A valid name is composed of ASCII letters, numbers, the hyphen, and possibly the underscore (controlled by the underscores_in_headers directive). merge_slashes Disables the removal of multiple slashes. The default value of on means that NGINX will compress two or more / characters into one. recursive_error_pages Enables doing more than one redirect using the error_page directive (default is off). types Sets up a map of MIME types to file name extensions. NGINX ships with a conf/mime.types file that contains most MIME type mappings. Using include to load this file should be sufficient for most purposes. underscores_in_headers Enables the use of the underscore character in client request headers. If left at the default value off , evaluation of such headers is subject to the value of the ignore_invalid_headers directive. The error_page directive is one of NGINX's most flexible. Using this directive, we may serve any page when an error condition presents. This page could be on the local machine, but could also be a dynamic page produced by an application server, and could even be a page on a completely different site. http { # a generic error page to handle any server-level errors error_page 500 501 502 503 504 share/examples/nginx/50x.html; server { server_name www.example.com; root /home/customer/html; # for any files not found, the page located at # /home/customer/html/404.html will be delivered error_page 404 /404.html; location / { # any server-level errors for this host will be directed # to a custom application handler error_page 500 501 502 503 504 = @error_handler; } location /microsite { # for any non-existent files under the /microsite URI, # the client will be shown a foreign page error_page 404 http://microsite.example.com/404.html; } # the named location containing the custom error handler location @error_handler { # we set the default type here to ensure the browser # displays the error page correctly default_type text/html; proxy_pass http://127.0.0.1:8080; } } } Using limits to prevent abuse We build and host websites because we want users to visit them. We want our websites to always be available for legitimate access. This means that we may have to take measures to limit access to abusive users. We may define "abusive" to mean anything from one request per second to a number of connections from the same IP address. Abuse can also take the form of a DDOS (distributed denial-of-service) attack, where bots running on multiple machines around the world all try to access the site as many times as possible at the same time. In this section, we will explore methods to counter each type of abuse to ensure that our websites are available. First, let's take a look at the different configuration directives that will help us achieve our goal: Table: HTTP limits directives Directive Explanation limit_conn Specifies a shared memory zone (configured with limit_conn_zone) and the maximum number of connections that are allowed per key value. limit_conn_log_level When NGINX limits a connection due to the limit_conn directive, this directive specifies at which log level that limitation is reported. limit_conn_zone Specifies the key to be limited in limit_conn as the first parameter. The second parameter, zone, indicates the name of the shared memory zone used to store the key and current number of connections per key and the size of that zone (name:size). limit_rate Limits the rate (in bytes per second) at which clients can download content. The rate limit works on a connection level, meaning that a single client could increase their throughput by opening multiple connections. limit_rate_after Starts the limit_rate after this number of bytes have been transferred. limit_req Sets a limit with bursting capability on the number of requests for a specific key in a shared memory store (configured with limit_req_zone). The burst can be specified with the second parameter. If there shouldn't be a delay in between requests up to the burst, a third parameter nodelay needs to be configured. limit_req_log_level When NGINX limits the number of requests due to the limit_req directive, this directive specifies at which log level that limitation is reported. A delay is logged at a level one less than the one indicated here. limit_req_zone Specifies the key to be limited in limit_req as the first parameter. The second parameter, zone, indicates the name of the shared memory zone used to store the key and current number of requests per key and the size of that zone ( name:size). The third parameter, rate, configures the number of requests per second (r/s) or per minute (r/m) before the limit is imposed. max_ranges Sets the maximum number of ranges allowed in a byte-range request. Specifying 0 disables byte-range support. Here we limit access to 10 connections per unique IP address. This should be enough for normal browsing, as modern browsers open two to three connections per host. Keep in mind, though, that any users behind a proxy will all appear to come from the same address. So observe the logs for error code 503 (Service Unavailable), meaning that this limit has come into effect: http { limit_conn_zone $binary_remote_addr zone=connections:10m; limit_conn_log_level notice; server { limit_conn connections 10; } } Limiting access based on a rate looks almost the same, but works a bit differently. When limiting how many pages per unit of time a user may request, NGINX will insert a delay after the first page request, up to a burst. This may or may not be what you want, so NGINX offers the possibility to remove this delay with the nodelay parameter: http { limit_req_zone $binary_remote_addr zone=requests:10m rate=1r/s; limit_req_log_level warn; server { limit_req zone=requests burst=10 nodelay; } } Using $binary_remote_addr We use the $binary_remote_addr variable in the preceding example to know exactly how much space storing an IP address will take. This variable takes 32 bytes on 32-bit platforms and 64 bytes on 64-bit platforms. So the 10m zone we configured previously is capable of holding up to 320,000 states on 32-bit platforms or 160,000 states on 64-bit platforms. We can also limit the bandwidth per client. This way we can ensure that a few clients don't take up all the available bandwidth. One caveat, though: the limit_rate directive works on a connection basis. A single client that is allowed to open multiple connections will still be able to get around this limit: location /downloads { limit_rate 500k; } Alternatively, we can allow a kind of bursting to freely download smaller files, but make sure that larger ones are limited: location /downloads { limit_rate_after 1m; limit_rate 500k; } Combining these different rate limitations enables us to create a configuration that is very flexible as to how and where clients are limited: http { limit_conn_zone $binary_remote_addr zone=ips:10m; limit_conn_zone $server_name zone=servers:10m; limit_req_zone $binary_remote_addr zone=requests:10m rate=1r/s; limit_conn_log_level notice; limit_req_log_level warn; reset_timedout_connection on; server { # these limits apply to the whole virtual server limit_conn ips 10; # only 1000 simultaneous connections to the same server_name limit_conn servers 1000; location /search { # here we want only the /search URL to be rate-limited limit_req zone=requests burst=3 nodelay; } location /downloads { # using limit_conn to ensure that each client is # bandwidth-limited # with no getting around it limit_conn connections 1; limit_rate_after 1m; limit_rate 500k; } } } Restricting access In the previous section, we explored ways to limit abusive access to websites running under NGINX. Now we will take a look at ways to restrict access to a whole website or certain parts of it. Access restriction can take two forms here: restricting to a certain set of IP addresses, or restricting to a certain set of users. These two methods can also be combined to satisfy requirements that some users can access the website either from a certain set of IP addresses or if they are able to authenticate with a valid username and password. The following directives will help us achieve these goals: Table: HTTP access module directives Directive Explanation allow Allows access from this IP address, network, or all. auth_basic Enables authentication using HTTP Basic Authentication. The parameter string is used as the realm name. If the special value off is used, this indicates that the auth_basic value of the parent configuration level is negated. auth_basic_user_file Indicates the location of a file of username:password:comment tuples used to authenticate users. The password field needs to be encrypted with the crypt algorithm. The comment field is optional. deny Denies access from this IP address, network, or all. satisfy Allows access if all or any of the preceding directives grant access. The default value all indicates that a user must come from a specific network address and enter the correct password. To restrict access to clients coming from a certain set of IP addresses, the allow and deny directives can be used as follows: location /stats { allow 127.0.0.1; deny all; } This configuration will allow access to the /stats URI from the localhost only. To restrict access to authenticated users, the auth_basic and auth_basic_user_file directives are used as follows: server { server_name restricted.example.com; auth_basic "restricted"; auth_basic_user_file conf/htpasswd; } Any user wanting to access restricted.example.com would need to provide credentials matching those in the htpasswd file located in the conf directory of NGINX's root. The entries in the htpasswd file can be generated using any available tool that uses the standard UNIX crypt() function. For example, the following Ruby script will generate a file of the appropriate format: #!/usr/bin/env ruby # setup the command-line options require 'optparse' OptionParser.new do |o| o.on('-f FILE') { |file| $file = file } o.on('-u', "--username USER") { |u| $user = u } o.on('-p', "--password PASS") { |p| $pass = p } o.on('-c', "--comment COMM (optional)") { |c| $comm = c } o.on('-h') { puts o; exit } o.parse! if $user.nil? or $pass.nil? puts o; exit end end # initialize an array of ASCII characters to be used for the salt ascii = ('a'..'z').to_a + ('A'..'Z').to_a + ('0'..'9').to_a + [ ".", "/" ] $lines = [] begin # read in the current http auth file File.open($file) do |f| f.lines.each { |l| $lines << l } end rescue Errno::ENOENT # if the file doesn't exist (first use), initialize the array $lines = ["#{$user}:#{$pass}n"] end # remove the user from the current list, since this is the one we're editing $lines.map! do |line| unless line =~ /#{$user}:/ line end end # generate a crypt()ed password pass = $pass.crypt(ascii[rand(64)] + ascii[rand(64)]) # if there's a comment, insert it if $comm $lines << "#{$user}:#{pass}:#{$comm}n" else $lines << "#{$user}:#{pass}n" end # write out the new file, creating it if necessary File.open($file, File::RDWR|File::CREAT) do |f| $lines.each { |l| f << l} end Save this file as http_auth_basic.rb and give it a filename (-f), a user (-u), and a password (-p), and it will generate entries appropriate to use in NGINX's auth_ basic_user_file directive: $ ./http_auth_basic.rb -f htpasswd -u testuser -p 123456 To handle scenarios where a username and password should only be entered if not coming from a certain set of IP addresses, NGINX has the satisfy directive. The any parameter is used here for this either/or scenario: server { server_name intranet.example.com; location / { auth_basic "intranet: please login"; auth_basic_user_file conf/htpasswd-intranet; allow 192.168.40.0/24; allow 192.168.50.0/24; deny all; satisfy any; } If, instead, the requirements are for a configuration in which the user must come from a certain IP address and provide authentication, the all parameter is the default. So, we omit the satisfy directive itself and include only allow, deny, auth_basic, and auth_basic_user_file: server { server_name stage.example.com; location / { auth_basic "staging server"; auth_basic_user_file conf/htpasswd-stage; allow 192.168.40.0/24; allow 192.168.50.0/24; deny all; } Streaming media files NGINX is capable of serving certain video media types. The flv and mp4 modules, included in the base distribution, can perform what is called pseudo-streaming. This means that NGINX will seek to a certain location in the video file, as indicated by the start request parameter. In order to use the pseudo-streaming capabilities, the corresponding module needs to be included at compile time: --with-http_flv_module for Flash Video (FLV) files and/or --with-http_mp4_module for H.264/AAC files. The following directives will then become available for configuration: Table: HTTP streaming directives Directive Explanation flv Activates the flv  module for this location. mp4 Activates the mp4  module for this location. mp4_buffer_size Sets the initial buffer size for delivering MP4 files. mp4_max_buffer_size Sets the maximum size of the buffer used to process MP4 metadata. Activating FLV pseudo-streaming for a location is as simple as just including the flv keyword: location /videos { flv; } There are more options for MP4 pseudo-streaming, as the H.264 format includes metadata that needs to be parsed. Seeking is available once the "moov atom" has been parsed by the player. So to optimize performance, ensure that the metadata is at the beginning of the file. If an error message such as the following shows up in the logs, the mp4_max_buffer_size needs to be increased: mp4 moov atom is too large mp4_max_buffer_size can be increased as follows: location /videos { mp4; mp4_buffer_size 1m; mp4_max_buffer_size 20m; } Predefined variables NGINX makes constructing configurations based on the values of variables easy. Not only can you instantiate your own variables by using the set or map directives, but there are also predefined variables used within NGINX. They are optimized for quick evaluation and the values are cached for the lifetime of a request. You can use any of them as a key in an if statement, or pass them on to a proxy. A number of them may prove useful if you define your own log file format. If you try to redefine any of them, though, you will get an error message as follows: <timestamp> [emerg] <master pid>#0: the duplicate "<variable_name>" variable in <path-to-configuration-file>:<line-number> They are also not made for macro expansion in the configuration—they are mostly used at run time. Summary In this article, we have explored a number of directives used to make NGINX serve files over HTTP. Not only does the http module provide this functionality, but there are also a number of helper modules that are essential to the normal operation of NGINX. These helper modules are enabled by default. Combining the directives of these various modules enables us to build a configuration that meets our needs. We explored how NGINX finds files based on the URI requested. We examined how different directives control how the HTTP server interacts with the client, and how the error_page directive can be used to serve a number of needs. Limiting access based on bandwidth usage, request rate, and number of connections is all possible. We saw, too, how we can restrict access based on either IP address or through requiring authentication. We explored how to use NGINX's logging capabilities to capture just the information we want. Pseudo-streaming was examined briefly, as well. NGINX provides us with a number of variables that we can use to construct our configurations. Resources for Article : Further resources on this subject: Nginx HTTP Server FAQs [Article] Nginx Web Services: Configuration and Implementation [Article] Using Nginx as a Reverse Proxy [Article]

(For more resources related to this topic, see here.) Sure Step is the methodology created by Microsoft for implementing Dynamics projects. Although Sure Step includes content for Dynamics: NAV, AX, GP, SL, and CRM, our book is only focusing on AX. In some cases, one Dynamics project will be split into multiple Sure Step projects. This may be the case for projects covering one or more phases or for global rollouts, with sub-projects covering multiple territories or lines of business. Sure Step has evolved through several versions, and gains content and maturity through each one. There would be no need to use an earlier version, so we have only considered the current version in this book (Sure Step 2010). Sure Step is designed to be used by both customers and partners, and although it is currently most often used by partners, it is becoming more and more common for customers to follow Sure Step methodology when implementing ERP systems. In this book, we use "project" to describe an implementation of Microsoft Dynamics AX. In most cases, one Microsoft Dynamics AX project will be the subject of one Microsoft Sure Step project. For more comprehensive information on Sure Step, you should refer the Microsoft training materials, or the Packt Publishing title Microsoft Dynamics Sure Step 2010 ISBN 978-1-849681-10-0 by Chandru Shankar and Vincent Bellefroid. This article will introduce the Sure Step methodology, explain how to install the Sure Step client and create new projects, discuss the various engagement and project types that are available within Sure Step, and explore the steps that need to be taken during the project initiation phase. Installing Sure Step Sure Step is available to Microsoft Dynamics customers and partners via download from the Microsoft CustomerSource or PartnerSource portals respectively. The download and installation of Sure Step is simple and similar to installing other Microsoft desktop applications. For this reason, step-by-step instructions will not be detailed here, however, this information can be found in the instructions that accompany the download. Creating the Dynamics AX project Once the Sure Step client is installed, it is time to create a project for a Microsoft Dynamics AX implementation. To create a project, run the Sure Step client and select the Projects tab from the opening screen. Once the Project tab is selected there is a list of existing projects. Upon choosing the option Create New Project , a wizard guides us through the creation process. The first stage of the wizard asks us to enter the project data as shown in the following screenshot: The fields marked with a red asterisk (*) are mandatory. The solution and product are obvious and the project name is a matter of personal preference, but the engagement is worthy of an explanation and is detailed in the following sections. Engagement types (offerings) Engagement types refer to the choice of Diagnostic , Implementation, or Optimization offerings. Diagnostic phase offering Selecting a Diagnostic phase offering creates a Sure Step project, with content and guidance exclusively covering solution envisioning (Diagnostic activities). This allows the partner and customer to gain a high-level understanding of the fit of the solution to the customers' needs, which in turn allows the partner to be able to define the scope, cost, and risk of the implementation more effectively. This type of offering is therefore, intended to be used by a partner to increase their customers' confidence in, and commitment to, an implementation of a Dynamics product. If a Diagnostic offering is chosen, there is the option of including one or more of the Decision Accelerators pictured in the following screenshot. Decision Accelerators such as Requirements and Process Review , Fit Gap and Solution Blueprint, Architecture Assessment, or Scoping Assessment are often very useful in later phases of the implementation, and will provide important high-level information about the project. Each project will often require a different combination of Decision Accelerators depending on individual circumstances, Implementation offering Selecting an Implementation offering creates a project, with content and guidance covering the full project implementation including the envisioning (Diagnostics) phase. This includes the key phases, namely Analysis , Design, Development, and Deployment. The Sure Step implementation project is intended to be used to help a customer (who would normally be working with a partner) to successfully implement a Dynamics project, and is the case we are considering in this book. Optimization offering Selecting this option creates a Sure Step project, with content and guidance covering the upgrade or improvement of an existing system. As the fundamentals of the upgrade template are covered in the implementation template, we will not be specifically covering the optimization in this book. Project types Depending on which Engagement Type we have chosen, Sure Step will now require entry of a project type on the next step of the wizard. Typically, we will choose an Implementation offering, which is the template that will be covered in the scope of this book. The five project types (Rapid, Standard, Enterprise, Agile, and Optimization) are described in the following sections. This is a very important choice, because the content and guidance for the project types differ markedly. The typical project types are Rapid, Standard, and Enterprise, all of which are referred to as waterfall project types, as they follow discrete sequential phases as shown in the following screenshot: Choosing a project type requires both subjective and objective evaluation of the risk and complexity of a project. Subjective analysis should include an assessment of the capability of the project teams, the buy-in of senior management, budget and timescale constraints, and so on. When looking back at challenged projects, the choice of too lightweight a project type is often cited as the reason for failure. When choosing the project type, projects have generally not yet encountered difficulties, and often budget and timescale constraints can be given too much weight in making the choice of project type. As the decision includes subjective and objective evaluation, it is ultimately a subjective choice which the project manager or governance team needs to be prepared to be accountable for. Some objective analysis points are identified in the following table: Project type Business Process Analysis Lines Of Business Sites Users Fit to the Gap Fit ISV solutions Complexity Countries Rapid No 1 1 1-35 >90 percent 0 Low 1 Standard Some 1-2 1-3 25-250 >75 percent 1 Medium 1-5 Enterprise Extensive >1 3+ 250+ <75 percent >1 High 5+ Agile No 1+ 1+ 1+ <75 percent 1+ Medium 1+ Rapid project type A Rapid project type will be created using the minimum artifacts (objects such as documents, spreadsheets, and diagrams) and processes of a structured project. As the project has less complexity and less risk, a more pragmatic approach can be taken to manage the project. Rapid project types suit small systems with little customization and well understood business and technical requirements. A very good use of the Rapid project type is when implementing repeats or small variations of previously implemented small projects—as tends to occur when working in vertical markets. Standard project type A Standard project type will be created with more artifacts, particularly around business process analysis and project governance. These suit more typical AX projects. Standard projects suit typical AX implementations with one or two sites, perhaps an Independent Software Vendor (ISV) solution and some customization. Enterprise project type An Enterprise project type will have the maximum number of artifacts and content based around quality assurance, governance, communications management, and all the elements required for managing large teams delivering complicated projects. Enterprise project suit more complex projects. Complexity is a bit subjective as it is influenced by all of the factors in the following list, but most of them on their own don't necessarily make a project an Enterprise project. International Multi-site Complex modifications Multiple ISV solutions Organizational Change Management Business Process Reengineering It is simple in Sure Step to create fictitious projects of all these types to review the content differences, which can be done using the Sure Step client. Agile project type The Agile project type is based on a wholly different approach to projects than the previous types. The previous types are all waterfall methodologies, where one stage is completed before progressing to the next stage through a familiar model of diagnosis, analysis, design, development, and deployment. A typical Agile project type follows an iterative approach shown in the following screenshot: An Agile project does not follow the waterfall type of approach and lacks some of the structure and stages of these types of projects. In an Agile project, an initial Diagnostic activity will occur, followed by an Agile Preparation phase. This will be followed by iterative cycles of Analysis , Design, and Development, known as sprint cycles . Each sprint cycle will focus on a pre-agreed list of activities, and anything outstanding at the end of that sprint cycle will be assigned to a later sprint cycle. There will be major sprint cycles (typically monthly) and minor sprint cycles (typically daily). At the start of each sprint cycle, there will be a sprint cycle review meeting to agree a list of activities to include and to provide an update on progress. Agile projects tend to be used where requirements are not sufficiently detailed but the project needs to get started, or where requirements are frequently changing. They are therefore iterative and their duration is hard to predict. This makes them difficult to manage and they should be avoided by inexperienced project managers. Even in an Agile project, it is important to ensure that activities from a Standard project type (testing, training, data migration, and so on) are completed and managed properly. Upgrade project type It is slightly odd that having not chosen an Optimization Engagement Type, you are nevertheless offered an Upgrade project type even for an Implementation offering. We will ignore this for the purposes of this book, which is about implementing AX, not upgrading it. Project initiation Project initiation is expected to occur between the Diagnostic and Analysis phases. On a larger project, each phase requires some initiation activity of its own and it is particularly common to initiate the Diagnostic phase separately from the Implementation project as the Diagnostic phase is often part of the decision making process. The appointment of a project manager to commence end-to-end planning activities can occur anywhere in this cycle which obviously influences initiation as discussed in the following sections. Concluding pre-sales and sales activity It is desirable that all sales and pre-sales activity is concluded prior to kicking off the main project, although some overlap is common. It is, unfortunately, not rare for the pre-sales and sales activities to have not formally concluded, or to have been handed over from the sales team to the delivery team. This is the first thing a project manager should be chasing and why we have emphasized it here. See the relevant Sure Step details, but at this stage a Project Charter, Statement of Work, Implementation Contract, Proposal, and Requirements review are among the key minimum requirements to plan a project. Document repository As both the partner and client project teams need to share documents and files, we should provide a repository. Not all content should be visible to all users of the repository, so appropriate security controls or multiple repositories need to be considered. Setting up the repository early in the project provides useful structure and saves hunting for current versions of documents later in the project. Microsoft SharePoint and/or SharePoint Workspace should suffice, although there are numerous options. Although there are many Sure Step documents, in this book we want to consider some of the most important and most overlooked, that in our project experience, can help make the difference between managed expectations or a degree of failure. Accordingly, below we have identified some of the first items that belong in the document repository. Communication plan The communication plan should be one of the first documents in the repository and Sure Step provides a good template document for this. The plan identifies all key project documents and communication types (for example, meetings). From within the plan, we can see the current version of all documents and communications, who last edited them and who is responsible for them. In addition, adding a URL to anything that has an associated file is a good practice for easy reference. Early in the project, this provides an index to key project files and documentation and should be maintained throughout the project. Project hierarchy, communications, and meetings Project hierarchy refers to the tiers of management governing a project. It is vital to establish the governance at the outset of the project and clearly communicate it to the whole project team—one of the functions of the communication plan. Different projects will require different governance structures depending on risk, size of the project, project type, project duration, and so on. This should be established as soon as the project kicks off and evolved during the project, if necessary. Each of the governance groups should attend a kick-off meeting of the project, and in larger projects, kick-off meetings of every phase. These meetings can be separate or combined as is appropriate. It is most common to have three levels of governance, whose team members overlap a little as shown in the following screenshot: Statement of Work and Project Charter Several documents are required at the outset of the project. The Statement of Work (normally provided by the partner) lists the scope of the key deliverables of the project. The Project Charter lists the rules of engagement of a project. These are supplemented by the System Proposal and Project Contract, but most of the time, the high-level scope can be checked from these two documents. Sure Step provides good templates for these. Functional Requirements Document The Functional Requirements Document (FRD) is a more obvious document and is perhaps the most important project document which details scope at a functional level. Most projects do contain an FRD, but in retrospective evaluation, they often lack some of the detail required. The FRD should contain all of the customer's requirements listed from a process or functional perspective. Each requirement should be listed in sufficient detail to satisfy the project team members from the customer and partner that the requirement is well enough understood to agree scope against. Each requirement should be identifiable via a reference which can be cross-referenced from the Gap Fit, Software Design Specifications, and so on. Non-functional requirements Non-functional requirements are also captured such as technical environment, high availability, migration, and so on. Other documents The Gap Fit, which records the level of fit of requirements to the standard application, is a simple but important document; this is rarely lacking in a project, so we have not focused on it here. There are many other documents we could mention, but if we look at the Documents tab of the Sure Step projects created earlier in this article, we can see and learn more about them there. Work Breakdown Structure The Work Breakdown Structure (WBS) is an entity used to link activities for the purposes of measurement and management. Sure Step includes a template WBS, but we may make our own if we feel it would be more suitable to the project. The benefits of having a clear and defined WBS from the beginning of the project are often realized in later stages, when reporting on the progress and budget to key stakeholders becomes more important. Three examples of structures are provided in the following tables. WBS 3 is the only one extracted from Sure Step; the others are examples of alternatives. WBS 1 Worker Based Single Level: WBS Level Resource Name 1 Consultant 1 Andrew Birch 2 Consultant 2 Keith Dunkinson 3 Consultant 3 Andrew Dunkinson WBS 2 Worker Based Two Levels: WBS Level Resource Name Activity 1.1 Consultant 1 Andrew Birch Analysis 1.2 Consultant 1 Andrew Birch Design 2.1 Consultant 1 Andrew Birch Development 2.2 Consultant 2 Keith Dunkinson Analysis 2.3 Consultant 2 Keith Dunkinson Design 2.4 Consultant 2 Keith Dunkinson Development Sure Step provides good templates for WBS including the following one: WBS 3 Extract of Sure Step Enterprise 4 level WBS: WBS Level Activity 1 Sure Step Enterprise methodology 1.1 Analysis phase 1.1.0 Program management 1.1.1 Project planning 1.1.1.1 Conduct Executive kick-off E1.1.1.1 Executive kick-off meeting 1.1.1.2 Develop Organizational Change Management Strategy E1.1.1.2 Organizational Change Management Strategy 1.1.1.3 Develop leadership action plan E1.1.1.3 Leadership action plan 1.1.1.4 Finalize Project Charter E1.1.1.4 Project Charter 1.1.1.5 Finalize WBS and project plan E1.1.1.5 Project plan 1.1.1.6 Review and Approve Project Charter and project plan 1.1.2 Risks and issues management 1.1.2.1 Conduct Organization risk and readiness assessment E1.1.2.1 Organization risk and readiness assessment analysis E1.1.2.2 Risk and readiness assessment report Linking the WBS to planning activities, progress, budgets, and the timesheets enables close cost control and duration management of the project. WBS levels We may measure and manage at any level of the WBS, but what we must consider with using detailed levels is that activities and budgets then need to be planned and recorded at the same level of granularity. We can of course manage different things using different levels of WBS, but again, consideration is required to ensure they can be reconciled at an appropriate level of detail. Labor and materials tracking Reconciling progress and budgets requires collecting information at the appropriate level of detail. Therefore, the budgeting formats and timesheets need to be agreed before the project begins. All too often, this is not realized until after the project has commenced and reconciliation has already become futile. Time logs In particular, timesheets from the consulting team (or indeed any budgeted resource) need to be broken down into the level of reporting required. This can be difficult for resources who undertake a wide range of activities in a day, and this should also be considered when planning the granularity of measurement and management required. Initiating Cross Phase activities While we have covered some of the more important and arguably less common activities, there are of course many more we could have considered. The project type, our project's needs, and our own experience will influence choices, and adoption of methodology for the project. Sure Step includes good content for Cross Phase activities, project type selection, Organizational Change Management, and so on. Once we have made our decisions, we should look again at what Sure Step has suggested before setting them in stone. If we are not sure, we can seek peer review. There are numerous forums for Sure Step and experts are available to help if necessary. The Microsoft forums, LinkedIn Dynamics World, and Mibuso are all useful reference points. Summary Sure Step is readily available to Microsoft Dynamics partners and customers and can easily be installed and set up. Utilizing Sure Step appropriately requires experience of implementing Dynamics projects in a structured fashion. Project initiation activities should be considered very early in a project, as they are much harder to implement properly once a project has started without them. Every project is unique and while trying to conform to standards, we need to be flexible and dynamic in our approach to achieve the best results within a reasonable timescale and cost. In this article, we have introduced you to the Sure Step client and discussed the various Engagement and project types available. We have also outlined the activities needed to move an engagement from the pre-sales phase on to the implementation phase. Resources for Article : Further resources on this subject: Overview of Microsoft Dynamics CRM 2011 [Article] Adding a security role [Article] Using Processes in Microsoft Dynamics CRM 2011 [Article]

(For more resources related to this topic, see here.) EaselJS is part of the CreateJS suite, a JavaScript library for building rich and interactive experiences, such as web applications and web-based games that run on desktop and mobile web browsers. The standard HTML5 canvas' syntax can be very hard for beginners, especially if you need to animate and draw many objects. EaselJS greatly simplifies application development in HTML5 canvas using a syntax and an architecture very similar to the ActionScript 3.0 language. As a result, Flash/Flex developers will immediately feel at home, but it's very easy to learn even if you've never opened Flash in your life. CreateJS is currently supported by Adobe, AOL, and Microsoft, and it's developed by Grant Skinner, an internationally recognized leader in the field of rich Internet application development. Thanks to EaselJS, you can easily manage many types of graphic elements (vector shapes, bitmap, spritesheets, texts, and HTML elements) and it also supports touch events, animations, and many other interesting features in order to quickly develop cross-platform HTML5 games and applications, providing a look and feel as well as a behavior very similar to native applications for iOS and Android. Following are the five reasons to choose EaselJS and HTML5 canvas to build your applications: Cross-platform — Using this technology will help you create HTML5 canvas applications that will be supported from: Desktop browsers such as Chrome, Safari, Firefox, Opera, and IE9+ iPhone, iPad, and iPod 4+ (iOS 3.2+) Android smartphones and tablets (OS 2.1+) BlackBerry browser (7.0 and 10.0+) Every HTML5 browser (go to http://caniuse.com/canvas for more information) The following screenshot shows how the same application can run on different devices and resolutions: Easy Integration — EaselJS applications run on browsers and finally can be seen by almost every desktop and mobile user without any plugin installed. The HTML5 canvas element behaves just like any other HTML element. It can overlap other elements or become part of an existing HTML page. So, your canvas application can fill the entire browser area or just a small part of an existing HTML page. You can create amazing image galleries for your sites, product configurators, microsites, games, and interactive banners, and replicate a lot of features that used to be created with Adobe Flash or Apache Flex. One source code — A single codebase can be used to create a responsive application that works on almost all devices and resolutions. If you've ever created a liquid or fluid layout using HTML, Flash, or Flex then you already know this concept. As shown in the previous screenshot, you can also adapt UI and change behaviors according to the size of the device being used. No creativity limits — As in Flash, you can now forget HTML DOM compatibility issues. When you display a graphic element using EaselJS, you can be sure it will be placed at the same position in every browser, desktop and mobile (except for texts because every browser uses a different font renderer, and there may be some minor differences between them and of course Internet Explorer 8 and lower versions that do not support HTML5 syntax). Furthermore the CreateJS suite includes a lot of additional tools helping developers and designers to create amazing stuff: TweenJS: An useful tween engine to create runtime animations PreloadJS: To load assets and create nice preloaders Zoë: To convert SWF (Adobe Flash native web format) into spritesheets and JSON for EaselJS SoundJS: A library to play sounds (this topic is not covered in this book) CreateJS Toolkit for Flash CS6: To export Flash timeline animations in an EaselJS-compatible format Freedom — Developers can now create and publish games and applications skipping the App Store submission process. Of course, the performance of HTML5 applications are not comparable to those achieved by the native applications but can still be an alternative solution to many needs. From a business perspective, it's a great opportunity because it is now possible to avoid following the Apple guidelines that usually don't allow publishing applications that are primarily marketing material or advertisements, duplicated applications or applications that are not very useful, or simply websites bundled as applications. Users can now have a cool touch experience directly while navigating through a website, avoiding having to download, install, and open a native application. Furthermore, developers can also use PhoneGap (http://www.phonegap.com) and many other technologies to convert their HTML applications in native applications for iOS, Android, Windows Phones, BlackBerry, Bada, or WebOS. After the previous introduction you will be guided through the process of downloading, installing and configuring EaselJS in your local machine (this part of the book is not copied in this article). The book continues with the traditional "Hello World" example, as shown in the next paragraph: Quick start — creating your first canvas application Now we'll see how to create our first HTML5 canvas application with EaselJS. Step 1 — creating the HTML template Take a look at the following code that represents the boilerplate we'll use: <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> <title>EaselJS Starter: Template Page</title> <script src = "lib/easeljs-0.6.0.min.js"></script> <script> // Your code here function init() { } </script> </head> <body onload="init();" style="background-color:# ccc "> <h1> EaselJS Starter: Template page </h1> <canvas id="mycanvas" width="960" height="450" style="background-color:#fff"></canvas> </body> </html> The following are the most important steps of the previous code: Define an HTML5 <canvas> object with a width of 960 pixels and a height of 450 pixels. This represents the drawing area of your EaselJS application. When the page is completely loaded, the onload event is fired and the init() function is called. The <script> block is the place where you have to add the code but you should always wait for the onload events before you do anything. Set the <body> and <canvas> background CSS styles. The result is a white container inside an HTML page, as shown in the following screenshot: Step 2 – creating a "Hello World" example Now replace the init() function with the following code: function init() { var canvas = document.getElementById("mycanvas"); var stage = new createjs.Stage(canvas); var text = new createjs.Text("Hello World!", "36px Arial", "#777"); stage.addChild(text); text.x = 360; text.y = 200; stage.update(); } Congrats! You have created your first canvas application! The following screenshot shows the output of the previous code, with a text field at the center of the canvas: The following are the most important steps of the previous code: Use the getElementById method to get a canvas reference. In order to use EaselJS, create a Stage property, passing the canvas reference as a parameter. Create a new Text property and add it to the stage. Assign values for the x and y coordinates in order to see the text at the center of the stage. Call the update() method on the stage to render it to the canvas. The Stage property represents the root level for the display list, which is the main container for all the other graphic elements. Now you only need to know that every graphic element must be added to the Stage property, and that every time you need to update your content you have to refresh the stage calling the update() method. Summary After the previous "Hello World" example the book will help you to learn how to use the most important EaselJS topics with practical examples, technical information, and a lot of tip and tricks, creating a small advertising interactive web application. By the end of book you will be able to draw graphic primitives and texts, load and preload images, handle mouse events, add animations and spritesheets, use TweenJS, PreloadJS, and Zoe and optimize your code for desktop and mobile devices. This article helped you to learn what EaselJS actually is, what you can do with it, and why it's so great. It also helped on hoe to create your first HTML5 canvas application "Hello World". Resources for Article : Further resources on this subject: HTML5: Developing Rich Media Applications using Canvas [Article] HTML5 Games Development: Using Local Storage to Store Game Data [Article] HTML5: Getting Started with Paths and Text [Article]

(For more resources related to this topic, see here.) ForgedUI is a Titanium Studio third-party WYSIWYG (what you see is what you get) module created with the aim to make cross-platform app development quicker and easier, by providing a visual environment and a drag-and-drop style interface. Even though Titanium generates multiple platform apps with a single codebase and facilitates the maintenance and management of mobile projects, it still lacks a design interface tool, and for this reason, ForgedUI comes to the market with the aim to improve what has been holding back a large portion of productivity gains on Titanium SDK. With a steep learning curve and a straightforward interface, Titanium developers are able to reduce app development time by visually designing their apps and get more time to concentrate on other aspects of the project. It doesn't matter if you choose to go with iOS or Android, ForgedUI will give you a hand on screen design, and alongside Titanium Studio, generate the cross-platform code with one click. The ForgedUI interface has common UI elements of the Android and iOS platforms and allows you to design with a simple drag-and-drop process instead of hand written code. This is how ForgedUI looks within Titanium Studio: Once you are happy with your UI design, you can generate the Titanium JavaScript code through ForgedUI with just one click, and this code can be integrated into new or old projects. The generated code is stored in a separate resource file within the current Titanium project. ForgedUI allows you to specify parent-child relationships between compatible components and intelligently generates code that conforms to the required layout and relationship rules. The key features of ForgedUI are as follows: Visual graphical user interface (GUI) designer Supports iPhone and Android (no Tablets) platform projects One-click code generation Seamless integration with Titanium Studio Summary This article has thus explained what ForgedUI is and how it makes cross-platform app development easier. Resources for Article : Further resources on this subject: Animating Properties and Tweening Pages in Android 3-0 [Article] Securely Encrypt Removable Media with Ubuntu [Article] Appcelerator Titanium: Creating Animations, Transformations, and Understanding Drag-and-drop [Article]

(For more resources related to this topic, see here.) What is cloud computing? If you have been in the IT industry for some time, you probably know what cloud means. For the rest, it is used as a metaphor for the worldwide network or the Internet. Computing normally indicates the use of computer hardware and software. Combining these two terms, we get a simple definition—use of computer resources over the Internet (as a service). In other words, when the computing is delegated to resources available over the Internet, we get what is called cloud computing. As Wikipedia defines it: Cloud computing is the delivery of computing as a service rather than a product, whereby shared resources, software, and information are provided to computers and other devices as a utility (like the electricity grid) over a network (typically the Internet). Still confused? A simple example will help clarify it. Say you are managing the IT department of an organization, where you are responsible for purchasing hardware and software (licenses) for your employees and making sure they have the right resources to do their jobs. Whenever there is a new hire, you need to go through all the purchase formalities once again to get your user the necessary resources. Soon this turns out to be a nightmare of managing all your software licenses! Now, what if you could find an alternative where you host an application on the Web, which your users can access through their browsers and interact with it? You are freed from maintaining individual licenses and maintaining high-end hardware at the user machines. Voila, we just discovered cloud computing! Cloud computing is the logical conclusion drawn from observing the drawbacks of in-house solutions. The trend is now picking up and is quickly replacing the onpremise software application delivery models that are accompanied with high costs of managing data centers, hardware, and software. All users pay for is the quantum of the services that they use. That is why it's sometimes also known as utility-based computing, as the corresponding payment is resource usage based. Chances are that even before you ever heard of this term, you had been using it unknowingly. Have you ever used hosted e-mail services such as Yahoo, Hotmail, or Gmail where you accessed all of their services through the browser instead of an e-mail client on your computer? Now that is a typical example of cloud computing. Anything that is offered as a service (aaS) is usually considered in the realm of cloud computing. Everything in the cloud means no hardware, no software, so no maintenance and that is what the biggest advantage is. Different types of services that are most prominently delivered on the cloud are as follows: Infrastructure as a service (IaaS) Platform as a service (PaaS) Software as a service (SaaS) Infrastructure as a service (IaaS) Sometimes referred to hardware as a service, infrastructure as a service offers the IT infrastructure, which includes servers, routers, storages, firewalls, computing resources, and so on, in physical or virtualized forms as a service. Users can subscribe to these services and pay on the basis of need and usage. The key player in this domain is Amazon.com, with EC2 and S3 as examples of typical IaaS. Elastic Cloud Computing (EC2) is a web service that provides resizable computing capacity in the cloud. Computing resources can be scaled up or down within minutes, allowing users to pay for the actual capacity being used. Similarly, S3 is an online storage web service offered by Amazon, which provides 99.999999999 percent durability and 99.99 percent availability of objects over a given year and stores arbitrary objects (computer files) up to 5 terabytes in size! Platform as a service (PaaS) PaaS provides the infrastructure for development of software applications. Accessed over the cloud, it sits between IaaS and SaaS where it hides the complexities of dealing with underlying hardware and software. It is an application-centric approach that allows developers to focus more on business applications rather than infrastructure-level issues. Developers no longer have to worry about the server upgrades, scalability, load balancing, service availability, and other infrastructure hassles, as these are delegated to the platform vendors. Paas allows development of custom applications by providing the appropriate building blocks and the necessary infrastructure available as a service. An excellent example, in this category, is the Force.com platform, which is a game changer in the aaS, specially in the PaaS domain. It exposes a proprietary application development platform, which is woven around a relational database. It stands at a higher level than another key player in this domain, Google App Engine, which supports scalable web application development in Java and Python on the appropriate application server stack, but does not provide equivalent robust proprietary components or the building blocks as Force.com. Another popular choice (or perhaps not) is Microsoft's application platform called Widows Azure, which can be used to build websites (developed in ASP.NET, PHP, Node.JS), provision virtual machines, and provide cloud services (containers of hosted applications). A limitation with applications built on these platforms is the quota limits, or the strategy to prohibit the monopolization of the shared resources in the multitenant environment. Some developers see this as a restriction, which allows them to build applications with limited capability, but we reckon this as an opportunity to build highly efficient solutions to work within governor limits, while still maintaining the business process sanctity. Specificcally for the Force.com platform, some people consider shortage of skilled resources as a possible limitation, but we think the learning curve is steep on this platform and an experienced resource can pick proprietary languages pretty quickly, average ramp up time spanning anywhere from 15 to 30 days! Software as a service (SaaS) The opposite end of IaaS is SaaS. Business applications are offered as services over the Internet to users who don't have to go through the complex custom application development and implementation cycles. They also don't invest upfront on the IT infrastructure or maintain their software with regular upgrades. All this is taken care of by the SaaS vendors. These business applications normally provide the customization capability to accommodate specific business needs such as user interfaces, business workflows, and so on. Some good examples in this category are the Salesforce.com CRM system and Google Apps services. What is Force.com? Force.com is a natural progression from Salesforce.com, which was started as a sales force automation system offered as a service (SaaS). The need to go beyond the initially offered customizable CRM application and develop custom-based solutions, resulted in a radical shift of cloud delivery model from SaaS to PaaS. The technology that powers Salesforce CRM, whose design fulfills all the prerequisites of being a cloud application, is now available for developing enterprise-level applications. An independent study of the Force.com platform concluded that compared to the traditional Java-based application development platform, development with the Force.com platform is almost five times faster, with about a 40 percent smaller overall project cost and better quality due to rapid prototyping during the requirement gathering—thanks to the declarative aspect of the Force.com development—and less testing due to proven code re-use. What empowers Force.com? Why is Force.com application development so successful? Primarily because of its key architectural features, discussed in the following sections. Multitenancy Multitenancy is a concept that is the opposite of single-tenancy. In the Cloud Computing jargon, a customer or an organization is referred to as tenant. The various downsides and cost inefficiencies of single-tenant models are overcame by the multitenant model. A multitenant application caters to multiple organizations, each working in its own isolated virtual environment called org and sharing a single physical instance and version of the application hosted on the Force.com infrastructure. It is isolated because although the infrastructure is shared, every customer's data, customizations, and code remain secure and insulated from other customers. Multitenant applications run on a single physical instance and version of the application, providing the same robust infrastructure to all their customers. This also means freedom from upfront costs, ongoing upgrades, and maintenance costs. The test methods written by the customers on respective orgs ensure more than 75 percent code coverage and thus help Salesforce.com in regression testing of the Force.com upgrades, releases, and patches. The same is difficult to even visualize with an in-house software application development. Metadata What drives the multitenant applications on Force.com? Nothing else but the metadata-driven architecture of the platform! Think about the following: The platform allows all tenants to coexist at the same time Tenants can extend the standard common object model without affecting others Tenants' data is kept isolated from others in a shared database The platform customizes the interface and business logic without disrupting the services for others The platform's codebase can be upgraded to offer new features without affecting the tenants' customizations The platform scales up with rising demands and new customers To meet all the listed challenges, Force.com has been built upon a metadata-driven architecture, where the runtime engine generates application components from the metadata. All customizations to the standard platform for each tenant are stored in the form of metadata, thus keeping the core Force.com application and the client customizations distinctly separate, making it possible to upgrade the core without affecting the metadata. The core Force.com application comprises the application data and the metadata describing the base application, thus forming three layers sitting on top of each other in a common database, with the runtime engine interpreting all these and rendering the final output in the client browser. As metadata is a virtual representation of the application components and customizations of the standard platform, the statically compiled Force.com application's runtime engine is highly optimized for dynamic metadata access and advanced caching techniques to produce remarkable application response times. Understanding the Force.com stack A white paper giving an excellent explanation of the Force.com stack has been published. It describes various layers of technologies and services that make up the platform. We will also cover it here briefly. The application stack is shown in the following diagram: Infrastructure as a service Infrastructure is the first layer of the stack on top of which other services function. It acts as the foundation for securely and reliably delivering the cloud applications developed by the customers as well as the core Salesforce CRM applications. It powers more than 200 million transactions per day and more than 1.5 million subscribers. The highly managed data centers provide unparalleled redundancy with near-real-time replication, world class security at physical, network, host, data transmission, and database levels, and excellent design to scale both vertically and horizontally. Database as a service The powerful and reliable data persistence layer in the Force.com stack is known as the Force.com database. It sits on top of the infrastructure and provides the majority of the Force.com platform capabilities. The declarative web interface allows user to create objects and fields generating the native application UI around them. Users can also define relationships between objects, create validation rules to ensure data integrity, track history on certain fields, create formula fields to logically derive new data values, create fine-grained security access with the point and click operations, and all of this without writing a single line of code or even worrying about the database backup, tuning, upgrade, and scalability issues! As compared with the relational database, it is similar in the sense that the object (a data instance) and fields are analogous to tables and columns, and Force.com relationships are similar to the referential integrity constraints in a relation DB. But unlike physically separate tables with dedicated storage, Force.com objects are maintained as a set of metadata interpreted on the fly by the runtime engine and all of the application data is stored in a set of a few large database tables. This data is represented as virtual records based on the interpretation of tenants' customizations stored as metadata. Integration as a service Integration as a service utilizes the underlying Force.com database layer and provides the platform's integration capabilities through the open-standards-based web services API. In today's world, most organizations have their applications developed on disparate platforms, which have to work in conjunction to correctly represent and support their internal business processes. Customers' existing applications can connect with Force.com through the SOAP or REST web services to access data and create mashups to combine data from multiple sources. The Force.com platform also allows native applications to integrate with third-party web services through callouts to include information from external systems in organizations' business processes. These integration capabilities of the platform through API (for example, Bulk API, Chatter API, Metadata API, Apex REST API, Apex SOAP API, Streaming API, and so on) can be used by developers to build custom integration solutions to both produce and consume web services. Accordingly, it's been leveraged by many third parties such as Informatica, Cast Iron, Talend, and so on, to create prepackaged connectors for applications and systems such as Outlook, Lotus Notes, SAP, Oracle Financials, and so on. It also allows clouds such as Facebook, Google, and Amazon to talk to each other and build useful mashups. The integration ability is the key for developing mobile applications for various device platforms, which solely rely on the web services exposed by the Force.com platform. Logic as a service A development platform has to have the capability to create business processes involving complex logic. The Force.com platform oversimplifies this task to automate a company's business processes and requirements. The platform logic features can be utilized by both developers and business analysts to build smart database applications that help increase user productivity, improve data quality, automate manual processes, and adapt quickly to changing requirements. The platform allows creating the business logic either through a declarative interface in the form of workflow rules, approval processes, required and unique fields, formula fields, validation rules, or in an advanced form by writing triggers and classes in the platform's programming language—Apex—to achieve greater levels of flexibility, which help define any kind of functionality and business requirement that otherwise may not be possible through the point and click operations. User interface as a service The user interface of platform applications can be created and customized by either of the two approaches. The Force.com builder application, an interface based on point-and-click/drag-and-drop, allows users to build page layouts that are interpreted from the data model and validation rules with user defined customizations, define custom application components, create application navigation structures through tabs, and define customizable reports and user-specific views. For more complex pages and tighter control over the presentation layer, a platform allows users to build custom user interfaces through a technology called Visualforce (VF), which is based on the XML markup tags. The custom VF pages may or may not adopt the standard look and feel based on the stylesheet applied and present data returned from the controller or the logic layer in the structured format. The Visualforce interfaces are either public, private, or a mix of the two. Private interfaces require users to log in to the system before they can access resources, whereas public interfaces, called sites, can be made available on the Internet to anonymous users. Development as a service This a set of features that allow developers to utilize traditional practices for building cloud applications. These features include the following: Force.com Metadata API: Lets developers push changes directly into the XML files describing the organization's customizations and acts as an alternative to platform's interface to manage applications IDE (Integrated Development Environment): A powerful client application built on the Eclipse platform, allowing programmers to code, compile, test, package, and deploy applications A development sandbox: A separate application environment for development, quality assurance, and training of programmers Code Share: A service for users around the globe to collaborate on development, testing, and deployment of the cloud applications Force.com also allows online browser based development providing code assist functionality, repository search, debugging, and so on, thus eliminating the need of a local machine specific IDE. DaaS expands the Cloud Computing development process to include external tools such as integrated development environments, source control systems, and batch scripts to facilitate developments and deployments. Force.com AppExchange This is a cloud marketplace (accessible at http://appexchange.salesforce.com/) that helps commercial application vendors to publish their custom development applications as packages and then reach out to potential customers who can install them on their orgs with merely a button click through the web interface, without going through the hassles of software installation and configuration. Here, you may find good apps that provide functionality, that are not available in Salesforce, or which may require some heavy duty custom development if carried out on-premises! Introduction to governor limits Any introduction to Force.com is incomplete without a mention of governor limits. By nature, all multitenant architecture based applications such as Force.com have to have a mechanism that does not allow the code to abuse the shared resources so that other tenants in the infrastructure remain unaffected. In the Force.com world, it is the Apex runtime engine that takes care of such malicious code by enforcing runtime limits (called governor limits) in almost all areas of programming on the Force.com platform. If these governor limits had not been in place, even the simplest code, such as an endless loop, would consume enough resources to disrupt the service to the other users of the system, as they all share the same physical infrastructure. The concept of governor limits is not just limited to Force.com, but extends to all SaaS/PaaS applications, such as Google App Engine, and is critical for making the cloud-based development platform stable. This concept may prove to be very painful for some people, but there is a key logic to it. The platform enforces the best practices so that the application is practically usable and makes an optimal usage of resources, keeping the code well under governor limits. So the longer you work on Force.com, the more you become familiar with these limits, the more stable your code becomes over time, and the easier it becomes to work around these limits. In one of the forthcoming chapters, we will discover how to work with these governor limits and not against them, and also talk about ways to work around them, if required. Salesforce environments An environment is a set of resources, physical or logical, that let users build, test, deploy, and use applications. In the traditional development model, one would expect to have application servers, web servers, databases, and their costly provisioning and configuration. But in the Force.com paradigm, all that's needed is a computer and an Internet connection to immediately get started to build and test a SaaS application. An environment, or a virtual or logical instance of the Force.com infrastructure and platform, is also called an organization or just org, which is provisioned in the cloud on demand. It has the following characteristics: Used for development, testing, and/or production Contains data and customizations Based on the edition containing specific functionality, objects, storage, and limits Certain restricted functionalities, such as the multicurrency feature (which is not available by default), can be enabled on demand All environments are accessible through a web browser There are broadly three types of environments available for developing, testing, and deploying applications: Production environments: The Salesforce.com environments that have active paying users accessing the business critical data. Development environments: These environments are used strictly for the development and testing applications with data that is not business critical, without affecting production environment. Developer environments are of two types: Developer Edition: This is a free, full-featured copy of the Enterprise Edition, with less storage and users. It allows users to create packaged applications suitable for any Salesforce production environment. It can be of two types: Regular Developer Edition: This is a regular DE org whose sign up is free and the user can register for any number of DE orgs. This is suitable when you want to develop managed packages for distribution through AppExchange or Trialforce, when you are working with an edition where sandbox is not available, or if you just want to explore the Force.com platform for free. Partner Developer Edition: This is a regular DE org but with more storage, features, and licenses. This is suitable when you expect a larger team to work who need a bigger environment to test the application against a larger real-life dataset. Note that this org can only be created with the Salesforce Consulting partners or Force.com ISV. Sandbox: This is nearly an identical copy of the production environment available to Enterprise or Unlimited Edition customers, and can contain data and/or customizations. This is suitable when developing applications for production environments only with no plans to distribute applications commercially through AppExchange or Trialforce, or if you want to test the beta-managed packages. Note that sandboxes are completely isolated from your Salesforce production organization, so operations you perform in your sandboxes do not affect your Salesforce production organization, and vice versa. Types of sandboxes are as follows: Full copy sandbox: Nearly an identical copy of the production environment, including data and customizations Configuration-only sandbox: Contains only configurations and not data from the production environment Developer sandbox: Same as Configuration-only sandbox but with less storage Test environments: These can be either production or developer environments, used speficially for testing application functionality before deploying to production or releasing to customers. These environments are suitable when you want to test applications in production such as environments with more users and storage to run real-life tests. Summary This article talked about the basic concepts of cloud computing. The key takeaway items from this article are the explanations of the different types of cloud-based services such as IaaS, SaaS, and PaaS. We introduced the Force.com platform and its key architectural features that power the platform types, such as multitenant and metadata. We briefly covered the application stack—technology and services layers—that makes up the Force.com platform. We gave an overview of governor limits without going too much detail about their use. We discussed situations where adopting cloud computing may be beneficial. We also discussed the guidelines that help you decide whether your software project should be developed on the Force.com platform or not. Last, but not least, we discussed various environments available to developers and business users and their characteristics and usage. Resources for Article : Further resources on this subject: Monitoring and Responding to Windows Intune Alerts [Article] Sharing a Mind Map: Using the Best of Mobile and Web Featuressil [Article] Force.com: Data Management [Article]

(For more resources related to this topic, see here.) Constructing proper attribute query syntax The construction of property attribute queries is critical to your success in creating geoprocessing scripts that query data from feature classes and tables. All attribute queries that you execute against feature classes and tables will need to have the correct SQL syntax and also follow various rules depending upon the datatype that you execute the queries against. Getting ready Creating the syntax for attribute queries is one of the most difficult and time-consuming tasks that you'll need to master when creating Python scripts that incorporate the use of the Select by Attributes tool. These queries are basically SQL statements along with a few idiosyncrasies that you'll need to master. If you already have a good understanding of creating queries in ArcMap or perhaps an experience with creating SQL statements in other programming languages, then this will be a little easier for you. In addition to creating valid SQL statements, you also need to be aware of some specific Python syntax requirements and some datatype differences that will result in a slightly altered formatting of your statements for some datatypes. In this recipe, you'll learn how to construct valid query syntax and understand the nuances of how different datatypes alter the syntax as well as some Python-specific constructs. How to do it… Initially, we're going to take a look at how queries are constructed in ArcMap, so that you can get a feel of how they are structured. In ArcMap, open C:ArcpyBookCh8Crime_Ch8.mxd. Right-click on the Burglaries in 2009 layer and select Open Attribute Table. You should see an attribute table similar to the following screenshot. We're going to be querying the SVCAREA field: With the attribute table open, select the Table Options button and then Select by Attributes to display a dialog box that will allow you to construct an attribute query. Notice the Select * FROM Burglary WHERE: statement on the query dialog box (shown in the following screenshot). This is a basic SQL statement that will return all the columns from the attribute table for Burglary that meet the condition that we define through the query builder. The asterisk (*) simply indicates that all fields will be returned: Make sure that Create a new selection is the selected item in the Method dropdown list. This will create a new selection set. Double-click on SVCAREA from the list of fields to add the field to the SQL statement builder, as follows: Click on the = button. Click on the Get Unique Values button. From the list of values generated, double-click on 'North' to complete the SQL statement, as shown in the following screenshot: Click on the Apply button to execute the query. This should select 7520 records. Many people mistakenly assume that you can simply take a query that has been generated in this fashion and paste it into a Python script. That is not the case. There are some important differences that we'll cover next. Close the Select by Attributes window and the Burglaries in 2009 table. Clear the selected feature set by clicking on Selection | Clear Selected Features. Open the Python window and add the code to import arcpy. import arcpy Create a new variable to hold the query and add the exact same statement that you created earlier: qry = "SVCAREA" = 'North' Press Enter on your keyboard and you should see an error message similar to the following: Runtime error SyntaxError: can't assign to literal (<string>, line1) Python interprets SVCAREA and North as strings but the equal to sign between the two is not part of the string used to set the qry variable. There are several things we need to do to generate a syntactically correct statement for the Python interpreter. One important thing has already been taken care of though. Each field name used in a query needs to be surrounded by double quotes. In this case, SVCAREA is the only field used in the query and it has already been enclosed by double quotes. This will always be the case when you're working with shapefiles, file geodatabases, or ArcSDE geodatabases. Here is where it gets a little confusing though. If you're working with data from a personal geodatabase, the field names will need to be enclosed by square brackets instead of double quotes as shown in the following code example. This can certainly leads to confusion for script developers. qry = [SVCAREA] = 'North' Now, we need to deal with the single quotes surrounding 'North'. When querying data from fields that have a text datatype, the string being evaluated must be enclosed by quotes. If you examine the original query, you'll notice that we have in fact already enclosed the word North with quotes, so everything should be fine right? Unfortunately, it's not that simple with Python. Quotes, along with a number of other characters, must be escaped with a forward slash followed by the character being escaped. In this case, the escape sequence would be '. Alter your query syntax to incorporate the escape sequence: qry = "SVCAREA" = 'North' Finally, the entire query statement should be enclosed with quotes: qry = '"SVCAREA" = 'North'' In addition to the = sign, which tests for equality, there are a number of additional operators that you can use with strings and numeric data, including not equal (> <), greater than (<), greater than or equal to (<=), less than (>), and less than or equal to (>=). Wildcard characters including % and _ can also be used for shapefiles, file geodatabases, and ArcSDE geodatabases. These include % for representing any number of characters. The LIKE operator is often used with wildcard characters to perform partial string matching. For example, the following query would find all records with a service area that begins with N and has any number of characters after. qry = '"SVCAREA" LIKE 'N%'' The underscore character (_) can be used to represent a single character. For personal geodatabases the asterisk (*) is used to represent a wildcard character for any number of characters, while (?) represents a single character. You can also query for the absence of data, also known as NULL values. A NULL value is often mistaken for a value of zero, but that is not the case. NULL values indicate the absence of data, which is different from a value of zero. Null operators include IS NULL and IS NOT NULL. The following code example will find all records where the SVCAREA field contains no data: qry = '"SVCAREA" IS NULL' The final topic that we'll cover in this section are operators used for combining expressions where multiple query conditions need to be met. The AND operator requires that both query conditions be met for the query result to be true, resulting in selected records. The OR operator requires that at least one of the conditions be met. How it works… The creation of syntactically correct queries is one of the most challenging aspects of programming ArcGIS with Python. However, once you understand some basic rules, it gets a little easier. In this section, we'll summarize these rules. One of the more important things to keep in mind is that field names must be enclosed with double quotes for all datasets, with the exception of personal geodatabases, which require braces surrounding field names. There is also an AddFieldDelimiters() function that you can use to add the correct delimiter to a field based on the datasource supplied as a parameter to the function. The syntax for this function is as follows: AddFieldDelimiters(dataSource,field) Additionally, most people, especially those new to programming with Python, struggle with the issue of adding single quotes to string values being evaluated by the query. In Python, quotes have to be escaped with a single forward slash followed by the quote. Using this escape sequence will ensure that Python does in fact see that as a quote rather than the end of the string. Finally, take some time to familiarize yourself with the wildcard characters. For datasets other than personal geodatabases, you'll use the (%) character for multiple characters and an underscore (_) character for a single character. If you're using a personal geodatabase, the (*) character is used to match multiple characters and the (?) character is used to match a single character. Obviously, the syntax differences between personal geodatabases and all other types of datasets can lead to some confusion. Creating feature layers and table views Feature layers and table views serve as intermediate datasets held in memory for use specifically with tools such as Select by Location and Select Attributes. Although these temporary datasets can be saved, they are not needed in most cases. Getting ready Feature classes are physical representations of geographic data and are stored as files (shapefiles, personal geodatabases, and file geodatabases) or within a geodatabase. ESRI defines a feature class as "a collection of features that shares a common geometry (point, line, or polygon), attribute table, and spatial reference." Feature classes can contain default and user-defined fields. Default fields include the SHAPE and OBJECTID fields. These fields are maintained and updated automatically by ArcGIS. The SHAPE field holds the geometric representation of a geographic feature, while the OBJECTID field holds a unique identifier for each feature. Additional default fields will also exist depending on the type of feature class. A line feature class will have a SHAPE_LENGTH field. A polygon feature class will have both, a SHAPE_LENGTH and a SHAPE_AREA field. Optional fields are created by end users of ArcGIS and are not automatically updated by GIS. These contain attribute information about the features. These fields can also be updated by your scripts. Tables are physically represented as standalone DBF tables or within a geodatabase. Both, tables and feature classes, contain attribute information. However, a table contains only attribute information. There isn't a SHAPE field associated with a table, and they may or may not contain an OBJECTID field. Standalone Python scripts that use the Select by Attributes or Select by Location tool require that you create an intermediate dataset rather than using feature classes or tables. These intermediate datasets are temporary in nature and are called Feature Layers or Table Views. Unlike feature classes and tables, these temporary datasets do not represent actual files on disk or within a geodatabase. Instead, they are "in memory" representations of feature classes and tables. These datasets are active only while a Python script is running. They are removed from memory after the tool has executed. However, if the script is run from within ArcGIS as a script tool, then the temporary layer can be saved either by right-clicking on the layer in the table of contents and selecting Save As Layer File or simply by saving the map document file. Feature layers and table views must be created as a separate step in your Python scripts, before you can call the Select by Attributes or Select by Location tools. The Make Feature Layer tool generates the "in-memory" representation of a feature class, which can then be used to create queries and selection sets, as well as to join tables. After this step has been completed, you can use the Select by Attributes or Select by Location tool. Similarly, the Make Table View tool is used to create an "in-memory" representation of a table. The function of this tool is the same as Make Feature Layer. Both the Make Feature Layer and Make Table View tools require an input dataset, an output layer name, and an optional query expression, which can be used to limit the features or rows that are a part of the output layer. In addition, both tools can be found in the Data Management Tools toolbox. The syntax for using the Make Feature Layer tool is as follows: arcpy.MakeFeatureLayer_management(<input feature layer>, <output layer name>,{where clause}) The syntax for using the Make Table View tool is as follows: Arcpy.MakeTableView_management(<input table>, <output table name>, {where clause}) In this recipe, you will learn how to use the Make Feature Layer and Make Table View tools. These tasks will be done inside ArcGIS, so that you can see the in-memory copy of the layer that is created. How to do it… Follow these steps to learn how to use the Make Feature Layer and Make Table View tools: Open c:ArcpyBookCh8Crime_Ch8.mxd in ArcMap. Open the Python window. Import the arcpy module: import arcpy Set the workspace: arcpy.env.workspace = "c:/ArcpyBook/data/CityOfSanAntonio.gdb" Start a try block: try: Make an in-memory copy of the Burglary feature class using the Make Feature Layer tool. Make sure you indent this line of code: flayer = arcpy.MakeFeatureLayer_management("Burglary","Burglary_ Layer") Add an except block and a line of code to print an error message in the event of a problem: except: print "An error occurred during creation" The entire script should appear as follows: import arcpy arcpy.env.workspace = "c:/ArcpyBook/data/CityOfSanAntonio.gdb" try: flayer = arcpy.MakeFeatureLayer_management("Burglary","Burglary_ Layer") except: print "An error occurred during creation" Save the script to c:ArcpyBookCh8CreateFeatureLayer.py. Run the script. The new Burglary_Layer file will be added to the ArcMap table of contents: The Make Table View tool functionality is equivalent to the Make Feature Layer tool. The difference is that it works against standalone tables instead of feature classes. Remove the following line of code: flayer = arcpy.MakeFeatureLayer_management("Burglary","Burglary_ Layer") Add the following line of code in its place: tView = arcpy.MakeTableView_management("Crime2009Table", "Crime2009TView") Run the script to see the table view added to the ArcMap table of contents. How it works... The Make Feature Layer and Make Table View tools create in-memory representations of feature classes and tables respectively. Both the Select by Attributes and Select by Location tools require that these temporary, in-memory structures be passed in as parameters when called from a Python script. Both tools also require that you pass in a name for the temporary structures. There's more... You can also apply a query to either the Make Feature Layer or Make Table View tools to restrict the records returned in the feature layer or table view. This is done through the addition of a where clause when calling either of the tools from your script. This query is much the same as if you'd set a definition query on the layer through Layer Properties | Definition Query. The syntax for adding a query is as follows: MakeFeatureLayer(in_features, out_layer, where_clause) MakeTableView(in_table, out_view, where_clause)

(For more resources related to this topic, see here.) Overview about portals Well, to understand more about what portals are, let me throw some familiar words at you. Have you used, heard, or seen iGoogle, the Yahoo! home page, or MSN? If the answer is yes, then you have been using portals already. All these websites have two things in common. A common dashboard Information from various sources shown on a single page, giving a uniform experience For example, on iGoogle, you can have a gadget showing the weather in Chicago, another gadget to play your favorite game of Sudoku, and a third one to read news from around the globe, everything on the same page without you knowing that all of these are served from different websites! That is what a portal is all about. So, a portal (or web portal) can be thought of as a website that shows, presents, displays, or brings together information or data from various sources and gives the user a uniform browsing experience. The small chunks of information that form the web page are given different names such as gadgets or widgets, portlets or dashlets. Introduction to Liferay Now that you have some basic idea about what portals are, let us revisit the initial statement I made about Liferay. Liferay is an open source portal solution. If you want to create a portal, you can use Liferay to do this. It is written in Java. It is an open source solution, which means the source code is freely available to everyone and people can modify and distribute it. With Liferay you can create basic intranet sites with minimal tweaking. You can also go for a full-fledged enterprise banking portal website with programming, and heavy customizations and integrations. Besides the powerful portal capabilities, Liferay also provides the following: Awesome enterprise and web content management capabilities Robust document management which supports protocols such as CMIS and WebDAV Good social collaboration features Liferay is backed up by a solid and active community, whose members are ever eager to help. Sounds good? So what are we waiting for? Let's take a look at Liferay and its features. Installation and setup In four easy steps, you can install Liferay and run it on your system. Step 1 – Prerequisites Before we go and start our Liferay download, we need to check if we have the requirements for the installation. They are as follows: Memory: 2 GB (minimum), 4 GB (recommended). Disk space: Around 5 GB of free space should be more than enough for the exercises mentioned in the book. The exercises performed in this book are done on Windows XP. So you can use the same or any subsequent versions of Windows OS. Although Liferay can be run on Mac OSX and Linux, it is beyond the scope of this book how to set up Liferay on them. The MySQL database should be installed. As with the OS, Liferay can be run on most of the major databases out there in the market. Liferay is shipped with the Hypersonic database by default for demo purpose, which should not be used for a production environment. Unzip tools such as gzip or 7-Zip. Step 2 – Downloading Liferay You can download the latest stable version of Liferay from https://www.liferay.com/downloads/liferay-portal/available-releases. Liferay comes in the following two versions: Enterprise Edition: This version is not free and you would have to purchase it. This version has undergone rigorous testing cycles to make sure that all the features are bug free, providing the necessary support and patches. Community Edition: This is a free downloadable version that has all the features but no enterprise support provided. Liferay is supported by a lot of open source application servers and the folks at Liferay have made it easy for end users by packaging everything as a bundle. What this means is that if you are asked to have Liferay installed in a JBoss application server, you can just go to the URL previously mentioned and select the Liferay-JBoss bundle to download, which gives you the JBoss Application server installed with Liferay. We will download the Community Edition of the Liferay-Tomcat bundle, which has Liferay preinstalled in the Tomcat server. The stable version at the time of writing this book was Liferay 6.1 GA2. As shown in the following screenshot, just click on Download after making sure that you have selected Liferay bundled with Tomcat and save the ZIP file at an appropriate location: Step 3 – Starting the server After you have downloaded the bundle, extract it to the location of your choice on your machine. You can see a folder named liferay-portal-6.1.1-ce-ga2. The latter part of the name can change based on the version that you download. Let us take a moment to have a look at the folder structure as shown in the following screenshot: The liferay-portal-6.1.1-ce-ga2 folder is what we will refer to as LIFERAY_HOME. This folder contains the server, which in our case is tomcat-7.0.27. Let's refer to this folder as SERVER_HOME. Liferay is created using Java, so to run Liferay we need Java Runtime Environment (JRE). The Liferay bundle is shipped with a JRE by default (as you can see inside our SERVER_HOME). So if you are running a Windows OS, you can directly start and run Liferay. If you are using any other OS, you need to set the JAVA_HOME environment variable. Navigate to SERVER_HOME/webapps. This is where all the web applications are deployed. Delete everything in this folder except marketplace-portlet and ROOT. Now go to SERVER/bin and double-click on startup.bat, since we are using Windows OS. This will bring up a console showing the server startup. Wait till you see the Server Startup message in the console, after which you can access Liferay from the browser. Step 4 – Doing necessary first-time configurations Once the server is up, open your favorite browser and type in http://localhost:8080. You will be shown a screen that performs basic configurations, such as changing the database and name of your portal, deciding what should be the admin name and e-mail address, or changing the default locale. This is a new feature introduced in Liferay 6.1 to ease the first-time setup, which on previous versions had to be done using the property file. Go change the name of the portal, administrator username, and e-mail address. Keep the locale as it is. As I stated earlier, Liferay is shipped with a default Hypersonic database which is normally used for demo purposes. You can change it to MySQL if you want, by selecting the database type from the drop-down list presented, and typing in the necessary JDBC details. I have created a database in MySQL by the name Portal Starter; hence my JDBC URL would contain that. You can create a blank database in MySQL and accordingly change the JDBC URL. Once you are done making your changes, click on the Finish Configuration button as shown in the following screenshot: This will open up a screen, which will show the path where this configuration is saved. What Liferay does behind the scenes is creates a property file named portal-setup-wizard.properties and put all the configurations in that. This, as I said earlier, was created manually in the previous versions of Liferay. Clicking on the Go to my portal button on this screen will take the user to the Terms of Use page. Agree to the terms and proceed further. A screen will be shown to change the password for your admin user that you earlier specified in the Basic Configuration screen. After you change the password, you will be presented with a screen to select a password reminder question. Select a question or create your own question from the drop-down list, set the password reminder, and move on. And that's it!! Finally, you can see the home page of Liferay. That's it and you are done setting up your very first Liferay instance. Summary So, we just gained a quick understanding about portals and Liferay and its installation and setup that teaches you how to set up Liferay on your local machine. Resources for Article : Further resources on this subject: Vaadin Portlets in Liferay User Interface Development [Article] Setting up and Configuring a Liferay Portal [Article] User Interface in Production [Article]