How-To Tutorials

Three.js is an awesome library. It makes complicated things such as 3D graphics and shaders easy for the average frontend developer, which opens up a lot of previously inaccessible avenues for web development. You can check out this repository of examples to see what's possible (basically, the sky is the limit). Even though Three.js provides all of this funcitonality, it is one of the lesser-documented libraries, and hence can be a little bit overwhelming for a newcomer. This tutorial will take you through the steps required to render a nice little scene with the Three.js library. Project Structure and Prerequisites We will be using npm and webpack to make our application, along with ES6 syntax. Initialize a new project in a new folder: npm init After that, install Three.js: npm install --save three And we are all set! Making a Cubemap A cubemap, if you haven't heard of it before, is precisely what its name suggests. Think of six enormous square pictures, all joined together to form a cube, with you being inside of the cube. The six pictures then form a cubemap. It is used to make 3D background sceneries and fillers. Every rendering in 3D graphics has two elements: the scene and the camera. The renderer then renders the scene relative to the camera. In this way, you can move through a scene by adjusting its camera, and at the same time stay still with respect to another scene because of its camera. This is the basic principle used while making movement-based 3D graphics. You (or in this case, your camera) are standing still with respect to the background (or in some cases, moving really slowly), considering it to be at a near-infinite distance from you. At the same time, you will be moving with respect to the objects around you, since they are considered to be within your immediate distance. First, we have to import our dependencies and initialize important objects. As mentioned before, each rendering will have a scene and a camera. Three.js provides constructors for both of these things. The arguments we see for the camera are parameters such as frustum vertical, aspect ratio, near frame, and far frame, which are much beyond the scope of this post. For almost all cases, these numbers can be consdered as defaults and need not be changed. import THREE from 'three'; let sceneCube = new THREE.Scene(); let cameraCube = new THREE.PerspectiveCamera( 60, window.innerWidth / window.innerHeight, 1, 100000 ); Each cubemap is composed of six images. In the snippet below, we are just making an array of paths, which is where all our images are kept. In this case, the image for one side of our cube (the positive x side) would be located at the path '/cubemap/cm_x_p.jpg': let path = '/cubemap/cm'; let format = '.jpg'; let urls = [ path + '_x_p' + format, path + '_x_n' + format, path + '_y_p' + format, path + '_y_n' + format, path + '_z_p' + format, path + '_z_n' + format ]; But where do I find cubemap images from? Normally, you would have to use Google for cubemap images, but those are not the best quality. You can make your own cubemap from normal images and some respectable photoshop skills, or you can take images from some of the examples that already exist. This is the part where we actually "create" the cubemap from the images we have. Each scene is defined by a number of "meshes." Each mesh is defined by a geometry, which specifies the shape of the mesh, and a material, which specifies the appearence and coloring of the mesh. We now load the six images defined previously into a "texture cube," which is then used to define our material. The geometry we used is called a box geometry, and we are defining the length, width, and breadth of this box as 100 units each: let textureCube = THREE.ImageUtils.loadTextureCube(urls, THREE.CubeRefractionMapping); let shader = THREE.ShaderLib.cube; shader.uniforms.tCube.value = textureCube; let material = new THREE.ShaderMaterial({ fragmentShader: shader.fragmentShader, vertexShader: shader.vertexShader, uniforms: shader.uniforms, depthWrite: false, side: THREE.BackSide }), mesh = new THREE.Mesh(new THREE.BoxGeometry(100, 100, 100), material); Finally, our cubemap is added to the scene: sceneCube.add(mesh); Of course, we want to keep our code modular, so all of the above code for making a cubemap should ideally be wrapped in its own function and used in our main program as and when it is needed. The final cubemap "module" would look something like this: 'use strict'; import THREE from 'three'; let Cubemap = function () { let sceneCube = new THREE.Scene(); let cameraCube = new THREE.PerspectiveCamera( 60, window.innerWidth / window.innerHeight, 1, 100000 ); let path = '/cubemap/cm'; let format = '.jpg'; let urls = [ path + '_x_p' + format, path + '_x_n' + format, path + '_y_p' + format, path + '_y_n' + format, path + '_z_p' + format, path + '_z_n' + format ]; let textureCube = THREE.ImageUtils.loadTextureCube(urls, THREE.CubeRefractionMapping); let shader = THREE.ShaderLib.cube; shader.uniforms.tCube.value = textureCube; let material = new THREE.ShaderMaterial({ fragmentShader: shader.fragmentShader, vertexShader: shader.vertexShader, uniforms: shader.uniforms, depthWrite: false, side: THREE.BackSide }), mesh = new THREE.Mesh(new THREE.BoxGeometry(100, 100, 100), material); sceneCube.add(mesh); return { scene : sceneCube, camera : cameraCube }; }; module.exports = Cubemap; Making Our Core Module Now, we will have to write the core of our little app to actually render the cubemap onto an element in the web browser: We import the dependencies to be used and initialize our WebGL renderer:   'use strict'; import THREE from 'three'; import Cubemap from './Cubemap'; let scene = new THREE.Scene(); let camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 10000); let renderer = new THREE.WebGLRenderer(); renderer.autoClear = false; renderer.setSize(window.innerWidth, window.innerHeight); document.body.appendChild(renderer.domElement); The renderer returns a canvas element, which you can fix to your DOM. This canvas element is where all the magic happens. As usual, we are creating another scene and camera, different from the ones in our cubemap. All objects that are not the cubemap will be included in this scene. Next, we add ambient light to our scene. This is so that we can actually see the other objects we add in the scene: let lightAmbient = new THREE.AmbientLight(0x202020); // soft white light scene.add(lightAmbient); We define and start our rendering function: let cubemap = Cubemap(); let render = function () { requestAnimationFrame(render); renderer.render(cubemap.scene, cubemap.camera); renderer.render(scene, camera); cubemap.camera.rotation.copy(camera.rotation); }; render(); In case you are using movement (which is most often the case with WebGL), you will want to render your scene a number of times a second. The requestAnimationFrame function is a native browser function that calls the function you pass to it after a set time. To really see your cubemapscene come alive, add a moving element as shown in the Hello World example from the Three.js website. Three.js may seem overwhelming at first, but it's a huge improvement over the otherwise steep learning curve for GLSL. If you want to see a slightly more complex example of using cubemaps and objects in Three.js, you can go here. About the author Soham Kamai is a Fullstack web developer and electronics hobbyist. He is especially interested in JavaScript, Python, and IOT. He can be found on Twitter @sohamkamani and GitHub at https://github.com/sohamkamani.

In this article by Rihards Olups, the author of the book Zabbix Network Monitoring, Second Edition, we will learn how communication between Zabbix components is done in plaintext by default. In many environments, this is not a significant problem, but monitoring over the Internet in plaintext is likely not a good approach. In previous Zabbix versions, there was no built-in solution, and various VPN, stunnel, and SSH port forwarding solutions were used. Such solutions can still be used, but version 3.0 is the first Zabbix version to provide built-in encryption. In this article, we will set up several of the components to use different methods of encryption. (For more resources related to this topic, see here.) Overview For Zabbix communication encryption, two methods are supported: Preshared Key Certificate-based encryption The Preshared Key (PSK) method is very easy to set up but is likely harder to scale. Certificate-based encryption can be more complicated to set up but easier to manage on a larger scale and potentially more secure. This encryption is supported between all Zabbix components: server, proxy, agent, and even zabbix_sender and zabbix_get. For outgoing connections (such as server-to-agent or proxy-to-server), one method may be used (no encryption, PSK or certificate-based). For incoming connections, multiple methods may be allowed. This way, an agent could work with encryption by default and then turn off encryption with zabbix_get for debugging. Backend libraries Behind the scenes, Zabbix encryption can use one of three different libraries: OpenSSL, GnuTLS, or mbedTLS. Which one to choose? If using packages, the easiest and safest method is to start with whichever one the packages are compiled with. If compiling from source, choose the one that is easiest to compile with. The Zabbix team has made a significant effort of implementing support for all three libraries to look as similar as possible from the user's perspective. There could be differences regarding support for some specific features, but those are likely to be more obscure ones; if such problems do come up later, switching from one library to another should be fairly easy. While in most cases it would likely not matter much which library you are using, it's a good idea to know it—one good reason for supporting these three different libraries is also the ability to switch to a different library if the currently used one has a security vulnerability. These libraries are used in a generic manner, and you don't need to use the same library for different Zabbix components—it's totally fine to use one library on the Zabbix server, another on the Zabbix proxy, and yet another with zabbix_sender. In this article, we will try out encryption with the Zabbix server and zabbix_sender first, then move on to encrypting agent traffic using both PSK and certificate-based encryption. If you have installed from the packages, your server most likely already supports encryption. Verify this by looking at the server and agent startup messages: TLS support:         YES If you compiled from source and there is no TLS support, recompile the server and agent adding one of these parameters: --with-openssl, --with-gnutls, or --with-mbedtls. Preshared key encryption Let's start with a simple situation: a single new host that will accept PSK-encrypted connections only, to which we will send some values using zabbix_sender. For this to work, both the Zabbix server and zabbix_sender must be compiled with Transport Layer Security (TLS) support. The PSK configuration consists of a PSK identity and key. The identity is a string that is not considered to be secret—it is not encrypted during the communication, so do not put sensitive information in the identity string. The key is a hexadecimal string. Zabbix requires the key it to be at least 32 characters long. The maximum in Zabbix is 512 characters, but it might depend on the specific version of the backend library you are using. We could just type the key in manually, but a slightly easier method is using the openssl command: $ openssl rand -hex 64 This will generate a 512-byte key, which we will use in a moment. Navigate to Configuration | Hosts, click on Create host, and fill in these values: Hostname: An encrypted host Groups: Have only Linux Servers in the In Groups block Switch to the Encryption tab, and in the Connections from host section, leave only PSK marked. In the PSK identity field, enter secret, and paste the key we generated earlier in the PSK field:   When done, click on the Add button at the bottom. Take a look at the AGENT ENCRYPTION column for this host: The first block has only one field and currently says NONE. For connections to the agent, only one method is possible; thus, this column must be showing the currently selected method for outgoing connections from the server's perspective. The second block has three fields. We can choose a combination of incoming connection methods; thus, this column must be showing which types of incoming connections from the server's perspective are accepted for this host. Now, click on Items next to Encrypted host, and click on Create item. Fill in these values: Name: Beers in the fridge Type: Zabbix trapper Key: fridge.beers Click on the Add button at the bottom. Let's try to send a value now using following command: $ zabbix_sender -z 127.0.0.1 -s "Encrypted host" -k fridge.beers -o 1 It should fail and show you the following message: info from server: "processed: 0; failed: 1; total: 1; seconds spent: 0.000193" Notice how the processed count is 0 and the failed count is 1. Let's check the Zabbix server logfile: 12254:20160122:231030.702 connection of type "unencrypted" is not allowed for host "Encrypted host" item "fridge.beers" (not every rejected item might be reported) Now this is actually quite a helpful message—we did not specify any encryption for zabbix_sender, but we did require encrypted connection for our host. Notice the text in parentheses—if multiple items on the same host fail because of this reason, we might only see some of them, and searching the logfile by item key only might not reveal the reason. Now is the time to get the PSK working for zabbix_sender. Run it with the --help parameter and look at the TLS connection options section. Oh yes, there's quite a lot of those. Luckily, for PSK encryption, we only need three of them: --tls-connect, --tls-psk-identity, and tls-psk-file. Before running the command, create a file called zabbix_encrypted_host_psk.txt and paste the hex key we generated earlier into it. It is more secure to create an empty file first, change its permissions to 400 or 600, and paste the key in the file afterwards. This way, another user won't have a chance to snatch the key from the file. Run zabbix_sender again, but with the three additional encryption parameters we just set: $ zabbix_sender -z 127.0.0.1 -s "Encrypted host" -k fridge.beers -o 1 --tls-connect psk --tls-psk-identity secret --tls-psk-file zabbix_encrypted_host_psk.txt With this command, we set the connection method to psk with the --tls-connect flag and specified the PSK identity and key file. Zabbix does not support specifying the PSK key on the command line for security reasons. It must be passed in from a file. This time, the value should be sent successfully: info from server: "processed: 1; failed: 0; total: 1; seconds spent: 0.000070" To be sure, verify that this item now has data in the frontend. Certificate-based encryption With PSK-based encryption protecting our sensitive Zabbix trapper item, let's move to the certificates. We will generate certificates for the Zabbix server and agent and require encrypted connections on the Zabbix agent side for passive items. You might have a certificate infrastructure in your organization, but for our first test, we will generate all required certificates ourselves. We will need a new Certificate Authority (CA) that will sign our certificate. Zabbix does not support self-signed certificates. It is highly recommended to use intermediate certificate authorities to sign client and server certificates. We will not use them in the following simple example. Being our own authority We'll start by creating the certificates in a separate directory. For simplicity's sake, let's do this on a test host. The following is not intended to be a good practice. It is actually doing quite a few bad and insecure things to obtain the certificates faster. Do not follow these steps for any production setup: $ mkdir zabbix_ca $ chmod 700 zabbix_ca $ cd zabbix_ca Generate the root CA key: $ openssl genrsa -aes256 -out zabbix_ca.key 4096 When prompted, enter a password twice to protect the key. Generate and self-sign the root certificate like this: $ openssl req -x509 -new -key zabbix_ca.key -sha256 -days 3560 -out zabbix_ca.crt When prompted, enter the password you used for the key before. Fill in the values as prompted—the easiest way is supplying empty values for most except the country code and common name. The common name does not have to be anything too meaningful for our test, so using a simple string such as zabbix_ca will suffice. Now, let's move on to creating a certificate we will use for the Zabbix server; first, let's generate server key and Certificate Signing Request (CSR): $ openssl genrsa -out zabbix_server.key 2048 $ openssl req -new -key zabbix_server.key -out zabbix_server.csr When prompted, enter the country code and the common name string as before. The common name does not have to match the server or agent name or anything else, so using a simple string such as zabbix_server will suffice. Now, let's sign this request: $ openssl x509 -req -in zabbix_server.csr -CA zabbix_ca.crt -CAkey zabbix_ca.key -CAcreateserial -out zabbix_server.crt -days 1460 -sha256 When prompted, enter the CA passphrase. Let's continue with the certificate we will use for the Zabbix agent. Generate the agent key and certificate signing request: $ openssl genrsa -out zabbix_agent.key 2048 $ openssl req -new -key zabbix_agent.key -out zabbix_agent.csr Again, when prompted, enter the country code and common name string as before. The common name does not have to match the server or agent name or anything else, so using a simple string such as zabbix_agent will suffice. Let's sign the request: $ openssl x509 -req -in zabbix_agent.csr -CA zabbix_ca.crt -CAkey zabbix_ca.key -CAcreateserial -out zabbix_agent.crt -days 1460 -sha256 When prompted, enter the CA passphrase. We're done with creating our test certificates. Both keys were created unencrypted. Zabbix does not support prompting for the key password at this time. Setting up Zabbix with certificates Now, we move on to making the passive items on our test host using the certificates we just generated. We must provide the certificates to the Zabbix agent. In the directory where the Zabbix agent configuration file is located, create a new directory called zabbix_agent_certs. Restrict access to it like this: # chown zabbix zabbix_agent_certs # chmod 500 zabbix_agent_certs From the directory where we generated the certificates, copy the relevant certificate files over to the new directory: # cp zabbix_ca.crt /path/to/zabbix_agent_certs/ # cp zabbix_agent.crt /path/to/zabbix_agent_certs/ # cp zabbix_agent.key /path/to/zabbix_agent_certs/ Edit zabbix_agentd.conf and modify these parameters: TLSAccept=cert TLSConnect=unencrypted TLSCAFile=/path/to/zabbix_agent_certs/zabbix_ca.crt TLSCertFile=/path/to/zabbix_agent_certs/zabbix_agent.crt TLSKeyFile=/path/to/zabbix_agent_certs/zabbix_agent.key This will make the agent only accept connections when they are encrypted and use a certificate signed by that CA, either directly or through intermediates. We'll still use unencrypted connections for active items. A user could supply certificates and expect all communication to be encrypted now, which would not be the case if both the TLSAccept and TLSConnect parameters did not require it; thus, Zabbix enforces them when certificates are supplied. Restart the Zabbix agent. Let's take a look at the host configuration list in the Zabbix frontend: Looks like connections to our test host do not work anymore. Let's check the agent logfile: 2820:20160125:194427.623 failed to accept an incoming connection: from 127.0.0.1: unencrypted connections are not allowed Looks like we broke it. We did set up encryption on the agent but did not get to configuring the server side. What if we would like to roll out encryption to all the agents and deal with the server later? In that case, it would be best to set TLSAccept=cert, unencrypted. Then, the agent would still accept unencrypted connections from our server. Once the certificates were deployed and configured on the Zabbix server, we'd only have to remove unencrypted from that parameter and restart the Zabbix agents. Let's try this out—change zabbix_agentd.conf again: TLSAccept=cert, unencrypted Restart the agent daemon and observe the monitoring resuming from the Zabbix server. Now let's make the server use its certificate. We'll place the certificate in a place where the Zabbix server can use it. In the directory where the Zabbix server configuration file is located, create a new directory called zabbix_server_certs. Restrict access to it using these commands: # chown zabbix zabbix_server_certs # chmod 500 zabbix_server_certs If using packages that run the Zabbix server with a different username such as zabbixs or zabbixserv, replace the username in these two commands. From the directory where we generated the certificates, copy them over to the new directory: # cp zabbix_ca.crt /path/to/zabbix_server_certs/ # cp zabbix_server.crt /path/to/zabbix_server_certs/ # cp zabbix_server.key /path/to/zabbix_server_certs/ Edit zabbix_server.conf and modify these parameters: TLSCAFile=/path/to/zabbix_server_certs/zabbix_ca.crt TLSCertFile=/path/to/zabbix_server_certs/zabbix_server.crt TLSKeyFile=/path/to/zabbix_server_certs/zabbix_server.key Now, restart the Zabbix server. Although we have specified the certificates in both the agent and server, the passive items still work in unencrypted mode. Let's proceed with making them encrypted. In the Zabbix frontend, navigate to Configuration | Hosts, click on A test host, and switch to the Encryption tab. In the Connections to host selection, choose Certificate, then click on the Update button. After the server configuration cache is updated, it will switch to using certificate-based encryption for this host. Going back to our scenario where we slowly rolled out certificate-based configuration to our agents and added it to the server later, we can now disable unencrypted connections on the agent side. Change this line in zabbix_agentd.conf: TLSAccept=cert Restart the agent. If we had followed this process from the very beginning, monitoring would have continued uninterrupted. Let's try to use zabbix_get: $ zabbix_get -s 127.0.0.1 -k system.cpu.load zabbix_get [5746]: Check access restrictions in Zabbix agent configuration That fails because agent only accepts encrypted connections now. Like we did for zabbix_sender, we can specify the certificate, but we must use the Zabbix server certificate now: $ zabbix_get -s 127.0.0.1 -k system.cpu.load --tls-connect cert --tls-ca-file /path/to/zabbix_server_certs/zabbix_ca.crt --tls-cert-file /path/to/zabbix_server_certs/zabbix_server.crt --tls-key-file /path/to/zabbix_server_certs/zabbix_server.key 0.030000 Certainly, this results in a more secure environment. It is not enough to spoof the IP address to access this agent. It is not enough to have an account on the Zabbix server to have access to all agents—access to the server certificate is required, too. On the other hand, it makes debugging a bit more complicated. We used PSK and certificate-based encryption with zabbix_sender and passive agent, but the same principles apply for active agents and zabbix_get. As an exercise, try to get the active agent items working with encryption too. Concerns and further reading At this time, encryption is a very new feature in Zabbix. While it has been developed and tested extremely carefully and pedantically, it is likely to receive further improvements. Make sure to read through the official documentation on encryption for more details and in case of changes being made. Right now, we will touch upon basic concerns and features that are missing. In this article, we covered the Zabbix server, agent, zabbix_get and zabbix_sender—what about Zabbix proxies? Zabbix proxies fully support encryption. The configuration on the proxy side is very similar to the agent configuration, and the configuration on the server or frontend side is done in a similar way to host encryption configuration, too. Keep in mind that all involved components must be compiled with TLS support—any proxies you have might have to be recompiled. When considering encryption, think about the areas where it's needed most: maybe you have Zabbix server and proxy communicating over the Internet while all other connections are in local networks. In that case, it might make sense to set up encryption only for server-proxy communication at first. Note that encryption is not supported when communicating with the Zabbix Java gateway, but one could easily have the gateway communicate with the Zabbix proxy on the localhost, which in turn provides encryption for the channel to the Zabbix server. We already figured out how the upgrading and transitioning to encryption can happen seamlessly without interrupting data collection—the ability of all components to accept various connection types allows us to roll the changes out sequentially. Another reason why one might want to implement encryption only partially is performance. Currently, Zabbix does not reuse connections, implement TLS session caches, or use any other mechanism that would avoid setting up encrypted connections from scratch every time. This can be especially devastating if you have lots of passive agent items. Make sure to understand the potential impact before reconfiguring it all. Encryption is not currently supported for authentication purposes. That is, we can not omit active agent hostnames and figure out which host it is based on the certificate alone. Similarly, we can not allow only encrypted connections for active agent auto-registration. For certificate-based encryption, we only specified the certificates and CA information. If the CA used is large enough, it would not be very secure: any certificate signed by that CA would be accepted. Zabbix also allows verifying both the issuer and subject of the remote certificate. Unless you are using an internal CA that is used for Zabbix only, it is highly recommended you limit issuers and subjects. Summary In this article, we explored Zabbix's built-in encryption, which is supported between all components: the server, proxy, agent, zabbix_sender, and zabbix_get. While not supported for the Java gateway, a Zabbix proxy could easily be put in front of the gateway to provide encryption back to the Zabbix server. Zabbix supports PSK and TLS certificate-based encryption and can use one of three different backend libraries: OpenSSL, GnuTLS, or mbedTLS. In case of security or other issues with one library, users have an option of switching to another library. The upgrade and encryption deployment can be done in steps. All Zabbix components can accept multiple connection methods at the same time. In our example, the agent would be set up to accept both encrypted and unencrypted connections, and when we would be done with all agents, we would switch to encrypted connections on the server side. Once that would be verified to work as expected, unencrypted connections could be disabled on the agents. With the encryption being built-in and easy to set up, it is worth remembering that encrypted connections will need more resources and that Zabbix does not support connection pooling or other methods that could decrease the load. It might be worth securing the most important channels first, leaving endpoints for later. For example, encrypting the communication between the Zabbix server and proxies would likely be a priority over connections to individual agents. Resources for Article:   Further resources on this subject: Deploying a Zabbix proxy [article] Zabbix Configuration [article] Monitoring Windows with Zabbix 1.8 [article]

 In this article by Gregg Marshall, the author of Mastering Drupal 8 Views, we will get introduced to the world of Views in Drupal. Drupal 8 was released November 19, 2015, after almost 5 years of development by over 3,000 members of the Drupal community. Drupal 8 is the largest refactoring in the project's history. One of the most important changes in Drupal 8 was the inclusion of the most popular contributed module, Views. Similar to including CCK in Drupal 7, adding Views to Drupal 8 influenced how Drupal operates as many of the administration pages, such as the content list page, are now Views that can be modified or extended by site builders. Every site builder needs to master the Views module to really take advantage of Drupal's content structuring capabilities by giving site builders the ability to create lists of content formatted in many different ways. A single piece of content can be used for different displays, and all the content in each View is dynamically created when a visitor comes to a page. It was the only contributed module included in the Acquia Site Builder certification examination for Drupal 7. In this article, we will discuss the following topics: Looking at the Views administration page Reviewing the general Views module settings Modifying one of the views from Drupal core to create a specialized administrative page (For more resources related to this topic, see here.) Drupal 8 is here, should I upgrade? "Jim, this is Lynn, how are things at Fancy Websites?" "I read that Drupal 8 is being released on November 19. From our conversations this year, I guess that means it is time to upgrade our current Drupal 6 site. Should I upgrade to Drupal 7 or Drupal 8?" "Lynn, we're really excited that Drupal 8 is finally ready. It is a game changer, and I can name 10 reasons why Drupal 8 is the way to go": Mobile device compatibility is built into Drupal 8's DNA. Analytics show that 32% of your site traffic is coming from buyers using phones, and that's up from only 19% compared to last year. Multilingual is baked in and really works, so we can go ahead and add the Spanish version of the site we have been talking about. There's a new theme engine that will make styling the new site much easier. It's time to update the look of your site; it's looking pretty outdated compared to the competition. Web services is built in. When you're ready to add an app for your customer's phones, Drupal 8 will be ready. There are lots of new fields, so we won't need to add half a dozen contributed modules to let you build your content types. Drupal 8 is built using industry standards. This was a huge change you won't see, but it means that our shop will be able to recruit new developers more easily. The configuration is now stored in code. Finally, we'll have a way for you to develop on your local computer and move your changes to staging and then to production without having to rebuild content types and Views manually over and over. The WYSIWYG editor is built in. The complex setup we went through to get the right buttons and make the output work won't be necessary in Drupal 8. There's a nice tour capability built in so that you can set up custom "how to" demonstrations for your new users. This should free up a lot of your time, which is good given how you are growing. I've saved the best for last. Your favorite module, Views, is now built into core! Between Fields in Drupal 7 and now Views in Drupal 8, you've got the tools to extend your site built right into core. The bottom line is I can't imagine not going ahead and upgrading to Drupal 8. Views in core is reason enough. Why don't I set up a Drupal 8 installation on your development server so that you can start playing with Drupal 8? We're not doing any development work on your site right now, and we still have staging to test any updates." "That sounds great, Jim! Let me know when I can log in." Less than an hour later, the e-mail arrived; the Drupal 8 development site was set up and ready for Lynn to start experimenting. Based on the existing Drupal 6 site, Lynn set up four content types with the same fields she had on the current site. Jim was able to use the built-in migrate module to move some of her data to the new site. Lynn was ready to start exploring Views in Drupal 8. Looking at the Views administration page That evening, Lynn logged into the new site. Clicking on the Manage menu item, she then clicked on the Structure submenu item, and at the bottom of the list displayed on the Structure page, she clicked on the Views option. About that time, Jackson came in and settled into his spot near her terminal. "Hi Jackson, ready to explore Views with me?" Looking at the Views administration page, Lynn noticed there were already a number of Views defined. Scanning the list, she said "Look Jackson, Drupal 8 uses Views for administration pages. This means we can customize them to fit our way of doing things. I like Drupal 8 already!". Jackson purred. Lynn studied the Views administration page shown here: Views administration page As Lynn looked at each view, the listing looked familiar; she had seen the same kind of listing on her Drupal 6 site. Trying the OPERATIONS pull-down menu on the first View, she saw that the options were Edit, Duplicate, Disable, and Delete. "That's pretty clear; I guess Duplicate is the same as Clone on my old version of Views. I can change a View, create a new one using this one as a template, make it temporarily unavailable, or wipe it completely off the face of the earth." "I wonder what kind of settings there are on the Settings tab of this listing page. Look, Jackson, there's a couple of subtabs hiding on the Settings page." As Lynn didn't want to mess up her new Drupal site, she called Jim. "Hi, Jim. Can you give me a quick rundown on the Views Settings tab?" "Sure," he replied. Views settings "Looking at the Views Settings tab, you'll notice two subtabs, Basic and Advanced. Select the advanced settings tab by clicking on Advanced to show the following display: The Views advanced settings configuration page Views advanced settings Let's look at the Advanced tab first since you'll probably never use these settings. The first option, Disable views data caching, shouldn't be checked unless you are having issues with Views not updating when the data changes. Even then, you should probably disable caching on a per-View basis using the caching setting in the View's edit page in the third column, labeled Advanced, near the bottom of the column. Disabling Views' data caching can really slow down the page loads on your site. You might actually use the Advanced settings tab if you need to clear all the Views' caches, which you would do by clicking on the Clear Views' cache button. Views basic settings The other advanced setting is DEBUGGING with a Add Views signature to all SQL queries checkbox. Unless you are using MySQL's logs to debug queries, which only an advanced developer would do, you aren't going to want this overhead added to Views queries, so just leave it unselected. Moving to the Basic tab, there are a number of settings that might be handy, and I'd recommend changing the default settings. Click on Basic to show the following display: The Views basic settings configuration page The first option, Always show the master (default) display, might or might not be useful. If you create a new View and don't select either create a page or create a block (or provide a REST export if this module is enabled), then a default View display is created called master. If you select either option or both, then page and/or block View displays are created, and generally, you won't see master. It's there; it's just hidden. Sometimes, it is handy to be able to edit or use the master display. While I don't like creating a lot of displays in each View, sometimes, I do create two or three if the content being displayed is very similar. An obvious example is when you want to display the same blog listing as either a page or in a block on other pages. The same teaser information is displayed, just in different ways. So, having the two displays in the same View makes sense. Just make sure when you customize each display that any changes you make are set to only apply to the current display and not all displays. Otherwise, you might make changes you hadn't planned on in the other displays. Most of the time, you will see a pull-down menu that defaults to All displays, but you can select This page (override) to have the setting change apply only to this display. Having the master display show lets you create the information that will be the same in all the displays you are creating; then, you can create and customize the different displays. Using our blog example, you may create a master display that has a basic list of titles, with the titles linking to the full blog post. Then, you can create a blog display page, and using the This page (override) option, you can add summaries, add more links, and set the results to 10 per page. Using the master display, you can go back and add a display block that shows only the last five blog posts without any pager, again applying each setting only to the block display. You might then go back to the master display and create a second block that uses the tags to select five blog posts that are related, again making sure that the changes are applied to the current block and not all displays. Finally, when you want to change something that will affect all the displays, make the change on the master display, and this time, use the All displays option to make sure the other displays are updated. In our blog example, you might decide to change the CSS class used to display the titles to apply formatting from the theme; you probably want this to look the same in every possible display of the blog posts. The next basic setting for Views is Allow embedded displays. You will not enable this option; it is for developers who will use Views-generated content in their custom code. However, if you see it enabled, don't disable it; doing this would likely break something on your site using this feature. The last setting before the LIVE PREVIEW SETTINGS field set is Label for "Any" value on non-required single-select exposed filters, which lets you pick either <Any> or -Any- as the format for exposed filters that would allow a user to ignore the filter. Live Preview Settings There are several LIVE PREVIEW SETTINGS field sets I like to enable because they make debugging your Views easier. If the LIVE PREVIEW SETTINGS field set is closed (that is, the options are not showing), click on the title next to the arrow, and it will open. It will look similar to this: Live Preview Settings I generally enable the Automatically update preview on changes option. This way, any change I make to the View when I edit it shows the results that would occur after each change. Seeing things change right away gives me a clue whether a change will have an effect I'm not expecting. A lot of Views options can be tricky to understand, so a bit of trial and error is often required. Hence, expect to make a change and not see what you expect; just change the setting back, rethink the problem, and try again. Almost always, you'll get the answer eventually. If you have a View that is really complex and very slow, you can always disable the live preview while you edit the View by selecting the Auto preview option in the grey Preview bar just under all the View's settings. The next two options control whether Views will display the SQL query generated by the Views options you selected in the edit screen. I like to display the SQL query, so I will select the Above the preview option under Show SQL query and then select the Show the SQL query checkbox that follows it. If you don't check the Show the SQL query option, it doesn't matter what you select for above or below the preview, and if you expect to see the SQL queries and don't, it is likely that you set one option and not the other. Showing the SQL query can be confusing at first, but after a while, you'll find it handy to figure out what is going on, especially if you have relationships (or should have relationships and don't realize it). And, of course, if you can't read the query, you can always e-mail me for a translation to English. The next option, Show performance statistics, is handy when trying to figure out why some Views-generated page is loading slowly. But usually, this isn't an issue you'd be thinking of, so I'd leave it off. You want to focus on getting the right information to display exactly the way you want without thinking about the performance. If we later decide it's too slow, the developer we'll assign to it will use this information and turn the option on in development. The same is true about Show other queries run during render during live preview. This information is handy to figure out performance issues and occasionally a display formatting issue during theming, but it isn't something you as a nonprogrammer should be worried about. Seeing all the extra queries can be confusing and intimidating, yet it doesn't really offer you any help creating a View. "Oh, don't forget to click on Save configuration if you change any settings. I don't know how many times I've forgotten to save a configuration change in Drupal and then wondered why my change hasn't stuck. Does this help?" "Thanks Jim, that is great. I owe you a coffee next time we get together." Hanging up the phone, Lynn said, "What do you think, Jackson? Let's start off by creating a property maintenance page for our salespeople to use? I think I'll get a quick win by modifying one of Drupal's core views." Adapting an existing View Lynn will use her knowledge from using Views on her existing Drupal site, and so move quickly. The existing content page provided by Views is general purpose and offers lots of options, and not all these options are appropriate for all content editors. This page looks similar to the following one: Drupal's standard content listing page Lynn started creating her property maintenance page by going to the Views listing page (Manage | Structure | Views) and selecting Duplicate from the OPERATIONS pull-down menu on the right-hand side of this row. On the next screen, she named the Property Maintenance view and clicked on the Duplicate button. When the View edit screen appeared, she was ready to adapt it to her need. First, she selected the Page display, assuming the Always show the master (default) display setting was already selected; otherwise, the Page display will be selected by default as it is the only display in this View. Remember that any change made in the View edit page isn't saved until you click on the Save button. Also, unsaved changes won't show up when the page/block is displayed. If you make a change, look at it using another browser or tab, and if you don't see the change reflected, it is likely that you didn't save the change you just made. The Property Maintenance screen before making any changes Editing the Property Maintenance view Starting with the left-hand side column of the View edit screen, Lynn changed the title by clicking on the Content link next to the Title label. She changed the title to Property Maintenance. Moving down the column, Lynn decided that the table display and settings were okay on the original screen and skipped them. Under the FIELDS section, Lynn decided to delete the Content: Node operations bulk form, Content: Type (Content Type), and (author) User: Name (Author) fields/columns as they weren't useful to the real estate salespeople who would be using this page. To do this, she clicked on Content: Node operations bulk form and then on the Remove link at the bottom of the Configure field modal that appeared. She repeated the removing of the field for the Content: Type (Content Type) and (author) User: Name (Author) fields. Lynn noted that the username field appeared to be the only field reference to the author entity, so she could delete the relationship later. Moving on to FILTER CRITERIA, Lynn was a bit confused by the first two filters. When she clicked on Content: Published status or admin user, the description said "Filters out unpublished content if the current user cannot view it". "This seems reasonable, let's keep this filter," she thought, and she clicked on Cancel. Next was Content: Publishing status (grouped), an exposed filter that lets the user filter by either published or unpublished. This seemed useful, so Lynn kept it and clicked on Cancel. The next filter, Content: Type (exposed), is necessary but shouldn't be selectable by the user, so Lynn clicked on it to edit the filter, unselected the Expose this filter to visitors option, and selected just the Property content type, making the filter only select content that are properties. The next filter, Content: Title (exposed), is handy, so Lynn left it as is. The final filter, Content: Translation language (exposed), isn't needed as Lynn's site isn't multilingual, so Lynn deleted the filter. Moving on to the center column of the View edit page, under the PAGE SETTINGS heading, Lynn changed the path for the View to /admin/property-maintenance by clicking on the existing /admin/content/node path, making the change, and clicking on the Apply button. Next in this column was the menu setting. Lynn doesn't want the property maintenance page to be part of the administration content page, so she clicked on Tab: Content and changed the menu type to Normal menu entry. This changed the fields displayed on the right-hand side of the modal, so Lynn changed the Menu link title to Property Maintenance, left the description blank, and left Show as expanded unselected. In the Parent pull-down menu, she selected the <Tools> menu. Tools is the default Drupal menu for site tools that is only shown to authorized users, who are logged into the site and can view the page linked to, which real estate salespeople will be able to view. She left the weight at -10, planning on reorganizing this menu when she has most of it configured. As this is the last option, she clicked on Apply to exit the modal. The last setting in the PAGE SETTINGS section is Access. Lynn knew she needed to change the required permission as she didn't plan on giving real estate salespeople access to the main content page, but she wasn't sure which permission to give them. Looking through the permissions page (the People | Permissions tab), Lynn didn't see any permission that made sense for who should be able to see this maintenance page. So, she clicked on the Permission link in the center column of the View edit page and changed the Access value from Permission to Role, and when she clicked on the Apply (all displays) button, she could select the role(s) she wanted to be able to see on this page. She selected the Administrator, Real Estate Salesperson, and Office Administrator roles. One way to test access while you develop is to use a second browser and log in as the other kind of user. A common mistake in Drupal is to see content while logged in as an administrator that can't be seen by other users. This can also be done using a second tab opened in "incognito" mode, but I find it easier to use a different browser (for example, Chrome and Firefox). You can even have three browsers open to the same page to test a third kind of user. Continuing down the column, Lynn decided she didn't need a header or footer on this administration page at least for now, but she did want to change the NO RESULTS BEHAVIOR message. Drupal has a text message defined, so she clicked on the Global: Unfiltered text (Global: Unfiltered text) link, changed the Content field to No properties meeting your filter criteria are available., and clicked on the Apply (all displays) button. The final section, PAGER, seemed fine, so Lynn skipped over it and moved to the third column of the view edit page, ADVANCED SETTINGS. As Lynn had changed the setting to always show the advanced settings, Lynn noticed that there was a relationship for author. As she had deleted displaying the author name, there wasn't any reason to keep the relationship because she wasn't using any of the author's details. She clicked on the author link and then on the Remove link at the bottom of the modal. Reviewing the results of the live preview, Lynn was satisfied and clicked on the Save button to save her modified view. There is a maxim in computers, Save Early, Save Often. As you develop or modify your View, when you reach a point where your progress so far is okay, click on the Save button. Then, if you make a terrible mistake in the next change, you can click on the Cancel button and then click on Edit to resume from where you last saved. Before saving the View, the result looked similar to the following screen: The resulting Property Maintenance View edit screen with all the changes Debugging – Live Preview is your friend Assuming you enabled Live Preview in your Views settings earlier in this article, as you are building your View, Views will show what will be displayed. Formatting and some JavaScript displays, such as Google mapping, can't be displayed in Live Preview, but to debug, you generally don't need them. Many Views challenges are getting the data that you want to display or getting data to be displayed the way you want. Many Views are created using the fields content display. Often, you will see fields that you don't want displayed when reviewing Live Preview because you didn't check the Exclude from display option in the field configuration. Or, you will select a field from the Add fields list that isn't actually the field you want to display the data you want—for instance, do you want article tags or article tags (field_tags: delta)? Sometimes you have to just try one and see what happens. If it isn't the right option, delete the field and try another. Experience will guide you as you use Views, but even the most experienced site builders wonder what some field or field option does in the context of the View they are building. Remember to save the View before you experiment with this next idea. Then, if it doesn't work out, you can just click on Cancel and not lose all the previous work you put in. If you disabled Live Preview, hopefully, you have decided to go back and enable it; seeing the output and looking at the generated SQL queries is really very useful in trying to figure out what might be going wrong. "Okay, Jackson, I see that a lot of what I knew from the previous versions of Views applies to the version in Drupal 8. Now that I've quickly gone through the edit screen to modify a core View, let's get serious and really learn the ins and outs of this version of Views." Summary In this article, we covered the Views administration page, where you can add, delete, edit, and duplicate views. Then, we reviewed all the general Views module settings. Finally, we modified a core View, quickly going through several configuration options. If you have used Views in older versions of Drupal, you should feel comfortable. If this is your first introduction to Views, don't panic that we glossed over a lot or if you felt lost. Resources for Article: Further resources on this subject: Working with Drupal Audio in Flash (part 2) [article] Modular Programming in ECMAScript 6 [article] Using NoSQL Databases [article]

In this article by Adam Boduch, author of Flux Architecture covers the basic idea of Flux. Flux is supposed to be this great new way of building complex user interfaces that scale well. At least that's the general messaging around Flux, if you're only skimming the Internet literature. But, how do we define this great new way of building user interfaces? What makes it superior to other more established frontend architectures? The aim of this article is to cut through the sales bullet points and explicitly spell out what Flux is, and what it isn't, by looking at the patterns that Flux provides. And since Flux isn't a software package in the traditional sense, we'll go over the conceptual problems that we're trying to solve with Flux. Finally, we'll close the article by walking through the core components found in any Flux architecture, and we'll install the Flux npm package and write a hello world Flux application right away. Let's get started. (For more resources related to this topic, see here.) Flux is a set of patterns We should probably get the harsh reality out of the way first—Flux is not a software package. It's a set of architectural patterns for us to follow. While this might sound disappointing to some, don't despair—there's good reasons for not implementing yet another framework. Throughout the course of this book, we'll see the value of Flux existing as a set of patterns instead of a de facto implementation. For now, we'll go over some of the high-level architectural patterns put in place by Flux. Data entry points With traditional approaches to building frontend architectures, we don't put much thought into how data enters the system. We might entertain the idea of data entry points, but not in any detail. For example, with MVC (Model View Controller) architectures, the controller is supposed control the flow of data. And for the most part, it does exactly that. On the other hand, the controller is really just about controlling what happens after it already has the data. How does the controller get data in the first place? Consider the following illustration: At first glance, there's nothing wrong with this picture. The data flow, represented by the arrows, is easy to follow. But where does the data originate? For example, the view can create new data and pass it to the controller, in response to a user event. A controller can create new data and pass it to another controller, depending on the composition of our controller hierarchy. What about the controller in question—can it create data itself and then use it? In a diagram such as this one, these questions don't have much virtue. But, if we're trying to scale an architecture to have hundreds of these components, the points at which data enters the system becomes very important. Since Flux is used to build architectures that scale, it considers data entry points an important architectural pattern. Managing state State is one of those realities we need to cope with in frontend development. Unfortunately, we can't compose our entire application of pure functions with no side effects for two reasons. First, our code needs to interact with the DOM interface, in one way or another. This is how the user sees changes in the UI. Second, we don't store all our application data in the DOM (at least we shouldn't do this). As time passes and the user interacts with the application, this data will change. There's no cut-and-dry approach to managing state in a web application, but there are several ways to limit the amount of state changes that can happen, and enforce how they happen. For example, pure functions don't change the state of anything, they can only create new data. Here's an example of what this looks like: As you can see, there's no side effects with pure functions because no data changes state as a result of calling them. So why is this a desirable trait, if state changes are inevitable? The idea is to enforce where state changes happen. For example, perhaps we only allow certain types of components to change the state of our application data. This way, we can rule out several sources as the cause of a state change. Flux is big on controlling where state changes happen. Later on in the article, we'll see how Flux stores manage state changes. What's important about how Flux manages state is that it's handled at an architectural layer. Contrast this with an approach that lays out a set of rules that say which component types are allowed to mutate application data—things get confusing. With Flux, there's less room for guessing where state changes take place. Keeping updates synchronous Complimentary to data entry points is the notion of update synchronicity. That is, in addition to managing where the state changes originate from, we have to manage the ordering of these changes relative to other things. If the data entry points are the what of our data, then synchronously applying state changes across all the data in our system is the when. Let's think about why this matters for a moment. In a system where data is updated asynchronously, we have to account for race conditions. Race conditions can be problematic because one piece of data can depend on another, and if they're updated in the wrong order, we see cascading problems, from one component to another. Take a look at this diagram, which illustrates this problem: When something is asynchronous, we have no control over when that something changes state. So, all we can do is wait for the asynchronous updates to happen, and then go through our data and make sure all of our data dependencies are satisfied. Without tools that automatically handle these dependencies for us, we end up writing a lot of state-checking code. Flux addresses this problem by ensuring that the updates that take place across our data stores are synchronous. This means that the scenario illustrated in the preceding diagram isn't possible. Here's a better visualization of how Flux handles the data synchronization issues that are typical of JavaScript applications today: Information architecture It's easy to forget that we work in information technology and that we should be building technology around information. In recent times, however, we seem to have moved in the other direction, where we're forced to think about implementation before we think about information. More often than not, the data exposed by the sources used by our application, don't have what the user needs. It's up to our JavaScript to turn this raw data into something consumable by the user. This is our information architecture. Does this mean that Flux is used to design information architectures as opposed to a software architecture? This isn't the case at all. In fact, Flux components are realized as true software components that perform actual computations. The trick is that Flux patterns enable us to think about information architecture as a first-class design consideration. Rather than having to sift through all sorts of components and their implementation concerns, we can make sure that we're getting the right information to the user. Once our information architecture takes shape, the larger architecture of our application follows, as a natural extension to the information we're trying to communicate to our users. Producing information from data is the difficult part. We have to distill many sources of data into not only information, but information that's also of value to the user. Getting this wrong is a huge risk for any project. When we get it right, we can then move on to the specific application components, like the state of a button widget, and so on. Flux architectures keep data transformations confined to their stores. A store is an information factory—raw data goes in and new information comes out. Stores control how data enters the system, the synchronicity of state changes, and they define how the state changes. When we go into more depth on stores as we progress through the book, we'll see how they're the pillars of our information architecture. Flux isn't another framework Now that we've explored some of the high-level patterns of Flux, it's time to revisit the question: what is Flux again? Well, it is just a set of architectural patterns we can apply to our frontend JavaScript applications. Flux scales well because it puts information first. Information is the most difficult aspect of software to scale; Flux tackles information architecture head on. So, why aren't Flux patterns implemented as a Framework? This way, Flux would have a canonical implementation for everyone to use; and like any other large scale open source project, the code would improve over time as the project matures. The main problem is that Flux operates at an architectural level. It's used to address information problems that prevent a given application from scaling to meet user demand. If Facebook decided to release Flux as yet another JavaScript framework, it would likely have the same types of implementation issues that plague other frameworks out there. For example, if some component in a framework isn't implemented in a way that best suits the project we're working on, then it's not so easy to implement a better alternative, without hacking the framework to bits. What's nice about Flux is that Facebook decided to leave the implementation options on the table. They do provide a few Flux component implementations, but these are reference implementations. They're functional, but the idea is that they're a starting point for us to understand the mechanics of how things such as dispatchers are expected to work. We're free to implement the same Flux architectural pattern as we see it. Flux isn't a framework. Does this mean we have to implement everything ourselves? No, we do not. In fact, developers are implementing Flux frameworks and releasing them as open source projects. Some Flux libraries stick more closely to the Flux patterns than others. These implementations are opinionated, and there's nothing wrong with using them if they're a good fit for what we're building. The Flux patterns aim to solve generic conceptual problems with JavaScript development, so you'll learn what they are before diving into Flux implementation discussions. Flux solves conceptual problems If Flux is simply a collection of architectural patterns instead of a software framework, what sort of problems does it solve? In this section, we'll look at some of the conceptual problems that Flux addresses from an architectural perspective. These include unidirectional data flow, traceability, consistency, component layering, and loosely coupled components. Each of these conceptual problems pose a degree of risk to our software, in particular, the ability to scale it. Flux helps us get out in front of these issues as we're building the software. Data flow direction We're creating an information architecture to support the feature-rich application that will ultimately sit on top of this architecture. Data flows into the system, and will eventually reach an endpoint, terminating the flow. It's what happens in between the entry point and the termination point that determines the data flow within a Flux architecture. This is illustrated here: Data flow is a useful abstraction, because it's easy to visualize data as it enters the system and moves from one point to another. Eventually, the flow stops. But before it does, several side effects happen along the way. It's that middle block in the preceding diagram that's concerning, because we don't know exactly how the data-flow reached the end. Let's say that our architecture doesn't pose any restrictions on data flow. Any component is allowed to pass data to any other component, regardless of where that component lives. Let's try to visualize this setup: As you can see, our system has clearly defined entry and exit points for our data. This is good because it means that we can confidently say that the data flows through our system. The problem with this picture is with how the data flows between the components of the system. There's no direction, or rather, it's multidirectional. This isn't a good thing. Flux is a unidirectional data flow architecture. This means that the preceding component layout isn't possible. The question is—why does this matter? At times, it might seem convenient to be able to pass data around in any direction, that is, from any component to any other component. This in and of itself isn't the issue—passing data alone doesn't break our architecture. However, when data moves around our system in more than one direction, there's more opportunity for components to fall out of sync with one another. This simply means that if data doesn't always move in the same direction, there's always the possibility of ordering bugs. Flux enforces the direction of data flows, and thus eliminates the possibility of components updating themselves in an order that breaks the system. No matter what data has just entered the system, it'll always flow through the system in the same order as any other data, as illustrated here: Predictable root cause With data entering our system and flowing through our components in one direction, we can more easily trace any effect to it's cause. In contrast, when a component sends data to any other component residing in any architectural layer, it's a lot more difficult to figure how the data reached it's destination. Why does this matter? Debuggers are sophisticated enough that we can easily traverse any level of complexity during runtime. The problem with this notion is that it presumes we only need to trace what's happening in our code for the purposes of debugging. Flux architectures have inherently predictable data flows. This is important for a number of design activities and not just debugging. Programmers working on Flux applications will begin to intuitively sense what's going to happen. Anticipation is key, because it let's us avoid design dead-ends before we hit them. When the cause and effect are easy to tease out, we can spend more time focusing on building application features—the things the customers care about. Consistent notifications The direction in which we pass data from component to component in Flux architectures should be consistent. In terms of consistency, we also need to think about the mechanism used to move data around our system. For example, publish/subscribe (pub/sub) is a popular mechanism used for inter-component communication. What's neat about this approach is that our components can communicate with one another, and yet, we're able to maintain a level of decoupling. In fact, this is fairly common in frontend development because component communication is largely driven by user events. These events can be thought of as fire-and-forget. Any other components that want to respond to these events in some way, need to take it upon themselves to subscribe to the particular event. While pub/sub does have some nice properties, it also poses architectural challenges, in particular, scaling complexities. For example, let's say that we've just added several new components for a new feature. Well, in which order do these components receive update messages relative to pre-existing components? Do they get notified after all the pre-existing components? Should they come first? This presents a data dependency scaling issue. The other challenge with pub-sub is that the events that get published are often fine grained to the point where we'll want to subscribe and later unsubscribe from the notifications. This leads to consistency challenges because trying to code lifecycle changes when there's a large number of components in the system is difficult and presents opportunities for missed events. The idea with Flux is to sidestep the issue by maintaining a static inter-component messaging infrastructure that issues notifications to every component. In other words, programmers don't get to pick and choose the events their components will subscribe to. Instead, they have to figure out which of the events that are dispatched to them are relevant, ignoring the rest. Here's a visualization of how Flux dispatches events to components: The Flux dispatcher sends the event to every component; there's no getting around this. Instead of trying to fiddle with the messaging infrastructure, which is difficult to scale, we implement logic within the component to determine whether or not the message is of interest. It's also within the component that we can declare dependencies on other components, which helps influence the ordering of messages. Simple architectural layers Layers can be a great way to organize an architecture of components. For one thing, it's an obvious way to categorize the various components that make up our application. For another thing, layers serve as a means to put constraints around communication paths. This latter point is especially relevant to Flux architectures since it's imperative that data flow in one direction. It's much easier to apply constraints to layers than it is to individual components. Here is an illustration of Flux layers: This diagram isn't intended to capture the entire data flow of a Flux architecture, just how data flows between the main three layers. It also doesn't give any detail about what's in the layers. Don't worry, the next section gives introductory explanations of the types of Flux components and the communication that happens between the layers is the focus of this entire book. As you can see, the data flows from one layer to the next, in one direction. Flux only has a few layers, and as our applications scale in terms of component counts, the layer counts remains fixed. This puts a cap on the complexity involved with adding new features to an already large application. In addition to constraining the layer count and the data flow direction, Flux architectures are strict about which layers are actually allowed to communicate with one another. For example, the action layer could communicate with the view layer, and we would still be moving in one direction. We would still have the layers that Flux expects. However, skipping a layer like this is prohibited. By ensuring that layers only communicate with the layer directly beneath it, we can rule out bugs introduced by doing something out-of-order. Loosely coupled rendering One decision made by the Flux designers that stands out is that Flux architectures don't care how UI elements are rendered. That is to say, the view layer is loosely coupled to the rest of the architecture. There are good reasons for this. Flux is an information architecture first, and a software architecture second. We start with the former and graduate toward the latter. The challenge with view technology is that it can exert a negative influence on the rest of the architecture. For example, one view has a particular way of interacting with the DOM. Then, if we've already decided on this technology, we'll end up letting it influence the way our information architecture is structured. This isn't necessarily a bad thing, but it can lead to us making concessions about the information we ultimately display to our users. What we should really be thinking about is the information itself and how this information changes over time. What actions are involved that bring about these changes? How is one piece of data dependent on another piece of data? Flux naturally removes itself from the browser technology constraints of the day so that we can focus on the information first. It's easy to plug views into our information architecture as it evolves into a software product. Flux components In this section, we'll begin our journey into the concepts of Flux. These concepts are the essential ingredients used in formulating a Flux architecture. While there's no detailed specifications for how these components should be implemented, they nevertheless lay the foundation of our implementation. This is a high-level introduction to the components we'll be implementing throughout this book. Action Actions are the verbs of the system. In fact, it's helpful if we derive the name of an action directly from a sentence. These sentences are typically statements of functionality; something we want the application to do. Here are some examples: Fetch the session Navigate to the settings page Filter the user list Toggle the visibility of the details section These are simple capabilities of the application, and when we implement them as part of a Flux architecture, actions are the starting point. These human-readable action statements often require other new components elsewhere in the system, but the first step is always an action. So, what exactly is a Flux action? At it's simplest, an action is nothing more than a string—a name that helps identify the purpose of the action. More typically, actions consist of a name and a payload. Don't worry about the payload specifics just yet—as far as actions are concerned, they're just opaque pieces of data being delivered into the system. Put differently, actions are like mail parcels. The entry point into our Flux system doesn't care about the internals of the parcel, only that they get to where they need to go. Here's an illustration of actions entering a Flux system: This diagram might give the impression that actions are external to Flux when in fact, they're an integral part of the system. The reason this perspective is valuable is because it forces us to think about actions as the only means to deliver new data into the system. Golden Flux Rule: If it's not an action, it can't happen. Dispatcher The dispatcher in a Flux architecture is responsible for distributing actions to the store components (we'll talk about stores next). A dispatcher is actually kind of like a broker—if actions want to deliver new data to a store, they have to talk to the broker, so it can figure out the best way to deliver them. Think about a message broker in a system like RabbitMQ. It's the central hub where everything is sent before it's actually delivered. Here is a diagram depicting a Flux dispatcher receiving actions and dispatching them to stores: In a Flux application, there's only one dispatcher. It can be thought of more as a pseudo layer than an explicit layer. We know the dispatcher is there, but it's not essential to this level of abstraction. What we're concerned about at an architectural level, is making sure that when a given action is dispatched, we know that it's going to make it's way to every store in the system. Having said that, the dispatcher's role is critical to how Flux works. It's the place where store callback functions are registered. And it's how data dependencies are handled. Stores tell the dispatcher about other stores that it depends on, and it's up to the dispatcher to make sure these dependencies are properly handled. Golden Flux Rule: The dispatcher is the ultimate arbiter of data dependencies. Store Stores are where state is kept in a Flux application. Typically, this means the application data that's sent to the frontend from the API. However, Flux stores take this a step further and explicitly model the state of the entire application. For now, just know that stores are where state that matters can be found. Other Flux components don't have state—they have implicit state at the code level, but we're not interested in this, from an architectural point of view. Actions are the delivery mechanism for new data entering the system. The term new data doesn't imply that we're simply appending it to some collection in a store. All data entering the system is new in the sense that it hasn't been dispatched as an action yet—it could in fact result in a store changing state. Let's look at a visualization of an action that results in a store changing state: The key aspect of how stores change state is that there's no external logic that determines a state change should happen. It's the store, and only the store, that makes this decision and then carries out the state transformation. This is all tightly encapsulated within the store. This means that when we need to reason about a particular information, we need not look any further than the stores. They're their own boss—they're self-employed. Golden Flux Rule: Stores are where state lives, and only stores themselves can change this state. View The last Flux component we're going to look at in this section is the view, and it technically isn't even a part of Flux. At the same time, views are obviously a critical part of our application. Views are almost universally understood as the part of our architecture that's responsible for displaying data to the user—it's the last stop as data flows through our information architecture. For example, in MVC architectures, views take model data and display it. In this sense, views in a Flux-based application aren't all that different from MVC views. Where they differ markedly is with regard to handling events. Let's take a look at the following diagram: Here we can see the contrasting responsibilities of a Flux view, compared with a view component found in your typical MVC architecture. The two view types have similar types of data flowing into them—application data used to render the component and events (often user input). What's different between the two types of views is what flows out of them. The typical view doesn't really have any constraints in how it's event handler functions communicate with other components. For example, in response to a user clicking a button, the view could directly invoke behavior on a controller, change the state of a model, or it might query the state of another view. On the other hand, the Flux view can only dispatch new actions. This keeps our single entry point into the system intact and consistent with other mechanisms that want to change the state of our store data. In other words, an API response updates state in the exact same way as a user clicking a button does. Given that views should be restricted in terms of how data flows out of them (besides DOM updates) in a Flux architecture, you would think that views should be an actual Flux component. This would make sense insofar as making actions the only possible option for views. However, there's also no reason we can't enforce this now, with the benefit being that Flux remains entirely focused on creating information architectures. Keep in mind, however, that Flux is still in it's infancy. There's no doubt going to be external influences as more people start adopting Flux. Maybe Flux will have something to say about views in the future. Until then, views exist outside of Flux but are constrained by the unidirectional nature of Flux. Golden Flux Rule: The only way data flows out of a view is by dispatching an action. Installing the Flux package We'll get some of our boilerplate code setup tasks out of the way too, since we'll be using a similar setup throughout the book. We'll skip going over Node + NPM installation since it's sufficiently covered in great detail all over the Internet. We'll assume Node is installed and ready to go from this point forward. The first NPM package we'll need installed is Webpack. This is an advanced module bundler that's well suited for modern JavaScript applications, including Flux-based applications. We'll want to install this package globally so that the webpack command gets installed on our system: npm install webpack -g With Webpack in place, we can build each of the code examples that ship with this book. However, our project does require a couple local NPM packages, and these can be installed as follows: npm install flux babel-core babel-loader babel-preset-es2015 --save-dev The --save-dev option adds these development dependencies to our file, if one exists. This is just to get started—it isn't necessary to manually install these packages to run the code examples in this book. The examples you've downloaded already come with a package.json, so to install the local dependencies, simply run the following from within the same directory as the package.json file: npm install Now the webpack command can be used to build the example. Alternatively, if you plan on playing with the code, which is obviously encouraged, try running webpack --watch. This latter form of the command will monitor for file changes to the files used in the build, and run the build whenever they change. This is indeed a simple hello world to get us off to a running start, in preparation for the remainder of the book. We've taken care of all the boilerplate setup tasks by installing Webpack and it's supporting modules. Let's take a look at the code now. We'll start by looking at the markup that's used. <!doctype html> <html>   <head>     <title>Hello Flux</title>     <script src="main-bundle.js" defer></script>   </head>   <body></body> </html> Not a lot to it is there? There isn't even content within the body tag. The important part is the main-bundle.js script—this is the code that's built for us by Webpack. Let's take a look at this code now: // Imports the "flux" module. import * as flux from 'flux'; // Creates a new dispatcher instance. "Dispatcher" is // the only useful construct found in the "flux" module. const dispatcher = new flux.Dispatcher(); // Registers a callback function, invoked every time // an action is dispatched. dispatcher.register((e) => {   var p;   // Determines how to respond to the action. In this case,   // we're simply creating new content using the "payload"   // property. The "type" property determines how we create   // the content.   switch (e.type) {     case 'hello':       p = document.createElement('p');       p.textContent = e.payload;       document.body.appendChild(p);       break;     case 'world':       p = document.createElement('p');       p.textContent = `${e.payload}!`;       p.style.fontWeight = 'bold';       document.body.appendChild(p);       break;     default:       break;   } });   // Dispatches a "hello" action. dispatcher.dispatch({   type: 'hello',   payload: 'Hello' }); // Dispatches a "world" action. dispatcher.dispatch({   type: 'world',   payload: 'World' }); As you can see, there's not much too this hello world Flux application. In fact, the only Flux-specific component this code creates is a dispatcher. It then dispatches a couple of actions and the handler function that's registered to the store processes the actions. Don't worry that there's no stores or views in this example. The idea is that we've got the basic Flux NPM package installed and ready to go. Summary This article introduced you to Flux. Specifically, we looked at both what Flux is and what it isn't. Flux is a set of architectural patterns, that when applied to our JavaScript application, help with getting the data flow aspect of our architecture right. Flux isn't yet another framework used for solving specific implementation challenges, be it browser quirks or performance gains—there's a multitude of tools already available for these purposes. Perhaps the most important defining aspect of Flux are the conceptual problems it solves—things like unidirectional data flow. This is a major reason that there's no de facto Flux implementation. We wrapped the article up by walking through the setup of our build components used throughout the book. To test that the packages are all in place, we created a very basic hello world Flux application. Resources for Article: Further resources on this subject: Reactive Programming and the Flux Architecture [article] Advanced React [article] Qlik Sense's Vision [article]  

In this article by Valentin Bojinov, the author of the book RESTful Web API Design with Node.JS, Second Edition, we willlook for a better storage solution, which can be scalable easily, together with our REST-enabled application. These days, the so-called NoSQL databases are used heavily in cloud environments. They have the following advantages over traditional transactional SQL databases: They are schemaless; that is, they work with object representations rather than store the object state in one or several tables, depending on their complexity. They are extendable, because they store an actual object. Data evolution is supported implicitly, so all you need to do is just call the operation that stores the object. They are designed to be highly distributed and scalable. Nearly all modern NoSQL solutions out there support clustering and can scale further, along with the load of your application. Additionally, most of them have REST-enabled interfaces over HTTP, which eases their usage over a load balancer in high-availability scenarios. Classical database drivers are usually not available for traditional client-side languages, such as JavaScript, because they require native libraries or drivers. However, the idea of NoSQL originated from using document data stores. Thus, most of them support the JSON format, which is native to JavaScript. Last but not least, most NoSQL solutions are open source and are available for free, with all the benefits that open source projects offer: community, examples, and freedom! In this article, we will take a look at two NoSQL solutions: LevelDB and MongoDB. We will see how to design and test our database models, and finally, we will take a brief look at the content delivery network (CDN) infrastructures (For more resources related to this topic, see here.) Key/value store – LevelDB The first data store we will look at is LevelDB. It is an open source implementation developed by Google and written in C++. It is supported by a wide range of platforms, including Node.js. LevelDB is a key/value store; both the key and value are represented as binary data, so their content can vary from simple strings to binary representations of serialized objects in any format, such as JSON or XML. As it is a key/value data store, working with it is similar to working with an associative array—a key identifies an object uniquely within the store. Furthermore, the keys are stored as sorted for better performance. But what makes LevelDB perform better than an arbitrary file storage implementation? Well, it uses a "log-structured merge" topology, which stores all write operations in an in-memory log, transferred (flushed) regularly to a permanent storage called Sorted String Table (SST) files. Read operations first attempt to retrieve entries from a cache containing the most commonly returned results. The size of the reading cache and the flush interval of the writing log are configurable parameters, which can be further adjusted in order to be adequate for the application load. The following image shows this topology: The storage is a collection of string-sorted files with a maximum size of about 2 MB. Each file consists of 4 KB segments that are readable by a single read operation. The table files are not sorted in a straightforward manner, but are organized into levels. The log level is on top, before all other levels. It is always flushed to level 0, which consists of at most four SST files. When filled, one STS file is compacted to a lower level, that is, level 1. The maximum size of level 1 is 10 MB. When it gets filled, a file goes from level 1 to level 2. LevelDB assumes that the size of each lower level is ten times larger than the size of the previous level. So we have the following level structure: Log with a configurable size Level 0, consisting of four SST files Level 1, with a maximum size of 10 MB Level 2, with a maximum size of 100 MB Level 3, with a maximum size of 1000 MB Level n, with a maximum size of the previous level multiplied by 10 – (n-1)*10 MB The hierarchical structure of this topology assures that newer data stays in the top levels, while older data is somewhere in the lower levels. A read operation always starts searching for a given key in the cache, and if it is not found there, the operation traverses through each level until the entry is found. An entry is considered non-existing if its key is not found anywhere within all levels. LevelDB provides get, put, and delete operations to manipulate data records as well as a batch operation that can be used to perform multiple data manipulations atomically; that is, either all or none of the operations in the batch are executed successfully. LevelDB can optionally use a compression library in order to reduce the size of the stored values. This compression is provided by Google's Snappy compression library. It is highly optimized for fast compression with low performance impact, so too high expectations should not be expected for a large compression ratio. There are two popular libraries that enable LevelDB usage in Node: LevelDOWN and LevelUP. Initially, LevelDOWN was acting as foundation binding, implicitly provided with LevelUP, but after version 0.9, it had been extracted out of it and became available as a standalone binding for LevelDB. Currently, LevelUP has no explicit dependency on LevelDOWN defined. It needs to be installed separately, as LevelUP expects it to be available on its Node's require() path. LevelDOWN is a pure C++ interface used to bind Node and LevelDB. Though it is slightly faster than LevelUP, it has some state safety and API considerations, which make it less preferable than LevelUP. To be concrete, LevelDOWN does not keep track of the state of the underlying instance. Thus, it is up to the developers themselves not to open a connection more than once or use a data manipulating operation against a closed database connection, as this will cause errors. LevelUP provides state-safe operations out the box. Thus, it prevents out-of-state operations from being sent to its foundation—LevelDOWN. Let's move on to installing LevelUP by executing the following npm command: npm install levelup leveldown Even though the LevelUP module can be installed without LevelDOWN, it will not work at runtime, complaining that it can't find an underlying dependency. Enough theory! Let's see what the LevelUP API looks like. The following code snippet instantiates LevelDB and inserts a dummy contact record into it. It also exposes a /contacts/:number route so that this very record can be returned as a JSON output if queried appropriately. Let's use it in a new project in the Enide studio, in a file named levelup.js: var express = require('express') , http = require('http') , path = require('path') , bodyParser = require('body-parser') , logger = require('morgan') , methodOverride = require('method-override') , errorHandler = require('errorhandler') , levelup = require('levelup'); var app = express(); var url = require('url'); // all environments app.set('port', process.env.PORT || 3000); app.set('views', __dirname + '/views'); app.set('view engine', 'jade'); app.use(methodOverride()); app.use(bodyParser.json()); // development only if ('development' == app.get('env')) { app.use(errorHandler()); } var db = levelup('./contact', {valueEncoding: 'json'}); db.put('+359777123456', { "firstname": "Joe", "lastname": "Smith", "title": "Mr.", "company": "Dev Inc.", "jobtitle": "Developer", "primarycontactnumber": "+359777123456", "othercontactnumbers": [ "+359777456789", "+359777112233"], "primaryemailaddress": "joe.smith@xyz.com", "emailaddresses": [ "j.smith@xyz.com"], "groups": ["Dev","Family"] }); app.get('/contacts/:number', function(request, response) { console.log(request.url + ' : querying for ' + request.params.number); db.get(request.params.number, function(error, data) { if (error) { response.writeHead(404, { 'Content-Type' : 'text/plain'}); response.end('Not Found'); return; } response.setHeader('content-type', 'application/json'); response.send(data); }); }); console.log('Running at port ' + app.get('port')); http.createServer(app).listen(app.get('port')); As the contact is inserted into LevelDB before the HTTP server is created, the record identified with the +359777123456 key will be available in the database when we execute our first GET request. But before requesting any data, let's take a closer look at the usage of LevelUP. The get() function of LevelDB takes two arguments: The first argument is the key to be used in the query. The second argument is a handler function used to process the results. It also has two additional arguments: A Boolean value, specifying whether an error has occurred during the query The actual result entity from the database. Let's start it with Node's levelup.js and execute some test requests with the REST Client tool to http://localhost:3000/contacts/%2B359777123456. This can be seen in the following screenshot: Expectedly the response is a JSON representation of the contact inserted statically in LevelUP during the initialization of the application. Requesting any other key will result in an "HTTP 404 Not found" response. This example demonstrates how to bind a LevelUP operation to an HTTP operation and process its results, but currently, it lacks support for inserting, editing, and deleting data. We will improve that with the next sample. It binds the HTTP's GET, PUT, and DELETE operations exposed via an express route, /contacts/:number, to the LevelDB's get, put, and del handlers: var express = require('express') , http = require('http') , path = require('path') , bodyParser = require('body-parser') , logger = require('morgan') , methodOverride = require('method-override') , errorHandler = require('errorhandler') , levelup = require('levelup'); var app = express(); var url = require('url'); // all environments app.set('port', process.env.PORT || 3000); app.set('views', __dirname + '/views'); app.set('view engine', 'jade'); app.use(methodOverride()); app.use(bodyParser.json()); // development only if ('development' == app.get('env')) { app.use(errorHandler()); } var db = levelup('./contact', {valueEncoding: 'json'}); app.get('/contacts/:number', function(request, response) { console.log(request.url + ' : querying for ' + request.params.number); db.get(request.params.number, function(error, data) { if (error) { response.writeHead(404, { 'Content-Type' : 'text/plain'}); response.end('Not Found'); return; } response.setHeader('content-type', 'application/json'); response.send(data); }); }); app.post('/contacts/:number', function(request, response) { console.log('Adding new contact with primary number' + request.params.number); db.put(request.params.number, request.body, function(error) { if (error) { response.writeHead(500, { 'Content-Type' : 'text/plain'}); response.end('Internal server error'); return; } response.send(request.params.number + ' successfully inserted'); }); }); app.del('/contacts/:number', function(request, response) { console.log('Deleting contact with primary number' + request.params.number); db.del(request.params.number, function(error) { if (error) { response.writeHead(500, { 'Content-Type' : 'text/plain'}); response.end('Internal server error'); return; } response.send(request.params.number + ' successfully deleted'); }); }); app.get('/contacts', function(request, response) { console.log('Listing all contacts'); var is_first = true; response.setHeader('content-type', 'application/json'); db.createReadStream() .on('data', function (data) { console.log(data.value); if (is_first == true) { response.write('['); } else { response.write(','); } response.write(JSON.stringify(data.value)); is_first = false; }) .on('error', function (error) { console.log('Error while reading', error) }) .on('close', function () { console.log('Closing db stream');}) .on('end', function () { console.log('Db stream closed'); response.end(']'); }) }); console.log('Running at port ' + app.get('port')); http.createServer(app).listen(app.get('port')); Perhaps the most interesting part of the preceding sample is the handler of the /contacts route. It writes a JSON array of all the contacts available in the database to the output stream of the HTTP response. LevelUP's createInputStream method exposes a data handler for every key/value pair available. As LevelDB is not aware of the format of its values, we have to use the native JSON.stringify method to convert each value to a JSON object, based on which we can implement any kind of login. Let's assume we want a function that flushes to the HTTP response only those contacts whose first name is Joe. Then we will need to add filtering logic to the data handler: db.createReadStream() .on('data', function (data) { if (is_first == true) { response.write('['); } else { response.write(','); } if (data.value.lastname.toString() == 'Smith') { var jsonString = JSON.stringify(data.value) console.log('Adding Mr. ' + data.value.lastname + ' to the response'); response.write(jsonString); is_first = false; } else{ console.log('Skipping Mr. ' + data.value.lastname); } }) .on('error', function (error) { console.log('Error while reading', error) }) .on('close', function () { console.log('Closing db stream'); }) .on('end', function () { console.log('Db stream closed'); response.end(']'); }) This looks a bit artificial, doesn't it? Well, this is all that LevelDB can possibly offer us, since LevelDB can search only by a single key. This makes it an inappropriate option for data that has to be indexed by several different attributes. This is where document stores come into play. Summary In this article, we looked at one type of NoSQL database: LevelDB, a key/value datastore. We utilized it to implement automated test for the database layer. Resources for Article: Further resources on this subject: Node.js Fundamentals and Asynchronous JavaScript [article] An Introduction to Node.js Design Patterns [article] Making a Web Server in Node.js [article]

In this article by Thomas Bitterman, the author of the book Mastering IPython 4.0, we will look at the tools the IPython interactive shell provides. With the split of the Jupyter and IPython projects, the command line provided by IPython will gain importance. This article covers the following topics: What is IPython? Installing IPython Starting out with the terminal IPython beyond Python Magic commands (For more resources related to this topic, see here.) What is IPython? IPython is an open source platform for interactive and parallel computing. It started with the realization that the standard Python interpreter was too limited for sustained interactive use, especially in the areas of scientific and parallel computing. Overcoming these limitations resulted in a three-part architecture: An enhanced, interactive shell Separation of the shell from the computational kernel A new architecture for parallel computing This article will provide a brief overview of the architecture before introducing some basic shell commands. Before proceeding further, however, IPython needs to be installed. Those readers with experience in parallel and high-performance computing but new to IPython will find the following sections useful in quickly getting up to speed. Those experienced with IPython may skim the next few sections, noting where things have changed now that the notebook is no longer an integral part of development. Installing IPython The first step in installing IPython is to install Python. Instructions for the various platforms differ, but the instructions for each can be found on the Python home page at http://www.python.org. IPython requires Python 2.7 or ≥ 3.3. This article will use 3.5. Both Python and IPython are open source software, so downloading and installation are free. A standard Python installation includes the pip package manager. pip is a handy command-line tool that can be used to download and install various Python libraries. Once Python is installed, IPython can be installed with this command: pip install ipython IPython comes with a test suite called iptest. To run it, simply issue the following command: iptest A series of tests will be run. It is possible (and likely on Windows) that some libraries will be missing, causing the associated tests to fail. Simply use pip to install those libraries and rerun the test until everything passes. It is also possible that all tests pass without an important library being installed. This is the readline library (also known as PyReadline). IPython will work without it but will be missing some features that are useful for the IPython terminal, such as command completion and history navigation. To install readline, use pip: pip install readline pip install gnureadline At this point, issuing the ipython command will start up an IPython interpreter: ipython IPython beyond Python No one would use IPython if it were not more powerful than the standard terminal. Much of IPython's power comes from two features: Shell integration Magic commands Shell integration Any command starting with ! is passed directly to the operating system to be executed, and the result is returned. By default, the output is then printed out to the terminal. If desired, the result of the system command can be assigned to a variable. The result is treated as a multiline string, and the variable is a list containing one string element per line of output. For example: In [22]: myDir = !dir In [23]: myDir Out[23]: [' Volume in drive C has no label.', ' Volume Serial Number is 1E95-5694', '', ' Directory of C:\Program Files\Python 3.5', '', '10/04/2015 08:43 AM <DIR> .', '10/04/2015 08:43 AM <DIR> ..',] While this functionality is not entirely absent in straight Python (the OS and subprocess libraries provide similar abilities), the IPython syntax is much cleaner. Additional functionalities such as input and output caching, directory history, and automatic parentheses are also included. History The previous examples have had lines that were prefixed by elements such as In[23] and Out[15]. In and Out are arrays of strings, where each element is either an input command or the resulting output. They can be referred to using the arrays notation, or "magic" commands can accept the subscript alone. Magic commands IPython also accepts commands that control IPython itself. These are called "magic" commands, and they start with % or %%. A complete list of magic commands can be found by typing %lsmagic in the terminal. Magics that start with a single % sign are called "line" magics. They accept the rest of the current line for arguments. Magics that start with %% are called "cell" magics. They accept not only the rest of the current line but also the following lines. There are too many magic commands to go over in detail, but there are some related families to be aware of: OS equivalents: %cd, %env, %pwd Working with code: %run, %edit, %save, %load, %load_ext, %%capture Logging: %logstart, %logstop, %logon, %logoff, %logstate Debugging: %debug, %pdb, %run, %tb Documentation: %pdef, %pdoc, %pfile, %pprint, %psource, %pycat, %%writefile Profiling: %prun, %time, %run, %time, %timeit Working with other languages: %%script, %%html, %%javascript, %%latex, %%perl, %%ruby With magic commands, IPython becomes a more full-featured development environment. A development session might include the following steps: Set up the OS-level environment with the %cd, %env, and ! commands. Set up the Python environment with %load and %load_ext. Create a program using %edit. Run the program using %run. Log the input/output with %logstart, %logstop, %logon, and %logoff. Debug with %pdb. Create documentation with %pdoc and %pdef. This is not a tenable workflow for a large project, but for exploratory coding of smaller modules, magic commands provide a lightweight support structure. Creating custom magic commands IPython supports the creation of custom magic commands through function decorators. Luckily, one does not have to know how decorators work in order to use them. An example will explain. First, grab the required decorator from the appropriate library: In [1]: from IPython.core.magic import(register_line_magic) Then, prepend the decorator to a standard IPython function definition: In [2]: @register_line_magic ...: def getBootDevice(line): ...: sysinfo = !systeminfo ...: for ln in sysinfo: ...: if ln.startswith("Boot Device"): ...: return(ln.split()[2]) ...: Your new magic is ready to go: In [3]: %getBootDevice Out[3]: '\Device\HarddiskVolume1' Some observations are in order: Note that the function is, for the most part, standard Python. Also note the use of the !systeminfo shell command. One can freely mix both standard Python and IPython in IPython. The name of the function will be the name of the line magic. The parameter, "line," contains the rest of the line (in case any parameters are passed). A parameter is required, although it need not be used. The Out associated with calling this line magic is the return value of the magic. Any print statements executed as part of the magic are displayed on the terminal but are not part of Out (or _). Cython We are not limited to writing custom magic commands in Python. Several languages are supported, including R and Octave. We will look at one in particular, Cython. Cython is a language that can be used to write C extensions for Python. The goal for Cython is to be a superset of Python, with support for optional static type declarations. The driving force behind Cython is efficiency. As a compiled language, there are performance gains to be had from running C code. The downside is that Python is much more productive in terms of programmer hours. Cython can translate Python code into compiled C code, achieving more efficient execution at runtime while retaining the programmer-friendliness of Python. The idea of turning Python into C is not new to Cython. The default and most widely used interpreter (CPython) for Python is written in C. In some sense then, running Python code means running C code, just through an interpreter. There are other Python interpreter implementations as well, including those in Java (Jython) and C# (IronPython). CPython has a foreign function interface to C. That is, it is possible to write C language functions that interface with CPython in such a way that data can be exchanged and functions invoked from one to the other. The primary use is to call C code from Python. There are, however, two primary drawbacks: writing code that works with the CPython foreign function interface is difficult in its own right; and doing so requires knowledge of Python, C, and CPython. Cython aims to remedy this problem by doing all the work of turning Python into C and interfacing with CPython internally to Cython. The programmer writes Cython code and leaves the rest to the Cython compiler. Cython is very close to Python. The primary difference is the ability to specify C types for variables using the cdef keyword. Cython then handles type checking and conversion between Python values and C values, scoping issues, marshalling and unmarshalling of Python objects into C structures, and other cross-language issues. Cython is enabled in IPython by loading an extension. In order to use the Cython extension, do this: In [1]: %load_ext Cython At this point, the cython cell magic can be invoked: In [2]: %%cython ...: def sum(int a, int b): ...: cdef int s = a+b ...: return s And the Cython function can now be called just as if it were a standard Python function: In [3]: sum(1, 1) Out[3]: 2 While this may seem like a lot of work for something that could have been written more easily in Python in the first place, that is the price to be paid for efficiency. If, instead of simply summing two numbers, a function is expensive to execute and is called multiple times (perhaps in a tight loop), it can be worth it to use Cython for a reduction in runtime. There are other languages that have merited the same treatment, GNU Octave and R among them. Summary In this article, we covered many of the basics of using IPython for development. We started out by just getting an instance of IPython running. The intrepid developer can perform all the steps by hand, but there are also various all-in-one distributions available that will include popular modules upon installation. By default, IPython will use the pip package managers. Again, the all-in-one distributions provide added value, this time in the form of advanced package management capability. At that point, all that is obviously available is a terminal, much like the standard Python terminal. IPython offers two additional sources of functionality, however: configuration and magic commands. Magic commands fall into several categories: OS equivalents, working with code, logging, debugging, documentation, profiling, and working with other languages among others. Add to this the ability to create custom magic commands (in IPython or another language) and the IPython terminal becomes a much more powerful alternative to the standard Python terminal. Also included in IPython is the debugger—ipdb. It is very similar to the Python pdb debugger, so it should be familiar to Python developers. All this is supported by the IPython architecture. The basic idea is that of a Read-Eval-Print loop in which the Eval section has been separated out into its own process. This decoupling allows different user interface components and kernels to communicate with each other, making for a flexible system. This flexibility extends to the development environment. There are IDEs devoted to IPython (for example, Spyder and Canopy) and others that originally targeted Python but also work with IPython (for example, Eclipse). There are too many Python IDEs to list, and many should work with an IPython kernel "dropped in" as a superior replacement to a Python interpreter. Resources for Article: Further resources on this subject: Python Data Science Up and Running [article] Scientific Computing APIs for Python [article] Overview of Process Management in Microsoft Visio 2013 [article]

In this article by Nathan Kozyra, the author of the book Learning Go Web Development, one of the most common things you'll hear being said is that it's a systems language. Indeed, one of the earlier descriptions of Go, by the Go team itself, was that the language was built to be a modern systems language. It was constructed to combine the speed and the power of languages, such as C with the syntactical elegance and thrift of modern interpreted languages, such as Python. You can see that the goal is realized when you look at just a few snippets of the Go code. The Go FAQ, on why Go was created: "Go was born out of frustration with existing languages and environments for systems programming." Perhaps the largest part of the present-day systems programming comprises of designing backend servers. Obviously, the Web comprises a huge, but not exclusive, percentage of that world. Go hasn't been considered a web language until recently. Unsurprisingly, it took a few years of developers dabbling, experimenting, and finally embracing the language to start taking it to new avenues. While Go is web-ready out of the box, it lacks a lot of the critical frameworks and tools people so often take for granted with web development now. As the community around Go grew, the scaffolding began to manifest in a lot of new and exciting ways. Combined with existing ancillary tools, Go is now a wholly viable option for end-to-end web development. However, lets get back to the primary question: Why Go? To be fair, it's not right for every web project, but any application that can benefit from a high-performance, secure web-serving out of the box with the added benefits of a beautiful concurrency model would make for a good candidate. We're not going to deal with a lot of low-level aspects of the Go language. For example, we assume that you're familiar with variable and constant declaration. We assume that you understand control structures. In this article, we will cover the following topics: Installing Go Structuring a project Importing packages (For more resources related to this topic, see here.) Installing Go The most critical first step is, of course, making sure that Go is available and ready to start our first web server. While one of Go's biggest selling points is its cross-platform support (both building and using locally while targeting other operating systems), your life will be much more easier on a Nix compatible platform. If you're on Windows, don't fear. Natively, you may run into incompatible packages and firewall issues when running using Go run and some other quirks, but 95% of the Go ecosystem will be available to you. You can also, very easily, run a virtual machine and in fact that is a great way to simulate a potential production environment. In-depth installation instructions are available at https://golang.org/doc/install, but we'll talk about a few quirky points here before moving on. For OS X and Windows, Go is provided as a part of a binary installation packages. For any Linux platform with a package manager, things can be pretty easy. To install via common Linux package managers: Ubuntu: sudo apt-get golang CentOS: sudo yum install golang On both OS X and Linux, you'll need to add a couple of lines to your path—the GOPATH and PATH. First, you'll have to find the location of your Go binary's installation. This varies from distribution to distribution. Once you've found that, you can configure the PATH and GOPATH, as follows: export PATH=$PATH:/usr/local/go/bin export GOPATH="/usr/share/go" While the path to be used is not defined rigidly, some convention has coalesced around; starting at a subdirectory directly under your user's home directory, such as $HOME/go or ~Home/go. As long as this location is set perpetually and doesn't change, you won't run into issues with conflicts or missing packages. You can test the impact of these changes by running the go env command. If you see any issues with this, it means that your directories are not correct. Note that this may not prevent Go from running—depending on whether the GOBIN directory is properly set—but will prevent you from installing packages globally across your system. To test the installation, you can grab any Go package using a go get command and create a Go file somewhere. As a quick example, first get a package at random, we'll use a package from the Gorilla framework. go get github.com/gorilla/mux If this runs without any issue, Go is finding your GOPATH correctly. To make sure that Go is able to access your downloaded packages, draw up a very quick package that will attempt to utilize Gorilla's mux package and run it to verify whether the packages are found: package main import ( "fmt" "github.com/gorilla/mux" "net/http" ) func TestHandler(w http.ResponseWriter, r *http.Request) { } func main() { router := mux.NewRouter() router.HandleFunc("/test", TestHandler) http.Handle("/", router) fmt.Println("Everything is set up!") } Run go run test.go in the command line. It won't do much, but it will deliver the good news, as shown in the following screenshot: Structuring a project When you're first getting started and mostly playing around, there's no real problem with setting your application lazily. For example, to get started as quickly as possible, you can create a simple hello.go file anywhere you like and compile without any issue. But when you get into environments that require multiple or distinct packages (more on that shortly) or have more explicit cross-platform requirements, it makes sense to design your projects in a way that will facilitate the use of the Go build tool. The value of setting up your code in this manner lies in the way that the Go build tool works. If you have local (to your project) packages, the build tool will look in the src directory first and then in your GOPATH. When you're building for other platforms, Go build will utilize the local bin folder to organize the binaries. While building packages that are intended for mass use, you may also find that either starting your application under your GOPATH directory and then symbolically linking it to another directory or doing the opposite will allow you to develop without the need to subsequently go get your own code. Code conventions As with any language, being a part of the Go community means perpetual consideration of the way others create their code. Particularly if you're going to work in open source repositories, you'll want to generate your code the way others do, to reduce the amount of friction when people get or include your code. One incredibly helpful piece of tooling that the Go team has included is go fmt. fmt here, of course, means format and that's exactly what this tool does, it automatically formats your code according to the designed conventions. By enforcing style conventions, the Go team has helped to mitigate one of the most common and pervasive debates that exist among a lot of other languages. While the language communities tend to drive coding conventions, there are always little idiosyncrasies in the way individuals write programs. Let's use one of the most common examples around—where to put the opening bracket. Some programmers like it on the same line as the statement: for (int i = 0; i < 100; i++) { // do something } While others prefer it in the subsequent line: for (int i = 0; i < 100; i++) { // do something } These types of minor differences spark major, near-religious debates. The Gofmt tool helps alleviate this by allowing you to yield to Go's directive. Now, Go bypasses this obvious source of contention at the compiler, by formatting your code similar to the latter example discussed earlier. The compiler will complain and all you'll get is a fatal error. However, the other style choices have some flexibility, which are enforced when you use the tool to format. Here, for example, is a piece of code in Go before go fmt: func Double(n int) int { if (n == 0) { return 0 } else { return n * 2 } } Arbitrary whitespace can be the bane of a team's existence when it comes to sharing and reading code, particularly when every team member is not on the same IDE. By running go fmt, we clean this up, thereby translating our whitespace according to Go's conventions: func Double(n int) int { if n == 0 { return 0 } else { return n * 2 } } Long story short: always run go fmt before shipping or pushing your code. Importing packages Beyond the absolute and the most trivial application—one that cannot even produce a Hello World output—you must have some imported package in a Go application. To say Hello World, for example, we'd need some sort of a way to generate an output. Unlike in many other languages, even the core language library is accessible by a namespaced package. In Go, namespaces are handled by a repository endpoint URL, which is github.com/nkozyra/gotest, which can be opened directly on Github (or any other public location) for the review. Handling private repositories The go get tool easily handles packages hosted at the repositories, such as Github, Bitbucket, and Google Code (as well as a few others). You can also host your own projects, ideally a git project, elsewhere; although it might introduce some dependencies and sources for errors, which you'd probably like to avoid. But, what about the private repos? While go get is a wonderful tool, you'll find yourself looking at an error without some additional configuration with an SSH agent forwarding and so on. You can work around this in a couple of ways, but one very simple method is to clone the repository locally, using your version control software directly. Summary This article serves as an introduction to the most basic concepts of Go and producing for the Web in Go, but these points are critical foundational elements for being productive in the language and in the community. We've looked at coding conventions and package design and organization. Obviously, we're a long way from a real, mature application for the Web, but the building blocks are essential to getting there. Resources for Article: Further resources on this subject: ASP.NET 3.5 CMS: Adding Security and Membership (Part 2) [article] Working with Drupal Audio in Flash (part 2) [article] Posting on Your WordPress Blog [article]

In this article by Heather Mahalik and Rohit Tamma, authors of the book Practical Mobile Forensics, Second Edition, we will cover the following topics: Introduction to mobile forensics Challenges in mobile forensics (For more resources related to this topic, see here.) Why do we need mobile forensics? In 2015, there were more than 7 billion mobile cellular subscriptions worldwide, up from less than 1 billion in 2000, says International Telecommunication Union (ITU). The world is witnessing technology and user migration from desktops to mobile phones. The following figure sourced from statista.com shows the actual and estimated growth of smartphones from the year 2009 to 2018. Growth of smartphones from 2009 to 2018 in million units Gartner Inc. reports that global mobile data traffic reached 52 million terabytes (TB) in 2015, an increase of 59 percent from 2014, and the rapid growth is set to continue through 2018, when mobile data levels are estimated to reach 173 million TB. Smartphones of today, such as the Apple iPhone, Samsung Galaxy series, and BlackBerry phones, are compact forms of computers with high performance, huge storage, and enhanced functionalities. Mobile phones are the most personal electronic device that a user accesses. They are used to perform simple communication tasks, such as calling and texting, while still providing support for Internet browsing, e-mail, taking photos and videos, creating and storing documents, identifying locations with GPS services, and managing business tasks. As new features and applications are incorporated into mobile phones, the amount of information stored on the devices is continuously growing. Mobiles phones become portable data carriers, and they keep track of all your moves. With the increasing prevalence of mobile phones in peoples' daily lives and in crime, data acquired from phones become an invaluable source of evidence for investigations relating to criminal, civil, and even high-profile cases. It is rare to conduct a digital forensic investigation that does not include a phone. Mobile device call logs and GPS data were used to help solve the attempted bombing in Times Square, New York, in 2010. The details of the case can be found at http://www.forensicon.com/forensics-blotter/cell-phone-email-forensics-investigation-cracks-nyc-times-square-car-bombing-case/. The science behind recovering digital evidence from mobile phones is called mobile forensics. Digital evidence is defined as information and data that is stored on, received, or transmitted by an electronic device that is used for investigations. Digital evidence encompasses any and all digital data that can be used as evidence in a case. Mobile forensics Digital forensics is a branch of forensic science focusing on the recovery and investigation of raw data residing in electronic or digital devices. The goal of the process is to extract and recover any information from a digital device without altering the data present on the device. Over the years, digital forensics grew along with the rapid growth of computers and various other digital devices. There are various branches of digital forensics based on the type of digital device involved such as computer forensics, network forensics, mobile forensics, and so on. Mobile forensics is a branch of digital forensics related to the recovery of digital evidence from mobile devices. Forensically sound is a term used extensively in the digital forensics community to qualify and justify the use of particular forensic technology or methodology. The main principle for a sound forensic examination of digital evidence is that the original evidence must not be modified. This is extremely difficult with mobile devices. Some forensic tools require a communication vector with the mobile device, thus a standard write protection will not work during forensic acquisition. Other forensic acquisition methods may involve removing a chip or installing a bootloader on the mobile device prior to extract data for forensic examination. In cases where the examination or data acquisition is not possible without changing the configuration of the device, the procedure and the changes must be tested, validated, and documented. Following proper methodology and guidelines is crucial in examining mobile devices as it yields the most valuable data. As with any evidence gathering, not following the proper procedure during the examination can result in loss or damage of evidence or render it inadmissible in court. The mobile forensics process is broken into three main categories: seizure, acquisition, and examination/analysis. Forensic examiners face some challenges while seizing the mobile device as a source of evidence. At the crime scene, if the mobile device is found switched off, the examiner should place the device in a faraday bag to prevent changes should the device automatically power on. As shown in the following figure, Faraday bags are specifically designed to isolate the phone from the network. A Faraday bag (Image courtesy: http://www.amazon.com/Black-Hole-Faraday-Bag-Isolation/dp/B0091WILY0) If the phone is found switched on, switching it off has a lot of concerns attached to it. If the phone is locked by a PIN or password or encrypted, the examiner will be required to bypass the lock or determine the PIN to access the device. Mobile phones are networked devices and can send and receive data through different sources, such as telecommunication systems, Wi-Fi access points, and Bluetooth. So, if the phone is in a running state, a criminal can securely erase the data stored on the phone by executing a remote wipe command. When a phone is switched on, it should be placed in a faraday bag. If possible, prior to placing the mobile device in the faraday bag, disconnect it from the network to protect the evidence by enabling the flight mode and disabling all network connections (Wi-Fi, GPS, Hotspots, and so on). This will also preserve the battery, which will drain while in a faraday bag and protect against leaks in the faraday bag. Once the mobile device is seized properly, the examiner may need several forensic tools to acquire and analyze the data stored on the phone. Mobile phones are dynamic systems that present a lot of challenges to the examiner in extracting and analyzing digital evidence. The rapid increase in the number of different kinds of mobile phones from different manufacturers makes it difficult to develop a single process or tool to examine all types of devices. Mobile phones are continuously evolving as existing technologies progress and new technologies are introduced. Furthermore, each mobile is designed with a variety of embedded operating systems. Hence, special knowledge and skills are required from forensic experts to acquire and analyze the devices. Challenges in mobile forensics One of the biggest forensic challenges when it comes to the mobile platform is the fact that data can be accessed, stored, and synchronized across multiple devices. As the data is volatile and can be quickly transformed or deleted remotely, more effort is required for the preservation of this data. Mobile forensics is different from computer forensics and presents unique challenges to forensic examiners. Law enforcement and forensic examiners often struggle to obtain digital evidence from mobile devices. The following are some of the reasons: Hardware differences: The market is flooded with different models of mobile phones from different manufacturers. Forensic examiners may come across different types of mobile models, which differ in size, hardware, features, and operating system. Also, with a short product development cycle, new models emerge very frequently. As the mobile landscape is changing each passing day, it is critical for the examiner to adapt to all the challenges and remain updated on mobile device forensic techniques across various devices. Mobile operating systems: Unlike personal computers where Windows has dominated the market for years, mobile devices widely use more operating systems, including Apple's iOS, Google's Android, RIM's BlackBerry OS, Microsoft's Windows Mobile, HP's webOS, Nokia's Symbian OS, and many others. Even within these operating systems, there are several versions which make the task of forensic investigator even more difficult. Mobile platform security features: Modern mobile platforms contain built-in security features to protect user data and privacy. These features act as a hurdle during the forensic acquisition and examination. For example, modern mobile devices come with default encryption mechanisms from the hardware layer to the software layer. The examiner might need to break through these encryption mechanisms to extract data from the devices. Lack of resources: As mentioned earlier, with the growing number of mobile phones, the tools required by a forensic examiner would also increase. Forensic acquisition accessories, such as USB cables, batteries, and chargers for different mobile phones, have to be maintained in order to acquire those devices. Preventing data modification: One of the fundamental rules in forensics is to make sure that data on the device is not modified. In other words, any attempt to extract data from the device should not alter the data present on that device. But this is practically not possible with mobiles because just switching on a device can change the data on that device. Even if a device appears to be in an off state, background processes may still run. For example, in most mobiles, the alarm clock still works even when the phone is switched off. A sudden transition from one state to another may result in the loss or modification of data. Anti-forensic techniques: Anti-forensic techniques, such as data hiding, data obfuscation, data forgery, and secure wiping, make investigations on digital media more difficult. Dynamic nature of evidence: Digital evidence may be easily altered either intentionally or unintentionally. For example, browsing an application on the phone might alter the data stored by that application on the device. Accidental reset: Mobile phones provide features to reset everything. Resetting the device accidentally while examining may result in the loss of data. Device alteration: The possible ways to alter devices may range from moving application data, renaming files, and modifying the manufacturer's operating system. In this case, the expertise of the suspect should be taken into account. Passcode recovery: If the device is protected with a passcode, the forensic examiner needs to gain access to the device without damaging the data on the device. While there are techniques to bypass the screen lock, they may not work always on all the versions. Communication shielding: Mobile devices communicate over cellular networks, Wi-Fi networks, Bluetooth, and Infrared. As device communication might alter the device data, the possibility of further communication should be eliminated after seizing the device. Lack of availability of tools: There is a wide range of mobile devices. A single tool may not support all the devices or perform all the necessary functions, so a combination of tools needs to be used. Choosing the right tool for a particular phone might be difficult. Malicious programs: The device might contain malicious software or malware, such as a virus or a Trojan. Such malicious programs may attempt to spread over other devices over either a wired interface or a wireless one. Legal issues: Mobile devices might be involved in crimes, which can cross geographical boundaries. In order to tackle these multijurisdictional issues, the forensic examiner should be aware of the nature of the crime and the regional laws. Summary Mobile devices store a wide range of information such as SMS, call logs, browser history, chat messages, location details, and so on. Mobile device forensics includes many approaches and concepts that fall outside of the boundaries of traditional digital forensics. Extreme care should be taken while handling the device right from evidence intake phase to archiving phase. Examiners responsible for mobile devices must understand the different acquisition methods and the complexities of handling the data during analysis. Extracting data from a mobile device is half the battle. The operating system, security features, and type of smartphone will determine the amount of access you have to the data. It is important to follow sound forensic practices and make sure that the evidence is unaltered during the investigation. Resources for Article: Further resources on this subject: Forensics Recovery [article] Mobile Phone Forensics – A First Step into Android Forensics [article] Mobility [article]

In this article by Yogesh Patel, the author of the book, Sitecore Cookbook for Developers, we will discuss about the importance of Sitecore and its good features. (For more resources related to this topic, see here.) Why Sitecore? Sitecore Experience Platform (XP) is not only an enterprise-level content management system (CMS), but rather a web framework or web platform, which is the global leader in experience management. It continues to be very popular because of its highly scalable and robust architecture, continuous innovations, and ease of implementations compared to other CMSs available. It also provides an easier integration with many external platforms such as customer relationship management (CRM), e-commerce, and so on. Sitecore architecture is built with the Microsoft .NET framework and provides greater depth of APIs, flexibility, scalability, performance, and power to developers. It has great out-of-the-box capabilities, but one of its great strengths is the ease of extending these capabilities; hence, developers love Sitecore! Sitecore provides many features and functionalities out of the box to help content owners and marketing teams. These features can be extended and highly customized to meet the needs of your unique business rules. Sitecore provides these features with different user-friendly interfaces for content owners that helps them manage content and media easily and quickly. Sitecore user interfaces are supported on almost every modern browser. In addition, fully customized web applications can be layered in and integrated with other modules and tools using Sitecore as the core platform. It helps marketers to optimize the flow of content continuously for better results and more valuable outcomes. It also provides in-depth analytics, personalized experience to end users, and marketing automation tools, which play a significant role for marketing teams. The following are a few of the many features of Sitecore. CMS based on the .NET Framework Sitecore provides building components on ASP.NET Web forms as well as ASP.NET Model-View-Controller (MVC) frameworks, so developers can choose either approach to match the required architecture. Sitecore provides web controls and sublayouts while working with ASP.NET web forms and view rendering, controller rendering, and models and item rendering while working with the ASP.NET MVC framework. Sitecore also provides two frameworks to prepare user interface (UI) applications for Sitecore clients—Sheer UI and SPEAK. Sheer UI applications are prepared using Extensible Application Markup Language (XAML) and most of the Sitecore applications are prepared using Sheer UI. Sitecore Process Enablement and Accelerator Kit (SPEAK) is the latest framework to develop Sitecore applications with a consistent interface quickly and easily. SPEAK gives you a predefined set of page layouts and components: Component-based architecture Sitecore is built on a component-based architecture, which provides us with loosely coupled independent components. The main advantage of these components is their reusability and loosely coupled independent behaviour. It aims to provide reusability of components at the page level, site level, and Sitecore instance level to support multisite or multitenant sites. Components in Sitecore are built with the normal layered approach, where the components are split into layers such as presentation, business logic, data layer, and so on. Sitecore provides different presenation components, including layouts, sublayouts, web control renderings, MVC renderings, and placeholders. Sitecore manages different components in logical grouping by their templates, layouts, sublayouts, renderings, devices, media, content items, and so on: Layout engine The Sitecore layout engine extends the ASP.NET web application server to merge content with presentation logic dynamically when web clients request resources. A layout can be a web form page (.aspx) or MVC view (.cshtml) file. A layout can have multiple placeholders to place content on predefined places, where the controls are placed. Controls can be HTML markup controls such as a sublayout (.ascx) file, MVC view (.cshtml) file, or other renderings such as web control, controller rendering, and so on, which can contain business logic. Once the request criteria are resolved by the layout engine, such as item, language, and device, the layout engine creates a platform to render different controls and assemble their output to relevant placeholders on the layout. Layout engine provides both static and dynamic binding. So, with dynamic binding, we can have clean HTML markups and reusability of all the controls or components. Binding of controls, layouts, and devices can be applied on Sitecore content items itself, as shown in the following screenshot: Once the layout engine renders the page, you can see how the controls will be bound to the layout, as shown in the following image: The layout engine in Sitecore is reponsible for layout rendering, device detection, rule engine, and personalization: Multilingual support In Sitecore, content can be maintained in any number of languages. It provides easier integration with external translation providers for seamless translation and also supports the dynamic creation of multilingual web pages. Sitecore also supports the language fallback feature on the field, item, and template level, which makes life easier for content owners and developers. It also supports chained fallback. Multi-device support Devices represent different types of web clients that connect to the Internet and place HTTP requests. Each device represents a different type of web client. Each device can have unique markup requirements. As we saw, the layout engine applies the presentation components specified for the context device to the layout details of the context item. In the same way, developers can use devices to format the context item output using different collections of presentation components for various types of web clients. Dynamically assembled content can be transformed to conform to virtually any output format, such as a mobile, tablet, desktop, print, or RSS. Sitecore also supports the device fallback feature so that any web page not supported for the requesting device can still be served through the fallback device. It also supports chained fallback for devices. Multi-site capabilities There are many ways to manage multisites on a single Sitecore installation. For example, you can host multiple regional domains with different regional languages as the default language for a single site. For example, http://www.sitecorecookbook.com will serve English content, http://www.sitecorecookbook.de will serve German content of the same website, and so on. Another way is to create multiple websites for different subsidiaries or franchise of a company. In this approach, you can share some common resources across all the sites such as templates, renderings, user interface elements, and other content or media items, but have unique content and pages so that you can find a separate existence of each website in Sitecore. Sitecore has security capabilities so that each franchise or subsidiary can manage their own website independently without affecting other websites. Developers have full flexibility to re-architect Sitecore's multisite architecture as per business needs. Sitecore also supports multitenant multisite architecture so that each website can work as an individual physical website. Caching Caching plays a very important role in website performance. Sitecore contains multiple levels of caching such as prefetch cache, data cache, item cache, and HTML cache. Apart from this, Sitecore creates different caching such as standard values cache, filtered item cache, registry cache, media cache, user cache, proxy cache, AccessResult cache, and so on. This makes understanding all the Sitecore caches really important. Sitecore caching is a very vast topic to cover; you can read more about it at http://sitecoreblog.patelyogesh.in/2013/06/how-sitecore-caching-work.html. Configuration factory Sitecore is configured using IIS's configuration file, Web.config. Sitecore configuration factory allows you to configure pipelines, events, scheduling agents, commands, settings, properties, and configuration nodes in Web.config files, which can be defined in the /configuration/sitecore path. Configurations inside this path can be spread out between multiple files to make it scalable. This process is often called config patching. Instead of touching the Web.config file, Sitecore provides the Sitecore.config file in the App_ConfigInclude directory, which contains all the important Sitecore configurations. Functionality-specific configurations are split into the number of .config files based, which you can find in its subdirectories. These .config files are merged into a single configuration file at runtime, which you can evaluate using http://<domain>/sitecore/admin/showconfig.aspx. Thus, developers create custom .config files in the App_ConfigInclude directory to introduce, override, or delete settings, properties, configuration nodes, and attributes without touching Sitecore's default .config files. This makes managing .config files very easy from development to deployment. You can learn more about file patching from https://sdn.sitecore.net/upload/sitecore6/60/include_file_patching_facilities_sc6orlater-a4.pdf. Dependency injection in .NET has become very common nowadays. If you want to build a generic and reusable functionality, you will surely go for the inversion of control (IoC) framework. Fortunately, Sitecore provides a solution that will allow you to easily use different IoC frameworks between projects. Using patch files, Sitecore allows you to define objects that will be available at runtime. These nodes are defined under /configuration/sitecore and can be retrieved using the Sitecore API. We can define types, constructors, methods, properties, and their input parameters in logical nodes inside nodes of pipelines, events, scheduling agents, and so on. You can learn more examples of it from http://sitecore-community.github.io/docs/documentation/Sitecore%20Fundamentals/Sitecore%20Configuration%20Factory/. Pipelines An operation to be performed in multiple steps can be carried out using the pipeline system, where each individual step is defined as a processor. Data processed from one processor is then carried to the next processor in arguments. The flow of the pipeline can be defined in XML format in the .config files. You can find default pipelines in the Sitecore.config file or patch file under the <pipelines> node (which are system processes) and the <processors> node (which are UI processes). The following image visualizes the pipeline and processors concept: Each processor in a pipeline contains a method named Process() that accepts a single argument, Sitecore.Pipelines.PipelineArgs, to get different argument values and returns void. A processor can abort the pipeline, preventing Sitecore from invoking subsequent processors. A page request traverses through different pipelines such as <preProcessRequest>, <httpRequestBegin>, <renderLayout>, <httpRequestEnd>, and so on. The <httpRequestBegin> pipeline is the heart of the Sitecore HTTP request execution process. It defines different processors to resolve the site, device, language, item, layout, and so on sequentially, which you can find in Sitecore.config as follows: <httpRequestBegin>   ...   <processor type="Sitecore.Pipelines.HttpRequest.SiteResolver,     Sitecore.Kernel"/>   <processor type="Sitecore.Pipelines.HttpRequest.UserResolver,     Sitecore.Kernel"/>   <processor type="     Sitecore.Pipelines.HttpRequest.DatabaseResolver,     Sitecore.Kernel"/>   <processor type="     Sitecore.Pipelines.HttpRequest.BeginDiagnostics,     Sitecore.Kernel"/>   <processor type="     Sitecore.Pipelines.HttpRequest.DeviceResolver,     Sitecore.Kernel"/>   <processor type="     Sitecore.Pipelines.HttpRequest.LanguageResolver,     Sitecore.Kernel"/>   ... </httpRequestBegin> There are more than a hundred pipelines, and the list goes on increasing after every new version release. Sitecore also allows us to create our own pipelines and processors. Background jobs When you need to do some long-running operations such as importing data from external services, sending e-mails to subscribers, resetting content item layout details, and so on, we can use Sitecore jobs, which are asynchronous operations in the backend that you can monitor in a foreground thread (Job Viewer) of Sitecore Rocks or by creating a custom Sitecore application. The jobs can be invoked from the user interface by users or can be scheduled. Sitecore provides APIs to invoke jobs with many different options available. You can simply create and start a job using the following code: public void Run() {   JobOptions options = new JobOptions("Job Name", "Job Category",     "Site Name", "current object", "Task Method to Invoke", new     object[] { rootItem })   {     EnableSecurity = true,     ContextUser = Sitecore.Context.User,     Priority = ThreadPriority.AboveNormal   };   JobManager.Start(options); } You can schedule tasks or jobs by creating scheduling agents in the Sitecore.config file. You can also set their execution frequency. The following example shows you how Sitecore has configured PublishAgent, which publishes a site every 12 hours and simply executes the Run() method of the Sitecore.Tasks.PublishAgent class: <scheduling>   <agent type="Sitecore.Tasks.PublishAgent" method="Run"     interval="12:00:00">     <param desc="source database">master</param>     <param desc="target database">web</param>     <param desc="mode (full or smart or       incremental)">incremental</param>     <param desc="languages">en, da</param>   </agent> </scheduling> Apart from this, Sitecore also provides you with the facility to define scheduled tasks in the database, which has a great advantage of storing tasks in the database, so that we can handle its start and end date and time. We can use it once or make it recurring as well. Workflow and publishing Workflows are essential to the content author experience. Workflows ensure that items move through a predefined set of states before they become publishable. It is necessary to ensure that content receives the appropriate reviews and approvals before publication to the live website. Apart from workflow, Sitecore provides highly configurable security features, access permissions, and versioning. Sitecore also provides full workflow history like when and by whom the content was edited, reviewed, or approved. It also allows you to restrict publishing as well as identify when it is ready to be published. Publishing is an essential part of working in Sitecore. Every time you edit or create new content, you have to publish it to see it on your live website. When publishing happens, the item is copied from the master database to the web database. So, the content of the web database will be shown on the website. When multiple users are working on different content pages or media items, publishing restrictions and workflows play a vital role to make releases, embargoed, or go-live successful. There are three types of publishing available in Sitecore: Republish: This publishes every item even though items are already published. Smart Publish: Sitecore compares the internal revision identifier of the item in the master and web databases. If both identifiers are different, it means that the item is changed in the master database, hence Sitecore will publish the item or skip the item if identifiers are the same. Incremental Publish: Every modified item is added to the publish queue. Once incremental publishing is done, Sitecore will publish all the items found in the publish queue and clear it. Sitecore also supports the publishing of subitems as well as related items (such as publishing a content item will also publish related media items). Search Sitecore comes with out-of-the-box Lucene support. You can also switch your Sitecore search to Solr, which just needs to install Solr and enable Solr configurations already available. Sitecore by default indexes Sitecore content in Lucene index files. The Sitecore search engine lets you search through millions of items of the content tree quickly with the help of different types of queries with Lucene or Solr indexes. Sitecore provides you with the following functionalities for content search: We can search content items and documents such as PDF, Word, and so on. It allows you to search content items based on preconfigured fields. It provides APIs to create and search composite fields as per business needs. It provides content search APIs to sort, filter, and page search results. We can apply wildcards to search complex results and autosuggest. We can apply boosting to influence search results or elevate results by giving more priority. We can create custom dictionaries and index files, using which we can suggest did you mean kind of suggestions to users. We can apply facets to refine search results as we can see on e-commerce sites. W can apply different analyzers to hunt MoreLikeThis results or similar results. We can tag content or media items to categorize them so that we can use features such as a tag cloud. It provides a scalable user interface to search content items and apply filters and operations to selected search results. It provides different indexing strategies to create transparent and diverse models for index maintenance. In short, Sitecore allows us to implement different searching techniques, which are available in Google or other search engines. Content authors always find it difficult while working with a big number of items. You can read more about Sitecore search at https://doc.sitecore.net/sitecore_experience_platform/content_authoring/searching/searching. Security model Sitecore has the reputation of being very easy to set up the security of users, roles, access rights, and so on. Sitecore follows the .NET security model, so we get all the basic information of the .NET membership in Sitecore, which offers several advantages: A variety of plug-and-play features provided directly by Microsoft The option to replace or extend the default configuration with custom providers It is also possible to store the accounts in different storage areas using several providers simultaneously Sitecore provides item-level and field-level rights and an option to create custom rights as well Dynamic user profile structure and role management is possible just through the user interface, which is simpler and easier compared to pure ASP.NET solutions It provides easier implementation for integration with external systems Even after having an extended wrapper on the .NET solution, we get the same performance as a pure ASP.NET solution Experience analytics and personalization Sitecore contains state-of-the-art Analysis, Insights, Decisions, Automation (AIDA) framework, which is the heart for marketing programs. It provides comprehensive analytics data and reports, insights from every website interaction with rules, behavior-based personalization, and marketing automation. Sitecore collects all the visitor interactions in a real-time, big data repository—Experience Database (xDB)—to increase the availability, scalability, and performance of website. Sitecore Marketing Foundation provides the following features: Sitecore uses MongoDB, a big marketing data repository that collects all customer interactions. It provides real-time data to marketers to automate interactions across all channels. It provides a unified 360 degree view of the individual website visitors and in-depth analytics reports. It provides fundamental analytics measurement components such as goals and events to evaluate the effectiveness of online business and marketing campaigns. It provides comprehensive conditions and actions to achieve conditional and behavioral or predictive personalization, which helps show customers what they are looking for instead of forcing them to see what we want to show. Sitecore collects, evaluates, and processes Omnichannel visitor behavioral patterns, which helps better planned effective marketing campaigns and improved user experience. Sitecore provides an engagement plan to control how your website interacts with visitors. It helps nurture relationships with your visitors by adapting personalized communication based on which state they are falling. Sitecore provides an in-depth geolocation service, helpful in optimizing campaigns through segmentation, personalization, and profiling strategies. The Sitecore Device Detection service is helpful in personalizing user experience or promotions based on the device they use. It provides different dimensions and reports to reflect data on full taxonomy provided in Marketing Control Panel. It provides different charting controls to get smart reports. It has full flexibility for developers to customize or extend all these features. High performance and scalability Sitecore supports heavy content management and content delivery usage with a large volume of data. Sitecore is architected for high performance and unlimited scalability. Sitecore cache engine provides caching on the raw data as well as rendered output data, which gives a high-performance platform. Sitecore uses the event queue concept for scalability. Theoretically, it makes Sitecore scalable to any number of instances under a load balancer. Summary In this article, we discussed about the importance of Sitecore and its good features. We also saw that Sitecore XP is not only an enterprise-level CMS, but also a web platform, which is the global leader in experience management. Resources for Article: Further resources on this subject: Building a Recommendation Engine with Spark [article] Configuring a MySQL linked server on SQL Server 2008 [article] Features and utilities in SQL Developer Data Modeler [article]

In this article by Venkat Ankam, author of the book, Big Data Analytics with Spark and Hadoop, we will understand the features of Hadoop and Spark and how we can combine them. (For more resources related to this topic, see here.) This article is divided into the following subtopics: Introducing Apache Spark Why Hadoop + Spark? Introducing Apache Spark Hadoop and MapReduce have been around for 10 years and have proven to be the best solution to process massive data with high performance. However, MapReduce lacked performance in iterative computing where the output between multiple MapReduce jobs had to be written to Hadoop Distributed File System (HDFS). In a single MapReduce job as well, it lacked performance because of the drawbacks of the MapReduce framework. Let's take a look at the history of computing trends to understand how computing paradigms have changed over the last two decades. The trend was to reference the URI when the network was cheaper (in 1990), Replicate when storage became cheaper (in 2000), and Recompute when memory became cheaper (in 2010), as shown in Figure 1: Figure 1: Trends of computing So, what really changed over a period of time? Over a period of time, tape is dead, disk has become tape, and SSD has almost become disk. Now, caching data in RAM is the current trend. Let's understand why memory-based computing is important and how it provides significant performance benefits. Figure 2 indicates that data transfer rates from various mediums to the CPU. Disk to CPU is 100 MB/s, SSD to CPU is 600 MB/s, and over a network to CPU is 1 MB to 1 GB/s. However, the RAM to CPU transfer speed is astonishingly fast, which is 10 GB/s. So, the idea is to cache all or partial data in memory so that higher performance can be achieved. Figure 2: Why memory? Spark history Spark started in 2009 as a research project in the UC Berkeley RAD Lab, that later became AMPLab. The researchers in the lab had previously been working on Hadoop MapReduce and observed that MapReduce was inefficient for iterative and interactive computing jobs. Thus, from the beginning, Spark was designed to be fast for interactive queries and iterative algorithms, bringing in ideas such as support for in-memory storage and efficient fault recovery. In 2011, AMPLab started to develop high-level components in Spark, such as Shark and Spark Streaming. These components are sometimes referred to as Berkeley Data Analytics Stack (BDAS). Spark was first open sourced in March 2010 and transferred to the Apache Software Foundation in June 2013, where it is now a top-level project. In February 2014, it became a top-level project at the Apache Software Foundation. Spark has since become one of the largest open source communities in big data. Now, over 250+ contributors in 50+ organizations are contributing to Spark development. User base has increased tremendously from small companies to Fortune 500 companies.Figure 3 shows the history of Apache Spark: Figure 3: The history of Apache Spark What is Apache Spark? Let's understand what Apache Spark is and what makes it a force to reckon with in big data analytics: Apache Spark is a fast enterprise-grade large-scale data processing, which is interoperable with Apache Hadoop. It is written in Scala, which is both an object-oriented and functional programming language that runs in a JVM. Spark enables applications to distribute data in-memory reliably during processing. This is the key to Spark's performance as it allows applications to avoid expensive disk access and performs computations at memory speeds. It is suitable for iterative algorithms by having every iteration access data through memory. Spark programs perform 100 times faster than MapReduce in-memory or 10 times faster on the disk (http://spark.apache.org/). It provides native support for Java, Scala, Python, and R languages with interactive shells for Scala, Python, and R. Applications can be developed easily and often 2 to 10 times less code is needed. Spark powers a stack of libraries including Spark SQL and DataFrames for interactive analytics, MLlib for machine learning, GraphX for graph processing, and Spark Streaming for real-time analytics. You can combine these features seamlessly in the same application. Spark runs on Hadoop, Mesos, standalone resource managers, on-premise hardware, or in the cloud. What Apache Spark is not Hadoop provides us with HDFS for storage and MapReduce for compute. However, Spark does not provide any specific storage medium. Spark is mainly a compute engine, but you can store data in-memory or on Tachyon to process it. Spark has the ability to create distributed datasets from any file stored in the HDFS or other storage systems supported by Hadoop APIs (including your local filesystem, Amazon S3, Cassandra, Hive, HBase, Elasticsearch, and so on). It's important to note that Spark is not Hadoop and does not require Hadoop to run. It simply has support for storage systems implementing Hadoop APIs. Spark supports text files, SequenceFiles, Avro, Parquet, and any other Hadoop InputFormat. Can Spark replace Hadoop? Spark is designed to interoperate with Hadoop. It's not a replacement for Hadoop but for the MapReduce framework on Hadoop. All Hadoop processing frameworks (Sqoop, Hive, Pig, Mahout, Cascading, Crunch, and so on) using MapReduce as the engine now use Spark as an additional processing engine. MapReduce issues MapReduce developers faced challenges with respect to performance and converting every business problem to a MapReduce problem. Let's understand the issues related to MapReduce and how they are addressed in Apache Spark: MapReduce (MR) creates separate JVMs for every Mapper and Reducer. Launching JVMs takes time. MR code requires a significant amount of boilerplate coding. The programmer needs to think and design every business problem in terms of Map and Reduce, which makes it a very difficult program. One MR job can rarely do a full computation. You need multiple MR jobs to finish the complete task and the programmer needs to design and keep track of optimizations at all levels. An MR job writes the data to the disk between each job and hence is not suitable for iterative processing. A higher level of abstraction, such as Cascading and Scalding, provides better programming of MR jobs, but it does not provide any additional performance benefits. MR does not provide great APIs either. MapReduce is slow because every job in a MapReduce job flow stores data on the disk. Multiple queries on the same dataset will read the data separately and create a high disk I/O, as shown in Figure 4: Figure 4: MapReduce versus Apache Spark Spark takes the concept of MapReduce to the next level to store the intermediate data in-memory and reuse it, as needed, multiple times. This provides high performance at memory speeds, as shown in Figure 4. If I have only one MapReduce job, does it perform the same as Spark? No, the performance of the Spark job is superior to the MapReduce job because of in-memory computations and shuffle improvements. The performance of Spark is superior to MapReduce even when the memory cache is disabled. A new shuffle implementation (sort-based shuffle instead of hash-based shuffle), a new network module (based on netty instead of using block manager to send shuffle data), and a new external shuffle service make Spark perform the fastest petabyte sort (on 190 nodes with 46TB RAM) and terabyte sort. Spark sorted 100 TB of data using 206 EC2 i2.8x large machines in 23 minutes. The previous world record was 72 minutes, set by a Hadoop MapReduce cluster of 2,100 nodes. This means that Spark sorted the same data 3x faster using 10x less machines. All the sorting took place on the disk (HDFS) without using Spark's in-memory cache (https://databricks.com/blog/2014/10/10/spark-petabyte-sort.html). To summarize, here are the differences between MapReduce and Spark: MapReduce Spark Ease of use Not easy to code and use Spark provides a good API and is easy to code and use Performance Performance is relatively poor when compared with Spark In-memory performance Iterative processing Every MR job writes the data to the disk and the next iteration reads from the disk Spark caches data in-memory Fault Tolerance Its achieved by replicating the data in HDFS Spark achieves fault tolerance by resilient distributed dataset (RDD) lineage Runtime Architecture Every Mapper and Reducer runs in a separate JVM Tasks are run in a preallocated executor JVM Shuffle Stores data on the disk Stores data in-memory and on the disk Operations Map and Reduce Map, Reduce, Join, Cogroup, and many more Execution Model Batch Batch, Interactive, and Streaming Natively supported Programming Languages Java Java, Scala, Python, and R Spark's stack Spark's stack components are Spark Core, Spark SQL and DataFrames, Spark Streaming, MLlib, and Graphx, as shown in Figure 5: Figure 5: The Apache Spark ecosystem Here is a comparison of Spark components versus Hadoop components: Spark Hadoop Spark Core MapReduce Apache Tez Spark SQL and DataFrames Apache Hive Impala Apache Tez Apache Drill Spark Streaming Apache Storm Spark MLlib Apache Mahout Spark GraphX Apache Giraph To understand the framework at a higher level, let's take a look at these core components of Spark and their integrations: Feature Details Programming languages Java, Scala, Python, and R. Scala, Python, and R shell for quick development. Core execution engine Spark Core: Spark Core is the underlying general execution engine for the Spark platform and all the other functionality is built on top of it. It provides Java, Scala, Python, and R APIs for the ease of development. Tungsten: This provides memory management and binary processing, cache-aware computation and code generation. Frameworks Spark SQL and DataFrames: Spark SQL is a Spark module for structured data processing. It provides a programming abstraction called DataFrames and can also act as a distributed SQL query engine. Spark Streaming: Spark Streaming enables us to build scalable and fault-tolerant streaming applications. It integrates with a wide variety of data sources, including filesystems, HDFS, Flume, Kafka, and Twitter. MLlib: MLlib is a machine learning library to create data products or extract deep meaning from the data. MLlib provides a high performance because of in-memory caching of data. Graphx: GraphX is a graph computation engine with graph algorithms to build graph applications. Off-heap storage Tachyon: This provides reliable data sharing at memory speed within and across cluster frameworks/jobs. Spark's default OFF_HEAP (experimental) storage is Tachyon. Cluster resource managers Standalone: By default, applications are submitted to the standalone mode cluster and each application will try to use all the available nodes and resources. YARN: YARN controls the resource allocation and provides dynamic resource allocation capabilities. Mesos: Mesos has two modes, Coarse-grained and Fine-grained. The coarse-grained approach has a static number of resources just like the standalone resource manager. The fine-grained approach has dynamic resource allocation just like YARN. Storage HDFS, S3, and other filesystems with the support of Hadoop InputFormat. Database integrations HBase, Cassandra, Mongo DB, Neo4J, and RDBMS databases. Integrations with streaming sources Flume, Kafka and Kinesis, Twitter, Zero MQ, and File Streams. Packages http://spark-packages.org/ provides a list of third-party data source APIs and packages. Distributions Distributions from Cloudera, Hortonworks, MapR, and DataStax. The Spark ecosystem is a unified stack that provides you with the power of combining SQL, streaming, and machine learning in one program. The advantages of unification are as follows: No need of copying or ETL of data between systems Combines processing types in one program Code reuse One system to learn One system to maintain An example of unification is shown in Figure 6: Figure 6: Unification of the Apache Spark ecosystem Why Hadoop + Spark? Apache Spark shines better when it is combined with Hadoop. To understand this, let's take a look at Hadoop and Spark features. Hadoop features The Hadoop features are described as follows: Feature Details Unlimited scalability Stores unlimited data by scaling out HDFS Effectively manages the cluster resources with YARN Runs multiple applications along with Spark Thousands of simultaneous users Enterprise grade Provides security with Kerberos authentication and ACLs authorization Data encryption High reliability and integrity Multitenancy Wide range of applications Files: Strucutured, semi-structured, or unstructured Streaming sources: Flume and Kafka Databases: Any RDBMS and NoSQL database Spark features The Spark features are described as follows: Feature Details Easy development No boilerplate coding Multiple native APIs: Java, Scala, Python, and R REPL for Scala, Python, and R In-memory performance RDDs Direct Acyclic Graph (DAG) to unify processing Unification Batch, SQL, machine learning, streaming, and graph processing When both frameworks are combined, we get the power of enterprise-grade applications with in-memory performance, as shown in Figure 7: Figure 7: Spark applications on the Hadoop platform Frequently asked questions about Spark The following are the frequent questions that practitioners raise about Spark: My dataset does not fit in-memory. How can I use Spark? Spark's operators spill data to the disk if it does not fit in-memory, allowing it to run well on data of any size. Likewise, cached datasets that do not fit in-memory are either spilled to the disk or recomputed on the fly when needed, as determined by the RDD's storage level. By default, Spark will recompute the partitions that don't fit in-memory. The storage level can be changed to MEMORY_AND_DISK to spill partitions to the disk. Figure 8 shows the performance difference in fully cached versus on the disk:Figure 8: Spark performance: Fully cached versus on the disk How does fault recovery work in Spark? Spark's in-built fault tolerance based on RDD lineage will automatically recover from failures. Figure 9 shows the performance over failure in the 6th iteration in a k-means algorithm: Figure 9: Fault recovery performance Summary In this article, we saw an introduction to Apache Spark and the features of Hadoop and Spark and discussed how we can combine them together. Resources for Article: Further resources on this subject: Adding a Spark to R[article] Big Data Analytics[article] Big Data Analysis (R and Hadoop)[article]

In this article by Jaime Soriano Pastor and Allesandro Frachezi, the authors of Extending Puppet - Second Edition, you will learn that the modules reusability is a topic that has got more and more attention in the past few years; as more people started to use Puppet, more evident became the need of having some common and shared code to manage common things. The main characteristics of reusable modules are: They can be used by different people without the need to modify their content They support different OSes, and allow easy extension to new ones They allow users to override the default files provided by the module They might have an opinionated approach to the managed resources but don't force it They follow a single responsibility principle and should manage only the application they are made for Reusability, we must underline, is not an all-or-nothing feature; we might have different levels of reusability to fulfill the needs of a variant percentage of users. For example, a module might support Red Hat and Debian derivatives, but not Solaris or AIX; Is it reusable? If we use the latter OSes, definitively not, if we don't use them, yes, for us it is reusable. I am personally a bit extreme about reusability, and according to me, a module should also: Allow users to provide alternative classes for eventual dependencies from other modules, to ease interoperability Allow any kind of treatment of the managed configuration files, be that file- or setting-based Allow alternative installation methods Allow users to provide their own classes for users or other resources, which could be managed in custom and alternative ways Allow users to modify the default settings (calculated inside the module according to the underlining OS) for package and service names, file paths, and other more or less internal variables that are not always exposed as parameters. Expose parameters that allow removal of the resources provided by the module (this is a functionality feature more than a reusability one) Abstract monitoring and firewalling features so that they are not directly tied to specific modules or applications Managing files Everything is a file in UNIX, and Puppet most of the times manages files. A module can expose parameters that allow its users to manipulate configuration files and it can follow one or both the file/setting approaches, as they are not alternative and can coexist. To manage the contents of a file, Puppet provides different alternative solutions: Use templates, populated with variables that come from parameters, facts, or any scope (argument for the File type: content => template('modulename/path/templatefile.erb') Use static files, served by the Puppet server Manage the file content via concat (https://github.com/puppetlabs/puppetlabs-concat) a module that provides resources that allow to build a file joining different fragments. Manage the file contents via augeas, a native type that interfaces with the Augeas configuration editing tool (http://augeas.net/) Manage the contents with alternative in-file line editing tools For the first two cases, we can expose parameters that allow to define the module's main configuration file either directly via the source and content arguments, or by specifying the name of the template to be passed to the template() function: class redis (   $config_file           = $redis::params::file,   $config_file_source    = undef,   $config_file_template  = undef,   $config_file_content   = undef,   ) { Manage the configuration file arguments with: $managed_config_file_content = $config_file_content ? {     undef   => $config_file_template ? {       undef   => undef,       default => template($config_file_template),     },     default => $config_file_content,   } The $managed_config_file_content variable computed here takes the value of the $config_file_content, if present; otherwise, it uses the template defined with $config_file_template. If also this parameter is unset, the value is undef: if $redis::config_file {     file { 'redis.conf':       path    => $redis::config_file,       source  => $redis::config_file_source,       content => $redis::managed_config_file_content,     }   } } In this way, users can populate redis.conf via a custom template (placed in the site module): class { 'redis':   config_file_template => 'site/redis/redis.conf.erb', } Otherwise, they can also provide the content attribute directly: class { 'redis':   config_file_content => template('site/redis/redis.conf.erb'), } Finally, they can also provide a fileserver source path: class { 'redis':   config_file_source => 'puppet:///modules/site/redis/redis.conf', } In case users prefer to manage the file in other ways (augeas, concat, and so on), they can just include the main class, which, by default, does not manage the configuration file's contents and uses whatever method to alter them: class { 'redis': } A good module could also provide custom defines that allow easy and direct ways to alter configuration files' single lines, either using Augeas or other in-file line management tools. Managing configuration hash patterns If we want a full infrastructure as data setup and be able to manage all our configuration settings as data, we can follow two approaches, regarding the number, name, and kind of parameters to expose: Provide a parameter for each configuration entry we want to manage Provide a single parameter that expects a hash where any configuration entry may be defined The first approach requires a substantial and ongoing effort, as we have to keep our module's classes updated with all the current and future configuration settings our application may have. Its benefit is that it allows us to manage them as plain and easily readable data on, for example, Hiera YAML files. Such an approach is followed, for example, by the OpenStack modules (https://github.com/stackforge) where the configurations of the single components of OpenStack are managed on a settings-based approach, which is fed by the parameters of various classes and subclasses. For example, the Nova module (https://github.com/stackforge/puppet-nova) has many subclasses where the parameters that map to Nova's configuration entries are exposed and are applied via the nova_config native type, which is a basically a line editing tool that works line by line. An alternative and quicker approach is to just define a single parameter, like config_file_options_hash that accepts any settings as a hash: class openssh (   $config_file_options_hash   = { }, } Then, manage in a custom template the hash, either via a custom function, like the hash_lookup() provided by the stdmod shared module (https://github.com/stdmod/stdmod): # File Managed by Puppet [...]   Port <%= scope.function_hash_lookup(['Port','22']) %>   PermitRootLogin <%= scope.function_hash_lookup(['PermitRootLogin','yes']) %>   UsePAM <%= scope.function_hash_lookup(['UsePAM','yes']) %> [...] Otherwise, refer directly to a specific key of the config_file_options_hash parameter:  Port <%= scope.lookupvar('openssh::config_file_options_hash')['Port'] ||= '22' %>   PermitRootLogin <%= scope.lookupvar('openssh::config_file_options_hash')['PermitRootLogin'] ||= 'yes' %>   UsePAM <%= scope.lookupvar('openssh::config_file_options_hash')['UsePAM'] ||= 'yes' %> [...] Needless to say that Hiera is a good place to define these parameters; on a YAML-based backend, we can set these parameters with:  --- openssh::config_file_template: 'site/openssh/sshd_config.erb' 

openssh::config_file_options_hash:   Port: '22222'   PermitRootLogin: 'no' Otherwise, if we prefer to use an explicit parameterized class declaration: class { 'openssh':   config_file_template     => 'site/openssh/sshd_config.erb'   config_file_options_hash => {     Port            => '22222',     PermitRootLogin => 'no',   } } Managing multiple configuration files An application may have different configuration files and our module should provide ways to manage them. In these cases, we may have various options to implement in a reusable module: Expose parameters that let us configure the whole configuration directory Expose parameters that let us configure specific extra files Provide a general purpose define that eases management of configuration files To manage the whole configuration directory these parameters should be enough: class redis (   $config_dir_path            = $redis::params::config_dir,   $config_dir_source          = undef,   $config_dir_purge           = false,   $config_dir_recurse         = true,   ) {   $config_dir_ensure = $ensure ? {     'absent'  => 'absent',     'present' => 'directory',   }   if $redis::config_dir_source {     file { 'redis.dir':       ensure  => $redis::config_dir_ensure,       path    => $redis::config_dir_path,       source  => $redis::config_dir_source,       recurse => $redis::config_dir_recurse,       purge   => $redis::config_dir_purge,       force   => $redis::config_dir_purge,     }   } } Such a code would allow providing a custom location, on the Puppet Master, to use as source for the whole configuration directory:  class { 'redis':   config_dir_source => 'puppet:///modules/site/redis/conf/', } Provide a custom source for the whole config_dir_path and purge any unmanaged config file; all the destination files not present on the source directory would be deleted. Use this option only when we want to have complete control on the contents of a directory:  class { 'redis':   config_dir_source => [                   "puppet:///modules/site/redis/conf--${::fqdn}/", 
                  "puppet:///modules/site/redis/conf-${::role}/",                   'puppet:///modules/site/redis/conf/' ],   config_dir_purge  => true, } Consider that the source files, in this example, placed in the site module according to a naming hierarchy that allows overrides per node or role name, can only be static and not templates. If we want to provide parameters that allow direct management of alternative extra files, we can add parameters such as the following (stdmod compliant): class postgresql (   $hba_file_path             = $postgresql::params::hba_file_path,   $hba_file_template         = undef,   $hba_file_content          = undef,   $hba_file_options_hash     = { } ,   ) { […] } Finally, we can place in our module a general purpose define that allows users to provide the content for any file in the configuration directory. Here is an example https://github.com/example42/puppet-pacemaker/blob/master/manifests/conf.pp The usage is as easy as: pacemaker::conf { 'authkey':   source => 'site/pacemaker/authkey', } Managing users and dependencies Sometimes a module has to create a user or have some prerequisite packages installed in order to have its application running correctly. These are the kind of "extra" resources that can create conflicts among modules, as we may have them already defined somewhere else in the catalog via other modules. For example, we may want to manage users in our own way and don't want them to be created by an application module, or we may already have classes that manage the module's prerequisite. There's not a universally defined way to cope with these cases in Puppet, if not the principle of single point of responsibility, which might conflict with the need to have a full working module, when it requires external prerequisites. My personal approach, which I've not seen being used around, is to let the users define the name of alternative classes, if any, where such resources can be managed. On the code side, the implementation is quite easy: class elasticsearch (   $user_class          = 'elasticsearch::user',   ) { [...]   if $elasticsearch::user_class {     require $elasticsearch::user_class   } Also, of course, in elasticsearch/manifests/user.pp, we can define the module's default elasticsearch::user class. Module users can provide custom classes with: class { 'elasticsearch':   user_class => 'site::users::elasticsearch', } Otherwise, they decide to manage users in other ways and unset any class name: class { 'elasticsearch':   user_class => '', } Something similar can be done for a dependency class or other classes. In an outburst of a reusability spree, in some cases, I added parameters to let users define alternative classes for the typical module classes: class postgresql (   $install_class             = 'postgresql::install',   $config_class              = 'postgresql::config',   $setup_class               = 'postgresql::setup',   $service_class             = 'postgresql::service',   [… ] ) { […] } Maybe this is really too much, but, for example, giving users the option to define the install class to use, and have it integrated in the module's own relationships logic, may be useful for cases where we want to manage the installation in a custom way. Managing installation options Generally, it is recommended to always install applications via packages, eventually to be created onsite when we can't find fitting public repositories. Still, sometimes, we might need to, have to, or want to install an application in other ways; for example just downloading its archive, extracting it, and eventually compiling it. It may not be a best practice, but still it can be done, and people do it. Another reusability feature a module may provide is alternative methods to manage the installation of an application. Implementation may be as easy as: class elasticsearch (   $install_class       = 'elasticsearch::install',   $install             = 'package',   $install_base_url    = $elasticsearch::params::install_base_url,   $install_destination = '/opt',   ) { These options expose both the install method to be used, the name of the installation class (so that it can be overridden), the URL from where to retrieve the archive, and the destination at which to install it. In init.pp, we can include the install class using the parameter that sets its name: include $install_class In the default install class file (here install.pp) manage the install parameter with a case switch: class elasticsearch::install {   case $elasticsearch::install {     package: {       package { $elasticsearch::package:         ensure   => $elasticsearch::managed_package_ensure,         provider => $elasticsearch::package_provider,       }     }     upstream: {       puppi::netinstall { 'netinstall_elasticsearch':         url             => $elasticsearch::base_url,         destination_dir => $elasticsearch::install_destination,         owner           => $elasticsearch::user,         group           => $elasticsearch::user,       }     }     default: { fail('No valid install method defined') }   } } The puppi::netinstall defined in the preceding code comes from a module of mine (https://github.com/example42/puppi) and it's used to download, extract, and eventually execute custom commands on any kind of archive. Users can therefore define which installation method to use with the install parameter and they can even provide another class to manage in a custom way the installation of the application. Managing extra resources Many times, we have in our environment some customizations that cannot be managed just by setting different parameters or names. Sometimes, we have to create extra resources, which no public module may provide as they are too custom and specific. While we can place these extra resources in any class, we may include in our nodes; it may be useful to link this extra class directly to our module, providing a parameter that lets us specify the name of an extra custom class, which, if present, is included (and contained) by the module: class elasticsearch (   $my_class            = undef,   ) { [...]   if $elasticsearch::my_class {     include $elasticsearch::my_class     Class[$elasticsearch::my_class] ->      Anchor['elasticsearch::end']   } } Another method to let users create extra resources by passing a parameter to a class is based on the create_resources function. We have already seen it; it creates all the resources of a given type from a nested hash where their names and arguments can be defined. Here is an example from https://github.com/example42/puppet-network: class network (   $interfaces_hash           = undef,   […] ) { […]   if $interfaces_hash {     create_resources('network::interface', $interfaces_hash)   } } In this case, the type used is network::interface (provided by the same module) and it can be fed with a hash. On Hiera, with the YAML backend, it could look like this: ---   network::interfaces_hash:     eth0:       method: manual       bond_master: 'bond3'       allow_hotplug: 'bond3 eth0 eth1 eth2 eth3'     eth1:       method: manual       bond_master: 'bond3'     bond3:       ipaddress: '10.10.10.3'       netmask: '255.255.255.0'       gateway: '10.10.10.1'       dns_nameservers: '8.8.8.8 8.8.4.4'       bond_mode: 'balance-alb'       bond_miimon: '100'       bond_slaves: 'none' Summary As we can imagine, the usage patterns that such a function allows are quite wide and interesting. Being able to base, on pure data, all the information we need to create a resource may definitively shift most of the logic and the implementation that is done with Puppet code and normal resources to the data backend.

In this article by Nicolae Tarla, the author of Microsoft Dynamics CRM 2016 Customization, you will learn about the Customer Relationship Management (CRM) market and the huge uptake it has seen in the last few years. Some of the drivers for this market are the need to enhance customer experience, provide faster and better services, and adapting to the customer’s growing digital presence. CRM systems, in general, are taking a central place in the new organizational initiatives. (For more resources related to this topic, see here.) Dynamics CRM is Microsoft’s response to a growing trend. The newest version is Dynamics CRM 2016. It is being offered in a variety of deployment scenarios. From the standard on-premise deployment to a private cloud or an online cloud offering from Microsoft, the choice depends on each customer, their type of project, and a large number of requirements, policies, and legal restrictions. We’ll first look at what environment we need to complete the examples presented. We will create a new environment based on a Microsoft Dynamics CRM Online trial. This approach will give us 30-day trial to experiment with an environment for free. The following topics will be covered: Introducing Dynamics CRM Dynamics CRM features Deployment models Global datacenter locations Customization requirements Getting setup Dynamics CRM 2016 is the current version of the popular Customer Relationship Management platform offered by Microsoft. This platform offers users the ability to integrate and connect data across their sales, marketing, and customer service activities, and to give staff an overall 360-degree view of all interactions and activities as they relate to a specific customer. Along with the standard platform functionality provided, we have a wide range of customization options, allowing us to extend and further customize solutions to solve a majority of other business requirements. In addition, we can integrate this platform with other applications, and create a seamless solution. While by no means the only available CRM platform on the market today, Microsoft Dynamics CRM 2016, is one of the fastest growing, gaining large acceptance at all levels from small to mid-size and enterprise level organizations. This is due to a multitude of reasons, some of which include the variety of deployment options, the scalability, the extensibility, the ease of integration with other systems, and the ease of use. Microsoft Dynamics CRM can be deployed in a variety of options. Starting with the offering from Microsoft, you can get CRM Online. Once we have a 30-day trial active, this can be easily turned into a full production environment by providing payment information and keeping the environment active. The data will live in the cloud, on one of the data centers provided by Microsoft. Alternatively, you can obtain hosting with a third-party provider. The whole environment can be hosted by a third party, and the service can be offered either as a SaaS solution, or a fully hosted environment. Usually, there is a difference in the way payment is processed, with a SaaS solution in most cases being offered in a monthly subscription model. Another option is to have the environment hosted in-house. This option is called on premise deployment and carries the highest up-front cost but gives you the ability to customize the system extensively. In addition to the higher up-front cost, the cost to maintain the environment, the hardware, and skilled people required to constantly administer the environment can easily add-up. As of recently, we now have the ability to host a virtual CRM environment in Azure. This offloads the cost of maintaining the local infrastructure in a fashion similar to a third-party-hosted solution but takes advantage of the scalability and performance of a large cloud solution maintained and supported fully by Microsoft. The following white paper released by Microsoft describes the deployment model using Azure Virtual Machines: http://www.microsoft.com/en-us/download/details.aspx?id=49193 Features of Dynamics CRM Some of the most notable features of the Dynamics CRM platform include: Scalability Extensibility Ability to integrate with other systems Ease of use Let’s look at each of the features in more detail. Scalability Dynamics CRM can scale over a wide range of deployment options. From a single box deployment, used mostly for development, all the way to a cloud offering that can span over a large number of servers, and host a large number of environments, the same base solution can handle all the scenarios in between with ease. Extensibility Dynamics CRM is a platform in which the base offering comes with prepackaged functionality for Sales, Service, and Marketing; a large variety of solutions can be built on top of Dynamics CRM. The extensibility model is called xRM and allows power users, non-developers, and developers alike to build custom solutions to handle various other business scenarios or integrate with other third-party platforms. The Dynamics CRM Marketplace is a great example of such solutions that are built to extend the core platform, and are offered for sale by various companies. These companies are called Independent Software Vendors (ISVs) and play a very important role in the ecosystem created by Microsoft. In time and with enough experience, some of them become the go-to partners for various implementations. If nothing else, the Dynamics Marketplace is a cool place to look at some of the solutions created, and search for specific applications. The idea of the marketplace became public sometime around 2010 and was integrated into Dynamics CRM 2011. At launch, it was designed as a searchable repository of solutions. It is a win-win for both solution providers and customers alike. Solutions can also be rated, thus giving customers better community feedback before committing to purchasing and implementing a foreign solution into their organization. The Dynamics Marketplace is hosted on Pinpoint, Microsoft’s online directory of software applications and professional services. On this platform, independent companies and certified partners offer their products and services. At the time of this writing, Pinpoint hosts a few marketplaces, including Office, Azure, Dynamics, and Cloud, and is available at the following location: https://pinpoint.microsoft.com/en-CA/Home/Dynamics Navigating to the Dynamics page you are presented with a search option as seen in the following screenshot: You now have the option to filter your results by Solution providers, Services, or Apps (Applications). In addition, you can further filter your results by distance to a geo-location derived from an address or postal code, as well as other categories as illustrated in the following screenshot: When searching for a solution provider, the results provide a high-level view of the organization, with a logo and a high-level description. The Ratings and Competencies count are displayed for easy visibility as shown here: Drilling down into the partner profile page, you can find additional details on the organization, the industries focus, details on the competencies, as well as a way to connect with the organization. Navigation to additional details, including Reviews and Locations is available on the profile page. The Dynamics Marketplace is also available, starting with Dynamics CRM 2011, as a part of the organization. A user with necessary permission can navigate to Settings | Dynamics Marketplace. This presents the user with a view by solutions available. Options for sorting and filtering include Popular, Newest, and Featured. Community rating is clearly visible and provides the necessary feedback to consider when evaluating new solutions. Ability to integrate with other systems There is a large variety of integration options available when working with Dynamics CRM. In addition, various deployment options offer more or fewer integration features. With CRM Online, you tend to get more integration options into cloud services; whereas, the on-premise solution has a limited number of configurable integration options, but can provide more integration using various third-party tools. The base solution comes with the ability to configure integration with the following common services: SharePoint for document management Yammer for social features In addition, you can use specific connectors provided by either Microsoft or other third-party providers for integration with specific solutions. When the preceding options are not available, you can still integrate with other solutions using a third-party integration tool. This allows real-time integration into legacy systems. Some of the most popular tools used for integration include, but are not limited to: Kingsway Software (https://www.kingswaysoft.com/) Scribe (http://www.scribesoft.com/) BizTalk (http://www.microsoft.com/en-us/server-cloud/products/biztalk/) Ease of use Dynamics CRM offers users a variety of options to interact with the system. You can access Dynamics CRM either through a browser, with support for all recent versions of the major browsers. The following browsers and versions are supported: Internet Explorer—versions 10 and above Edge—latest version Chrome—latest version on Windows 7 and above Firefox—latest version on Windows 7 and above Safari on Mac—using the latest publicly released version on OS x 10.8 and above In addition, a user can interact with the system directly from the very familiar interface of Outlook. The Dynamics CRM connector for Outlook allows users to get access to all the system data and features from within Outlook. In addition, a set of functions built specifically for Outlook allows users to track and interact with e-mails, tasks, and events from within Outlook. Further to the features provided through the Outlook integration, users of CRM for Outlook have the ability to work offline. Data can be taken offline, work can be done when disconnected, and can be synchronized back into the system when connectivity resumes. For mobile users, Dynamics CRM can be accessed from mobile devices and tablets. Dynamics CRM provides a standard web-based interface for most mobile devices, as well as specific applications for various platforms including Windows-based tablets, iPads, and Android tablets. With these apps, you can also take a limited sub-set of cached data offline, as well as have the ability to create new records and synchronize them back to CRM next time you go online. The quality of these mobile offerings has increased exponentially over the last few versions, and new features are being added with each new release. In addition, third-party providers have also built mobile solutions for Dynamics CRM. A quick search in the application markets for each platform will reveal several options for each platform. Global Data Centre Locations for Dynamics CRM Online Dynamics CRM Online is hosted at various locations in the world. Preview organizations can be created in all available locations, but features are sometimes rolled out on a schedule, in some locations faster than others. The format of the Dynamics CRM Online Organization URL describes the data center location. As such, the standard format is as follows: https://OrganizationName.crm[x].dynamics.com The OrganizationName is the name you have selected for your online organization. This is customizable, and is validated for uniqueness within the respective data center. The [x] represents a number. As of this writing, this number can be anywhere between 2, 4, 5, 6, 7, 9, or no number at all. This describes the global data center used to host your organization. The following table maps the data center to the URL format: URL Format: crm[x].dynamics.com Global Data Centre Location crm.dynamics.com NAM crm2.dynamics.com SAM crm4.dynamics.com EMEA crm5.dynamics.com APAC crm6.dynamics.com OCE crm7.dynamics.com JPN crm9.dynamics.com GCC Out of these global locations, usually the following get a preview of the new features first: Organization Global Location crm.dynamics.com North America crm4.dynamics.com Europe, the Middle East and Africa crm5.dynamics.com Asia-Pacific New data centers are being added on a regular basis. As of this writing, new data centers are being added in Europe and Canada, with others to follow as needed. Some of the drivers behind adding these new data centers revolve around not only performance improvements, as a data center located closer to a customer will provide theoretically better performance, but also a need for privacy and localization of data. Strict legislation around data residency has a great impact on the selection of the deployment model by customers who are bound to store all data local to the country of the operation. Overall, by the end of 2016, the plan is to have Dynamics CRM Online available in 105 markets. These markets (countries) will be served by data centers spread across five generic global regions. These data centers share services between Dynamics CRM Online and other services such as Azure and Office 365. Advantages of choosing Dynamics CRM online Choosing one of the available hosting models for Dynamics CRM is now not only a matter of preference. The decision can be driven by multiple factors. During the last few years, there has been a huge push for the cloud. Microsoft has been very focused on enhancing their online offering, and has continued to push more functionality and more resources in supporting the cloud model. As such, Dynamics CRM Online has become a force to reckon with. It is hosted on a very modern and high performing infrastructure. Microsoft has pushed literally billions of dollars in new data centers and infrastructure. This allows new customers to forego the necessary expenses on infrastructure associated with an on-premise deployment. Along with investments on infrastructure, the SLA (service level agreement) offered by Dynamics CRM Online is financially backed by Microsoft. Depending on the service selected, the uptime is guaranteed and backed financially. Application and Infrastructure are automatically handled for you by Microsoft so you don’t have to. This translates in much lower upfront costs, as well as reduced costs around ongoing maintenance and upgrades. The Dynamics CRM Online offering is also compliant with various regulatory requirements, and backed and verified through various third-party tests. Various rules, regulations, and policies in various locales are validated and certified by various organizations. Some of the various compliance policies evaluated include but are not limited to: Data Privacy and Confidentiality Policies Data Classification Information Security Privacy Data Stewardship Secure Infrastructure Identity and Access Control All these compliance requirements are in conformance with regulations stipulated by the International Standard Organization and other international and local standards. Independent auditors validate standards compliance. Microsoft is ISO 27001 certified. The Microsoft Trust Center website located at http://www.microsoft.com/en-us/trustcenter/CloudServices/Dynamics provides additional information on compliance, responsibilities, and warranties. Further to the aforementioned benefits, choosing cloud over a standard on-premise deployment offers other advantages around scalability, faster time to market, and higher value proposition. In addition to the standard benefits of an online deployment, one other great advantage is the ability to spin-up a 30-day trial instance of Dynamics CRM Online and convert it to a paid instance only when ready to go to production. This allows customizers and companies to get started and customize their solution in a free environment, with no additional costs attached. The 30-day trial instance gives us a 25-license instance, which allows us to not only customize the organization, but also test various roles and restrictions. Summary We learned to create a new environment based on a Microsoft Dynamics CRM Online trial Resources for Article: Further resources on this subject: Customization in Microsoft Dynamics CRM[article] Introduction to Reporting in Microsoft Dynamics CRM[article] Using Processes in Microsoft Dynamics CRM 2011[article]

In this article by Manoj Kumar, author of the book Learning Sinatra, we will write an application. Make sure that you have Ruby installed. We will get a basic skeleton app up and running and see how to structure the application. (For more resources related to this topic, see here.) In this article, we will discuss the following topics: A project that will be used to understand Sinatra Bundler gem File structure of the application Responsibilities of each file Before we begin writing our application, let's write the Hello World application. Getting started The Hello World program is as follows: 1require 'sinatra'23 get '/' do4 return 'Hello World!'5 end The following is how the code works: ruby helloworld.rb Executing this from the command line will run the application and the server will listen to the 4567 port. If we point our browser to http://localhost:4567/, the output will be as shown in the following screenshot: The application To understand how to write a Sinatra application, we will take a small project and discuss every part of the program in detail. The idea We will make a ToDo app and use Sinatra along with a lot of other libraries. The features of the app will be as follows: Each user can have multiple to-do lists Each to-do list will have multiple items To-do lists can be private, public, or shared with a group Items in each to-do list can be assigned to a user or group The modules that we build are as follows: Users: This will manage the users and groups List: This will manage the to-do lists Items: This will manage the items for all the to-do lists Before we start writing the code, let's see what the file structure will be like, understand why each one of them is required, and learn about some new files. The file structure It is always better to keep certain files in certain folders for better readability. We could dump all the files in the home folder; however, that would make it difficult for us to manage the code: The app.rb file This file is the base file that loads all the other files (such as, models, libs, and so on) and starts the application. We can configure various settings of Sinatra here according to the various deployment environments. The config.ru file The config.ru file is generally used when we need to deploy our application with different application servers, such as Passenger, Unicorn, or Heroku. It is also easy to maintain the different deployment environment using config.ru. Gemfile This is one of the interesting stuff that we can do with Ruby applications. As we know, we can use a variety of gems for different purposes. The gems are just pieces of code and are constantly updated. Therefore, sometimes, we need to use specific versions of gems to maintain the stability of our application. We list all the gems that we are going to use for our application with their version. Before we discuss how to use this Gemfile, we will talk about gem bundler. Bundler The gem bundler manages the installation of all the gems and their dependencies. Of course, we would need to install the gem bundler manually: gem install bundler This will install the latest stable version of bundler gem. Once we are done with this, we need to create a new file with the name Gemfile (yes, with a capital G) and add the gems that we will use. It is not necessary to add all the gems to Gemfile before starting to write the application. We can add and remove gems as we require; however, after every change, we need to run the following: bundle install This will make sure that all the required gems and their dependencies are installed. It will also create a 'Gemfile.lock' file. Make sure that we do not edit this file. It contains all the gems and their dependencies information. Therefore, we now know why we should use Gemfile. This is the lib/routes.rb path for folder containing the routes file. What is a route? A route is the URL path for which the application serves a web page when requested. For example, when we type http://www.example.com/, the URL path is / and when we type http://www.example.com/something/, /something/ is the URL path. Now, we need to explicitly define all the routes for which we will be serving requests so that our application knows what to return. It is not important to have this file in the lib folder or to even have it at all. We can also write the routes in the app.rb file. Consider the following examples: get '/' do # code end post '/something' do # code end Both of the preceding routes are valid. The get and post method are the HTTP methods. The first code block will be executed when a GET request is made on / and the second one will be executed when a POST request is made on /something. The only reason we are writing the routes in a separate file is to maintain clean code. The responsibility of each file will be clearly understood in the following: models/: This folder contains all the files that define model of the application. When we write the models for our application, we will save them in this folder. public/: This folder contains all our CSS, JavaScript, and image files. views/: This folder will contain all the files that define the views, such as HTML, HAML, and ERB files. The code Now, we know what we want to build. You also have a rough idea about what our file structure would be. When we run the application, the rackup file that we load will be config.ru. This file tells the server what environment to use and which file is the main application to load. Before running the server, we need to write a minimum code. It includes writing three files, as follows: app.rb config.ru Gemfile We can, of course, write these files in any order we want; however, we need to make sure that all three files have sufficient code for the application to work. Let's start with the app.rb file. The app.rb file This is the file that config.ru loads when the application is executed. This file, in turn, loads all the other files that help it to understand the available routes and the underlying model: 1 require 'sinatra' 2 3 class Todo < Sinatra::Base 4 set :environment, ENV['RACK_ENV'] 5 6 configure do 7 end 8 9 Dir[File.join(File.dirname(__FILE__),'models','*.rb')].each { |model| require model } 10 Dir[File.join(File.dirname(__FILE__),'lib','*.rb')].each { |lib| load lib } 11 12 end What does this code do? Let's see what this code does in the following: 1 require 'sinatra' //This loads the sinatra gem into memory. 3 class Todo < Sinatra::Base 4 set :environment, ENV['RACK_ENV'] 5 6 configure do 7 end 8 9 Dir[File.join(File.dirname(__FILE__),'models','*.rb')].each { |model| require model } 10 Dir[File.join(File.dirname(__FILE__),'lib','*.rb')].each { |lib| load lib } 11 12 end This defines our main application's class. This skeleton is enough to start the basic application. We inherit the Base class of the Sinatra module. Before starting the application, we may want to change some basic configuration settings such as logging, error display, user sessions, and so on. We handle all these configurations through the configure blocks. Also, we might need different configurations for different environments. For example, in development mode, we might want to see all the errors; however, in production we don’t want the end user to see the error dump. Therefore, we can define the configurations for different environments. The first step would be to set the application environment to the concerned one, as follows: 4 set :environment, ENV['RACK_ENV'] We will later see that we can have multiple configure blocks for multiple environments. This line reads the system environment RACK_ENV variable and sets the same environment for the application. When we discuss config.ru, we will see how to set RACK_ENV in the first place: 6 configure do 7 end The following is how we define a configure block. Note that here we have not informed the application that to which environment do these configurations need to be applied. In such cases, this becomes the generic configuration for all the environments and this is generally the last configuration block. All the environment-specific configurations should be written before this block in order to avoid code overriding: 9 Dir[File.join(File.dirname(__FILE__),'models','*.rb')].each { |model| require model } If we see the file structure discussed earlier, we can see that models/ is a directory that contains the model files. We need to import all these files in the application. We have kept all our model files in the models/ folder: Dir[File.join(File.dirname(__FILE__),'models','*.rb')] This would return an array of files having the .rb extension in the models folder. Doing this, avoids writing one require line for each file and modifying this file again: 10 Dir[File.join(File.dirname(__FILE__),'lib','*.rb')].each { |lib| load lib } Similarly, we will import all the files in the lib/ folder. Therefore, in short, the app.rb configures our application according to the deployment environment and imports the model files and the other library files before starting the application. Now, let's proceed to write our next file. The config.ru file The config.ru is the rackup file of the application. This loads all the gems and app.rb. We generally pass this file as a parameter to the server, as follows: 1 require 'sinatra' 2 require 'bundler/setup' 3 Bundler.require 4 5 ENV["RACK_ENV"] = "development" 6 7 require File.join(File.dirname(__FILE__), 'app.rb') 8 9 Todo .start! W Working of the code Let's go through each of the lines, as follows: 1 require 'sinatra' 2 require 'bundler/setup' The first two lines import the gems. This is exactly what we do in other languages. The gem 'sinatra' command will include all the Sinatra classes and help in listening to requests, while the bundler gem will manage all the other gems. As we have discussed earlier, we will always use bundler to manage our gems. 3 Bundler.require This line of the code will check Gemfile and make sure that all the gems available match the version and all the dependencies are met. This does not import all the gems as all gems may not be needed in the memory at all times: 5 ENV["RACK_ENV"] = "development" This code will set the system environment RACK_ENV variable to development. This will help the server know which configurations does it need to use. We will later see how to manage a single configuration file with different settings for different environments and use one particular set of configurations for the given environment. If we use version control for our application, config.ru is not version controlled. It has to be customized on whether our environment is development, staging, testing, or production. We may version control a sample config.ru. We will discuss this when we talk about deploying our application. Next, we will require the main application file, as follows: 7 require File.join(File.dirname(__FILE__), 'app.rb') We see here that we have used the File class to include app.rb: File.dirname(__FILE__) It is a convention to keep config.ru and app.rb in the same folder. It is good practice to give the complete file path whenever we require a file in order to avoid breaking the code. Therefore, this part of the code will return the path of the folder containing config.ru. Now, we know that our main application file is in the same folder as config.ru, therefore, we do the following: File.join(File.dirname(__FILE__), 'app.rb') This would return the complete file path of app.rb and the line 7 will load the main application file in the memory. Now, all we need to do is execute app.rb to start the application, as follows: 9 Todo .start! We see that the start! method is not defined by us in the Todo class in app.rb. This is inherited from the Sinatra::Base class. It starts the application and listens to incoming requests. In short, config.ru checks the availability of all the gems and their dependencies, sets the environment variables, and starts the application. The easiest file to write is Gemfile. It has no complex code and logic. It just contains a list of gems and their version details. Gemfile In Gemfile, we need to specify the source from where the gems will be downloaded and the list of the gems. Therefore, let's write a Gemfile with the following lines: 1 source 'https://rubygems.org' 2 gem 'bundler', '1.6.0' 3 gem 'sinatra', '1.4.4' The first line specifies the source. The https://rubygems.org website is a trusted place to download gems. It has a large collection of gems hosted. We can view this page, search for gems that we want to use, read the documentation, and select the exact version for our application. Generally, the latest stable version of bundler is used. Therefore, we search the site for bundler and find out its version. We do the same for the Sinatra gem. Summary In this article, you learned how to build a Hello World program using Sinatra. Resources for Article: Further resources on this subject: Getting Ready for RubyMotion[article] Quick start - your first Sinatra application[article] Building tiny Web-applications in Ruby using Sinatra[article]

In this article by Steven Daniel, the author of Apple Watch App Development, we will introduce ourselves to Apple's Swift programming language. At WWDC 2014, Apple introduced a brand new programming language called Swift. The Swift programming language brings concise syntax, type safety, and modern programming language features to Mac and iOS developers. (For more resources related to this topic, see here.) Since its release, the Apple developer community has responded with great excitement, and developers are rapidly starting to adopt this new language within their own applications. The Swift language is the future of developing on Apple's platforms. This article includes the following topics: Learning how to register as an Apple developer Learning how to download and install Xcode development tools Introduction to the Swift programming language Learning how to work with Xcode playgrounds Introduction to the newest additions in Swift 2.0 Registering as an Apple developer Before you can begin building iOS applications for your iOS devices, you must first join as a registered user of Apple Developer Program in order to download all of the necessary components to your computer. The registration process is free and provides you with access to the iOS SDK and other developer resources that are really useful to get you started. The following short list outlines some of the things that you will be able to access once you become a registered member of Apple Developer Program: It provides helpful “Getting Started” guides to help you get up and running quickly It gives you helpful tips that show you how to submit your apps to App Store It provides the ability to download the current releases of iOS software It provides the ability to beta test the releases of iOS and the iOS SDK It provides access to Apple Developer Forums Whether you develop applications for the iPhone or iPad, these use the same OS and iOS SDK that allows you to create universal apps that will work with each of these devices. On the other hand, Apple Watch uses an entirely different OS called watchOS. To prepare your computer for iOS development, you need to register as an Apple developer. This free process gives you access to the basic levels of development that allow you to test your app using iOS Simulator without the ability to sell your app on Apple App Store. The steps are as follows: To sign up to Apple Developer Program, you will need to go to https://developer.apple.com/programs/ and then click on the Enroll button to proceed, as shown in the following screenshot: Next, click on the Start Your Enrollment button, as shown in the following screenshot: Once you sign up, you will then be able to download the iOS SDK and proceed with installing it onto your computer. You will then become an official member of Apple Developer Program. You will then be able to download beta software so that you can test them on your actual device hardware as well as having the freedom to distribute your apps to your end users. In the next section, we will look at how to download and install Xcode development tools. Getting and installing Xcode development tools In this section, we will take a look at what Integrated Development Environments (IDEs) and Software Development Kits (SDKs) are needed to develop applications for the iOS platform, which is Apple's operating system for mobile devices. We will explain the importance of each tool's role in the development cycle and the tools required to develop applications for the iOS platform, which are as follows: An Intel-based Mac computer running OS X Yosemite (10.10.2) or later with the latest point release and security patches installed is required. This is so that you can install the latest version of the Xcode development tool. Xcode 6.4 or later is required. Xcode is the main development tool for iOS. You need Xcode 6.4 minimum as this version includes Swift 1.2, and you must be registered as an Apple developer. The iOS SDK consists of the following components: Component Description Xcode This is the main IDE that enables you to develop, edit, and debug your native applications for the iOS and Mac platforms using the Objective-C or Swift programming languages. iOS Simulator This is a Cocoa-based application that enables you to debug your iOS applications on your computer without the need of having an iOS device. There are many iOS features that simply won't work within Simulator, so a device is required if an application uses features such as the Core Location and MapKit frameworks. Instruments These are the analysis tools that help you optimize your applications and monitor memory leaks during the execution of your application in real time. Dashcode This enables you to develop web-based iOS applications and dashboard widgets. Once you are registered, you will need to download and install Xcode developer tools by performing the following steps: Begin by downloading and installing Xcode from Mac App Store at https://itunes.apple.com/au/app/xcode/id497799835?mt=12. Select either the Free or Install button on the App Store page. Once it completes the installation process, you will be able to launch Xcode.app from your Applications folder. You can find additional development tools from the Apple developer website at https://developer.apple.com/. In the next section, we will be look at what exactly Xcode playgrounds are and how you can use them to experiment with designing code algorithms prior to incorporating the code into your project. So, let's get started. Introduction to Xcode playgrounds A playground is basically an interactive Swift coding environment that displays the results of each statement as updates are made without having the need to compile and run a project. You can use playgrounds to learn and explore Swift, prototype parts of your app, and create learning environments for others. The interactive Swift code environment lets you experiment with algorithms, explore system APIs, and even create your very own custom views without the need of having to create a project. Once you perfect your code in the playground, simply move this code into your project. Given that playgrounds are highly interactive, they are a wonderful vehicle for distributing code samples with instructive documentation and can even be used as an alternative medium for presentations. With the new Xcode 7 IDE, you can incorporate rich text comments with bold, italic, and bulleted lists with the addition of having the ability to embed images and links. You can even embed resources and support Swift source code in the playground to make the experience incredibly powerful and engaging, while the visible code remains simple. Playgrounds provide you with the ability to do the following: Share curriculum to teach programming with beautiful text and interactive code Design a new algorithm and watch its results every step of the way Create new tests and verify that they work before promoting them into your test suite Experiment with new APIs to hone your Swift coding skills Turn your experiments into documentation with example code that runs directly within the playground Let's begin by opening the Xcode IDE and explore how to create a new playground file for the first time. Perform the following steps: Open the Xcode.app application either using the finder in your Applications directory or using Apple's Launchpad. If you've never created or opened an Xcode project before, you will be presented with the following screen: In the Welcome to Xcode dialog, select the Get started with a playground option. If this dialog doesn't appear, you can navigate to File | New | Playground… or simply press Shift + Option + Command + N. Next, enter SwiftLanguageBasics as the name of your playground. Then, ensure that you choose iOS as the platform that we will target. Click on the Next button to proceed to the next step in the wizard. Specify the location where you would like to save your project. Then, click on the Create button to save your playground at the specified location. Once your project is created, you will be presented with the default playground template, as shown in the following screenshot: In the next section, you will begin learning about some of the Swift language basics, start adding lines of code within this playground file, and see the results that we get when they are executed. Introduction to the Swift language In this section, we will introduce some of the new and exciting features of the Swift programming language. So, let's get started. Variables, constants, strings, and semicolons Our next step is to familiarize ourselves with the differences between variables, constants, strings, and semicolons in a bit more detail. We will work with and use Xcode playgrounds to put each of these into practice. Variables A variable is a value that can change. Every variable contains a name, called the variable name, and must contain a data type. The data type indicates what sort of value the variable represents, such as whether it is an integer, a floating point number, or a string. Let's take a look at how we can put this into practice and create a variable in Swift. First, let's start by revealing the console output window by navigating to View | Debug Area | Show Debug Area. Next, clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics – Variables : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ var myGreeting = "Welcome to Learning the basics of Swift Programming" print(myGreeting, terminator: "") As you begin to type in the code, you should immediately see the Welcome to Learning the basics of Swift text magically appear in the right-hand pane in which the assignment takes place and it appears once more for the print statement. The right-hand pane is great to show you smaller output, but for longer debugging output, you would normally take a look at the Xcode console. Constants A constant is basically a value that cannot be changed. Creating these constant variables prevents you from performing accidental assignments and can even improve performance. Let's take a look at how we can put this into practice and create a constant variable in Swift. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics – Constants : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ let myGreeting = "Welcome to Learning the basics" print(myGreeting, terminator: "") myGreeting += " of Swift Programming" Take a look at the screenshot now: As you begin to type in the code, you will immediately receive an error message stating that you cannot assign myGreeting to our let value because the object is not mutable. In Swift, you can control the mutability of the built-in Swift types by using either the let or var keywords during declaration. Strings A string is basically an ordered collection of characters—for example "hello, world". In Swift, strings are represented by the String data type, which represents a collection of values of the char data type. You can use strings to insert constants, variables, literals, and expressions into longer strings in a process known as string interpolation, which we will cover later on in this article. This makes it easy to create custom string values for display, storage, and printing. Let's take a look at how we can put this into practice, create a String variable in Swift, and utilize some of the string methods. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics – Strings : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation let myGreeting = "Welcome to Swift Language Basics, working with Strings" // Make our String uppercase print(myGreeting.uppercaseString) // Append exclamation mark at the end of the string var newGreeting = myGreeting.stringByAppendingString("!!!") print(newGreeting) Take a look at the screenshot now: As you can note in the preceding code snippet, we began by importing the Foundation framework class, which contains several APIs to deal with objects such as strings and dates. Next, we declared our myGreeting constant variable and then assigned a default string. We then used the uppercaseString method of the string object to perform a function to make all of the characters within our string uppercase. In our next step, we will declare a new variable called newGreeting and call the stringByAppendingString method to append additional characters at the end of our string. For more information on using the String class, you can consult the Swift programming language documentation at https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/StringsAndCharacters.html. Semicolons As you would have probably noticed so far, the code you wrote doesn't contain any semicolons. This is because in Swift, these are only required if you want to write multiple statements on a single line. Let's take a look at a code example to see how we can put this into practice. Delete the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics – Semicolons : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation var myGreeting = "Welcome to Swift Language" let newString = myGreeting + " Basics, ".uppercaseString + "working with semicolons"; print(newString) Take a look the following screenshot now: As you can note in the preceding code snippet, we began by declaring our myGreeting variable and then assigned a default string. In our next step, we declared a new variable called newString, concatenated the details from our myGreeting string, and used the uppercaseString method, which cycles through each character within our string, making our characters uppercase. Next, we appended the additional working with semicolons string to the end of our string and finally used the print statement to output the contents of our newString variable to the console window. As you must have noticed, we included a semicolon to the end of the statement; this is because in Swift, you are required to include semicolons if you want to write multiple statements on a single line. Numeric types and conversion In this section, we will take a look at how we can perform arithmetic operations on our Swift variables. In this example, we will look at how to calculate the area of a triangle, given a base and height value. Let's take a look at a code example to see how we can put this into practice. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics - Numeric Types and Conversion : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation // method to calculate the area of a triangle func calcTriangleArea(triBase: Double, triHeight: Double) -> Double { return (triBase * triHeight) / 2 } // Declare our base and height of our triangle let base = 20.0 let height = 120.0 // Calculate and display the area of the triangle and print ("The calculated Area is: " + String(calcTriangleArea(base, triHeight: height))); Take a look at the following screenshot now: As you can note in the preceding code snippet, we started by creating our calcTriangleArea function method, which accepts a base and a height parameter value in order to calculate the area of the triangle. In our next step, we declared two variables, base and height, which contain the assigned values that will be used to calculate the base and the height of our triangle. Next, we made a call to our calcTriangleArea method, passing in the values for our base and height before finally using the print statement to output the calculated area of our triangle to the console window. An important feature of the Swift programming language is that all numeric data type conversions must be explicit, regardless of whether you want to convert to a data type containing more of less precision. Booleans, tuples, and string interpolation In this section, we will look at the various features that come with the Swift programming language. We will look at the improvements that Swift has over Objective-C when it comes to using Booleans and string interpolation before finally discussing how we can use tuples to access elements from a string. Booleans Boolean variables in Swift are basically defined using the Bool data type. This data type can only hold values containing either true or false. Let's take a look at a code example to see how we can put this into practice. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics – Booleans : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation let displaySettings : Bool = true print("Display Settings is: " + (displaySettings ? "ON" : "OFF")) Take a look at the following screenshot now: As you can note from the preceding code snippet, we started by declaring our constant variable called displaySettings and assigned it a default Boolean value of true. Next, we performed a check to see whether the value of our displaySettings variable is set to true and called our print statement to output the Display Settings is: ON value to the console window. In Objective-C, you would assign a value of 1 and 0 to denote true and false; this is no longer the case with Swift because Swift doesn't treat 1 as true and 0 as false. You need to explicitly use the actual Boolean values to stay within Swift's data type system. Let's replace the existing playground code with the following code snippet to take a look at what would happen if we changed our value from true to 1: /*: # Swift Language Basics – Booleans : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation let displaySettings : Bool = 1 // This will cause an error!!! print("Display Settings is: " + (displaySettings ? "ON" : "OFF")) Take a look at the following screenshot now: As you can note from the previous screenshot, Swift detected that we were assigning an integer value to our Boolean data type and threw an error message. Tuples Tuples provide you with the ability to group multiple values into a single compound value. The values contained within a tuple can be any data type, and therefore are not required to be of the same type. Let's take a look at a code example, to see how we can put this into practice. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics – Tuples : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation // Define our Address Details var addressDetails = ("Apple Inc.", "1 Infinite Loop", "Cupertino, California", "United States"); print(addressDetails.0) // Get the Name print(addressDetails.1) // Address print(addressDetails.2) // State print(addressDetails.3) // Country Take a look at the following screenshot now: As you can note from the preceding code snippet, we started by declaring a tuple variable called addressDetails that contains a combination of strings. Next, we accessed each of the tuple elements by referencing their index values and displayed each of these elements in the console window. Let's say that you want to modify the contents of the first element within your tuple. Add the following code snippet after your var addressDetails variable: // Modify the element within our String addressDetails.0 = "Apple Computers, Inc." Take a look at the following screenshot now: As you can note from the preceding screenshot, we modified our first component within our tuple to the Apple Computers, Inc value. If you do not want modifications to be made to your variable, you can just change the var keyword to let, and the assignment would result in a compilation error. You can also express your tuples by referencing them using their named elements. This makes it really useful as you can ensure that your users know exactly what the element refers to. If you express your tuples using their named elements, you will still be able to access your elements using their index notation, as can be seen in the following highlighted code snippet: /*: # Swift Language Basics – Tuples : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation // Define our Address Details var addressDetails = (company:"Apple Inc.", Address:"1 Infinite Loop", City:"Cupertino, California", Country:"United States"); // Accessing our Tuple by using their NAMES print("Accessing our Tuple using their NAMESn") print(addressDetails.company) // Get the Name print(addressDetails.Address) // Address print(addressDetails.City) // State print(addressDetails.Country) // Country // Accessing our Tuple by using their index notation print("nAccess our Tuple using their index notation:n") print(addressDetails.0) // Get the Name print(addressDetails.1) // Address print(addressDetails.2) // State print(addressDetails.3) // Country Take a look at the following screenshot now: As you can note from what we covered so far about tuples, these are really cool and are basically just like any other data type in Swift; they can be really powerful to use within your own programs. String interpolation String interpolation means embedding constants, variables, as well as expressions within your string literals. In this section, we will take a look at an example of how you can use this. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics - String Interpolation : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation // Define our Address Details var addressDetails = (company:"Apple Inc.", Address:"1 Infinite Loop", City:"Cupertino, California", Country:"United States"); // Use String Interpolation to format output print("Apple Headquarters are located at: nn" + addressDetails.company + ",n" + addressDetails.Address + "n" + addressDetails.City + "n" + addressDetails.Country); Take a look at the following screenshot now: As you can note from the preceding code snippet, we started by declaring a tuple variable called addressDetails that contains a combination of strings. Next, we performed a string concatenation to generate our output in the format that we want by accessing each of the tuple elements using their index values and displaying each of these elements in the console window. Let's take this a step further and use string interpolation to place our address detail information into string variables. The result will still be the same, but I just want to show you the power of using tuples with the Swift programming language. Clear the contents of the playground template and replace them with the following highlighted code snippet: /*: # Swift Language Basics - String Interpolation : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation // Use String Interpolation to place elements into string initializers var addressDetails = ("Apple Inc.", "1 Infinite Loop", "Cupertino, California", "United States"); let (Company, Address, City, Country) = addressDetails print("Apple Headquarters are located at: nn" + Company + ",n" + Address + "n" + City + "n" + Country); Take a look at the following screenshot now: As you can note from the preceding code snippet, we removed the named types from our addressDetails string contents, created a new type using the let keyword, and assigned placeholders for each of our tuple elements. This is very handy as it not only makes your code a lot more readable but you can also continue to create additional placeholders for the additional fields that you create. Controlling the flow In this section, we will take a look at how to use the for…in loop to iterate over a set of statements within the body of the loop until a specific condition is met. The for…in loops The for…in loops basically perform a set of statements over a certain number of times until a specific condition is met, which is typically handled by incrementing a counter each time until the loop ends. Let's take a look at a code example to see how we can put this into practice. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics - Control Flow : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation // Perform a fibonacci loop using the For-In Loop var fibonacci = 0 var iTemp = 1 var jTemp = 0 for iterator in 0...19 { jTemp = fibonacci fibonacci += iTemp iTemp = jTemp print("Fibonacci: " + String(fibonacci), terminator: "n") } Take a look at the following screenshot now: The preceding code demonstrates the for…in loop and the closed range operator (...). These are often used together, but they are entirely independent. As you can note from the preceding code snippet, we declared the exact same variables: fibonacci, iTemp, and jTemp. Next, we used the for…in loop to iterate over our range, which is from 0 to 19, while displaying the current Fibonacci value in the console window. What's new in Swift 2.0 In this section, we will take a look at some of the new features that come as part of the Swift 2.0 programming language. Error handling Error handling is defined as the process of responding to and recovering from error conditions within your program. The Swift language provides first-class support for throwing, catching, propagating, and manipulating recoverable errors at runtime. In Swift, these are referred to as throwing functions and throwing methods. In Swift 2.0, error handling has vastly improved and adds an additional layer of safety to error checking. You can use the throws keyword to specify which functions and method are most likely to cause an error. You can implement and use the do, try, and catch keywords to handle something that could likely throw an error. Let's take a look at a code example to see how we can put this into practice. Clear the contents of the playground template and replace them with the following code snippet: /*: # Swift Language Basics - What's new in Swift 2.0 : Created by Steven F. Daniel : Copyright © 2015 GENIESOFT STUDIOS. All Rights Reserved. */ import Foundation enum EncryptionError: ErrorType { case Empty case Short } // Method to handle the Encryption func encryptString(str: String, withPassword password: String) throws -> String { if password.characters.count > 0 { // Password is valid } else { throw EncryptionError.Empty } if password.characters.count >= 5 { // Password is valid } else { throw EncryptionError.Short } // Begin constructing our encrypted string let encrypted = password + str + password return String(encrypted.characters.reverse()) } // Call our method to encrypt our string do { let encrypted = try encryptString("Encrypted String Goes Here", withPassword: "123") print(encrypted) } catch EncryptionError.Empty { print("You must provide a password.") } catch EncryptionError.Short { print("Passwords must be at least five characters.") } catch { print("An error occurred!") } Take a look at the following screenshot now: As you can note in the preceding code, we began by creating an enum object that derives from the ErrorType class so that we could create and throw an error. Next, we created a method called encryptString that takes two parameters: str and password. This method performed a check to ensure that we didn't pass an empty password. If our method determines that we did not specify a valid password, we will automatically throw an error using EncryptionError.Empty and exit from this method. Alternatively, if we provide a valid password and string to encrypt, our string will be encrypted. Binding Binding in Swift is something new and provides a means of checking whether a variable contains a valid value prior to continuing and exiting from the method otherwise. Fortunately, Swift 2.0 provides you with exactly this, and it is called the guard keyword. Let's go back to our previous code snippet and take a look at how we can implement the guard statement to our conditional checking within our encryptedString method. Modify the contents of the playground template and replace them with the following highlighted sections: // Method to handle the Encryption func encryptString(str: String, withPassword password: String) throws -> String { guard password.characters.count > 0 else { throw EncryptionError.Empty } guard password.characters.count >= 5 else { throw EncryptionError.Short } // Begin constructing our encrypted string let encrypted = password + str + password return String(encrypted.characters.reverse()) } As you can note in the preceding code snippet, using the guard keyword, you can provide a code block to perform a conditional check within the else statement that will run if the condition fails. This will make your code cleaner as the guard statement lets you trap invalid parameters from being passed to a method. Any conditions you would have checked using if before you can now check using guard. Protocol extensions In Swift 2.0, you have the ability to extend protocols and add additional implementations for properties and methods. For example, you can choose to add additional methods to the String or Array classes, as follows: /* # What's new in Swift 2.0 - Protocol Extensions The first content line displayed in this block of rich text. */ import Foundation let greeting = "Working with Swift Rocks!" // Extend the String class to include additional methods extension CustomStringConvertible { var uCaseString: String { return "(self.description.uppercaseString)!!!" } } print(greeting.uCaseString) Take a look at the following screenshot now: As you can note in the preceding code, we extended the String class using the CustomStringConvertible protocol, which most of the Foundation class objects conform to. Using protocol extensions, they provide you with a wide variety of ways to extend the base classes so that you can add and implement your very own custom functionalities. Summary In this article, we explored how to go about downloading and installing Xcode development tools and then moved on to discussing and using playgrounds to write Swift code to get to grips with some of the Swift programming language features. Next, we looked at some of the newest features that come as part of the Swift 2.0 language. Resources for Article: Further resources on this subject: Exploring Swift[article] Playing with Swift[article] Introduction to WatchKit[article]

In this article by Abhijeet Shriram Janwalkar, the author of VMware vRealize Configuration Manager Cookbook, we will discuss how to check compliance, create exceptions so that we don't get any false positives, and finally, create some Alert Rules that will alert us when non-compliant rules are found. (For more resources related to this topic, see here.) Checking the compliance of the infrastructure After creating all the rules, rule groups, and templates, we need to check the compliance of the infrastructure. We will learn how to check how compliant we are against internal standards, or we can directly use standard compliance packs we have already downloaded and imported. We will use a standard imported template for this recipe. Getting ready All the heavy lifting should have been done on the VCM server: it should be ready with the templates and at least one machine group, which will have the machines for whom we need to check the compliance, or we can use the default machine groups available. Using our own machine group is preferable. How to do it... As mentioned earlier, we will use an imported standard template, International Organization for Standardization 27001-27002- Windows 2008 R2 Mbr Server Controls, and we will run this against the default machine group, All Machines. Follow these steps to check the compliance of the Windows servers: Once logged in to VCM, go to Compliance | Templates Make sure the correct machine group is selected from the top; this is how VCM decides which machines to apply the template to measure the compliance. If you want to change the machine group, click on the Machine Group, and from the popup, select the correct machine group. Select the required template from the right-hand side and click on Run.   Depending upon the organization policies, decide to enforce or not enforce the compliance. Not all rules are enforceable; also, we can cause issues such as breaking a working application; for example, if the print spooler service is required to be disabled and we disable the service when we enforce the compliance, this will create an issue on the printer farm as it will stop functioning. So it is better that we first learn what is non-compliant and then make necessary exceptions. We can then enforce compliance from VCM or can ask the respective server owners to take the necessary action.   In a few minutes, depending on how many machines there are to check in the machine group, the compliance run will finish. Click on Close.   The compliance status can be viewed by navigating to the template on the left-hand side and selecting the correct machine group from the top.   In our case, our support team needs to work a lot as we are non-compliant. How it works... When we ask VCM to check compliance, it first applies the filters available in the rule groups, and then, only the machines that pass those filters are considered. The compliance checks are performed on the data collected by VCM and are available in the database, unlike some other tools that perform the checks at the client end, after which the client submits the data to VCM. The process followed by VCM is better as this can be performed on servers that are offline at that time, and when we check the result, we get the value because of which the machine is non-compliant for a rule. Again, this has some issues as well: first, we need to make sure our VCM is clean. By this, I mean whether a machine is purged from VCM when it is decommissioned, or else we will have details of machines that are not present in the infrastructure, and that could affect our final compliance score. The second issue is that it does not give us live details as it works on the data in its database; again, this can produce false positives. To counter this issue, we can schedule a compliance check after a full data collection for that machine group, in which way we will not have stale data to process. Once the compliance has been checked, and if we have chosen to enforce the compliance, it will create jobs to enforce them and will start executing on the managed machines; for example, if we have rules to check the status of a service and expect certain services in the running state, then VCM will start those services. Creating compliance exceptions As you know, every rule has an exception, and this is applicable to compliance as well: you create a rule for blocking the SMTP port on all the servers, and then, you have mail servers that need this port active. Now, we can't block the port, but at the same time, we know this is a known and accepted deviation; hence, we don't want our compliance score to suffer a hit because of this. To solve this, what we can do is add an exception so that this will not create issues while checking compliance. Getting ready Our organization has a policy to disable unwanted services on servers, and the print spooler is considered an unwanted service, so it must be disabled on all the servers but, of course, the exceptions are the print servers. We will create an exception for the print server machine group to be excused from this mandate. We will need rules created in VCM along with a machine group that will include all the print servers. How to do it... Let's create an exception for our print servers by following these steps: Log in to VCM and go to Compliance | Machine Group Compliance | Exceptions. Click on Add.   Provide a descriptive Name and Description, and click on Next.   Select the template for which you want this machine group to be excluded. In this case, we are selecting the one created by our organization rules. Click on Next.   Select the machine group created for this exception; in our case, it is named Print Servers. Click on Next. Select Override non-compliant results to compliant.  I really don't know why there is another option, but there must be a use case that I am not aware of.   We want this exception only for our rule for the print spooler server, called Service_Print_Spooler; so select that rule. Depending upon you requirement, you can have the exception for a complete rule group as well. But having exception for a single rule is sufficient in our case. Click on Finish.   You can enable/disable this exception as per requirement.   How it works... Exceptions are considered when we do a compliance check, and a final score is calculated. By creating an exception, we make sure that we don't get a bad score just because we need to have some things non-compliant. Also, this helps when we are enforcing compliance like in the earlier case, where we enforced Service Status to be disabled then VCM disabled the print spooler service on all the servers including the print servers, and that would have affected productivity. So, creating compliance exception is a win-win situation for both teams: the security team has a nice compliant environment and the printer admin team has a working print farm. Creating compliance alert rules Nobody likes to wait and nobody likes to work on Excel, so what if we get a ticket in our ticketing tool if a managed machine is non-complaint. We can create alerts and then maybe integrate them with a ticketing tool that can create a ticket for us, or VCM can send an e-mail to configured e-mail IDs. Getting ready We will need a working VCM server that is configured to check compliance. How to do it... This is a two-step process; first, we need to create an alert rule and then associate the created alert with the machine group. So, let's begin creating a compliance alert: Log in to VCM and go to Administration | Alerts | Rules. Click on Add.   In the wizard, give descriptive name and add a description. Click on Next. Select Compliance Results Data as the Data Type and click on Next.   As we want to create an alert for the non-compliancy of the rules created for our organization standards, select the appropriate compliance template. If we want an alert for the ISO 27001-27002 standard, we should have opted for that template.   On the next page, accept the newly create rule by clicking on Finish (the button is not in the screenshot).   The next process is to associate this Rule to correct Machine group. So we will continue to step 6. Now move to Administration | AlertsàMachine Group Configuration, and select the Machine group for which you would like the alert to be generated and click Add.   Select the alert we created and click on Next.   Select Severity and click on Next (not shown in the screenshot).   Select the actions that need to be done when the alert is created: we can send an e-mail, we can send an SNMP trap to a monitoring system or VCO that will create an alert in the organization ticketing system, or we can write the log to the Windows event log, and then, from there it will be picked up by the monitoring system to create a ticket. We are choosing to send an e-mail to the concerned people or teams.   Provide the details of who should receive the e-mail, the sender's e-mail ID, the SMTP server, e-mail subject, and modify the message body.   If required, you can check for alerts and click on Finish to close the wizard (the button is not shown in the screenshot).   The alerts can be seen at Console| Alerts.   How it works... We can't just depend on reports for checking compliance, even though that is a good way to check the status, but getting alerts for a non-compliant machine can be more proactive than going through a report. When we check or schedule a check for compliance, the result can be stored in the VCM database and is fetched when we visit the Compliance tab on the VCM console, Alerts provide a more proactive approach: they tell you that there is something wrong and you need to check it, so after every compliance check, if that machine group has something non-compliant, an alert will be created and that will take configured actions like sending an e-mail, sending an SNMP trap, or writing an event to the Windows logs. Those can be proactively worked upon rather than going to the console and checking the reports.   Summary In this article, we learned how to check compliance using a standard imported template. We also learned how to create exceptions for our compliance rules so that standard services can be run without causing our score to go down. Finally, we looked at alerts and ticketing systems. Resources for Article: Further resources on this subject: VM, It Is Not What You Think! [article] vRealize Automation and the Deconstruction of Components [article] Deploying New Hosts with vCenter [article]