How-To Tutorials

In this article by the author, Daniel Hall, of the book, Ansible Configuration Management - Second Edition, we learn to start digging a bit deeper into playbooks. We will be covering the following topics: External data lookups Storing results Processing data Debugging playboks (For more resources related to this topic, see here.) External data lookups Ansible introduced the lookup plugins in version 0.9. These plugins allow Ansible to fetch data from outside sources. Ansible provides several plugins, but you can also write your own. This really opens the doors and allows you to be flexible in your configuration. Lookup plugins are written in Python and run on the controlling machine. They are executed in two different ways: direct calls and with_* keys. Direct calls are useful when you want to use them like you would use variables. Using the with_* keys is useful when you want to use them as loops. In the next example, we use a lookup plugin directly to get the http_proxy value from environment and send it through to the configured machine. This makes sure that the machines we are configuring will use the same proxy server to download the file. --- - name: Downloads a file using a proxy hosts: all tasks:    - name: Download file      get_url:        dest: /var/tmp/file.tar.gz         url: http://server/file.tar.gz      environment:        http_proxy: "{{ lookup('env', 'http_proxy') }}" You can also use lookup plugins in the variable section. This doesn't immediately lookup the result and put it in the variable as you might assume; instead, it stores it as a macro and looks it up every time you use it. This is good to know if you are using something, the value of which might change over time. Using lookup plugins in the with_* form will allow you to iterate over things you wouldn't normally be able to. You can use any plugin like this, but ones that return a list are most useful. In the following code, we show how to dynamically register a webapp farm. --- - name: Registers the app server farm hosts: localhost connection: local vars:    hostcount: 5 tasks:    - name: Register the webapp farm      local_action: add_host name={{ item }} groupname=webapp      with_sequence: start=1 end={{ hostcount }} format=webapp%02x If you were using this example, you would append a task to create each as a virtual machine and then a new play to configure each of them. Situations where lookup plugins are useful are as follows: Copying a whole directory of Apache config to a conf.d style directory Using environment variables to adjust what the playbooks does Getting configuration from DNS TXT records Fetching the output of a command into a variable Storing results Almost every module outputs something, even the debug module. Most of the time, the only variable used is the one named changed. The changed variable helps Ansible decide whether to run handlers or not and which color to print the output in. However, if you wish to, you can store the returned values and use them later in the playbook. In this example, we look at the mode in the /tmp directory and create a new directory named /tmp/subtmp with the same mode as shown here. --- - name: Using register hosts: ansibletest user: root tasks:    - name: Get /tmp info    file:        dest: /tmp        state: directory      register: tmp      - name: Set mode on /var/tmp      file:        dest: /tmp/subtmp        mode: "{{ tmp.mode }}"        state: directory Some modules, such as the file module in the previous example, can be configured to simply give information. By combining this with the register feature, you can create playbooks that can examine the environment and calculate how to proceed. Combining the register feature and the set_fact module allows you to perform data processing on data you receive back from the modules. This allows you to compute values and perform data processing on these values. This makes your playbooks even smarter and more flexible than ever. Register allows you to make your own facts about hosts from modules already available to you. This can be useful in many different circumstances: Getting a list of files in a remote directory and downloading them all with fetch Running a task when a previous task changes, before the handlers run Getting the contents of the remote host SSH key and building a known_hosts file Processing data Ansible uses Jinja2 filters to allow you to transform data in ways that aren't possible with basic templates. We use filters when the data available to us in our playbooks is not in the format we want, or require further complex processing before it can be used with modules or templates. Filters can be used anywhere we would normally use a variable, such as in templates, as arguments to modules, and in conditionals. Filters are used by providing the variable name, a pipe character, and then the filter name. We can use multiple filter names separated with pipe characters to use multiple pipes, which are then applied left to right. Here is an example where we ensure that all users are created with lowercase usernames: --- - name: Create user accounts hosts: all vars:    users: tasks:    - name: Create accounts      user: name={{ item|lower }} state=present      with_items:        - Fred        - John        - DanielH Here are a few popular filters that you may find useful: Filter Description min When the argument is a list it returns only the smallest value. max When the argument is a list it returns only the largest value. random When the argument is a list it picks a random item from the list. changed When used on a variable created with the register keyword, it returns true if the task changed anything; otherwise, it returns false. failed When used on a variable created with the register keyword, it returns true if the task failed; otherwise, it returns false. skipped When used on a variable created with the register keyword, it returns true if the task changed anything; otherwise, it returns false. default(X) If the variable does not exist, then the value of X will be used instead. unique When the argument is a list, return a list without any duplicate items. b64decode Convert the base64 encoded string in the variable to its binary representation. This is useful with the slurp modules, as it returns its data as a base64 encoded string. replace(X, Y) Return a copy of the string with any occurrences of X replaced by Y. join(X) When the variable is a list, return a string with all the entries separated by X. Debugging playbooks There are a few ways in which you can debug a playbook. Ansible includes both a verbose mode and a debug module specifically for debugging. You can also use modules such as fetch and get_url for help. These debugging techniques can also be used to examine how modules behave when you wish to learn how to use them. The debug module Using the debug module is really quite simple. It takes two optional arguments, msg and fail.msg to set the message that will be printed by the module and fail, if set to yes, indicates a failure to Ansible, which will cause it to stop processing the playbook for that host. We used this module earlier in the skipping modules section to bail out of a playbook if the operating system was not recognized. In the following example, we will show how to use the debug module to list all the interfaces available on the machine: --- - name: Demonstrate the debug module hosts: ansibletest user: root vars:    hostcount: 5 tasks:    - name: Print interface      debug:        msg: "{{ item }}"      with_items: ansible_interfaces The preceding code gives the following output: PLAY [Demonstrate the debug module] *********************************   GATHERING FACTS ***************************************************** ok: [ansibletest]   TASK: [Print interface] ********************************************* ok: [ansibletest] => (item=lo) => {"item": "lo", "msg": "lo"} ok: [ansibletest] => (item=eth0) => {"item": "eth0", "msg": "eth0"}   PLAY RECAP ********************************************************** ansibletest               : ok=2   changed=0   unreachable=0   failed=0 As you can see, the debug module is easy to use to see the current value of a variable during the play. The verbose mode Your other option for debugging is the verbose option. When running Ansible with verbose, it prints out all the values that were returned by each module after it runs. This is especially useful if you are using the register keyword introduced in the previous section. To run ansible-playbook in verbose mode, simply add --verbose to your command line as follows: ansible-playbook --verbose playbook.yml The check mode In addition to the verbose mode, Ansible also includes a check mode and a diff mode. You can use the check mode by adding --check to the command line, and --diff to use the diff mode. The check mode instructs Ansible to walk through the play without actually making any changes to remote systems. This allows you to obtain a listing of the changes that Ansible plans to make to the configured system. It is important here to note that the check mode of Ansible is not perfect. Any modules that do not implement the check feature are skipped. Additionally, if a module is skipped that provides more variables, or the variables depend on a module actually changing something (such as file size), then they will not be available. This is an obvious limitation when using the command or shell modules The diff mode shows the changes that are made by the template module. This limitation is because the template file only works with text files. If you were to provide a diff of a binary file from the copy module, the result would almost be unreadable. The diff mode also works with the check mode to show you the planned changes that were not made due to being in check mode. The pause module Another technique is to use the pause module to pause the playbook while you examine the configured machine as it runs. This way, you can see changes that the modules have made at the current position in the play, and then watch while it continues with the rest of the play. Summary In this article, we explored the more advanced details of writing playbooks. You should now be able to use features such as delegation, looping, conditionals, and fact registration to make your plays much easier to maintain and edit. We also looked at how to access information from other hosts, configure the environment for a module, and gather data from external sources. Finally, we covered some techniques for debugging plays that are not behaving as expected. Resources for Article: Further resources on this subject: Static Data Management [article] Drupal 8 and Configuration Management [article] Introduction to Drupal Web Services [article]

In this article, Dmitry Anoshin, the author of Learning Hunk will talk about Hadoop—how to extract Hunk to VM to set up a connection with Hadoop to create dashboards. We are living in a century of information technology. There are a lot of electronic devices around us that generate a lot of data. For example, you can surf on the Internet, visit a couple news portals, order new Airmax on the web store, write a couple of messages to your friend, and chat on Facebook. Every action produces data; we can multiply these actions with the number of people who have access to the Internet or just use a mobile phone and we will get really big data. Of course, you have a question, how big is it? I suppose, now it starts from terabytes or even petabytes. The volume is not the only issue; we struggle with a variety of data. As a result, it is not enough to analyze only structure data. We should dive deep into the unstructured data, such as machine data, that are generated by various machines. World famous enterprises try to collect this extremely big data in order to monetize it and find business insights. Big data offers us new opportunities, for example, we can enrich customer data via social networks using the APIs of Facebook or Twitter. We can build customer profiles and try to predict customer wishes in order to sell our product or improve customer experience. It is easy to say, but difficult to do. However, organizations try to overcome these challenges and use big data stores, such as Hadoop. (For more resources related to this topic, see here.) The big problem Hadoop is a distributed file system and framework to compute. It is relatively easy to get data into Hadoop. There are plenty of tools to get data into different formats. However, it is extremely difficult to get value out of these data that you put into Hadoop. Let's look at the path from data to value. First, we have to start at the collection of data. Then, we also spend a lot of time preparing and making sure this data is available for analysis while being able to ask questions to this data. It looks as follows: Unfortunately, the questions that you asked are not good or the answers that you got are not clear, and you have to repeat this cycle over again. Maybe, you have transformed and formatted your data. In other words, it is a long and challenging process. What you actually want is something to collect data; spend some time preparing the data, then you would able to ask question and get answers from data repetitively. Now, you can spend a lot of time asking multiple questions. In addition, you are able to iterate with data on those questions to refine the answers that you are looking for. The elegant solution What if we could take Splunk and put it on top of all these data stored in Hadoop? And it was, what the Splunk company actually did. The following figure shows how we got Hunk as name of the new product: Let's discuss some solution goals Hunk inventors were thinking about when they were planning Hunk: Splunk can take data from Hadoop via the Splunk Hadoop Connection app. However, it is a bad idea to copy massive data from Hadoop to Splunk. It is much better to process data in place because Hadoop provides both storage and computation and why not take advantage of both. Splunk has extremely powerful Splunk Processing Language (SPL) and it is a kind of advantage of Splunk, because it has a wide range of analytic functions. This is why it is a good idea to keep SPL in the new product. Splunk has true schema on the fly. The data that we store in Hadoop changes constantly. So, Hunk should be able to build schema on the fly, independent from the format of the data. It's a very good idea to have the ability to make previews. As you know, when a search is going on, you would able to get incremental results. It can dramatically reduce the outage. For example, we don't need to wait till the MapReduce job is finished. We can look at the incremental result and, in the case of a wrong result, restart a search query. The deployment of Hadoop is not easy; Splunk tries to make the installation and configuration of Hunk easy for us. Getting up Hunk In order to start exploring the Hadoop data, we have to install Hunk on the top of our Hadoop cluster. Hunk is easy to install and configure. Let's learn how to deploy Hunk Version 6.2.1 on top of the existing CDH cluster. It's assumed that your VM is up and running. Extracting Hunk to VM To extract Hunk to VM, perform the following steps: Open the console application. Run ls -la to see the list of files in your Home directory: [cloudera@quickstart ~]$ cd ~ [cloudera@quickstart ~]$ ls -la | grep hunk -rw-r--r--   1 root     root     113913609 Mar 23 04:09 hunk-6.2.1-249325-Linux-x86_64.tgz Unpack the archive: cd /opt sudo tar xvzf /home/cloudera/hunk-6.2.1-249325-Linux-x86_64.tgz -C /opt Setting up Hunk variables and configuration files Perform the following steps to set up the Hunk variables and configuration files It's time to set the SPLUNK_HOME environment variable. This variable is already added to the profile; it is just to bring to your attention that this variable must be set: export SPLUNK_HOME=/opt/hunk Use default splunk-launch.conf. This is the basic properties file used by the Hunk service. We don't have to change there something special, so let's use the default settings: sudocp /opt/hunk/etc/splunk-launch.conf.default /opt/hunk//etc/splunk-launch.conf Running Hunk for the first time Perform the following steps to run Hunk: Run Hunk: sudo /opt/hunk/bin/splunk start --accept-license Here is the sample output from the first run: sudo /opt/hunk/bin/splunk start --accept-license This appears to be your first time running this version of Splunk. Copying '/opt/hunk/etc/openldap/ldap.conf.default' to '/opt/hunk/etc/openldap/ldap.conf'. Generating RSA private key, 1024 bit long modulus Some output lines were deleted to reduce amount of log text Waiting for web server at http://127.0.0.1:8000 to be available.... Done If you get stuck, we're here to help. Look for answers here: http://docs.splunk.com The Splunk web interface is at http://vm-cluster-node1.localdomain:8000 Setting up a data provider and virtual index for the CDR data We need to accomplish two tasks: provide a technical connector to the underlying data storage and create a virtual index for the data on this storage. Log in to http://quickstart.cloudera:8000. The system would ask you to change the default admin user password. I did set it to admin: Setting up a connection to Hadoop Right now, we are ready to set up the integration between Hadoop and Hunk. At first, we need to specify the way Hunk connects to the current Hadoop installation. We are using the most recent way: YARN with MR2. Then, we have to point virtual indexes to the data stored on Hadoop. To do this, perform the following steps: Click on Explore Data. Click on Create a provider: Let's fill the form to create the data provider: Property name Value Name hadoop-hunk-provider Java home /usr/java/jdk1.7.0_67-cloudera Hadoop home /usr/lib/hadoop Hadoop version Hadoop 2.x, (Yarn) filesystem hdfs://quickstart.cloudera:8020 Resource Manager Address quickstart.cloudera:8032 Resource Scheduler Address quickstart.cloudera:8030 HDFS Working Directory /user/hunk Job Queue default You don't have to modify any other properties. The HDFS working directory has been created for you in advance. You can create it using the following command: sudo -u hdfshadoop fs -mkdir -p /user/hunk If you did everything correctly, you should see a screen similar to the following screenshot: Let's discuss briefly what we have done: We told Hunk where Hadoop home and Java are. Hunk uses Hadoop streaming internally, so it needs to know how to call Java and Hadoop streaming. You can inspect the submitted jobs from Hunk (discussed later) and see the following lines: /opt/hunk/bin/jars/sudobash /usr/bin/hadoop jar "/opt/hunk/bin/jars/SplunkMR-s6.0-hy2.0.jar" "com.splunk.mr.SplunkMR" MapReduce JAR is submitted by Hunk. Also, we need to tell Hunk where the YARN Resource Manager and Scheduler are located. These services allow us to ask for cluster resources and run jobs. Job queue could be useful in the production environment. You could have several queues for cluster resource distribution in real life. We would set queue name as default, since we are not discussing cluster utilization and load balancing. Setting up a virtual index for the data stored in Hadoop Now it's time to create virtual index. We are going to add the dataset with the avro files to the virtual index as an example data. Click on Explore Data and then click on Create a virtual index: You'll get a message telling that there are no indexes: Just click on New Virtual Index. A virtual index is a metadata. It tells Hunk where the data is located and what provider should be used to read the data. Property name Value Name milano_cdr_aggregated_10_min_activity Path to data in HDFS /masterdata/stream/milano_cdr Here is an example screen you should see after you create your first virtual index: Accessing data through the virtual index To access data through the virtual index, perform the following steps: Click on Explore Data and select a provider and virtual index: Select part-m-00000.avro by clicking on it. The Next button will be activated after you pick up a file: Preview data in the Preview Data tab. You should see how Hunk automatically for timestamp from our CDR data: Pay attention to the Time column and the field named Time_interval from the Event column. The time_interval column keeps the time of record. Hunk should automatically use that field as a time field: Save the source type by clicking on Save As and then Next: In the Entering Context Settings page, select search in the App context drop-down box. Then, navigate to Sharing context | All apps and then click on Next. The last step allows you to review what we've done: Click on Finish to create the finalized wizard. Creating a dashbord Now it's time to see how the dashboards work. Let's find the regions where the visitors face problems (status = 500) while using our online store: index="digital_analytics" status=500 | iplocation clientip | geostats latfield=lat longfield=lon count by Country You should see the map and the portions of error for the countries: Now let's save it as dashboard. Click on Save as and select Dashboard panel from drop-down menu. Name it as Web Operations. You should get a new dashboard with a single panel and our report on it. We have several previously created reports. Let's add them to the newly created dashboard using separate panels: Click on Edit and then Edit panels. Select Add new panel and then New from report, and add one of our previous reports. Summary In this article, you learned how to extract Hunk to VM. We also saw how to set up Hunk variables and configuration files. You learned how to run Hunk and how to set up the data provided and a virtual index for the CDR data. Setting up a connection to Hadoop and a virtual index for the data stored in Hadoop were also covered in detail. Apart from these, you also learned how to create a dashboard. Resources for Article: Further resources on this subject: Identifying Big Data Evidence in Hadoop [Article] Big Data [Article] Understanding Hadoop Backup and Recovery Needs [Article]

In this article by Russ McKendrick, the author of the book Docker Data Management with Flocker, we are going to look at what both what problems Flocker has been developed to resolve as well as jumping in at the deep end and perform our first installation of Flocker. (For more resources related to this topic, see here.) By the end of the article, you will have: Configured and used the default Docker volume driver Learned a little about how Flocker came about and who wrote it Installed and configured a Flocker control and storage node Integrated your Flocker installation with Docker Have visualized your volumes using the Volume Hub Interacted with your Flocker installation using the Flocker CLI Have installed and used dvol However, before we start to look at both Docker and Flocker we should talk about compute resource. Compute and Storage Resource In later articles, we will be looking at running Flocker on public cloud services such as Amazon Web Services and Google Cloud so we will not be using those services for the practical examples in our early articles, instead I will be server instances launched in DigitalOcean. DigitalOcean has quickly become the platform of choice for both developers & system administrators who want to experiment with new technologies and services. While their cloud service is simple to use their SSD backed virtual machines offer the performance required to run modern software stacks at a fraction of the price of other public clouds making them perfect for launching prototyping and sandbox environments. For more information on DigitalOcean, please see their website at: https://www.digitalocean.com/. The main reason I choose to use an external cloud provider rather than local virtual machine is that we will need to launch multiple hosts to run Flocker. You do not have to use DigitalOcean, as long as you are able to launch multiple virtual machines and attach some sort of block storage to your instances as a secondary device. Finally, where we are doing manual installations of Flocker and Docker I will give instructions, which cover both CentOS 7.2 and Ubuntu 16.04. Docker If you are reading this article then Docker does not need much of an introduction, it is one of the most talked about technologies of recent years quickly gaining support from pretty much all what I would consider the big players in software. Companies such as Google, Microsoft, Red Hat, VMWare, IBM, Amazon Web Services and Hewlett Packard all offer solutions based on Dockers container technology or have contributed towards its development. Rather than me give you a current state of the union on Docker I would recommend you watch the opening key note from the 2016 Docker Conference, it gives a very rundown of how far Docker has come over the last few years. The video can be found at the following URL https://www.youtube.com/watch?v=vE1iDPx6-Ok. Now you are all caught up with what's going in the world of Docker, let's take a look at what storage options you get out of the box with the core Docker Engine. Installing Docker Engine Let start with a small server instance, install Docker and then run a through a couple of scenarios where we may need storage for our containerized application: Before installing Docker, it is always best to check that you have the latest updates installed, to do this run one of the following commands. On CentOS you need to run: yum -y update and for Ubuntu run: apt-get -y update Both of these commands will check for and automatically install any available updates. Once your server instance is up-to-date, you can go-ahead and install Docker, the simplest way of doing this is to run the installation script provided by Docker by entering the following command: curl -fsSL https://get.docker.com/ | sh This method of installation works on both CentOS and Ubuntu, the script run checks to see which packages need to installed and then installs the latest version of the Docker Engine from Docker's own repositories. If you would prefer to check the contents of the script before running it on your server instance you can go to https://get.docker.com/ in your browser, this will display the full bash script so you can see exactly what the script is doing when executed on your server instance. If everything worked as expected, then you will have latest stable version of Docker installed. You check this by running the following command: docker --version This will return the version and build of the version of Docker which was installed, in my case this is Docker version 1.12.0, build 8eab29e. On both operating systems you can check that the Docker is running by using the following command: systemctl status docker If Docker is stopped then you can use the following command to start it: systemctl start docker The Docker local volume driver Now we have Docker installed and running on our server instance, we can start launching containers. From here, the instructions are the same if you are running CentOS or Ubuntu. We are going to be launching an application called Moby Counter, this was written by Kai Davenport to demonstrate maintaining state between containers. It is a Nodejs application which allows you draw (using Docker logos) in your browser window. The coordinates for your drawing are stored in a Redis key value store, the idea being if you stop, start and even remove the Redis service your drawing should persist: Let start by launching our Redis container, to do this run the following command: docker run -itd --name redis redis:alpine redis-server --appendonly Using a backslash: As we will sometimes have a lot options to pass to the commands we are going to be running, we are going to be using to split the command over multiple lines so that it's easier to follow what is going on, like in the command above. The Docker Hub is the registry service provided by Docker. The service allows you to distribute your container images, either by uploading images directly or by building your Docker files which are stored in either GitHub or BitBicket. We will be using the Docker Hub throughout the article, for more information on the Docker Hub see the official documentation at: https://docs.docker.com/docker-hub/. This will download the official Redis image (https://hub.docker.com/_/redis/) from the Docker Hub and then launch a container called Redis by running the redis-server --appendonly command. Adding the --appendonly flag to the Redis server command means that each time Redis receives a command such as SET it will not only execute the command in the dataset which is held in memory, it will also append it to what's called an AOF file. When Redis is restarted the AOF file is replayed to restore the state of the dataset held in RAM. Note that we are using the Alpine Linux version of the container to keep the download size down. Alpine Linux is a Linux distribution which is designed to be small, in fact the official Docker image based on Alpine Linux is only 5 MB in size! Compared to the sizes other base operating system images such as the official Debian, Ubuntu and CentOS which you can see from the following terminal session It is easy to see why the majority of official Docker images are using Alpine Linux as their base image. Now we have Redis up and running, let's launch our application container by running: docker run -itd --name moby-counter -p 80:80 --link redis:redis russmckendrick/moby-counter You check that your two containers are running using the docker ps command, you should see something like the following terminal session: Now you have the Moby Counter application container running, open your browser and go to the IP address of your server instance, in my case this was http://159.203.185.102 once the page loads you should see a white page which says Click to add logos… in the center. As per the instructions, click around the page to add some Docker logos: Now we have some data stored in the Redis store let's do the equivalent of yanking the power cord out of the container by running the following command: docker rm -f redis This will cause your browser to looking something like the following screen capture: Relaunching the Redis container using the following command: docker run -itd --name redis redis:alpine redis-server --appendonly and refreshing our browser takes us back to the page which says Click to add logos… So what gives, why didn't you see your drawing? It's quite simple, we never told Docker to start the container with any persistent storage. Luckily, the official Redis image is smart and assumes that you should really be using persistent storage to store your data in, even if you don't tell it you want to as no-one likes data loss. Using the docker volume ls command you should be able to see that there are two volumes, both with really long names, in my case the output of the command looked like the following terminal capture: So we now have two Docker Volumes, one which contains our original "master piece" and the second which is blank. Before we remove our Redis container again, let's check to see which of the two Docker Volumes is currently mounted on our Redis container, we can do this by running: docker inspect redis There quite a bit of information shown when the docker inspect command, there information we are after is in the Mounts section, and we need to know the Name: As you can see from the terminal output above, the currently mounted volume in my setup is called b76101d13afc2b33206f5a2bba9a3e9b9176f43ce57f74d5836c824c22c. Now we know the name of the blank volume, we know that the second volume has the data for our drawing. Let's terminate the running Redis container by running: docker rm -f redis and now launch our Redis container again, but this time telling Docker to use the local volume driver and also use volume we know contains the data for our drawing: docker run -itd --name redis --volume-driver=local -v 37cb395253624782836cc39be1aa815682b70f73371abb6d500a:/data redis:alpine redis-server --appendonly yes Make sure you replace the name of volume which immediately follows the -v on the fourth line with the name of the volume you know contains your own data. After a few moments, refresh your browser and you should see your original drawing. Our volume was automatically created by the Redis container as it launched which is why it has a Unique Identification Number (UID) as its name, if we wanted to we could give it more friendly name by running the following command to launch our Redis container: docker run -itd --name redis --volume-driver=local -v redis_data:/data redis:alpine redis-server --appendonly yes As you can when running the docker volume ls command again, this is a lot more identifiable: As we have seen with the simple example above, it is easy to create volumes to persist your data on using Docker Engine, however there is one drawback to using the default volume driver and that is your volumes are only available on a single Docker host. For most people just setting out on their journey into using containers this is fine, however for people who want to host their applications in production or would like a more resilient persistent data store using the local driver quickly becomes a single point of failure. This is where Flocker comes in. Summary In this article we have worked through the most basic Flocker installation possible, we have integrated the installation with Docker, the Volume Hub and also the Flocker command to interact with the volumes created and managed by Flocker. In the next article we are going to look at how Flocker can be used to provide more resilient storage for a cluster of Docker hosts rather than just the single host we have been using in this article. Resources for Article: Further resources on this subject: Docker Hosts [article] Hands On with Docker Swarm [article] Understanding Docker [article]

  IBM InfoSphere Replication Server and Data Event Publisher Design, implement, and monitor a successful Q replication and Event Publishing project Covers the toolsets needed to implement a successful Q replication project Aimed at the Linux, Unix, and Windows operating systems, with many concepts common to z/OS as well A chapter dedicated exclusively to WebSphere MQ for the DB2 DBA Detailed step-by-step instructions for 13 Q replication scenarios with troubleshooting and monitoring tips Written in a conversational and easy to follow manner Checking the state of a Q subscription The state of a Q subscription is recorded in the IBMQREP_SUBS table, and can be queried as follows: db2 "SELECT SUBSTR(subname,1,10) AS subname, state FROM asn.ibmqrep_subs" SUBNAME STATE -------- ----- DEPT0001 A XEMP0001 A Stopping a Q subscription The command to stop a Q subscription is STOP QSUB SUBNAME <qsubname>. Note that if Q Capture is not running, then the command will not take effect until Q Capture is started, because the STOP QSUB command generates an INSERT command into the IBMQREP_SIGNAL table: INSERT INTO ASN.IBMQREP_SIGNAL (signal_type, signal_subtype, signal_input_in) VALUES ('CMD', 'CAPSTOP', 'T10001'); In a unidirectional setup, to stop a Q subscription called T10001 where the Q Capture and Q Apply control tables have a schema of ASN, create a text file called SYSA_qsub_stop_uni.asnclp containing the following ASNCLP commands: ASNCLP SESSION SET TO Q REPLICATION; SET RUN SCRIPT NOW STOP ON SQL ERROR ON; SET SERVER CAPTURE TO DB DB2A; SET SERVER TARGET TO DB DB2B; SET CAPTURE SCHEMA SOURCE ASN; SET APPLY SCHEMA ASN; stop qsub subname T10001; In bidirectional or peer-to-peer two-way replication, we have to specify both Q subscriptions (T10001 and T10002) for the subscription group: ASNCLP SESSION SET TO Q REPLICATION; SET RUN SCRIPT NOW STOP ON SQL ERROR ON; SET CAPTURE SCHEMA SOURCE ASN; SET APPLY SCHEMA ASN; SET SERVER CAPTURE TO DB DB2A; SET SERVER TARGET TO DB DB2B; stop qsub subname T10001; SET SERVER CAPTURE TO DB DB2B; SET SERVER TARGET TO DB DB2A; stop qsub subname T10002; In a Peer-to-peer four-way setup, the commands would be in a file called qsub_stop_p2p4w.asnclp containing the following ASNCLP commands: ASNCLP SESSION SET TO Q REPLICATION; SET RUN SCRIPT NOW STOP ON SQL ERROR ON; SET CAPTURE SCHEMA SOURCE ASN; SET APPLY SCHEMA ASN; SET SERVER CAPTURE TO DB DB2A; SET SERVER TARGET TO DB DB2B; stop qsub subname T10001; SET SERVER CAPTURE TO DB DB2A; SET SERVER TARGET TO DB DB2C; stop qsub subname T10002; SET SERVER CAPTURE TO DB DB2A; SET SERVER TARGET TO DB DB2D; stop qsub subname T10003; Dropping a Q subscription The ASNCLP command to drop a Q subscription is: DROP QSUB (SUBNAME <qsubname> USING REPLQMAP <repqmapname>); In a unidirectional setup, to drop a Q subscription called T10001, which uses a Replication Queue Map called RQMA2B, create a file called drop_qsub_uni.asnclp containing the following ASNCLP commands: ASNCLP SESSION SET TO Q REPLICATION; SET RUN SCRIPT NOW STOP ON SQL ERROR ON; SET SERVER CAPTURE TO DB DB2A; SET SERVER TARGET TO DB DB2B; SET CAPTURE SCHEMA SOURCE ASN; SET APPLY SCHEMA ASN; drop qsub (subname TAB1 using replqmap RQMA2B); We can use the SET DROP command to specify whether for unidirectional replication the target table and its table space are dropped when a Q subscription is deleted: SET DROP TARGET [NEVER|ALWAYS] The default is not to drop the target table. In a multi-directional setup, there are three methods we can use: In the first method, we need to issue the DROP QSUB command twice, once for the Q subscription from DB2A to DB2B and once for the Q subscription from DB2B to DB2A. In this method, we need to know the Q subscription and Replication Queue Map names, which is shown in the qsub_drop_bidi0.asnclp file: ASNCLP SESSION SET TO Q REPLICATION; SET RUN SCRIPT NOW STOP ON SQL ERROR ON; SET CAPTURE SCHEMA SOURCE ASN; SET APPLY SCHEMA ASN; SET SERVER CAPTURE TO DB DB2A; SET SERVER TARGET TO DB DB2B; drop qsub (subname T10001 using replqmap RQMA2B); SET SERVER CAPTURE TO DB DB2B; SET SERVER TARGET TO DB DB2A; drop qsub (subname T10002 using replqmap RQMB2A); In the second method, we use the DROP SUBTYPE command, which is used to delete the multi-directional Q subscriptions for a single logical table. We use the DROP SUBTYPE command with the SET REFERENCE TABLE construct, which identifies a Q subscription for multi-directional replication. An example of using these two is shown in the following content file, which drops all the Q subscriptions for the source table eric.t1. This content file needs to be called from a load script file. SET SUBGROUP "TABT1"; SET SERVER MULTIDIR TO DB "DB2A"; SET SERVER MULTIDIR TO DB "DB2B"; SET REFERENCE TABLE USING SCHEMA "DB2A".ASN USES TABLE eric.t1; DROP SUBTYPE B QSUBS; The USING SCHEMA part of the SET REFERENCE TABLE command identifies the server that contains the table (DB2A) and the schema (ASN) of the control tables in which this table is specified as a source and target. The USES TABLE part specifies the table schema (eric) and table name (t1) to which the Q subscription applies. When we use this command, no tables or table spaces are ever dropped. The SUBGROUP name must be the valid for the tables whose Q subscriptions we want to drop. We can find the SUBGROUP name for a table using the following query: db2 "SELECT SUBSTR(subgroup,1,10) AS subsgroup, SUBSTR(source_ owner,1,10) as schema, SUBSTR(source_name,1,10) as name FROM asn. ibmqrep_subs" SUBSGROUP SCHEMA NAME ------ ------- ------ TABT2 DB2ADMIN DEPT TABT2 DB2ADMIN XEMP The preceding ASNCLP command generates the following SQL: -- CONNECT TO DB2B USER XXXX using XXXX; DELETE FROM ASN.IBMQREP_TRG_COLS WHERE subname = 'T10001' AND recvq = 'CAPA.TO.APPB.RECVQ'; DELETE FROM ASN.IBMQREP_TARGETS WHERE subname = 'T10001' AND recvq = 'CAPA.TO.APPB.RECVQ'; DELETE FROM ASN.IBMQREP_SRC_COLS WHERE subname = 'T10002'; DELETE FROM ASN.IBMQREP_SUBS WHERE subname = 'T10002'; -- CONNECT TO DB2A USER XXXX using XXXX; DELETE FROM ASN.IBMQREP_SRC_COLS WHERE subname = 'T10001'; DELETE FROM ASN.IBMQREP_SUBS WHERE subname = 'T10001'; DELETE FROM ASN.IBMQREP_TRG_COLS WHERE subname = 'T10002' AND recvq = 'CAPB.TO.APPA.RECVQ'; DELETE FROM ASN.IBMQREP_TARGETS WHERE subname = 'T10002' AND recvq = 'CAPB.TO.APPA.RECVQ'; A third method uses the DROP SUBGROUP command, as shown: SET SUBGROUP "TABT2"; SET SERVER MULTIDIR TO DB "DB2A"; SET SERVER MULTIDIR TO DB "DB2B"; SET MULTIDIR SCHEMA "DB2A".ASN ; DROP SUBGROUP; With this command, we just need to specify the Q subscription group name (SUBGROUP). The preceding ASNCLP command generates the following SQL: -- CONNECT TO DB2A USER XXXX using XXXX; DELETE FROM ASN.IBMQREP_TRG_COLS WHERE subname = 'T10002' AND recvq = 'CAPB.TO.APPA.RECVQ'; DELETE FROM ASN.IBMQREP_TARGETS WHERE subname = 'T10002' AND recvq = 'CAPB.TO.APPA.RECVQ'; DELETE FROM ASN.IBMQREP_SRC_COLS WHERE subname = 'T10001'; DELETE FROM ASN.IBMQREP_SUBS WHERE subname = 'T10001'; -- CONNECT TO DB2B USER XXXX using XXXX; DELETE FROM ASN.IBMQREP_SRC_COLS WHERE subname = 'T10002'; DELETE FROM ASN.IBMQREP_SUBS WHERE subname = 'T10002'; DELETE FROM ASN.IBMQREP_TRG_COLS WHERE subname = 'T10001' AND recvq = 'CAPA.TO.APPB.RECVQ'; DELETE FROM ASN.IBMQREP_TARGETS WHERE subname = 'T10001' AND recvq = 'CAPA.TO.APPB.RECVQ'; In a peer-to-peer three-way scenario, we would add a third SET SERVER MULTIDIR TO DB line pointing to the third server. If we use the second or third method, then we do not need to know the Q subscription names, just the table name in the second method and the Q subscription group name in the third method. Altering a Q subscription We can only alter Q subscriptions which are inactive. The following query shows the state of all Q subscriptions: db2 "SELECT SUBSTR(subname,1,10) AS subname, state FROM asn.ibmqrep_subs" SUBNAME STATE ---------- ----- DEPT0001 I At the time of writing, if we try and alter an active Q subscription, we will get the following error when we run the ASNCLP commands: ErrorReport : ASN2003I The action "Alter Subscription" started at "Friday, 22 January 2010 12:53:16 o'clock GMT". Q subscription name: "DEPT0001". Q Capture server: "DB2A". Q Capture schema: "ASN". Q Apply server: "DB2B". Q Apply schema: "ASN". The source table is "DB2ADMIN.DEPT". The target table or stored procedure is "DB2ADMIN.DEPT". ASN0999E "The attribute "erroraction" cannot be updated." : "The Subscription cannot be updated because it is in active state" : Error condition "*", error code(s): "*", "*", "*". This should be resolved in a future release. So now let's move on and look at the command to alter a Q subscription. To alter a Q subscription, we use the ALTER QSUB ASNCLP command. The parameters for the command depend on whether we are running unidirectional or multi-directional replication. We can change attributes for both the source and target tables, but what we can change depends on the type of replication (unidirectional, bidirectional, or peer-to-peer), as shown in the following table: ParameterUniBiP2PSource table:ALL CHANGED ROWS [N | Y]YY HAS LOAD PHASE [N | I |E]YYYTarget table:CONFLICT RULE [K | C | A] Y CONFLICT ACTION [I | F | D | S | Q] Y ERROR ACTION [Q | D | S]YYYLOAD TYPE [0 | 2 | 3 | 4 | 104 | 5 | 105]YYYOKSQLSTATES ["sqlstates"]YYY For unidirectional replication, the format of the command is: ALTER QSUB <subname> REPLQMAP <mapname> USING REPLQMAP <mapname> DESC <description> MANAGE TARGET CCD [CREATE SQL REGISTRATION|DROP SQL REGISTRATION|ALTER SQL REGISTRATION FOR Q REPLICATION] USING OPTIONS [other-opt-clause|add-cols-clause] other-opt-clause: SEARCH CONDITION "<search_condition>" ALL CHANGED ROWS [N|Y] HAS LOAD PHASE-- [N|I|E] SUPPRESS DELETES [N|Y] CONFLICT ACTION [I|F|D|S|Q] ERROR ACTION [S|D|Q] OKSQLSTATES "<sqlstates>" LOAD TYPE [0|1|2|3|4|104|5|105] add-cols-clause: ADD COLS (<trgcolname1> <srccolname1>,<trgcolname2> <srccolname2>) An example of altering a Q subscription to add a search condition is: ASNCLP SESSION SET TO Q REPLICATION; SET RUN SCRIPT NOW STOP ON SQL ERROR ON; SET CAPTURE SCHEMA SOURCE ASN; SET APPLY SCHEMA ASN; SET SERVER CAPTURE TO DB DB2A; SET SERVER TARGET TO DB DB2B; ALTER QSUB tab1 REPLQMAP rqma2b USING OPTIONS SEARCH CONDITION "WHERE :c1 > 1000" ; In multi-directional replication, the format of the command is: ALTER QSUB SUBTYPE B FROM NODE <svn.schema> SOURCE [src-clause] TARGET [trg-clause] FROM NODE <svn.schema> SOURCE [src-clause] TARGET [trg-clause] src-clause: ALL CHANGED ROWS [N/Y] HAS LOAD PHASE [N/I/E] trg-clause: CONFLICT RULE [K/C/A] +-' '-CONFLICT ACTION [I/F/D/S/Q] ERROR ACTION [Q/D/S] LOAD TYPE [0/2/3] OKSQLSTATES <"sqlstates"> If we are altering a Q subscription in a multi-directional environment, then we can use the SET REFERENCE TABLE construct. We need to specify the SUBTYPE parameter as follows: Bidirectional replication: ALTER QSUB SUBTYPE B Peer-to-peer replication: ALTER QSUB SUBTYPE P Let's look at a bidirectional replication example, where we want to change the ERROR ACTION to D for a Q subscription where the source table name is db2admin.dept. The content file (SYSA_cont_alter02.txt) will contain: SET SUBGROUP "TABT2"; SET SERVER MULTIDIR TO DB "DB2A"; SET SERVER MULTIDIR TO DB "DB2B"; SET REFERENCE TABLE USING SCHEMA "DB2A".ASN USES TABLE db2admin.dept; ALTER QSUB SUBTYPE B FROM NODE DB2A.ASN SOURCE TARGET ERROR ACTION D FROM NODE DB2B.ASN SOURCE TARGET ERROR ACTION D; We have to specify the SOURCE keyword even though we are only changing the target attributes. The ALTER QSUB statement spans the three last lines of the file. Starting a Q subscription An example of the ASNCLP command START QSUB to start a Q subscription can be found in the SYSA_qsub_start_db2ac.asnclp file. We just have to plug in the Q subscription name (T10002 in our example). ASNCLP SESSION SET TO Q REPLICATION; SET RUN SCRIPT NOW STOP ON SQL ERROR ON; SET CAPTURE SCHEMA SOURCE ASN; SET APPLY SCHEMA ASN; SET SERVER CAPTURE TO DB DB2A; SET SERVER TARGET TO DB DB2C; START QSUB SUBNAME T10002; Run the file as: asnclp -f SYSA_qsub_start_db2ac.asnclp We cannot put two START QSUB statements in the same file (as shown), even if they have their own section. So, we cannot code: ASNCLP SESSION SET TO Q REPLICATION; SET RUN SCRIPT NOW STOP ON SQL ERROR ON; SET SERVER CAPTURE TO DB DB2A; SET SERVER TARGET TO DB DB2D; SET CAPTURE SCHEMA SOURCE ASN; SET APPLY SCHEMA ASN; START QSUB SUBNAME T10003; SET SERVER CAPTURE TO DB DB2A; SET SERVER TARGET TO DB DB2C; START QSUB SUBNAME T10002; Sending a signal using ASNCLP For signals such as CAPSTART, CAPSTOP, and LOADDONE to be picked up, Q Capture needs to be running. Note that Q Capture does not have to be up for the signals to be issued, just picked up. As they are written to the DB2 log, Q Capture will see them when it reads the log and will action them in the order they were received. Summary In this article we took a look at how we can stop or drop or alter a Q subscription using ASNCLP commands and how we can issue a CAPSTART command. Further resources on this subject: Lotus Notes Domino 8: Upgrader's Guide [Book] Q Replication Components in IBM Replication Server [Article] IBM WebSphere MQ commands [Article] WebSphere MQ Sample Programs [Article] MQ Listener, Channel and Queue Management [Article]

Designing an API from scratch is a frustrating affair; there are a million best-practices to keep in mind, granularity to argue over, and the inherent struggle between consumer & developer. This article deals with all of the issues above, by concretely developing an API for a courier service–addressing the needs of the business, how they translate into data relationships & schema, and how they could/should be translated into consumable REST end-points. Make your life easy There is a shelf of software; take from it greedily! Instead of busting your hump to design your own API spec from scratch, make your life easy, and look to the world for already validated and accepted API specs. I advocate heavily for JSON:API, as I believe it offers the best combination of best-practices and granularity (I’d call it medium to medium-fine grained), thus we’ll be writing our API out according to the JSON:API spec. If you are a masochist and/or truly wish to design your own API spec then I would highly suggest wisely brushing up on the state-of-the-art, via the following articles: Using HTTP Methods for RESTful ServicesREST API Design - Resource ModelingCQRS summary Design step 1: determine business concerns Before we even begin coding, we need to identify our business concerns. These will lead us to our resources, and thus the resources our API consumers will want/need to deal with. It is very important that we do not confuse necessary business logic/resources with necessary consumer endpoints. As far as the API consumer is concerned, they should be given a tool to perform introspection on the state of their affairs, and no one else’s. This means that they should be able to command new deliveries and inspect the state of current & past deliveries; they should not have insight into any other aspects of our operation. Nouns and their actions Because we’re dealing with a courier service, try to estimate the consumer-facing nouns, and their relevant actions: Customers Command the delivery of a package, specifying origin and destination Couriers Receive and deliver packages Packages Get delivered from a sender to a recipient by a courier Senders Hand the package off to the Courier Recipients Receive the package from the Courier In performing this exercise, we’ve hopefully made clear the relationship between customers, couriers, packages, senders, and recipients. Not all nouns are equal You may have noticed that couriers, senders, and recipients appear to be cyclically linked. When relationships like this occur, it likely means that these nouns (pieces of data) do not deserve to be independent resources, but rather properties of another resource–specifically, packages. Be sure to keep an eye out for this, as you no doubt will be tempted to make all of your nouns resources (albeit heavily related ones), a practice that will bloat your API. Design step 2: formalize relationships Armed with your list of resources, nouns, and their actions, formalizing the relationship between resources should follow naturally. In our case, because we identified a cyclical relationship, we were able to consolidate our nouns into only two resources: customers and packages. The relationship between these (customers and packages) is thankfully quite obvious: one-to-many. That is because each customer can have any number of packages while a package may only belong to one customer. Internally, we’ve already collapsed the relationship between packages and couriers, senders, & recipients by making them simple properties–implicitly identifying these as one-to-one relationships. Design step 3: formalize the schema A dovetail to step 2, we now must concretely start listing the properties of each resource. customer: { id: "string", //UUID packages: ["string"] //Array of UUIDs } package: { id: "string", //UUID origin: "string", //Address destination: "string", //Address customer: "string", //UUID sender: "string", //Name recipient: "string", //Name courier: "string" //UUID } As we mentioned in step 2, the relationship between customers and packages is one-to-many–this is represented by the array of package ids–and each package has the consolidated nouns as properties. Design step 4: mock it out It is critical to use authorization tokens to avoid leaking package and customer information to the wrong parties. That said, for the case of this step, we are primarily concerned with mocking the API via JSON:API spec to see how it looks and feels. If you want a motivated example, just think of the case of whenever anyone queries http://example.com/customers. If the whole world has access to the collection of every customer you’ve got a huge problem. You can see an interactive example here–built off of a RAML file. Example: GET http://example.com/packages => { "links": { "self": "http://example.com/packages" }, "data": [{ "type": "packages", "id": "1", "origin": "1600 Pennsylvania Ave NW, Washington, DC 20500", "destination": "2 Lincoln Memorial Cir NW, Washington, DC 20037", "sender": "Barry Obama", "recipient": "Abraham Lincoln", "courier": "1", "links": { "self": "http://example.com/packages/1", "customer": { "self": "http://example.com/packages/1/links/customer", "related": "http://example.com/packages/1/customer", "linkage": { "type": "customers", "id": "1" } } } }, { "type": "packages", "id": "2", "origin": "437 N Wabash Ave., Chicago, 60611", "destination": "111 S Michigan Ave., Chicago, IL 60603", "sender": "Donald Trump", "recipient": "Marshal Fields", "courier": "1", "links": { "self": "http://example.com/packages/2", "customer": { "self": "http://example.com/packages/2/links/customer", "related": "http://example.com/packages/2/customer", "linkage": { "type": "customers", "id": "1" } } } }], "included": [{ "type": "customers", "id": "1", "links": { "self": "http://example.com/customers/1" } }] } GET http://example.com/customers => { "links": { "self": "http://example.com/customers" }, "data": [{ "type": "customers", "id": "1", "links": { "self": "http://example.com/customers/1", "packages": { "self": "http://example.com/customers/1/links/packages", "related": "http://example.com/customers/1/links/packages", "linkage": [ {"type": "packages", "id": 1}, {"type": "packages", "id": 2} ] } } }], "included": [{ "type": "packages", "id": "1", "origin": "1600 Pennsylvania Ave NW, Washington, DC 20500", "destination": "2 Lincoln Memorial Cir NW, Washington, DC 20037", "sender": "Barry Obama", "recipient": "Abraham Lincoln", "courier": "1", "links": { "self": "http://example.com/packages/1" } }, { "type": "packages", "id": "2", "origin": "437 N Wabash Ave., Chicago, 60611", "destination": "111 S Michigan Ave., Chicago, IL 60603", "sender": "Donald Trump", "recipient": "Marshal Fields", "courier": "1", "links": { "self": "http://example.com/packages/2" } }] } Yes, these responses seem fairly verbose, but they are designed to lower the number of necessary requests per interaction–making your server gather the necessary data you would have called for anyway, in anticipation of those requests rather than just in time. Believe me, after a few minutes of reading the JSON, you’re visceral reaction won’t lean so strongly towards disgust. Design step 5: get feedback Once you have a mock written out that people can try for themselves, like the interactive example in step 4, it’s time for you to ask the hard question: what are the pain points? Warning: unless you are the sole consumer of your API, try to avoid making changes that conflict with the JSON:API spec. Remember the whole point of using the spec was so to decrease your design overhead, and the consumer’s learning curve. Design step 6: finalize with tests While you are waiting for feedback, you should start writing tests. If you spec’d your API with something like RAML or API Blueprint it’s really easy to use your spec files to auto-generate mock API servers–so testing is a breeze. Conclusion By following the JSON:API spec we do away with a lot of the head scratching and frustration typically involved in designing an API. What’s left behind is simply the tedious task of implementation, and while that’s not a terribly exciting conclusion, I’ll take that over 10 hours of meetings on whether to use PUT or PATCH, and which color to paint the bike shed. And who knows? Maybe with your new-found time, you’ll write a JSON:API generator (connected to your favorite frame work) so that next time, you won’t even have to think about implementation either! About the author Jonathan Pollack is a full stack developer living in Berlin. He previously worked as a web developer at a public shoe company, and prior to that, worked at a start up that’s trying to build the world’s best pan-cloud virtualization layer. He can be found on Twitter @murphydanger.

In this article by Cyrus Dasadia, author of the book, MongoDB Cookbook Second Edition, we will cover the following recipes: Executing our first sample MapReduce job using the mongo-hadoop connector Writing our first Hadoop MapReduce job (For more resources related to this topic, see here.) Hadoop is a well-known open source software for the processing of large datasets. It also has an API for the MapReduce programming model, which is widely used. Nearly all the big data solutions have some sort of support to integrate them with Hadoop in order to use its MapReduce framework. MongoDB too has a connector that integrates with Hadoop and lets us write MapReduce jobs using the Hadoop MapReduce API, process the data residing in the MongoDB/MongoDB dumps, and write the result back to the MongoDB/MongoDB dump files. In this article, we will be looking at some recipes around the basic MongoDB and Hadoop integration. Executing our first sample MapReduce job using the mongo-hadoop connector In this recipe, we will see how to build the Mongo-Hadoop connector from the source and set up Hadoop just for the purpose of running the examples in a standalone mode. The connector is the backbone that runs Hadoop MapReduce jobs on Hadoop using the data in Mongo. Getting ready There are various distributions of Hadoop; however, we will use Apache Hadoop (http://hadoop.apache.org/). The installation will be done on Ubuntu Linux. For production, Apache Hadoop always runs on the Linux environment and Windows is not tested for production systems. For development purposes, however, Windows can be used. If you are a Windows user, I would recommend installing a virtualization environment such as VirtualBox (https://www.virtualbox.org/), set up a Linux environment, and then install Hadoop on it. Setting up VirtualBox and Linux on it is not shown in this recipe, but this is not a tedious task. The prerequisite for this recipe is a machine with a Linux operating system on it and an Internet connection. The version that we will set up here is 2.4.0 of Apache Hadoop. The latest version of Apache Hadoop that is supported by the mongo-hadoop connector is 2.4.0. A Git client is needed to clone the repository of the mongo-hadoop connector to the local filesystem. Refer to http://git-scm.com/book/en/Getting-Started-Installing-Git to install Git. You will also need MongoDB to be installed on your operating system. Refer to http://docs.mongodb.org/manual/installation/ and install it accordingly. Start the mongod instance listening to port 27017. It is not expected for you to be an expert in Hadoop but some familiarity with it will be helpful. Knowing the concept of MapReduce is important and knowing about the Hadoop MapReduce API will be an advantage. In this recipe, we will be explaining what is needed to get the work done. You can get more details on Hadoop and its MapReduce API from other sources. The Wikipedia page, http://en.wikipedia.org/wiki/MapReduce, gives good enough information about the MapReduce programming. How to do it… We will first install Java, Hadoop, and the required packages. We will start with installing JDK on the operating system.  Type the following on the command prompt of the operating system: $ javac –version If the program doesn't execute and you are told about the various packages that contain javac and program, then we need to install them as follows: $ sudo apt-get install default-jdk This is all we need to do to install Java Visit the URL, http://www.apache.org/dyn/closer.cgi/hadoop/common/, and download version 2.4 (or the latest mongo-hadoop connector supports). After the .tar.gz file has been downloaded, execute the following on the command prompt: $ tar –xvzf <name of the downloaded .tar.gz file> $ cd <extracted directory> Open the etc/hadoop/hadoop-env.sh file and replace export JAVA_HOME = ${JAVA_HOME} with export JAVA_HOME = /usr/lib/jvm/default-java. We will now get the mongo-hadoop connector code from GitHub on our local filesystem. Note that you don't need a GitHub account to clone a repository. Clone the git project from the operating system command prompt as follows: $git clone https://github.com/mongodb/mongo-hadoop.git $cd mongo-hadoop Create a soft link; the Hadoop installation directory is the same as the one that we extracted in step 3: $ln –s <hadoop installation directory> ~/hadoop-binaries For example, if Hadoop is extracted/installed in the home directory, then this is the command to be executed: $ln –s ~/hadoop-2.4.0 ~/hadoop-binaries By default, the mongo-hadoop connector will look for a Hadoop distribution in the ~/hadoop-binaries folder. So, even if the Hadoop archive is extracted elsewhere, we can create a soft link to it. Once this link is created, we should have the Hadoop binaries in the ~/hadoop-binaries/hadoop-2.4.0/bin path. We will now build the mongo-hadoop connector from the source for the Apache Hadoop version 2.4.0. The build-by-default builds for the latest version, so as of now, the -Phadoop_version parameter can be left out as 2.4 is the latest anyways. $./gradlew jar –Phadoop_version='2.4' This build process would take some time to get completed. Once the build is completed successfully, we are ready to execute our first MapReduce job. We will be doing it using a treasuryYield sample provided with the mongo-hadoop connector project. The first activity is to import the data to a collection in Mongo. Assuming that the mongod instance is up and running and listening to port 27017 for connections and the current directory is the root of the mongo-hadoop connector code base, execute the following command: $ mongoimport -c yield_historical.in -d mongo_hadoop --drop examples/treasury_yield/src/main/resources/yield_historical_in.json Once the import action is successful, we are left with copying two JAR files to the lib directory. Execute the following from the operating system shell: $ wget http://repo1.maven.org/maven2/org/mongodb/mongo-java-driver/2.12.0/mongo-java-driver-2.12.0.jar $ cp core/build/libs/mongo-hadoop-core-1.2.1-SNAPSHOT-hadoop_2.4.jar ~/hadoop-binaries/hadoop-2.4.0/lib/ $ mv mongo-java-driver-2.12.0.jar ~/hadoop-binaries/hadoop-2.4.0/lib The jar built for the mongo-hadoop core to be copied was named as above for the trunk version of the code and built for hadoop-2.4.0. Change the name of the JAR accordingly when you build it yourselves for a different version of the connector and Hadoop. The Mongo driver can be the latest version. The version 2.12.0 is the latest version. Now, execute the following command on the command prompt of the operating system shell:  ~/hadoop-binaries/hadoop-2.4.0/bin/hadoop     jar     examples/treasury_yield/build/libs/treasury_yield-1.2.1-SNAPSHOT-hadoop_2.4.jar  com.mongodb.hadoop.examples.treasury.TreasuryYieldXMLConfig  -Dmongo.input.split_size=8     -Dmongo.job.verbose=true  -Dmongo.input.uri=mongodb://localhost:27017/mongo_hadoop.yield_historical.in  -Dmongo.output.uri=mongodb://localhost:27017/mongo_hadoop.yield_historical.out The output should print out a lot of things; however, the following line in the output should tell us that the MapReduce job is successful:  14/05/11 21:38:54 INFO mapreduce.Job: Job job_local1226390512_0001 completed successfully Connect the mongod instance running on localhost from the mongo client and execute a find on the following collection: $ mongo > use mongo_hadoop switched to db mongo_hadoop > db.yield_historical.out.find() How it works… Installing Hadoop is not a trivial task and we don't need to get into this to try our samples for the mongo-hadoop connector. To learn about Hadoop and its installation, there are dedicated books and articles available. For the purpose of this article, we will simply be downloading the archive and extracting and running the MapReduce jobs in a standalone mode. This is the quickest way to get going with Hadoop. All the steps up to step 6 are needed to install Hadoop. In the next couple of steps, we will simply clone the mongo-hadoop connector recipe. You can also download a stable, built version for your version of Hadoop from https://github.com/mongodb/mongo-hadoop/releases if you prefer not to build from the source. We will then build the connector for our version of Hadoop (2.4.0) till step 13. From step 14 onwards is what we will do to run the actual MapReduce job in order to work on the data in MongoDB. We imported the data to the yield_historical.in collection, which would be used as an input to the MapReduce job. Go ahead and query the collection from the Mongo shell using the mongo_hadoop database to see a document. Don't worry if you don't understand the contents; we want to see in this example what we intend to do with this data. The next step was to invoke the MapReduce operation on the data. The hadoop command was executed giving one jar's path, (examples/treasury_yield/build/libs/treasury_yield-1.2.1-SNAPSHOT-hadoop_2.4.jar). This is the jar that contains the classes implementing a sample MapReduce operation for the treasury yield. The com.mongodb.hadoop.examples.treasury.TreasuryYieldXMLConfig class in this JAR file is the bootstrap class containing the main method. We will visit this class soon. There are lots of configurations supported by the connector.For now, we will just remember that mongo.input.uri and mongo.output.uri are the collections for the input and output for the MapReduce operations. With the project cloned, you can import it to any Java IDE of your choice. We are particularly interested in the project at /examples/treasury_yield and the core present in the root of the cloned repository. Let's look at the com.mongodb.hadoop.examples.treasury.TreasuryYieldXMLConfig class. This is the entry point to the MapReduce method and has a main method in it. To write MapReduce jobs for mongo using the mongo-hadoop connector, the main class always has to extend from com.mongodb.hadoop.util.MongoTool. This class implements the org.apache.hadoop.Tool interface, which has the run method and is implemented for us by the MongoTool class. All that the main method needs to do is execute this class using the org.apache.hadoop.util.ToolRunner class by invoking its static run method passing the instance of our main class (which is an instance of Tool). There is a static block that loads the configurations from two XML files, hadoop-local.xml and mongo-defaults.xml. The format of these files (or any XML file) is as follows. The root node of the file is the configuration node and multiple property nodes under it. <configuration>   <property>     <name>{property name}</name>     <value>{property value}</value>   </property>   ... </configuration> The property values that make sense in this context are all those that we mentioned in the URL provided earlier. We instantiate com.mongodb.hadoop.MongoConfig wrapping an instance of org.apache.hadoop.conf.Configuration in the constructor of the bootstrap class, TreasuryYieldXmlConfig. The MongoConfig class provides sensible defaults that are enough to satisfy the majority of the use cases. Some of the most important things that we need to set in the MongoConfig instance are to set the output and input format, mapper and reducer classes, output key, and value of the mapper, output key, and reducer. The input format and output format will always be the com.mongodb.hadoop.-MongoInputFormat and com.mongodb.hadoop.MongoOutputFormat classes, which are provided by the mongo-hadoop connector library. For the mapper and reducer output key and its value, we have the org.apache.hadoop.io.Writable implementation. Refer to the Hadoop documentation for different types of writable implementations in the org.apache.hadoop.io package. Apart from these, the mongo-hadoop connector also provides us with some implementations in the com.mongodb.hadoop.io package. For the treasury yield example, we used the BSONWritable instance. These configurable values can either be provided in the XML file that we saw earlier or be programmatically set. Finally, we have the option to provide them as vm arguments that we did for mongo.input.uri and mongo.output.uri. These parameters can be provided either in the XML or invoked directly from the code on the MongoConfig instance; the two methods are setInputURI and setOutputURI, respectively. We will now look at the mapper and reducer class implementation. We will copy the important portion of the class here to analyze. Refer to the cloned project for the entire implementation. public class TreasuryYieldMapper     extends Mapper<Object, BSONObject, IntWritable, DoubleWritable> {       @Override     public void map(final Object pKey,                     final BSONObject pValue,                     final Context pContext)         throws IOException, InterruptedException {         final int year = ((Date) pValue.get("_id")).getYear() + 1900;         double bid10Year = ((Number) pValue.get("bc10Year")).doubleValue();         pContext.write(new IntWritable(year), new DoubleWritable(bid10Year));     } } Our mapper extends the org.apache.hadoop.mapreduce.Mapper class. The four generic parameters are for the key class, type of the input value, type of the output key, and output value. The body of the map method reads the _id value from the input document, which is the date and extracts the year out of it. Then, it gets the double value from the document for the bc10Year field and simply writes to the context key value pair, where the key is the year and the value is the double. The implementation here doesn't rely on the value of the pKey parameter passed, which can be used as the key instead of hardcoding the _id value in the implementation. This value is basically the same field that would be set using the mongo.input.key property in XML or using the MongoConfig.setInputKey method. If none is set, _id is anyways the default value. Let's look at the reducer implementation (with the logging statements removed): public class TreasuryYieldReducer     extends Reducer<IntWritable, DoubleWritable, IntWritable, BSONWritable> {       @Override     public void reduce(final IntWritable pKey, final Iterable<DoubleWritable> pValues, final Context pContext)         throws IOException, InterruptedException {         int count = 0;         double sum = 0;         for (final DoubleWritable value : pValues) {             sum += value.get();             count++;         }         final double avg = sum / count;         BasicBSONObject output = new BasicBSONObject();         output.put("count", count);         output.put("avg", avg);         output.put("sum", sum);         pContext.write(pKey, new BSONWritable(output));     } } This class extends from org.apache.hadoop.mapreduce.Reducer and has four generic parameters again for the input key, input value, output key, and output value. The input to the reducer is the output from the mapper, and if you notice carefully, the type of the first two generic parameters are the same as the last two generic parameters of the mapper that we saw earlier. The third and fourth parameters in this case are the type of the key and the value emitted from the reducer. The type of the value is BSONDocument, and thus we have BSONWritable as the type. We now have the reduce method that has two parameters: the first one is the key, which is same as the key emitted from the map function, and the second parameter is java.lang.Iterable of the values emitted for the same key. This is how standard MapReduce functions work. For instance, if the map function gave the following key value pairs, (1950, 10), (1960, 20), (1950, 20), (1950, 30), then reduce will be invoked with two unique keys, 1950 and 1960, and the values for the key 1950 will be an iterable with (10, 20, 30), where the value of 1960 will be an iterable of a single element, (20). The reducer's reduce function simply iterates though this iterable of doubles, finds the sum and count of these numbers, and writes one key value pair, where the key is the same as the incoming key and the output value is BasicBSONObject with the sum, count, and average in it for the computed values. There are some good samples, including the enron dataset, in the examples of the cloned mongo-hadoop connector. If you would like to play around a bit, I would recommend that you to take a look at these example projects too and run them. There's more… What we saw here is a readymade sample that we executed. There is nothing like writing one MapReduce job ourselves for our understanding. In the next recipe, we will write one sample MapReduce job using the Hadoop API in Java and see it in action. See also If you're wondering what the writable interface is all about and why you should not use plain old serialization instead, then refer to this URL, which gives the explanation by the creator of Hadoop himself: http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg00378.html Writing our first Hadoop MapReduce job In this recipe, we will write our first MapReduce job using the Hadoop MapReduce API and run it using the mongo-hadoop connector getting the data from MongoDB. Getting ready Refer to the previous recipe, Executing our first sample MapReduce job using mongo-hadoop connector, for the setting up of the mongo-hadoop connector. This is a maven project and thus maven needs to be set up and installed. This project, however, is built on Ubuntu Linux and you need to execute the following command from the operating system shell to get maven: $ sudo apt-get install maven How to do it… We have a Java mongo-hadoop-mapreduce-test project that can be downloaded from the Packt site. We invoked that MapReduce job using the Python and Java client on previous occasions. With the current directory at the root of the project where the pom.xml file is present, execute the following command on the command prompt: $ mvn clean package The JAR file, mongo-hadoop-mapreduce-test-1.0.jar, will be built and kept in the target directory. With the assumption that the CSV file is already imported to the postalCodes collection, execute the following command with the current directory still at the root of the mongo-hadoop-mapreduce-test project that we just built: ~/hadoop-binaries/hadoop-2.4.0/bin/hadoop  jar target/mongo-hadoop-mapreduce-test-1.0.jar  com.packtpub.mongo.cookbook.TopStateMapReduceEntrypoint  -Dmongo.input.split_size=8 -Dmongo.job.verbose=true -Dmongo.input.uri=mongodb://localhost:27017/test.postalCodes -Dmongo.output.uri=mongodb://localhost:27017/test.postalCodesHadoopmrOut Once the MapReduce job is completed, open the Mongo shell by typing the following command on the operating system command prompt and execute the following query from the shell: $ mongo > db.postalCodesHadoopmrOut.find().sort({count:-1}).limit(5) Compare the output to the ones that we got earlier when we executed the MapReduce jobs using Mongo's MapReduce framework. How it works… We have kept the classes very simple and with the bare minimum things that we needed. We just have three classes in our project, TopStateMapReduceEntrypoint, TopStateReducer, and TopStatesMapper, all in the same com.packtpub.mongo.cookbook package. The mapper's map function just writes a key value pair to the context, where the key is the name of the state and value is an integer value, 1. The following is the code snippet from the mapper function: context.write(new Text((String)value.get("state")), new IntWritable(1)); What the reducer gets is the same key that is the list of states and an iterable of an integer value, 1. All we do is write the same name of the state and the sum of the iterables to the context. Now, as there is no size method in the iterable that can give the count in constant time, we are left with adding up all the ones that we get in the linear time. The following is the code in the reducer method: int sum = 0; for(IntWritable value : values) {   sum += value.get(); } BSONObject object = new BasicBSONObject(); object.put("count", sum); context.write(text, new BSONWritable(object)); We will write the text string that is the key and the value that is the JSON document containing the count to the context. The mongo-hadoop connector is then responsible for writing to the output collection that we have postalCodesHadoopmrOut, the document with the _id field same as the key emitted. Thus, when we execute the following, we get the top five states with the most number of cities in our database: > db. postalCodesHadoopmrOut.find().sort({count:-1}).limit(5) { "_id" : "Maharashtra", "count" : 6446 } { "_id" : "Kerala", "count" : 4684 } { "_id" : "Tamil Nadu", "count" : 3784 } { "_id" : "Andhra Pradesh", "count" : 3550 } { "_id" : "Karnataka", "count" : 3204 } Finally, the main method of the main entry point class is as follows: Configuration conf = new Configuration(); MongoConfig config = new MongoConfig(conf); config.setInputFormat(MongoInputFormat.class); config.setMapperOutputKey(Text.class); config.setMapperOutputValue(IntWritable.class); config.setMapper(TopStatesMapper.class); config.setOutputFormat(MongoOutputFormat.class); config.setOutputKey(Text.class); config.setOutputValue(BSONWritable.class); config.setReducer(TopStateReducer.class); ToolRunner.run(conf, new TopStateMapReduceEntrypoint(), args); All we do is wrap the org.apache.hadoop.conf.Configuration object with the com.mongodb.hadoop.MongoConfig instance to set the various properties and then submit the MapReduce job for the execution using ToolRunner. See also We executed a simple MapReduce job on Hadoop using the Hadoop API and sourcing the data from MongoDB and writing the data to the MongoDB collection in the recipe. What if we want to write the map and reduce the functions in a different language? Fortunately, this is possible by using a concept called Hadoop streaming, where stdout is used as a means to communicate between the program and the Hadoop MapReduce framework. Summary In this article, you learned about executing our first sample MapReduce job using the mongo-Hadoop connector and writing our first Hadoop MapReduce job. You can also refer to the following books related to MongoDB that are available on our website: MongoDB Cookbook: https://www.packtpub.com/big-data-and-business-intelligence/mongodb-cookbook Instant MongoDB: https://www.packtpub.com/big-data-and-business-intelligence/instant-mongodb-instant MongoDB High Availability: https://www.packtpub.com/big-data-and-business-intelligence/mongodb-high-availability Resources for Article: Further resources on this subject: About MongoDB [article] Ruby with MongoDB for Web Development [article] Sharding in Action [article]

(For more resources related to this topic, see here. Featured Image credit: Music Oomph website) The standard NAV 2013 distribution from Microsoft includes 21 different Role Center pages, identified for user roles such as Bookkeeper, Sales Manager, Shop Supervisor, Purchasing Agent, and so on. Some localized NAV distributions may have additional Role Center pages included. It is critical to realize that the Role Centers supplied out of the box are not generally intended to be used directly out of the box. According to the Help section, Working with Role Centers, only 3 of the 21 supplied Role Centers have been fully configured. These three are: 9004 — Bookkeeper 9005 — Sales Manager 9006 — Order Processor The 21 Role Centers should be used as templates or models for custom Role Centers tailored to the specific work role requirements of the individual customer implementation. One of the critical tasks of implementing a new system will be to analyze the workflow and responsibilities of the system's intended users and configure Role Centers to fit the users. In some cases, the supplied Role Centers can be used with minimal tailoring. Sometimes, it will be necessary to create complete new Role Centers. Even then, we will often be able to start with a copy of an existing Role Center Page, which we will modify as required. In any case, it is important to understand the structure of the Role Center Page and how it is built. Role Center structure The following screenshot shows Page 9006 — Order Processor Role Center: The components of the Role Center highlighted in the preceding screenshot are: Action Ribbon Navigation Pane Activity Pane Cue Group Actions (in Cue Groups) Cues (in Cue Groups) Page Parts User's Outlook System Parts A general representation of the structure of a Role Center Page is shown in the following outline: We need to understand the construction of a Role Center page so that we are prepared to modify an existing Role Center or create a new one. First we'll take a look at Page 9006 – Order Processor Role Center in the Page Designer. The Role Center page layout should look familiar, because it's very similar in structure to the pages we've designed previously. What's specific to a Role Center page? There is a Container control of the RoleCenterArea SubType. This is required for a Role Center page. There are two Group Controls which represent the two columns (left and right) of the Role Center page display. Each group contains several parts which show up individually in the Role Center display. Role Center page properties are accessed by highlighting the first blank line on the Page Designer form (the line below all of the defined controls), then clicking on the Properties icon, or we could right—click and choose the Properties option, or click on View | Properties or press Shift + F4. Note that the PageType is RoleCenter, and there is no Source Table . Role Center activities page Since the Group Control has no underlying code or settings, we'll examine the first Part Control's properties. The PagePartId property is SO Processor Activities . Cue Groups and Cues Now we'll focus on Page 9060 — SO Processor Activities. Designing that page, we see the following layout. Comparing the controls we see here to those of the Role Center, we can see this Page Part is the source of the Activities section of the Role Center Page. There are three CueGroup Controls – For Release, Sales Orders Released Not Shipped and Returns. In each CueGroup, there are the Field Controls for the individual Cues. An individual Cue is displayed as an iconic shortcut to a filtered list. The size of the stack of papers in the Cue icon represents the number of records in that list. The actual number of entries is also displayed as part of the icon (see the Sales Orders — Open example in following screenshot). The purpose of a Cue is to provide a focus on and single click access to a specific user task. The set of Cues is intended to represent the full set of primary activities for a user, based on their work Role. Cue source table In the Properties of the SO Processor Activities page, we see this is a PageType of CardPart tied to SourceTable Sales Cue . Next we want to Design the referenced table, Sales Cue, to see how it is constructed. There we see a simply structured table, with an integer field for each of the Cues that were displayed in the Role Center we are analyzing. There is also a key field and two fields identified as Date Filters. When we display the properties of one of these integer fields, Sales Orders — Open, we find it is a FlowField providing a Count of the Sales Orders with a Status of Open: If we inspect each of the other integer fields in this table, we will find a similar FlowField setup. Each is defined to fit the specific Cue to which it's tied. If we think about what the Cues show (a count) and how FlowFields work (a calculation based on a range of data records), we can see this is a simple, direct method of providing the information necessary to support Cue displays. Clicking on the Cue (the count) then opens up the list of records being counted. The following screenshot shows the list of Cue tables. Each of the Cue tables contains a series of FlowFields that support a set of Cues. Obviously, some Cue tables service more than one of the Role Center pages. Cue Group Actions Another set of Role Center page components to analyze are the Cue Group Actions . While the Cues are the primary tasks that are presented to the user, the Cue Group Actions are a related secondary set of tasks displayed to the right of the Cues. Cue Group Actions are defined in the Role Center essentially the same way as Actions are defined in other page types. As the name implies, Cue Group Actions are associated with a Control with the SubType CueGroup . If we right— click on the CueGroup Control, one of the options available is Control Actions (as shown in the following screenshot): When we choose Control Actions , the Action Designer form will be displayed showing the two CueGroup actions in the For Release CueGroup in the SO Processor Role Center page. If we open the Properties window we will see the "New" functionality is accomplished by setting the RunPageMode property to Create.

Opening the RadRails Views Some of the views that we will go through in this article are available as part of the Rails default perspective, which means you don't need to do anything special to open them; they will appear as tabbed views in a pane at the bottom of your workbench. Just look for the tab name of the view you want to see and click on it to make it visible. However, there are some views that are not opened by default, or maybe you closed them at some point accidentally, or maybe you changed to the Debug perspective and you want to display some of the RadRails views there. When you need to open a view whose tab is not displaying, you can go to the Window menu, and select the Show View option. If you are in the Rails perspective, all the available views will be displayed in that menu, as you can see in the screenshot above. When opening this menu from a different perspective, you will not see the RadRails views here, but you can select Other.... If this is the case, in the Show View dialog, most of the views will appear under the Ruby category, except for the Generators, Rails API, and Rake Tasks views, which are located under Rails. Documentation Views As happens with any modern programming language, Ruby has an extensive API. There are lots of libraries and classes and even with Ruby being an intuitive language with a neat consistent API, often we need to read the documentation. As you probably know, Ruby provides a standard documentation format called RDoc, which uses the comments in the source code to generate documentation. We can access this RDoc documentation in different ways, mainly in HTML format through a browser or by using the command-line tool RI. This produces a plain-text output directly at the command shell, in a similar way to the man command in a UNIX system. RadRails doesn't add any new functionality to the built-in documentation, but provides some convenient views so we can explore it without losing the context of our project's source. Ruby Interactive (RI) View This view provides a fast and comfortable way of browsing the local documentation in the same way as you would use RI from the command line. You can look either for a class or a method name. Just start typing at the input box at the top left corner of the view and the list below will display the matching entries. That's a nice improvement over the command line interface, since you can see the results as you type instead of having to run a complete search every time. If you know the name of both the class and the method you are looking for, then you can write them using the hash (pound) sign as a separator. For example, to get the documentation for the sum method of the class Enumerable you would write Enumerable#sum. The documentation will display in the right pane, with a convenient highlighting of the referenced methods and classes. Even if the search results of RI don't look very attractive compared to the output of the HTML-based documentation views, RI has the advantage of searching locally on your computer, so you can use it even when working off-line. Ruby Core, Ruby Standard Library, and Rails API There are three more views related to documentation in RadRails: Ruby Core API, Ruby Standard Library API, and Rails API. Unlike the RI view, these ones look for the information over the Internet, so you will not be able to use them unless you are on-line. On the other hand, the information is displayed in a more attractive way than with RI, and it provides links to the source code of the consulted methods, so if the documentation is not enough, you can always take a look at the inner details of the implementation. The Ruby Core API view displays the documentation of the classes included in Ruby's core. These are the classes you can directly use without a previous require statement. The documentation rendered is that at http://www.ruby-doc.org/core/. You are probably familiar with this type of layout, since it's the default RDoc output. The upper pane displays the navigation links, and the lower pane shows the detail of the documentation. The navigation is divided into three frames. The one to the left shows the files in which the source code is, the one in the middle shows the Classes and Modules, and in the third one you can find all the methods in the API. The Ruby Standard Library API is composed of all the classes and modules that are not a part of Ruby's core, but are typically distributed as a part of the Ruby installation. You can directly use these classes after a require statement in your code. The Ruby Standard Library API View displays the information from http://www.ruby-doc.org/stdlib. In this case, the navigation is the same as in Ruby Core, but with an additional area to the left, in which you can see all the available packages (the ones you would require for using the classes within your code). When you select a package link, you will see the files, classes, and methods for that single package. The last of the documentation views displays information about the Rails API. It includes the documentation of ActiveRecord, the ActionPack, ActiveSupport, and the rest of the Rails components. The information is obtained from http://api.rubyonrails.org. In this case the layout is slightly different because the information about the files, classes, and methods is displayed to the left instead at the top of the view. Apart from that, the behavior is identical to that of the Ruby Core API view. Since some of the API descriptions are fairly long, it can be convenient to maximize the documentation views when you are using them. Remember you can maximize any of the views by double-clicking its tab or by using the maximize icon on the view's toolbar. Double-clicking again will restore the view to the original size and position.

With DockerTM, containers are becoming mainstream and enterprises are ready to use them. Docker and its ecosystem are evolving at a very high pace, so it is very important to understand the basics and build group up to adopt to new concepts and tools. In this article, we will cover the following recipes: Setting up a Project Atomic host Doing atomic update/rollback with Project Atomic (For more resources related to this topic, see here.) Setting up a Project Atomic host Project Atomic facilitates application-centric IT architecture by providing an end-to-end solution to deploy containerized applications quickly and reliably, with atomic update and rollback for the application and host alike. This is achieved by running applications in containers on a Project Atomic host, which is a lightweight operating system specially designed to run containers. The hosts can be based on Fedora, CentOS, or Red Hat Enterprise Linux. Next, we will elaborate on the building blocks of the Project Atomic host. OSTree and rpm-OSTree OSTree (https://wiki.gnome.org/action/show/Projects/OSTree) is a tool to manage bootable, immutable, and versioned filesystem trees. Using this, we can build client-server architecture in which the server hosts an OSTree repository and the client subscribed to it can incrementally replicate the content. rpm-OSTree is a system to decompose RPMs on the server side into the OSTree repository to which the client can subscribe and perform updates. With each update, a new root is created, which is used for the next reboot. During updates, /etc is rebased and /var is untouched. Container runtime As of now Project Atomic only supports Docker as container runtime. systemd Project Atomic uses Kubernetes (http://kubernetes.io/) for application deployment over clusters of container hosts. Project Atomic can be installed on bare metal, cloud providers, VMs, and so on. In this recipe, let’s see how we can install it on a VM using virt-manager on Fedora. Getting ready Download the image: $ wget http://download.fedoraproject.org/pub/fedora/linux/releases/test/22_Beta/Cloud/x86_64/Images/Fedora-Cloud-Atomic-22_Beta-20150415.x86_64.raw.xz I have downloaded the beta image for Fedora 22 Cloud image For Containers. You should look for the latest cloud image For Containers at https://getfedora.org/en/cloud/download/. Uncompress this image by using the following command: $ xz -d Fedora-Cloud-Atomic-22_Beta-20150415.x86_64.raw.xz How to do it… We downloaded the cloud image that does not have any password set for the default user fedora. While booting the VM, we have to provide a cloud configuration file through which we can customize the VM. To do this, we need to create two files, meta-data and user-data, as follows: $ cat meta-data instance-id: iid-local01 local-hostname: atomichost $ cat user-data #cloud-config password: atomic ssh_pwauth: True chpasswd: { expire: False } ssh_authorized_keys: - ssh-rsa AAAAB3NzaC1yc......... In the preceding code, we need to provide the complete SSH public key. We then need to create an ISO image consisting of these files, which we will use to boot to the VM. As we are using a cloud image, our setting will be applied to the VM during the boot process. This means the hostname will be set to atomichost, the password will be set to atomic, and so on. To create the ISO, run the following command: $ genisoimage -output init.iso -volid cidata -joliet -rock user-data meta-data Start virt-manager. Select New Virtual Machine and then import the existing disk image. Enter the image path of the Project Atomic image we downloaded earlier. Select OS type as Linux and Version as Fedora 20/Fedora 21 (or later), and click on Forward. Next, assign CPU and Memory and click on Forward. Then, give a name to the VM and select Customize configuration before install. Finally, click on Finish and review the details. Next, click on Add Hardware, and after selecting Storage, attach the ISO (init.iso) file we created to the VM and select Begin Installation: Once booted, you can see that its hostname is correctly set and you will be able to log in through the password given in the cloud init file. The default user is fedora and password is atomic as set in the user-data file. How it works… In this recipe, we took a Project Atomic Fedora cloud image and booted it using virt-manager after supplying the cloud init file. There’s more… After logging in, if you do file listing at /, you will see that most of the traditional directories are linked to /var because it is preserved across upgrades. After logging in, you can run the Docker command as usual $sudo docker run -it fedora bash See also The virtual manager documentation at https://virt-manager.org/documentation/ More information on package systems, image systems, and RPM-OSTree at https://github.com/projectatomic/rpm-ostree/blob/master/doc/background.md The quick-start guide on the Project Atomic website at http://www.projectatomic.io/docs/quickstart/ The resources on cloud images at https://www.technovelty.org//linux/running-cloud-images-locally.html and http://cloudinit.readthedocs.org/en/latest/ How to set up Kubernetes with an Atomic host at http://www.projectatomic.io/blog/2014/11/testing-kubernetes-with-an-atomic-host/ and https://github.com/cgwalters/vagrant-atomic-cluster Doing atomic update/rollback with Project Atomic To get to the latest version or to roll back to the older version of Project Atomic, we use the atomic host command, which internally calls rpm-ostree. Getting ready Boot and log in to the Atomic host. How to do it… Just after the boot, run the following command: $ atomic host status You will see details about one deployment that is in use now. To upgrade, run the following command: This changes and/or adds new packages. After the upgrade, we will need to reboot the system to use the new update. Let’s reboot and see the outcome: As we can see, the system is now booted with the new update. The *, which is at the beginning of the first line, specifies the active build. To roll back, run the following command: $ sudo atomic host rollback We will have to reboot again if we want to use older bits. How it works… For updates, the Atomic host connects to the remote repository hosting the newer build, which is downloaded and used from the next reboot onwards until the user upgrades or rolls back. In the case rollback older build available on the system used after the reboot. See also The documentation Project Atomic website, which can be found at http://www.projectatomic.io/docs/os-updates/ Summary Docker and its ecosystem are evolving at a very high pace, so it is very important to understand the basics and build group up to adopt to new concepts and tools. Additional information on Docker can be gained be referring to: https://www.packtpub.com/networking-and-servers/docker-high-performance https://www.packtpub.com/virtualization-and-cloud/monitoring-docker Resources for Article: Further resources on this subject: Understanding Docker [article] Hands On with Docker Swarm [article] Introduction to Docker [article]

Using Multiple Templates Most advanced sites built today employ multiple page templates. In this section, we will look at the most common scenarios and how to address them with a PHPTemplate theme. While there are many good reasons for running multiple page templates, you should not create additional templates solely for the purpose of disabling regions to hide blocks. While the approach will work, it will result in a performance hit for the site, as the system will still produce the blocks, only to then wind up not displaying them for the pages. The better practice is to control your block visibility. Using a Separate Admin Theme With the arrival of Drupal 5, one of the most common Drupal user requests was satisfied; that is, the ability to easily designate a separate admin theme. In Drupal, designating a separate theme for your admin interface remains a simple matter that you can handle directly from within the admin system. To designate a separate theme for your admin section, follow these steps: Log in and access your site's admin system. Go to Administer | Site configuration | Administration theme. Select the theme you desire from the drop-down box listing all the installed themes. Click Save configuration, and your selected theme should appear immediately. Multiple Page or Section Templates In contrast to the complete ease of setting up a separate administration theme is the comparative difficulty of setting up multiple templates for different pages or sections. The bad news is that there is no admin system shortcut—you must manually create the various templates and customize them to suit your needs. The good news is that creating and implementing additional templates is not difficult and it is possible to attain a high degree of granularity with the techniques described below. Indeed, should you be so inclined, you could literally define a distinct template for each individual page of your site. Drupal employs an order of precedence based on a naming convention (or "suggestions" as they are now being called on the Drupal site). You can unlock the granularity of the system through proper application of the naming convention. It is possible, for example, to associate templates with every element on the path, or with specific users, or with a particular functionality—all through the simple process of creating a new template and naming it appropriately. The system will search for alternative templates, preferring the specific to the general, and failing to find a more specific template, will apply the default page.tpl.php. Consider the following example of the order of precedence and the naming convention in action. The custom templates above could be used to override the default page.tpl.php and theme either an entire node (page-node.tpl.php), or simply the node with an ID of 1 (page-node-1.tpl.php),or the node in edit mode (page-node-edit.tpl.php), depending on the name given the template. In the example above, the page-node templates would be applied to the node in full page view. In contrast, should you wish to theme the node in its entirety, you would need to intercept and override the default node.tpl.php. The fundamental methodology of the system is to use the first template file it finds and ignore other, more general templates (if any). This basic principle, combined with proper naming of the templates, gives you control over the template used in various situations. The default suggestions provided by the Drupal system should be sufficient for the vast majority of theme developers. However, if you find that you need additional suggestions beyond those provided by the system, it is possible to extend your site and add new suggestions. See http://drupal.org/node/223440 for a discussion of this advanced Drupal theming technique. Let's take a series of four examples to show how this feature can be used to provide solutions to common problems: Create a unique homepage template. Use a different template for a group of pages. Assign a specific template to a specific page. Designate a specific template for a specific user. Create a Unique Homepage Template Let's assume that you wish to set up a unique template for the homepage of a site. Employing separate templates for the homepage and the interior pages is one of the most common requests web developers hear. With Drupal, you can, without having to create a new template, achieve some variety within a theme by controlling the visibility of blocks on the homepage. If that simple technique does not give you enough flexibility, you will need to consider using a dedicated template that is purpose-built for your homepage content. The easiest way to set up a distinct front page template is to copy the existing page.tpl.php file, rename it, and make your changes to the new file. Alternatively, you can create a new file from scratch. In either situation, your front-page-specific template must be named page-front.tpl.php. The system will automatically display your new file for the site's homepage, and use the default page.tpl.php for the rest of the site. Note that page-front.tpl.php is whatever page you specify as the site's front page via the site configuration settings. To override the default homepage setting visit Administer | Site configuration | Site information, then enter the URL you desire into the field labeled Default home page. Use a Different Template for a Group of Pages Next, let's associate a template with a group of pages. You can provide a template to be used by any distinct group of pages, using as your guide the path for the pages. For example, to theme all the user pages you would create the template page-user.tpl.php. To theme according to the type of content, you can associate your page template with a specific node, for example, all blog entry pages can be controlled by the filepage-blog-tpl.php. The table below presents a list of suggestions you can employ to theme various pages associated with the default functionalities in the Drupal system. Suggestion Affected Page page-user.tpl.php user pages page-blog.tpl.php blog pages (but not the individual node pages) page-forum.tpl.php forum pages (but not the individual node pages) page-book.tpl.php book pages (but not the individual node pages) page-contact.tpl.php contact form (but not the form content)   Assign a Specific Template to a Specific Page Taking this to its extreme, you can associate a specific template with a specific page. By way of example, assume we wish to provide a unique template for a specific content item. Let's assume our example page is located at http://www.demosite.com/node/2/edit. The path of this specific page gives you a number of options. We could theme this page with any of the following templates (in addition to the default page.tpl.php): page-node.tpl.php page-node-2.tpl.php page-node-edit.tpl.php A Note on Templates and URLsDrupal bases the template order of precedence on the default path generated by the system. If the site is using a module like pathauto, which alters the path that appears to site visitors, remember that your templates will still be displayed based on the original paths. The exception here being page-front.tpl.php, which will be applied to whatever page you specify as the site's front page via the site configuration settings (Administer | Site configuration| Site information). Designate a Specific Template for a Specific User Assume that you want to add a personalized theme for the user with the ID of 1(the Drupal equivalent of a Super Administrator). To do this, copy the existing page.tpl.php file, rename it to reflect its association with the specific user, and make any changes to the new file. To associate the new template file with the user, name the file: page-user-1.tpl. Now, when user 1 logs into the site, they will be presented with this template. Only user 1 will see this template and only when he or she is logged in and visiting the account page. The official Drupal site includes a collection of snippets relating to the creation of custom templates for user profile pages. The discussion is instructive and worth review, though you should always be a bit cautious with user-submitted code snippets as they are not official releases from the Drupal Association. See, http://drupal.org/node/35728 Dynamically Theming Page Elements In addition to being able to style particular pages or groups of pages, Drupal and PHPTemplate make it possible to provide specific styling for different page elements. Associating Elements with the Front Page Drupal provides $is_front as a means of determining whether the page currently displayed is the front page. $is_front is set to true if Drupal is rendering the front page; otherwise it is set to false. We can use $is_front in our page.tpl.php file to help toggle display of items we want to associate with the front page. To display an element on only the front page, make it conditional on the state of $is_front. For example, to display the site mission on only the front page of the site, wrap $mission (in your page.tpl.php file) as follows: <?php if ($is_front): ?> <div id="mission"> <?php print $mission; ?> </div><?php endif; ?> To set up an alternative condition, so that one element will appear on the front page but a different element will appear on other pages, modify the statement like this: <?php if ($is_front): ?> //whatever you want to display on front page<?php else: ?> //what is displayed when not on the front page<?php endif; ?> $is_front is one of the default baseline variables available to all templates.

  MooTools 1.3 Cookbook Over 110 highly effective recipes to turbo-charge the user interface of any web-enabled Internet application and web page MooTroduction MooTools was conceived by Valerio Proietti and copy written under MIT License in 2006. We send a great round of roaring applause to Valerio for creating the Moo.FX (My Object Oriented Effects) plugin for Prototype, a JavaScript abstraction library. That work gave life to an arguably more effects-oriented (and highly extensible) abstraction layer of its own: MooTools (My Object Oriented Tools).   Knowing our MooTools version This recipe is an introduction to the different MooTools versions and how to be sure we are coding in the right version. Getting ready Not all are equal nor are backwards compatible! The biggest switch in compatibility came between MooTools 1.1 and MooTools 1.2. This minor version change caused clamor in the community given the rather major changes included. In our experience, we find that 1.2 and 1.3 MooTool scripts play well together while 1.0 and 1.1 scripts tend to be agreeable as well. However, Moo's popularity spiked with version 1.1, and well-used scripts written with 1.0, like MooTabs, were upgraded to 1.1 when released. The exact note in Google Libraries for the version difference between 1.1 and 1.2 reads: Since 1.1 versions are not compatible with 1.2 versions, specifying version "1" will map to the latest 1.1 version (currently 1.1.2). MooTools 1.1.1 has inline comments, which cause the uncompressed version to be about 180% larger than version 1.2.5 and 130% larger than the 1.3.0 release. When compressed, with YUI compression, 1.1 and 1.2 weigh in at about 65K while 1.3.0 with the CSS3 selectors is a modest 85K. In the code snippets, the compressed versions are denoted with a c.js file ending. Two great additions in 1.3.0 that account for most of the difference in size from 1.2.5 are Slick.Parser and Slick.Finder. We may not need CSS3 parsing; so we may download the MooTools Core with only the particular class or classes we need. Browse http://mootools.net/core/ and pick and choose the classes needed for the project. We should note that the best practice is to download all modules during development and pare down to what is needed when taking an application into production. When we are more concerned with functionality than we are with performance and have routines that require backwards compatibility with MooTools 1.1, we can download the 1.2.5 version with the 1.1 classes from the MooTools download page at http://mootools.net/download. The latest MooTools version as of authoring is 1.3.0. All scripts within this cookbook are built and tested using MooTools version 1.3.0 as hosted by Google Libraries. How to do it... This is the basic HTML framework within which all recipes will be launched. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html > <head> <title>MooTools Recipes</title> <meta http-equiv="content-type" content="text/html;charset=utf-8"/> Note that the portion above is necessary but is not included in the other recipes to save space. Please do always include a DOCTYPE, and opening HTML, HEAD, TITLE, and META tag for the HTTP-EQUIV and CONTENT. <script type="text/javascript" src="mootools-1.3.0.js"></script> </head> <body> <noscript>Your Browser has JavaScript Disabled. Please use industry best practices for coding in JavaScript; letting users know they are missing out is crucial!</noscript> <script type="text/javascript"> // best practice: ALWAYS include a NOSCRIPT tag! var mooversion = MooTools.version; var msg = 'version: '+mooversion; document.write(msg); // just for fun: var question = 'Use MooTools version '+msg+'?'; var yes = 'It is as you have requested!'; var no = "Please change the mootools source attribute in HTML->head->script."; // give 'em ham alert((confirm(question)?yes:no)); </script> </body> </html> How it works... Inclusion of external libraries like MooTools is usually handled within the HEAD element of the HTML document. The NOSCRIPT tag will only be read by browsers that have their JavaScript disabled. The SCRIPT tag may be placed directly within the layout of the page. There's more... Using the XHTML doctype (or any doctype for that matter) allows your HTML to validate, helps browsers parse your pages faster, and helps the Dynamic Object Model (DOM) behave consistently. When our HTML does not validate, our JavaScript errors will be more random and difficult to solve. Many seasoned developers have settled upon a favorite doctype. This allows them to become familiar with the ad-nauseam of cross browser oddities associated with that particular doctype. To further delve into doctypes, quirksmode, and other HTML specification esoterica, the heavily trafficked http://www.quirksmode.org/css/quirksmode.html provides an easy-to-follow and complete discourse.   Finding MooTools documentation both new and old Browsing http://mootools.net/docs/core will afford us the opportunity to use the version of our choice. The 1.2/1.3 demonstrations at the time of writing are expanding nicely. Tabs in the demonstrations at http://mootools.net/demos display each of the important elements of the demonstration. (Move the mouse over the image to enlarge.) MooTools had a major split at the minor revision number of 1.1. If working on a legacy project that still implements the deprecated MooTools version 1.1, take a shortcut to http://docs111.mootools.net. Copying the demonstrations line-for-line, without studying them to see how they work, may afford our project the opportunity to become malicious code.   Using Google Library's MooTools scripts Let Google maintain the core files and provide the bandwidth to serve them. Getting ready Google is leading the way in helping MooTools developers save time in the arenas of development, maintenance, and hosting by working together with the MooTools developers to host and deliver compressed and uncompressed versions of MooTools to our website visitors. Hosting on their servers eliminates the resources required to host, bandwidth required to deliver, and developer time required to maintain the requested, fully patched, and up-to-date version. Usually we link to a minor version of a library to prevent major version changes that could cause unexpected behavior in our production code. Google API keys that are required in the documentation to use Google Library can be easily and quickly obtained at: http://code.google.com/apis/libraries/devguide.html#sign_up_for_an_api_key. How to do it... Once you have the API Key, use the script tag method to include MooTools. For more information on loading the JavaScript API see http://code.google.com/apis/libraries/devguide.html#load_the_javascript_api_and_ajax_search_module. <!--script type="text/javascript" src="mootools-1.3.0.js"> </script--> <!--we've got ours commented out so that we can use google's here:--> <script src="https://www.google.com/jsapi?key=OUR-KEY-HERE" type="text/javascript"></script> // the full src path is truncated for display here <script src="https://ajax.googleapis.com/... /mootools-yui-compressed.js" type="text/javascript"></script> </head> <body> <noscript>JavaScript is disabled.</noscript> <script type="text/javascript"> var mooversion = MooTools.version; var msg = 'MooTools version: '+mooversion+' from Google'; // show the msg in two different ways (just because) document.write(msg); alert(msg); </script> Using google.load(), which is available to us when we include the Google Library API, we can make the inclusion code a bit more readable. See the line below that includes the string jsapi?key=. We replace OUR-KEY-HERE with our API key, which is tied to our domain name so Google can contact us if they detect a problem: <!--script type="text/javascript" src="mootools-1.3.0.js"> </script--> <!--we've got ours commented out so that we can use google's here:--> <script src="https://www.google.com/jsapi?key=OUR-KEY-HERE" type="text/javascript"></script> <script type="text/javascript"> google.load("mootools", "1.2.5"); </script> </head> <body> <noscript>JavaScript is disabled.</noscript> <script type="text/javascript"> var mooversion = MooTools.version; var msg = 'MooTools version: '+mooversion+' from Google'; // show the msg in two different ways (just because) document.write(msg); alert(msg); </script> How it works... There are several competing factors that go into the decision to use a direct load or dynamic load via google.load(): Are we loading more than one library? Are our visitors using other sites that include this dynamic load? Can our page benefit from parallel loading? Do we need to provide a secure environment? There's more... If we are only loading one library, a direct load or local load will almost assuredly benchmark faster than a dynamic load. However, this can be untrue when browser accelerator techniques, most specifically browser caching, come into play. If our web server is sending no-cache headers, then dynamic load, or even direct load, as opposed to a local load, will allow the browser to cache the Google code and reduce our page load time. If our page is making a number of requests to our web server, it may be possible to have the browser waiting on a response from the server. In this instance, parallel loading from another website can allow those requests that the browser can handle in parallel to continue during such a delay. We need to also take a look at how secure websites function with non-secure, external includes. Many of us are familiar with the errors that can occur when a secure website is loaded with an external (or internal) resource that is not provided via http. The browser can pop up an alert message that can be very concerning and lose the confidence of our visitors. Also, it is common to have some sort of negative indicator in the address bar or in the status bar that alerts visitors that not all resources on the page are secure. Avoid mixing http and https resources; if using a secure site, opt for a local load of MooTools or use Google Library over HTTPS.  

Service hosting and configuration is very important for building WCF services, especially at the service deployment stage. After developers complete the service development, we will need to deploy the service so as to make it available to all the client consumers. In the real world, there are various service deployment scenarios available, which will result in different deployment and configuration requirements on the service configuration or the hosting environment. As an enhanced service development platform, WCF provides rich, built-in support on service hosting and configuration that can fulfill most of the existing deployment demands and requirements. For example, the most popular IIS hosting approach can provide high availability and stable service for local intranet or public internet-based deployment cases. The Windows service-hosting approach makes WCF service hosting easier to integrate with existing background scheduled tasks, and the self-hosting approach provides the most flexibility and customization points for service deployment in a production environment. In this article, we will look at seven recipes on various WCF hosting and configuration scenarios. The recipes start with four typical hosting cases—self-hosting, Windows service hosting, IIS-based HTTP hosting, and IIS based non-HTTP hosting. This is followed by two customized service-hosting cases—including a custom ServiceHostFactory and a dedicated singleton-instance hosting. The last recipe demonstrates a more advanced WCF service-hosting scenario—Windows SharePoint Service hosting. Hosting a service in a console application When creating a simple demo program for .NET framework, we will probably choose a console application. At the same, when talking about WCF service hosting, the console-hosting scenario is the most convenient one, which is especially handy and useful when we want to do some quick demo or testing on some WCF functionality. How to do it... Create a .NET framework-based Console project through Visual Studio. Visual Studio provides various project templates for creating a .NET framework-based application. For our sample console-hosting service here, we will choose the Console Application project type from the Visual Studio New Project wizard. Add a new WCF service into the project. We can simply accomplish this by using the Add New Item function in Visual Studio and choose WCF Service as the item type from Visual Studio's Add New Item UI. Add code into the Main function to start up the WCF service. The following code shows the typical Main function that starts up a WCF service in a console application: static void Main(string[] args) { using (ServiceHost consoleHost = new ServiceHost(typeof(TestService))) { consoleHost.Open(); Console.WriteLine("press any key to stop service host..."); Console.ReadLine(); } } How it works... When you add a new WCF Service item in Visual Studio, the IDE actually helps you to finish the following three tasks: Creating a ServiceContract interface. Creating a Service class that implements the ServiceContract interface. The following code shows the sample ServiceContract and implementation class used in this recipe. [ServiceContract] public interface ITestService { [OperationContract] void DoWork(); } public class TestService : ITestService { public void DoWork() { } } Adding the service endpoint and binding configuration in the App.config file. In addition to the Contract and service type, the IDE will also insert a default configuration setting for the endpoint that can be exposed through the service. The following screenshot shows the sample service configuration section that contains a single endpoint, which uses WSHttpBinding. With the code and configuration entries as defined previously, we can start our service host by supplying the service type in the constructor of the ServiceHost class. using (ServiceHost consoleHost = new ServiceHost(typeof(TestService))) What the runtime will do is, it will lookup the configuration file and load the <service> entry that has the name identical to the type specified in the constructor, and launch the service and endpoints defined in it. The source code for this article can be found at http://www.packtpub.com/code_download/6034 Hosting a service in Windows Service Windows Services are widely used on Windows operating systems for hosting applications that will perform some long-run or scheduled tasks in the background. Applications hosted via Windows Service can be running under a specific user account and can choose the startup mode (manually or automatically). As a popular service-application-hosting scenario, it is also quite common to deploy a WCF service as a Windows Service. How to do it... In this recipe, we will use a typical .NET-based Windows Service to demonstrate how to host a WCF service in a Windows Service application. Let's go through the detailed steps: Create a Windows Service project. The first step is to create a new Windows Service project through the Visual Studio IDE. When creating the project, we simply choose the Windows Service project type. The following screenshot shows how we can select the Windows Service project type in the Visual Studio New Project wizard. Add a new WCF service item. As a WCF service hosting application, we certainly need to have a WCF service defined here. The steps for creating a WCF service are the same as what we've discussed in the Hosting a service in console application recipe. Add service hosting code into the service startup and shutdown event. As for the service-hosting code in the Windows Service, we need to put it in the correct place, since the .NET-based Windows Service type doesn't directly expose the Main function. The following code shows how the WCF service startup and shutdown code is defined: public partial class Service1 : ServiceBase { ServiceHost _svcHost = null; protected override void OnStart(string[] args) { // Start the service host here _svcHost = new ServiceHost(typeof(TestService)); _svcHost.Open(); } protected override void OnStop() { // Close the service host _svcHost.Close(); } } Add an installer for the Windows Service. Now the Windows Service class and WCF service types have been defined. However, we still need to add another component—the installer class for deploying the Windows Service into the Windows Service collection on the target operating system. In the Visual Studio IDE, we can simply add an installer type for the Windows Service by the context menu on the component designer. The following screenshot shows the context menu item for creating the installer class for the Windows Service. The IDE will help create two helper classes—one is of ServiceProcessInstaller type and another of ServiceInstaller type. We can specify many deployment parameters for the Windows Service in the Property panel of the two classes. The following screenshot shows the properties of the sample serviceProcessInstaller1 class. The next screenshot shows the properties of the sample serviceInstaller1 class. As with the screenshots displayed, Visual Studio will use standard Properties windows for displaying and configuring the individual properties of the Windows Service classes. Install the Windows Service through Installutil.exe. The last step is to install the Windows Service we have created (after building the project) into the operating system. This can be done by using the Installutil.exe tool provided by the .NET framework. You can directly execute the Installutil.exe command within the Visual Studio command-line prompt window or you can choose to launch the tool through its absolute path in the .NET framework folder such as C: WindowsMicrosoft.NETFrameworkv4.0.30319. The following statements show the complete commands for installing and uninstalling a .NET-based Windows Service application via the Installutil. exe tool. Install the Windows Service: InstallUtil.exe WCFNTService.exe Uninstall the Windows Service: Install Util.exe /u WCFNTService.exe The WCFNTService.exe mentioned earlier is the output assembly name of the sample Windows Service project. How it works... The OnStart event is fired when the Windows Service is starting, while the OnStop event is fired when the Windows Service is shutting down. Therefore, they are the best places for us to put the WCF service-hosting code. Sometimes, we may need to access some remote or protected resource in our Windows Service host program. In such cases, it is important to specify a proper service account, either at development time or in the Windows Service Configuration Manager. The following screenshot shows the service list, which contains the installed sample Windows Service in Windows Service Configuration Manager.

This function is only available to administrators, so use this role prudently. During the installation of the Oracle VM Manager, a default admin account is created. And with this admin's account we can go about managing the users and groups. Managing Users Here it is possible to create new users, delete older or unwanted ones, assign different roles to those users, reset user password, and so on. Let's break it up into a few topics and have a look at it. Creating a User Viewing or editing details Changing a role Deleting a User Creating a User To create a User, perform the following: On the Administrator's page click User tab and then click on the Create button: Enter the necessary information such as: Username (avoid using user, manager, and administrator as username) Password Retype Password First Name Last Name Valid Email address We can select the account status, it could either be locked or unlocked and is only accessible when it's unlocked. We can lock an account for security reasons by using the status Locked. We can grant the following roles to this newly created user—User, Manager, or Administrator. Then select the Server Pools for this user and also select the group to which this user should belong to. Click on the Confirm button to confirm the information and we will get this information: As we can see in the preceding screenshot this is a plain user and has no groups or servers assigned to it. However this was unlocked and was granted a User role. Viewing or editing a User Now let's view the User we just created. Click on the User tab on the Administrator page: Click on the Show link to view the Server Pools that the user is allowed to use: We can now edit account details such as change email address, change account status, and so on. Let's change the User's email address: Modifying the account status to either locked or unlocked: Changing the role: Next, add the User to the Server Pool: Removing a User from groups or Server Pools: Changing a User's role Lets change regular Users' role to Administrator: On the Administrator's page, select the newly created User and click on the Edit button. Select the role and click Apply to effectively assign the role to the User: Once applied, we will be presented with the following screen: Deleting User To delete a User, we need to do the following on the Administrator's page. We can carry out a search and then select the User that we want to delete. Click on the Delete button and confirm the User you want to delete:

What is Direct? Part of the power of any client-side library is its ability to tap nearly any server-side technology. That said, with so many server-side options available there were many different implementations being written for accessing the data. Direct is a means of marshalling those server-side connections, creating a 'one-stop-shop' for handling your basic Create, Read, Update, and Delete actions against that remote data. Through some basic configuration, we can now easily create entire server-side API's that we may programmatically expose to our Ext JS applications. In the process, we end up with one set of consistent, predefined methods for managing that data access. Building server-side stacks There are several examples of server-side stacks already available for Ext JS, directly from their site's Direct information. These are examples, showing you how you might use Direct with a particular server-side technology, but Ext provides us with a specification so that we might write our own. Current stack examples are available for: PHP .NET Java ColdFusion Ruby Perl These are examples written directly by the Ext team, as guides, as to what we can do. Each of us writes applications differently, so it may be that our application requires a different way of handling things at the server level. The Direct specification, along with the examples, gives us the guideposts we need for writing our own stacks when necessary. We will deconstruct one such example here to help illustrate this point. Each server-side stack is made up of three basic components: Configuration— denoting which components/classes are available to Ext JS API— client-side descriptors of our configuration Router— a means to 'route' our requests to their proper API counterparts To illustrate each of these pieces of the server-side stack we will deconstruct one of the example stacks provided by the Ext JS team. I have chosen the ColdFusion stack because: It is a good example of using a metadata configuration DirectCFM (the ColdFusion stack example) was written by Aaron Conran, who is the Senior Software Architect and Ext Services Team Leader for Ext, LLC Each of the following sections will contain a "Stack Deconstruction" section to illustrate each of the concepts. These are to show you how these concepts might be written in a server-side language, but you are welcome to move on if you feel you have a good grasp of the material. Configuration Ultimately the configuration must define the classes/objects being accessed, the functions of those objects that can be called, and the length (number) of arguments that the method is expecting. Different servers will allow us to define our configuration in different ways. The method we choose will sometimes depend upon the capabilities or deficiencies of the platform we're coding to. Some platforms provide the ability to introspect components/classes at runtime to build configurations, while others require a far more manual approach. You can also include an optional formHandler attribute to your method definitions, if the method can take form submissions directly. There are four basic ways to write a configuration. Programmatic A programmatic configuration may be achieved by creating a simple API object of key/value pairs in the native language. A key/value pair object is known by many different names, depending upon the platform to which we're writing for: HashMap, Structure, Object, Dictionary, or an Associative Array. For example, in PHP you might write something like this: $API = array( 'Authors'=>array( 'methods'=>array( 'GetAll'=>array( 'len'=>0 ), 'add'=>array( 'len'=>1 ), 'update'=>array( 'len'=>1 ) ) ) ); Look familiar? It should, in some way, as it's very similar to a JavaScript object. The same basic structure is true for our next two methods of configuration as well. JSON and XML For this configuration, we can pass in a basic JSON configuration of our API: { Authors:{ methods:{ GetAll:{ len:0 }, add:{ len:1 }, update:{ len:1 } } } } Or we could return an XML configuration object: <Authors> <methods> <method name="GetAll" len="0" /> <method name="add" len="1" /> <method name="update" len="1" /> </methods> </Authors> All of these forms have given us the same basic outcome, by providing a basic definition of server-side classes/objects to be exposed for use with our Ext applications. But, each of these methods require us to build these configurations basically by hand. Some server-side options make it a little easier. Metadata There are a few server-side technologies that allow us to add additional metadata to classes and function definitions, using which we can then introspect objects at runtime to create our configurations. The following example demonstrates this by adding additional metadata to a ColdFusion component (CFC): <cfcomponent name="Authors" ExtDirect="true"> <cffunction name="GetAll" ExtDirect="true"> <cfreturn true /> </cffunction> <cffunction name="add" ExtDirect="true"> <cfargument name="author" /> <cfreturn true /> </cffunction> <cffunction name="update" ExtDirect="true"> <cfargument name="author" /> <cfreturn true /> </cffunction> </cfcomponent> This is a very powerful method for creating our configuration, as it means adding a single name/value attribute (ExtDirect="true") to any object and function we want to make available to our Ext application. The ColdFusion server is able to introspect this metadata at runtime, passing the configuration object back to our Ext application for use. Stack deconstruction—configuration The example ColdFusion Component provided with the DirectCFM stack is pretty basic, so we'll write one slightly more detailed to illustrate the configuration. ColdFusion has a facility for attaching additional metadata to classes and methods, so we'll use the fourth configuration method for this example, Metadata. We'll start off with creating the Authors.cfc class: <cfcomponent name="Authors" ExtDirect="true"> </cfcomponent> Next we'll create our GetAll method for returning all the authors in the database: <cffunction name="GetAll" ExtDirect="true"> <cfset var q = "" /> <cfquery name="q" datasource="cfbookclub"> SELECT AuthorID, FirstName, LastName FROM Authors ORDER BY LastName </cfquery> <cfreturn q /> </cffunction> We're leaving out basic error handling and stuff, but these are the basics behind it. The classes and methods we want to make available will all contain the additional metadata. Building your API So now that we've explored how to create a configuration at the server, we need to take the next step by passing that configuration to our Ext application. We do this by writing a server-side template that will output our JavaScript configuration. Yes, we'll actually dynamically produce a JavaScript include, calling the server-side template directly from within our <script> tag: <script src="Api.cfm"></script> How we write our server-side file really depends on the platform, but ultimately we just want it to return a block of JavaScript (just like calling a .js file) containing our API configuration description. The configuration will appear as part of the actions attribute, but we must also pass the url of our Router, the type of connection, and our namespace. That API return might look something like this: Ext.ns("com.cc"); com.cc.APIDesc = { "url": "/remote/Router.cfm", "type": "remoting" "namespace": "com.cc", "actions": { "Authors": [{ "name": "GetAll", "len": 0 },{ "name": "add", "len": 1 },{ "name": "update", "len": 1 }] } }; This now exposes our server-side configuration to our Ext application. Stack deconstruction—API The purpose here is to create a JavaScript document, dynamically, of your configuration. Earlier we defined configuration via metadata. The DirectCFM API now has to convert that metadata into JavaScript. The first step is including the Api.cfm in a <script> tag on the page, but we need to know what's going on "under the hood." Api.cfm: <!--- Configure API Namespace and Description variable names ---> <cfset args = StructNew() /> <cfset args['ns'] = "com.cc" /> <cfset args['desc'] = "APIDesc" /> <cfinvoke component="Direct" method="getAPIScript" argumentcollection="#args#" returnVariable="apiScript" /> <cfcontent reset="true" /> <cfoutput>#apiScript#</cfoutput> Here we set a few variables, that will then be used in a method call. The getAPIScript method, of the Direct.cfc class, will construct our API from metadata. Direct.cfc getAPIScript() method: <cffunction name="getAPIScript"> <cfargument name="ns" /> <cfargument name="desc" /> <cfset var totalCFCs = '' /> <cfset var cfcName = '' /> <cfset var CFCApi = '' /> <cfset var fnLen = '' /> <cfset var Fn = '' /> <cfset var currFn = '' /> <cfset var newCfComponentMeta = '' /> <cfset var script = '' /> <cfset var jsonPacket = StructNew() /> <cfset jsonPacket['url'] = variables.routerUrl /> <cfset jsonPacket['type'] = variables.remotingType /> <cfset jsonPacket['namespace'] = ARGUMENTS.ns /> <cfset jsonPacket['actions'] = StructNew() /> <cfdirectory action="list" directory="#expandPath('.')#" name="totalCFCs" filter="*.cfc" recurse="false" /> <cfloop query="totalCFCs"> <cfset cfcName = ListFirst(totalCFCs.name, '.') /> <cfset newCfComponentMeta = GetComponentMetaData(cfcName) /> <cfif StructKeyExists(newCfComponentMeta, "ExtDirect")> <cfset CFCApi = ArrayNew(1) /> <cfset fnLen = ArrayLen(newCFComponentMeta.Functions) /> <cfloop from="1" to="#fnLen#" index="i"> <cfset currFn = newCfComponentMeta.Functions[i] /> <cfif StructKeyExists(currFn, "ExtDirect")> <cfset Fn = StructNew() /> <cfset Fn['name'] = currFn.Name/> <cfset Fn['len'] = ArrayLen(currFn.Parameters) /> <cfif StructKeyExists(currFn, "ExtFormHandler")> <cfset Fn['formHandler'] = true /> </cfif> <cfset ArrayAppend(CFCApi, Fn) /> </cfif> </cfloop> <cfset jsonPacket['actions'][cfcName] = CFCApi /> </cfif> </cfloop> <cfoutput><cfsavecontent variable="script">Ext.ns('#arguments. ns#');#arguments.ns#.#desc# = #SerializeJson(jsonPacket)#;</ cfsavecontent></cfoutput> <cfreturn script /> </cffunction> The getAPIScript method sets a few variables (including the 'actions' array), pulls a listing of all ColdFusion Components from the directory, loops over that listing, and finds any components containing "ExtDirect" in their root meta. With every component that does contain that meta, it then loops over each method, finds methods with "ExtDirect" in the function meta, and creates a structure with the function name and number of arguments, which is then added to an array of methods. When all methods have been introspected, the array of methods is added to the 'actions' array. Once all ColdFusion Components have been introspected, the entire packet is serialized into JSON, and returned to API.cfm for output. One item to note is that the script, when introspecting method metadata, also looks for a "ExtFormHandler" attribute. If it finds the attribute, it will include that in the method struct prior to placing the struct in the 'actions' array.

Combining Silverlight and Windows Azure projects Standard Silverlight applications require that they be hosted on HTML pages, so that they can be loaded in a browser. Developers who work with the .Net framework will usually host this page within an ASP.Net website. The easiest way to host a Silverlight application on Azure is to create a single web role that contains an ASP.Net application to host the Silverlight application. Hosting the Silverlight application in this way enables you, as a developer, to take advantage of the full .Net framework to support your Silverlight application. Supporting functionalities can be provided such as hosting WCF services, RIA services, Entity Framework, and so on. In the upcoming chapters, we will explore ways by which RIA services, OData, Entity Framework, and a few other technologies can be used together. For the rest of this chapter, we will focus on the basics of hosting a Silverlight application within Azure and integrating a hosted WCF service. Creating a Silverlight or Azure solution Your system should already be fully configured with all Silverlight and Azure tools. In this section, we are going to create a simple Silverlight application that is hosted inside an Azure web role. This will be the basic template that is used throughout the book as we explore different ways in which we can integrate the technologies together: Start Visual Studio as an administrator. You can do this by opening the Start Menu and finding Visual Studio, then right-clicking on it, and selecting Run as Administrator. This is required for the Azure compute emulator to run successfully. Create a new Windows Azure Cloud Service. The solution name used in the following example screenshot is Chapter3Exercise1: (Move the mouse over the image to enlarge.) Add a single ASP.Net Web Role as shown in the following screenshot. For this exercise, the default name of WebRole1 will be used. The name of the role can be changed by clicking on the pencil icon next to the WebRole1 name: Visual Studio should now be loaded with a single Azure project and an ASP. Net project. In the following screenshot, you can see that Visual Studio is opened with a solution named Chapter3Exercise1. The solution contains a Windows Azure Cloud project, also called Chapter3Exercise1. Finally, the ASP.Net project can be seen named as WebRole1: Right-click on the ASP.Net project named WebRole1 and select Properties. In the WebRole1 properties screen, click on the Silverlight Applications tab. Click on Add to add a new Silverlight project into the solution. The Add button has been highlighted in the following screenshot: For this exercise, rename the project to HelloWorldSilverlightProject. Click on Add to create the Silverlight project. The rest of the options can be left to their default settings, as shown in the following screenshot. Visual Studio will now create the Silverlight project and add it to the solution. The resulting solution should now have three projects as shown in the following screenshot. These include the original Azure project, Chapter3Exercise1; the ASP.Net web role, WebRole1; and the third new project HelloWorldSilverlightProject: Open MainPage.xaml in design view, if not already open. Change the grid to a StackPanel. Inside the StackPanel, add a button named button1 with a height of 40 and a content that displays Click me!. Inside the StackPanel, underneath button1, add a text block named textBlock1 with a height of 20 The final XAML should look similar to this code snippet: <UserControl> <StackPanel x_Name="LayoutRoot" Background="White"> <Button x_Name="button1" Height="40" Content="Click me!" /> <TextBlock x_Name="textBlock1" Height="20" /> </StackPanel> </UserControl> Double-click on button1 in the designer to have Visual Studio automatically create a click event. The final XAML in the designer should look similar to the following screenshot: Open the MainPage.xaml.cs code behind the file and find the button1_Click method. Add a code that will update textBlock1 to display Hello World and the current time as follows: private void button1_Click(object sender, RoutedEventArgs e) { textBlock1.Text = "Hello World at " + DateTime.Now.ToLongTimeString(); } Build the project to ensure that everything compiles correctly.Now that the solution has been built, it is ready to be run and debugged within the Windows Azure compute emulator. The next section will explore what happens while running an Azure application on the compute emulator. Running an Azure application on the Azure compute emulator With the solution built, it is ready to run on the Azure simulation: the compute emulator. The compute emulator is the local simulation of the Windows Azure compute emulator which Microsoft runs on the Azure servers it hosts. When you start debugging by pressing F5 (or by selecting Debug | Start Debugging from the menu), Visual Studio will automatically package the Azure project, then start the Azure compute emulator simulation. The package will be copied to a local folder used by the compute emulator. The compute emulator will then start a Windows process to host or execute the roles, one of which will be started as per the instance request for each role. Once the compute emulator has been successfully initialized, Visual Studio will then launch the browser and attach the debugger to the correct places. This is similar to the way Visual Studio handles debugging of an ASP.Net application with the ASP. Net Development Server. The following steps will take you through the process of running and debugging applications on top of the compute emulator: In Solution Explorer, inside the HelloWorldSilverlightProject, right-click on HelloWorldSilverlightProjectTestPage.aspx, and select Set as startup page. Ensure that the Azure project (Chapter3Exercise1) is still set as the start-up project. In Visual Studio, press F5 to start debugging (or from the menu select Debug | Start Debugging). Visual Studio will compile the project, and if successful, begins to launch the Azure compute emulator as shown in the following screenshot: Once the compute emulator has been started and the Azure package deployed to it, Visual Studio will launch Internet Explorer. Internet Explorer will display the page set as the start-up page (which was set to in an earlier step HelloWorldSilverlightProjectTestPage.aspx). Once the Silverlight application has been loaded, click on the Click me! button. The TextBlock should be updated with the current time, as shown in the following screenshot: Upon this completion, you should now have successfully deployed a Silverlight application on top of the Windows Azure compute emulator. You can now use this base project to build more advanced features and integration with other services. Consuming an Azure-hosted WCF service within a Silverlight application A standalone Silverlight application will not be able to do much by itself. Most applications will require that they consume data from a data source, such as to get a list of products or customer orders. A common way to send data between .Net applications is through WCF services.