Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-digital-and-mobile-forensics
Packt
14 Nov 2016
14 min read
Save for later

Digital and Mobile Forensics

Packt
14 Nov 2016
14 min read
In this article, Mattia Epifani and Pasquale Stirparo, co-authors of the book Learning iOS Forensics - Second Edition, would be talking mainly, if not solely, about computer forensics and computer crimes, such as when an attacker breaks into a computer network system and steals data. This would involve two types of offenses—unlawful/unauthorized access and data theft. As mobile phones became more popular, the new field of mobile forensics developed. (For more resources related to this topic, see here.) Nowadays, things have changed radically and they are still changing at quite a fast pace as technology evolves. Digital forensics, which includes all disciplines dealing with electronic evidence, is also being applied to common crimes, to those that, at least by definition, are not strictly IT crimes. Today, more than ever, we live in a society that is fully digitalized and people are equipped with all kinds of devices, which have different types of capabilities, but all of them process, store, and transmit information (mainly over the Internet). This means that forensic investigators have to be able to deal with all these devices. As defined at the first Digital Forensics Research Workshop (DFRWS) in 2001, digital forensics is: "The use of scientifically derived and proven methods toward the preservation, collection, validation, identification, analysis, interpretation, documentation, and presentation of digital evidence derived from digital sources for the purpose of facilitating or furthering the reconstruction of events found to be criminal, or helping to anticipate unauthorized actions shown to be disruptive to planned operations." As Casey asserted (Casey, 2011): "In this modern age, it is hard to imagine a crime that does not have a digital dimension." Criminals of all kinds use technology to facilitate their offenses, communicate with their peers, recruit other criminals, launder money, commit credit card fraud, gather information on their victims, and so on. This obviously creates new challenges for all the different actors involved, such as attorneys, judges, law enforcement agents, and forensic examiners. Among the cases solved in recent years, there were kidnappings where the kidnapper was caught—thanks to a request for ransom sent by e-mail from his mobile phone. There have been many cases of industrial espionage in which unfaithful employees were hiding projects in the memory cards of their smartphones, cases of drug dealing solved—thanks to the evidence found in the backup of mobile phones that was on computer, and many other such cases. Even the largest robberies of our time are now being conducted via computer networks. In this article, you will learn the following: Definition and principles of mobile forensics How to properly handle digital evidence Mobile forensics Mobile forensics is a field of study in digital forensics that focuses on mobile devices. Among the different digital forensics fields, mobile forensics is without doubt the fastest growing and evolving area of study, having an impact on many different situations from corporate to criminal investigations and intelligence gathering, which are on the rise. Moreover, the importance of mobile forensics is increasing exponentially due to the continuous fast growth of the mobile market. One of the most interesting peculiarities of mobile forensics is that mobile devices, particularly mobile phones, usually belong to a single individual, while this is not always the case with a computer that may be shared among employees of a company or members of a family. For this reason, the analysis of mobile phones gives access to plenty of personal information. Another important and interesting aspect that comes with mobile forensics, which is both challenging and frustrating at the same time for the analyst, is the multitude of different device models and the customized flavors of their operating systems available in the market. This makes it very difficult to have a single solution (either a tool or process) to address them all. Just think of all the applications people have installed on their smartphones: IM clients, web browsers, social network clients, password managers, navigation systems, and much more, other than the classic default ones, such as an address book, which can provide a lot more information than just the phone number for each contact that has been saved. Moreover, syncing such devices with a computer has become a very easy and smooth process, and all user activities, schedules, to-do lists, and everything else is stored inside a smartphone. Aren't these enough to profile a person and reconstruct all their recent activities, than building the network of contacts? Finally, in addition to a variety of smartphones and operating systems, such as Apple iOS, Google Android, Microsoft Windows Phone, and Blackberry OS, there is a massive number of so-called feature phones that use older mobile OS systems. Therefore, it's pretty clear that when talking about mobile/smartphone forensics, there is so much more than just printouts of phone calls. In fact, with a complete examination, we can retrieve SMSes/MMSes, pictures, videos, installed applications, e-mails, geolocation data, and so on—both present and deleted information. Digital evidence As mentioned earlier, on one hand the increasing involvement of mobile devices in digital forensics cases has brought a whole new series of challenges and complexities. However, on the other hand, this has also resulted in a much greater amount of evidence from criminals that it is now being used to reconstruct their activities with a more comprehensive level of detail. Moreover, while classical physical evidence may be destroyed, digital evidence, most of the time, leaves traces. Over the years, there have been several definitions of what digital evidence actually is, some of them focusing particularly on the evidentiary aspects of proof to be used in court, such as the one proposed by the Standard Working Group on Digital Evidence (SWGDE), stating that: "Digital evidence is any information of probative value that is either stored or transmitted in a digital form." The definition proposed by the International Organization of Computer Evidence (IOCE) states: "Digital evidence is information stored or transmitted in binary form that may be relied on in court." The definition given by E. Casey (Casey, 2000), refers to digital evidence as: "Physical objects that can establish that a crime has been committed, can provide a link between a crime and its victim, or can provide a link between a crime and its perpetrator." While all of these are correct, as previously said, all of these definitions focus mostly on proofs and tend to disregard data that is extremely useful for an investigation. For this reason, and for the purpose of this book, we will refer to the definition given by Carrier (Carrier, 2006), where digital evidence is defined as: "Digital data that supports or refutes a hypothesis about digital events or the state of digital data." This definition is a more general one, but better matches the current state of digital evidence and its value within the entire investigation process. Also from a standardization point of view, there have been, and still are, many attempts to define guidelines and best practices for digital forensics on how to handle digital evidence. Other than the several guidelines and special publications from NIST, there is a standard from ISO/IEC that was released in 2012, the ISO 27037 guidelines for identification, collection and/or acquisition, and preservation of digital evidence, which is not specific to mobile forensics, but is related to digital forensics in general, aiming to build a standard procedure for collecting and handling digital evidence, which will be legally recognized and accepted in court in different countries. This is a really important goal if you consider the lack of borders in the Internet era, particularly when it comes to digital crimes, where illicit actions can be perpetrated by attackers from anywhere in the world. Handling of mobile evidence In order to be useful not only in court but also during the entire investigation phase, digital evidence must be collected, preserved, and analyzed in a forensically sound manner. This means that each step, from the identification to the reporting, has to be carefully and strictly followed. Historically, we are used to referring to a methodology as forensically sound if, and only if, it would imply that the original source of evidence remains unmodified and unaltered. This was mostly true when talking about classical computer forensics, in scenarios where the forensic practitioner found the computer switched off or had to deal with external hard drives, although not completely true even in these situations. However, since the rise of live forensics, this concept has become more and more untrue. In fact, methods and tools for acquiring memory from live systems inevitably alter, even if just a little bit, the target system they are run on. The advent of mobile forensics stresses this concept even more, because mobile devices, and smartphones in particular, are networked devices that continuously exchange data through several communication protocols, such as GSM/CDMA, Wi-Fi, Bluetooth, and so on. Moreover, in order to acquire a mobile device, forensic practitioners need to have some degree of interaction with the device. Based on the type, a smartphone can need more or less interaction, altering in this way the original state of the device. All of this does not mean that preservation of the source evidence is useless, but that it is nearly impossible in the field of mobile devices. Therefore, it becomes a matter of extreme importance to thoroughly document every step taken during the collection, preservation, and acquisition phases. Using this approach, forensic practitioners will be able to demonstrate that they have been as unintrusive as possible. As Casey states (Casey, 2011): "One of the keys to forensic soundness is documentation. A solid case is built on supporting documentation that reports on where the evidence originated and how it was handled. From a forensic standpoint, the acquisition process should change the original evidence as little as possible and any changes should be documented and assessed in the context of the final analytical results." When in the presence of mobile devices to be collected, it is a good practice for the forensic practitioner to consider the following points: Take note of the current location where the device has been found. Report the device status (switched on or off, broken screen, and so on). Report date, time, and other information visible on the screen if the device is switched on, for example, by taking a picture of the screen. Look very carefully for the presence of memory cards. Although it is not the case with iOS devices, generally many mobile phones have a slot for an external memory card, where pictures, chat databases, and many other types of user data are usually stored. Look very carefully for the presence of cables related to the mobile phone that is being collected, especially if you don't have a full set of cables in your lab. Many mobile phones have their own cables to connect to the computer and to recharge the battery. Search for the original Subscriber Identity Module (SIM) package, because that is where the PIN and PIN unblocking key (PUK) codes are written. Take pictures of every item before collection. Modifications to mobile devices can happen not only because of interaction with the forensic practitioner, but also due to interaction with the network, voluntarily or not. In fact, digital evidence in mobile devices can be lost completely as they are susceptible to being overwritten by new data, for example, with the smartphone receiving an SMS while it is being collected, thus overwriting possible evidence previously stored in the same area of memory as the newly arrived SMS, or upon receiving a remote wiping command over a wireless network. Most of today's smartphones and iOS devices can be configured to be completely wiped remotely. From a real case: While searching inside the house of a person under investigation, law enforcement agents found and seized, among other things, computers and a smartphone. After cataloguing and documenting everything, they put all the material into boxes to bring them back to the laboratory. Once back in their laboratory, when acquiring the smart phone in order to proceed with the forensics analysis, they noticed that the smartphone was empty and it appeared to be brand new. The owner had wiped it remotely. Therefore, isolating the mobile device from all radio networks is a fundamental step in the process of preservation of evidence. There are several ways to achieve this, all with their own pros and cons, as follows: Airplane mode: Enabling Airplane mode on a device requires some sort of interaction, which may pose some risks of modification by the forensic practitioner. This is one of the best possible options since it implies that all wireless communication chips are switched off. In this case, it is always good to document the action taken with pictures and/or videos. Normally, this is possible only if the phone is not password-protected or the password is known. However, for devices with iOS 7 or higher, it is also possible to enable airplane mode by lifting the dock from the bottom, where there will be a button with the shape of a plane. This is possible only if the Access on Lock Screen option is enabled from Settings | Control Center. Faraday's bag: This item is a sort of envelope made of conducting material, which blocks out static electric fields and electromagnetic radiation completely isolating the device from communicating with external networks. It is based, as the name suggests, on Faraday's law. This is the most common solution, particularly useful when the device is being carried from the crime scene to the lab after seizure. However, the use of Faraday's bag will make the phone continuously search for a network, which will cause the battery to quickly drain. Unfortunately, it is also risky to plug the phone to a power cable outside that will go inside the bag, because this may act as antenna. Moreover, it is important to keep in mind that when you remove the phone from the bag (once arrived in the lab) it will again be exposed to the network. So, you would need either a shielded lab environment or a Faraday solution that would allow you to access the phone while it is still inside the shielded container, without the need for external power cables. Jamming: A jammer is used to prevent a wireless device from communicating by sending out radio waves along the same frequencies as that device. In our case, it would jam the GSM/UMTS/LTE frequencies that mobile phones use to connect with cellular base stations to send/receive data. Be aware that this practice may be considered illegal in some countries, since it will also interfere with any other mobile device in the range of the jammer, disrupting their communications too. Switching off the device: This is a very risky practice because it may activate authentication mechanisms, such as PIN codes or passcodes, that are not available to the forensic practitioner, or other encryption mechanisms that carry the risk of delaying or even blocking the acquisition of the mobile device. Removing the SIM card: In most mobile devices, this operation implies removing the battery and therefore all the risks and consequences we just mentioned regarding switching off the device; however, in iOS devices this task is quite straightforward and easy, and it does not imply removing the battery (in iOS devices this is not possible). Moreover, SIM cards can have PIN protection enabled; removing it from the phone may lock the SIM card, preventing its content from being displayed. However, bear in mind that removing the SIM card will isolate the device only from the cellular network, while other networks, such as Wi-Fi or Bluetooth, may still be active and therefore need to be addressed. The following image shows a SIM card extracted from an iPhone with just a clip; image taken from http://www.maclife.com/: Summary In this article, we gave a general introduction to digital forensics for those relatively new to this area of study and a good recap to those already in the field, keeping the mobile forensics field specifically in mind. We have shown what digital evidence is and how it should be handled, presenting several techniques to isolate the mobile device from the network. Resources for Article: Further resources on this subject: Mobile Forensics [article] Mobile Forensics and Its Challanges [article] Forensics Recovery [article]
Read more
  • 0
  • 0
  • 23716

article-image-planning-and-structuring-your-test-driven-ios-app
Packt
11 Nov 2016
13 min read
Save for later

Planning and Structuring Your Test-Driven iOS App

Packt
11 Nov 2016
13 min read
In this article written by Dr. Dominik Hauser, author of the book Test–Driven iOS Development with Swift 3.0, you will learn that when starting TDD, writing unit tests would be easy for most people. The hard part is to transfer the knowledge from writing the test to driving the development. What can be assumed? What should be done before one writes the first test? What should be tested to end up with a complete app? (For more resources related to this topic, see here.) As a developer, you are used to thinking in terms of code. When you see a feature on the requirement list for an app, your brain already starts to layout the code for this feature. And for recurring problems in iOS development (such as building table views), you most probably have already developed your own best practices. In TDD, you should not think about the code while working on the test. The tests have to describe what the unit under test should do and not how it should do it. It should be possible to change the implementation without breaking the tests. To practice this approach of development, we will develop a simple to-do list app in the remainder of this book. It is, on purpose, a boring and easy app. We want to concentrate on the TDD workflow, not complex implementations. An interesting app would distract from what is important in this book—how to do TDD. This article introduces the app that we are going to build, and it shows the views that the finished app will have. We will cover the following topics in this article: The task list view The task detail view The task input view The structure of an app Getting started with Xcode Setting up useful Xcode behaviors for testing The task list view When starting the app, the user sees a list of to-do items. The items in the list consist of a title, an optional location, and the due date. New items can be added to the list by an add (+) button, which is shown in the navigation bar of the view. The task list view will look like this: User stories: As a user, I want to see the list of to-do items when I open the app As a user, I want to add to-do items to the list In a to-do list app, the user will obviously need to be able to check items when they are finished. The checked items are shown below the unchecked items, and it is possible to uncheck them again. The app uses the delete button in the UI of UITableView to check and uncheck items. Checked items will be put at the end of the list in a section with the Finished header. The user can also delete all the items from the list by tapping the trash button. The UI for the to-do item list will look like this: User stories: As a user, I want to check a to-do item to mark it as finished As a user, I want to see all the checked items below the unchecked items As a user, I want to uncheck a to-do item As a user, I want to delete all the to-do items When the user taps an entry, the details of this entry is shown in the task detail view. The task detail view The task detail view shows all the information that's stored for a to-do item. The information consists of a title, due date, location (name and address), and a description. If an address is given, a map with an address is shown. The detail view also allows checking the item as finished. The detail view looks like this: User stories: As a user, given that I have tapped a to-do item in the list, I want to see its details As a user, I want to check a to-do item from its details view The task input view When the user selects the add (+) button in the list view, the task input view is shown. The user can add information for the task. Only the title is required. The Save button can only be selected when a title is given. It is not possible to add a task that is already in the list. The Cancel button dismisses the view. The task input view will look like this: User stories: As a user, given that I have tapped the add (+) button in the item list, I want to see a form to put in the details (title, optional date, optional location name, optional address, and optional description) of a to-do item As a user, I want to add a to-do item to the list of to-do items by tapping on the Save button We will not implement the editing and deletion of tasks. But when you have worked through this book completely, it will be easy for you to add this feature yourself by writing the tests first. Keep in mind that we will not test the look and design of the app. Unit tests cannot figure out whether an app looks like it was intended. Unit tests can test features, and these are independent of their presentation. In principle, it would be possible to write unit tests for the position and color of UI elements. But such things are very likely to change a lot in the early stages of development. We do not want to have failing tests only because a button has moved 10 points. However, we will test whether the UI elements are present on the view. If your user cannot see the information for the tasks, or if it is not possible to add all the information of a task, then the app does not meet the requirements. The structure of the app The following diagram shows the structure of the app: The Table View Controller, the delegate, and the data source In iOS apps, data is often presented using a table view. Table views are highly optimized for performance; they are easy to use and to implement. We will use a table view for the list of to-do items. A table view is usually represented by UITableViewController, which is also the data source and delegate for the table view. This often leads to a massive Table View Controller, because it is doing too much: presenting the view, navigating to other view controllers, and managing the presentation of the data in the table view. It is a good practice to split up the responsibility into several classes. Therefore, we will use a helper class to act as the data source and delegate for the table view. The communication between the Table View Controller and the helper class will be defined using a protocol. Protocols define what the interface of a class looks like. This has a great benefit: if we need to replace an implementation with a better version (maybe because we have learned how to implement the feature in a better way), we only need to develop against the clear interface. The inner workings of other classes do not matter. Table view cells As you can see in the preceding screenshots, the to-do list items have a title and, optionally, they can have a due date and a location name. The table view cells should only show the set data. We will accomplish this by implementing our own custom table view cell. The model The model of the application consists of the to-do item, the location, and an item manager, which allows the addition and removal of items and is also responsible for managing the items. Therefore, the controller will ask the item manager for the items to present. The item manager will also be responsible for storing the items on disc. Beginners often tend to manage the model objects within the controller. Then, the controller has a reference to a collection of items, and the addition and removal of items is directly done by the controller. This is not recommended, because if we decide to change the storage of the items (for example, by using Core Data), their addition and removal would have to be changed within the controller. It is difficult to keep an overview of such a class; and because of this reason, it is a source of bugs. It is much easier to have a clear interface between the controller and the model objects, because if we need to change how the model objects are managed, the controller can stay the same. We could even replace the complete model layer if we just keep the interface the same. Later in the article, we will see that this decoupling also helps to make testing easier. Other view controllers The application will have two more view controllers: a task detail View Controller and a View Controller for the input of the task. When the user taps a to-do item in the list, the details of the item are presented in the task detail View Controller. From the Details screen, the user will be able to check an item. New to-do items will be added to the list of items using the view presented by the input View Controller. The development strategy In this book, we will build the app from inside out. We will start with the model, and then build the controllers and networking. At the end of the book, we will put everything together. Of course, this is not the only way to build apps. But by separating on the basis of layers instead of features, it is easier to follow and keep an overview of what is happening. When you later need to refresh your memory, the relevant information you need is easier to find. Getting started with Xcode Now, let's start our journey by creating a project that we will implement using TDD. Open Xcode and create a new iOS project using the Single View Application template. In the options window, add ToDo as the product name, select Swift as language, choose iPhone in the Devices option, and check the box next to Include Unit Tests. Let the Use Core Data and Include UI Tests boxes stay unchecked. Xcode creates a small iOS project with two targets: one for the implementation code and the other for the unit tests. The template contains code that presents a single view on screen. We could have chosen to start with the master-detail application template, because the app will show a master and a detail view. However, we have chosen the Single View Application template because it comes with hardly any code. And in TDD, we want to have all the implementation code demanded by failing tests. To take a look at how the application target and test target fit together, select the project in Project Navigator, and then select the ToDoTests target. In the General tab, you'll find a setting for the Host Application that the test target should be able to test. It will look like this: Xcode has already set up the test target correctly to allow the testing of the implementations that we will write in the application target. Xcode has also set up a scheme to build the app and run the tests. Click on the Scheme selector next to the stop button in the toolbar, and select Edit Scheme.... In the Test action, all the test bundles of the project will be listed. In our case, only one test bundle is shown—ToDoTests. On the right-hand side of the shown window is a column named Test, with a checked checkbox. This means that if we run the tests while this scheme is selected in Xcode, all the tests in the selected test suite will be run. Setting up useful Xcode behaviors for testing Xcode has a feature called behaviors. With the use of behaviors and tabs, Xcode can show useful information depending on its state. Open the Behaviors window by going to Xcode | Behaviors | Edit Behaviors. On the left-hand side are the different stages for which you can add behaviors (Build, Testing, Running, and so on). The following behaviors are useful when doing TDD. The behaviors shown here are those that I find useful. Play around with the settings to find the ones most useful for you. Overall, I recommend using behaviors because I think they speed up development. Useful build behaviors When building starts, Xcode compiles the files and links them together. To see what is going on, you can activate the build log when building starts. It is recommended that you open the build log in a new tab because this allows switching back to the code editor when no error occurs during the build. Select the Starts stage and check Show tab named. Put in the Log name and select in active window. Check the Show navigator setting and select Issue Navigator. At the bottom of the window, check Navigate to and select current log. After you have made these changes, the settings window will look like this: Build and run to see what the behavior looks like. Testing behaviors To write some code, I have an Xcode tab called Coding. Usually, in this tab, the test is open on the left-hand side, and in the Assistant Editor which is on the right-hand side, there is the code to be tested (or in the case of TDD, the code to be written). It looks like this: When the test starts, we want to see the code editor again. So, we add a behavior to show the Coding tab. In addition to this, we want to see the Test Navigator and debugger with the console view. When the test succeeds, Xcode should show a bezel to notify us that all tests have passed. Go to the Testing | Succeeds stage, and check the Notify using bezel or system notification setting. In addition to this, it should hide the navigator and the debugger, because we want to concentrate on refactoring or writing the next test. In case the testing fails (which happens a lot in TDD), Xcode will show a bezel again. I like to hide the debugger, because usually, it is not the best place to figure out what is going on in the case of a failing test. And in most of the cases in TDD, we already know what the problem is. But we want to see the failing test. Therefore, check Show navigator and select Issue navigator. At the bottom of the window, check Navigate to and select first new issue. You can even make your Mac speak the announcements. Check Speak announcements using and select the voice you like. But be careful not to annoy your coworkers. You might need their help in the future. Now, the project and Xcode are set up, and we can start our TDD journey. Summary In this article, we took a look at the app that we are going to build throughout the course of this book. We took a look at how the screens of the app will look when we are finished with it. We created the project that we will use later on and learned about Xcode behaviors. Resources for Article: Further resources on this subject: Thinking Functionally [article] Hosting on Google App Engine [article] Cloud and Async Communication [article]
Read more
  • 0
  • 0
  • 27364

article-image-devops-tools-and-technologies
Packt
11 Nov 2016
15 min read
Save for later

DevOps Tools and Technologies

Packt
11 Nov 2016
15 min read
In this article by Ritesh Modi, the author of the book DevOps with Windows Server 2016, we will introduce foundational platforms and technologies instrumental in enabling and implementing DevOps practices. (For more resources related to this topic, see here.) These include: Technology stack for implementing Continuous Integration, Continuous Deployment, Continuous Deliver, Configuration Management, and Continuous Improvement. These form the backbone for DevOps processes and include source code services, build services, and release services through Visual Studio Team Services. Platform and technology used to create and deploy a sample web application. This includes technologies such as Microsoft .NET, ASP.NET and SQL Server databases. Tools and technology for configuration management, testing of code and application, authoring infrastructure as code, and deployment of environments. Examples of these tools and technologies are Pester for environment validation, environment provisioning through Azure Resource Manager (ARM) templates, Desired State Configuration (DSC) and Powershell, application hosting on containers through Windows Containers and Docker, application and database deployment through Web Deploy packages, and SQL Server bacpacs. Cloud technology Cloud is ubiquitous. Cloud is used for our development environment, implementation of DevOps practices, and deployment of applications. Cloud is a relatively new paradigm in infrastructure provisioning, application deployment, and hosting space. The only options prior to the advent of cloud was either self-hosted on-premises deployments or using services from a hosting service provider. However, cloud is changing the way enterprises look at their strategy in relation to infrastructure and application development, deployment, and hosting. In fact, the change is so enormous that it has found its way into every aspect of an organization's software development processes, tools, and practices. Cloud computing refers to the practice of deploying applications and services on the Internet with a cloud provider. A cloud provider provides multiple types of services on cloud. They are divided into three categories based on their level of abstraction and degree of control on services. These categories are as follows: Infrastructure as a Service (IaaS) Platform as a Service (PaaS) Software as a Service (SaaS) These three categories differ based on the level of control a cloud provider exercises compared to the cloud consumer. The services provided by a cloud provider can be divided into layers, with each layer providing a type of service. As we move higher in the stack of layers, the level of abstraction increases in line with the cloud provider's control over services. In other words, the cloud consumer starts to lose control over services as you move higher in each column: Figure 1: Cloud Services – IaaS, PaaS and SaaS Figure 1 shows the three types of service available through cloud providers and the layers that comprise these services. These layers are stacked vertically on each other and show the level of control a cloud provider has compared to a consumer. From Figure 1, it is clear that for IaaS, a cloud provider is responsible for providing, controlling, and managing layers from the network layer up to the virtualization layer. Similarly, for PaaS, a cloud provider controls and manages from the hardware layer up to the runtime layer, while the consumer controls only the application and data layers. Infrastructure as a Service (IaaS) As the name suggests, Infrastructure as a Service is an infrastructure service provided by a cloud provider. This service includes the physical hardware and its configuration, network hardware and its configuration, storage hardware and its configuration, load balancers, compute, and virtualization. Any layer above virtualization is the responsibility of the consumer to provision, configure, and manage. The consumer can decide to use the provided underlying infrastructure in whatever way best suits their requirements. Consumers can consume the storage, network, and virtualization to provision their virtual machines on top of. It is the consumer's responsibility to manage and control the virtual machines and the things deployed within it. Platform as a Service (PaaS) Platform as a Service enables consumers to deploy their applications and services on the provided platform, consuming the underlying runtime, middleware, and services. The cloud provider provides the services from infrastructure to runtime. The consumers cannot provision virtual machines as they cannot access and control them. Instead, they can only control and manage their applications. This is a comparatively faster method of development and deployment because now the consumer can focus on application development and deployment. Examples of Platform as a Service include Azure Automation, Azure SQL, and Azure App Services. Software as a Service (SaaS) Software as a Service provides complete control of the service to the cloud provider. The cloud provider provisions, configures, and manages everything from infrastructure to the application. It includes the provisioning of infrastructure, deployment and configuration of applications, and provides application access to the consumer. The consumer does not control and manage the application, and can use and configure only parts of the application. They control only their data and configuration. Generally, multi-tenant applications used by multiple consumers, such as Office 365 and Visual Studio Team Services, are examples of SaaS. Advantages of using cloud computing There are multiple distinct advantages of using cloud technologies. The major among them are as follows: Cost effective: Cloud computing helps organizations to reduce the cost of storage, networks, and physical infrastructure. It also prevents them from having to buy expensive software licenses. The operational cost of managing these infrastructures also reduces due to lesser effort and manpower requirements. Unlimited capacity: Cloud provides unlimited resources to the consumer. This ensures applications will never get throttled due to limited resource availability. Elasticity: Cloud computing provides the notion of unlimited capacity and applications deployed on it can scale up or down on an as-needed basis. When demand for the application increases, cloud can be configured to scale up the infrastructure and application by adding additional resources. At the same time, it can scale down unnecessary resources during periods of low demand. Pay as you go: Using cloud eliminates capital expenditure and organizations pay only for what they use, thereby providing maximum return on investment. Organizations do not need to build additional infrastructure to host their application for times of peak demand. Faster and better: Cloud provides ready-to-use applications and faster provisioning and deployment of environments. Moreover, organizations get better-managed services from their cloud provider with higher service-level agreements. We will use Azure as our preferred cloud computing provider for the purpose of demonstrating samples and examples. However, you can use any cloud provider that provides complete end-to-end services for DevOps. We will use multiple features and services provided by Azure across IaaS and PaaS. We will consume Operational Insights and Application Insights to monitor our environment and application, which will help capture relevant telemetry for auditing purposes. We will provision Azure virtual machines running Windows and Docker Containers as a hosting platform. We will use Windows Server 2016 as the target operating system for our applications on cloud. Azure Resource Manager (ARM). We will also use Desired State Configuration and PowerShell as our configuration management platform and tool. We will use Visual Studio Team Services (VSTS), a suite of PaaS services on cloud provided by Microsoft, to set up and implement our end-to-end DevOps practices. Microsoft also provides the same services as part of Team Foundation Services (TFS) as an on-premises solution. Technologies like Pester, DSC, and PowerShell can be deployed and configured to run on any platform. These will help both in the validation of our environment and in the configuration of both application and environment, as part of our Configuration management process. Windows Server 2016 is a breakthrough operating system from Microsoft also referred to as Cloud Operating System. We will look into Windows Server 2016 in the following section. Windows Server 2016 Windows Server 2016 has come a long way. All the way from Windows NT to Windows 2000 and 2003, then Windows 2008 (R2) and 2012 (R2), and now Windows Server 2016. Windows NT was the first popular Windows server among enterprises. However, the true enterprise servers were Windows 2000 and Windows 2003. The popularity of Windows Server 2003 was unprecedented and it was widely adopted. With Windows Server 2008 and 2008 R2, the idea of the data center took priority and enterprises with their own data center adopted it. Even the Windows Server 2008 series was quite popular among enterprises. In 2010, the Microsoft cloud, Azure, was launched. The first steps towards a cloud operating system were Windows Server 2012 and 2012 R2. They had the blueprints and technology to be seamlessly provisioned on Azure. Now, when Azure and cloud are gaining enormous popularity, Windows Server 2016 is released as a true cloud operating system. The evolution of Windows Server is shown in Figure 2: Figure 2: Windows Server evolution Windows Server 2016 is referred to as a cloud operating system. It is built with cloud in mind. It is also referred to as the first operating system that enables DevOps seamlessly by providing relevant tools and technologies. It makes implementing DevOps simpler and easier through its productivity tools. Let us look briefly into these tools and technologies. Multiple choices for Application platform Windows Server 2016 comes with many choices for application platform for applications. It provides the following: Windows Server 2016 Nano Server Windows and Docker Containers Hyper-V Containers Nested virtual machines Windows Server as a hosting platform Windows server 2016 can be used in the ways it has always been used, such as hosting applications and providing server functionalities. It provides the services necessary to make applications secure, scalable, and highly available. It also provides virtualization, directory services, certificate services, web server, databases, and more. These services can be consumed by the enterprise’s services and applications. Nano Server Windows Server provides a new option to host applications and services. This is a new variety of lightweight, scaled-down Windows server containing only the kernel and drivers necessary to run as an operating system. They are also known as headless servers. They do not have any graphical user interface and the only way to interact and manage them is through remote PowerShell. Out of the box, they do not contain any service or feature. The services need to be added to Nano servers explicitly before use. So far, they are the most secure servers from Microsoft. They are very lightweight and their resource requirements and consumption is less than 80% of a normal Windows server. The number of services running, the number of ports open, the number of processes running and the amount of memory and storage required, also are less than 80% compared to normal Windows server. Even though Nano Server out of box just has the kernel and drivers, its capabilities can be enhanced by adding features and deploying any Windows application on it. Windows Containers and Docker Containers are one of the most revolutionary features added to Windows Server 2016 after Nano Server. With the popularity and adoption of Docker Containers, which primarily run on Linux, Microsoft decided to introduce container services to Windows Server 2016. Containers are operating system virtualization. This means that multiple containers can be deployed on the same operating system and each one of them will share the host operating system kernel. It is the next level of virtualization after server virtualization (virtual machines). Containers generate the notion of complete operating system isolation and independence, even though it uses the same host operating system underneath it. This is possible through the use of namespace isolation and image layering. Containers are created from images. Images are immutable and cannot be modified. Each image has a base operating system and a series of instructions that are executed against it. Each instruction creates a new image on top of the previous image and contains only the modification. Finally, a writable image is stacked on top of these images. These images are combined into a single image, which can then be used for provisioning containers. A container made up of multiple image layers is shown in Figure 3: Figure 3: Containers made up of multiple image layers Namespace isolation helps provide containers with pristine new environments. The containers cannot see the host resources and the host cannot view the container resources. For the application within the container, a complete new installation of the operating system is available. The containers share the host's memory, CPU, and storage. Containers offer operating system virtualization, which means the containers can host only those operating systems supported by the host operating system. There cannot be a Windows container running on a Linux host, and a Linux container cannot run on a Windows host operating system. Hyper-V containers Another type of container technology Windows Server 2016 provides is Hyper-V Containers. These containers are similar to Windows Containers. They are managed through the same Docker client and extend the same Docker APIs. However, these containers contain their own scaled down operating system kernel. They do not share the host operating system but have their own dedicated operating system, and their own dedicated memory and CPU assigned in exactly the same way virtual machines are assigned resources. Hyper-V Containers brings in a higher level of isolation of containers from the host. While Windows Containers runs in full trust on the host operating system, Hyper-V Containers does not have full trust from the host’s perspective. It is this isolation that differentiates Hyper-V Containers from Windows Containers. Hyper-V Containers is ideal for hosting applications that might harm the host server affecting every other container and service on it. Scenarios where users can bring in and execute their own code are examples of such applications. Hyper-V Containers provides adequate isolation and security to ensure that applications cannot access the host resources and change them. Nested virtual machines Another breakthrough innovation of Windows Server 2016 is that now, virtual machines can host virtual machines. Now, we can deploy multiple virtual machines containing all tiers of an application within a single virtual machine. This is made possible through software-defined networks and storage. Enabling Microservices Nano Servers and Containers helps provide advanced lightweight deployment options through which we can now decompose the entire application into multiple smaller, independent services, each with their own scalability and high availability configuration, and deploy them independently of each other. Microservices helps in making the entire DevOps lifecycle agile. With Microservices, changes to services do not demand that every other Microservices undergo every test validation. Only the changed service needs to be tested rigorously, along with its integration with other services. Compare this to a monolithic application. Even a single small change will result in having to test the entire application. Microservices helps in that it requires smaller teams for its development, testing of a service can happen independently of other services, and deployment can be done for each service in isolation. Continuous Integration, Continuous Deployment, and Continuous Delivery for each service can be executed in isolation rather than compiling, testing, and deploying the whole application every time there is a change. Reduced maintenance Because of their intrinsic nature, Windows Nano Servers and Containers are lightweight and quick to provision. They help to quickly provision and configure environments, thereby reducing the overall time needed for Continuous Integration and deployment. Also, these resources can be provisioned on Azure on-demand without waiting for hours. Because of their small footprint in terms of size, storage, memory, and features, they need less maintenance. These servers are patched less often, with fewer fixes, they are secure by default, and have less chance of failing applications, which makes them ideal for operations. The operations team needs to spend fewer hours maintaining these servers compared to normal servers. This reduces the overall cost for the organization and helps DevOps ensure a high-quality delivery. Configuration management tools Windows Server 2016 comes with Windows Management Framework 5.0 installed by default. Desired State Configuration (DSC) is the new configuration management platform available out of the box in Windows Server 2016. It has a rich, mature set of features that enables configuration management for both environments and applications. With DSC, the desired state and configuration of environments are authored as part of Infrastructure as Code and executed on every server on a scheduled basis. They help check the current state of servers with the documented desired state and bring them back to the desired state. DSC is available as part of PowerShell and PowerShell helps with authoring these configuration documents. Windows server 2016 provides a PowerShell unit testing framework known as PESTER. Historically, unit testing for infrastructure environments was always missing as a feature. PESTER enables the testing of infrastructure provisioned either manually or through Infrastructure as Code using DSC configuration or ARM templates. These help with the operational validation of the entire environment, bringing in a high level of cadence and confidence in Continuous Integration and deployment processes. Deployment and packaging Package management and the deployment of utilities and tools through automation is a new concept in the Windows world. Package management has been ubiquitous in the Linux world for a long time. Packing management helps search, save, install, deploy, upgrade, and remove software packages from multiple sources and repositories on demand. There are public repositories such as Chocolatey and PSGallery available for storing readily deployable packages. Tools such as NuGet can connect these repositories and help with package management. They also help with the versioning of packages. Applications that rely on a specific package version can download it on an as-needed basis. Package management helps with the building of environments and application deployment. Package deployment is much easier and faster with this out-of-the-box Windows feature. Summary We have covered a lot of ground in this article. DevOps concepts were discussed mapping technology to those concepts. In this we saw the impetus DevOps can get from technology. We looked at cloud computing and the different services provided by cloud providers. From there, we went on to look at the benefits Windows Server 2016 brings to DevOps practices and how Windows Server 2016 makes DevOps easier and faster with its native tools and features. Resources for Article: Further resources on this subject: Introducing Dynamics CRM [article] Features of Dynamics GP [article] Creating Your First Plug-in [article]
Read more
  • 0
  • 0
  • 11686

article-image-manage-security-excel
Packt
11 Nov 2016
10 min read
Save for later

Manage Security in Excel

Packt
11 Nov 2016
10 min read
In this article by Belinda Allen, author of the book Building Dashboards with Microsoft Dynamics GP 2016 - Second Edition, we would learn how refreshable Excel reports are easy to deploy, easy to update, and easy to work with. That's why, they make a great foundation for an Excel 2016 dashboard. In this article, you will learn how to manage security to Excel reports and run these Excel reports from Dynamics GP 2016 as well as Excel 2016. (For more resources related to this topic, see here.) Security By default, users can view Excel reports and data connections only if they have administrative credentials on the server that is running the SQL Server and if they have access to the network share. Since this isn't a normal setup, users typically need reporting privileges in the SQL Server before they can view the Microsoft Dynamics GP data that is displayed in data connections and Excel reports. There are three areas of security around Excel reports deployed to a network share or local drive: Security to the network share/local folder Security at the database level Security around Excel We'll spend a few minutes on each one. Network share security Realistically, network share security is normally going to be set by a network administrator. To make a shortcut for administrators, the minimum required security on the shared folder is: Change option for the share tab Read option for the security tab Now, for those of you who want the version that is longer than a (as Mark Polino would say) Latvian wiener dog, follow these steps: In Windows Explorer, right-click on the folder where you deployed the Excel reports and then click on Sharing and Security… On the Sharing tab, click on Advanced Sharing… and select Share this folder. Click on Permissions. If the user or group already exists in this window, you can skip to the next step. Otherwise, follow these steps: Click on Add… In the Select Users, Computers, or Groups window, enter the group or users to whom you want to provide access to the shared reports Click on OK Select the user or group to apply permission to in the Group or user names area. Select the Allow checkbox for the Change permission and then click on OK. The Change permission is the minimum required permission. Click on the Security tab. In the Groups or user names area, click on Add. If the user or group already exists in this window, you can skip to the next step. Otherwise, follow these steps: In the Select Users, Computers, or Groups window, enter the group or the users to whom you want to provide access to the shared reports Click on OK In the Groups or user names area, select each group or user, and then click on the permission that you want the group or the user to have. The minimum required permission is Read. Click on OK. These instructions will vary depending on the version of the Windows server used on the network or the user's version of Windows on a local drive. If you are unsure about setting this up, consult your IT department. By default, Dynamics GP 2016 deploys reports related to each company and each functional area in their own network folder. This makes it easy to apply different permission levels to sensitive areas such as payroll. Database-level security Access to information in the Dynamics GP 2016 database is handled a little differently. A set of fixed security roles is created automatically in the SQL Server when Excel reports are deployed. All of these roles start with rpt_. These roles provide access to the underlying tables and views. The process to assign security is to add a user or group to the SQL Server and give them access to the appropriate roles. The users that get added are not Dynamics GP users. They are either SQL Server users (different from the GP login IDs) or active directory users and groups. To connect the SQL role with an Excel report to ensure that a user has appropriate access, you really need the spreadsheet from Microsoft that links the two together. You can find it at https://mbs.microsoft.com/fileexchange/?fileID=e4bb6958-0f07-4451-b72c-f02784e484df. This spreadsheet is from version GP 10, but it still works for GP 2016. In our example, we need access to the Account Summary Default Excel sheet. This sheet uses the Account Summary view. On the spreadsheet, we see a number of roles that include the appropriate access: For our example, we'll give a user access to the rpt_accounting manager role. In practice, it's not unusual to add all GP users to a single active directory group and give that group access to all the fixed reporting roles. This is particularly true for companies that don't use payroll and that don't have other sensitive reporting requirements. To grant database permission using the built-in roles, we have to add the user or group to the SQL Server and then assign the appropriate role(s). To add a user to SQL Server, follow these steps: Open SQL Server Management Studio and log in using either Windows Authentication or SQL Server Authentication. Go to Security | Logins. Right-click on Logins and select New Login… Click on Search. Enter the domain and user you want to add or enter the group that you want to add to the SQL Server. For my example, I'm entering my domain and user name—Njevityballen. This could also be a group of users such as GPUSERS, for example: Click on Check Names to validate the entry and click twice on OK to finish. The user has now been added to the SQL Server. Our example used a domain user, but you can also set up a SQL user. In general, a domain user is preferred, because it eliminates the need for the user to manage multiple logins and passwords for reporting. Using a domain login also provides additional control to administrators. If an employee leaves, for example, removing them from the domain removes both their network access and their reporting access in one step. To grant access to the reporting roles, follow these steps: Go to Security | Logins, double-click the user or group that you just created. Select User Mapping on the left-hand side. In the upper-center section labeled Users mapped to this login:, select the box next to the company that you want to grant report access to. For our example, select TWO. In the lower-center section named Database role membership for: TWO, select the box next to rpt_Accounting Manager: Click on OK to continue. The user now has rights to access the TWO AccountSummary default report that we've been working with and any other reports available as part of the rpt_Accounting Manager role. Excel 2016 security As you connect with database connections in Excel, a security bar may pop up with the message SECURITY WARNING External Data Connections have been disabled: This is an Excel security feature designed to prevent malicious code from running without the user's knowledge. In our case, however, we deployed the reports. We are now running them on our network and controlling access. This is about as secure as it's going to get, and the message is really annoying for users. Let's turn it off. To disable the Excel security message for these files, follow these steps: Open Microsoft Excel 2016 and go to File | Options | Trust Center. Go to Trusted Center Settings | Trusted Locations. Click on Add new location. Browse to the location where you deployed the Excel reports. In my example, I used C:GP2016XL. Click on OK. Select the box marked Subfolders of this location are also trusted and click on OK: Click on OK twice to exit. Now, when you run the Excel reports in the next section, the reports will open in Excel 2016 without the security warning. Microsoft offers a great knowledge base article on Excel reports and security at http://support.microsoft.com/kb/949524 for GP 10, but this portion of security remains the same. Running Excel reports Our next step is to run an Excel report. These reports can be run from Dynamics GP 2016, or they can be directly opened in Excel 2016. We will look at both these options. From Dynamics GP 2016 To run an Excel report from within Dynamics GP, follow these steps: In the navigation pane on the left-hand side, click on Financial. The list pane above will change to show financial items. In the list pane, click on Excel Reports. In the navigation list in the center, select TWO AccountSummary Default. Make sure that you select the Option column's options that includes Reports: Options that contain the word Reports open Excel reports. Options with Data Connections in the string indicate the data connector to build a new report, not an actual report. You can limit the Excel reports list to just Reports or Data Connections with the Add Filter button just above the Excel reports list. Double-click on the TWO AccountSummary Default item. We disabled the security warning earlier, but just in case, if Excel 2016 opens with a security warning at the top of the worksheet, click on Enable Content. Excel will open with live data from Microsoft Dynamics GP: As a test, highlight rows seven through 10 (7-10) on the left-hand side and press the Delete key. Go to Data | Refresh All on the ribbon. Excel 2016 will reconnect to Dynamics GP and bring back in the latest data. Saving the report with a different name in the same folder as the GP deployed reports will make that report visible in the list of Excel reports in GP. From Excel 2016 To accomplish this same task (run a GP Excel refreshable report) from Excel 2016, follow these steps: Open Windows Explorer and navigate to the location where you deployed the reports at the beginning of this article. In my example, the reports were deployed to C:GP2016XL. Drill down through the folders to Reports | TWO | Financial. This represents the report storage for the sample company's (TWO) financial reports: Double-click on TWO AccountSummary Default.xlsx. Excel 2016 will open with live data from Dynamics GP. Manual versus auto refresh Excel reports are refreshable, but that doesn't mean that they have to refresh automatically. Often accountants ask about saving a static version of the file. They love the idea of refreshing data, but they want it to happen on their terms. Most accountants prefer information that doesn't change once it's been finalized, so this request is perfectly natural. By default, the Dynamics GP 2016 connections are designed to refresh automatically when the file is opened, but you can control this. To understand how to control the refresh options, follow these steps: Start with the TWO AccountSummary Default Excel file that you already have open. In Excel, select the Data tab and then go to Connections | Properties: Uncheck the Refresh data when opening the file box and click on OK. Click on Close to return to the worksheet in Excel. To validate that this worked, select rows seven through 10 (7-10) in the Excel sheet and press Delete. Save the Excel sheet to your desktop as TWO AccountSummary Default Manual Refresh and close Excel 2016. To reopen the file, double-click on TWO AccountSummary Default Manual Refresh on the desktop. Excel will open with data, and rows seven through 10 (7-10) will be blank. The sheet did not refresh automatically. To manually refresh the sheet, right-click anywhere in the data area and click Refresh or select Data | Refresh All. Summary We've looked at one of the best methods for getting data for our dashboard. We've deployed, secured, run, and built Excel reports. Now that we've thoroughly explored one of the best ways to get real-time data out of Dynamics GP 2016 and into Microsoft Excel Resources for Article: Further resources on this subject: Planning: Microsoft Dynamics GP System [article] Microsoft Dynamics NAV: OS Integration [article] Microsoft Dynamics NAV 2009 Development Tools [article]
Read more
  • 0
  • 0
  • 11750

article-image-creating-reusable-generic-modals-react-and-redux
Mark Erikson
11 Nov 2016
6 min read
Save for later

Creating Reusable Generic Modals in React and Redux

Mark Erikson
11 Nov 2016
6 min read
Modal dialogs are a common part of user interface design. As with most other parts of a UI, modals in a given application probably fall into two general categories: modals that are specific to a given feature or task, and modals that are intended to be generic and reusable. However, defining generic reusable modal components in a React/Redux application presents some interesting challenges. Here's one approach you can use to create generic reusable modals that can be used in a variety of contexts throughout a React/Redux application. First, we need a way to manage modals in general. In a typical object-oriented widget API, we might manually create an instance of a modal class, and pass in some kind of callback function to do something when it's closed. Here's what this might look like for a ColorPicker modal in an OOP API: const colorPickerInstance = new ColorPicker({ initialColor : "red", onColorPicked(color) { // do something useful with the "returned" color value } }); colorPickerInstance.show(); This presents some problems, though. Who really "owns" the ColorPicker? What happens if you want to show multiple modals stacked on each other? What happens with the ColorPicker instance while it's being displayed? In a React/Redux application, we really want our entire UI to be declarative, and to be an output of our current state. Rather than imperatively creating modal instances and calling show(), we'd really like any nested part of our UI to be able to "request" that some modal be shown, and have the state and UI updated appropriately to show the modal. Dan Abramov describes a wonderful approach on Stack Overflow to React/Redux modal management, in response to a question about displaying modal dialogs in Redux. It's worth reading his answer in full, but here's a summary: Dispatch an action that indicates you want to show a modal. This includes some string that can be used to identify which modal component should be shown, and includes any arbitrary values we want to be passed along to the rendered modal component: dispatch({ type : 'SHOW_MODAL", payload : { modalType : "SomeModalComponentIdentifier", modalProps : { // any arbitrary values here that we want to be passed to the modal } } }); Have a reducer that simply stores the modalType and modalProps values for 'SHOW_MODAL', and clears them for 'HIDE_MODAL'. Create a central component that connects to the store, retrieves the details ofwhat modal is open and what its props should be, looks up the correct component type, and renders it: import FirstModal from "./FirstModal"; import SecondModal from "./SecondModal"; // lookup table mapping string identifiers to component classes const MODAL_COMPONENTS = { FirstModal, SecondModal }; const ModalRoot = ({modalType, modalProps}) => { if(!modalType) return null; const SpecificModal = MODAL_COMPONENTS[modalType]; return <SpecificModal {...modalProps} /> } const mapState = state => state.modal; export default connect(mapState)(ModalRoot); From there, each modal component class can be connected to the store, retrieve any other needed data, and dispatch specific actions for both internal behavior as well as ultimately dispatching a 'HIDE_MODAL' action when it's ready to close itself. This way, the handling of modal display is centralized, and nested components don't have to "own" the details of showing a modal. Unfortunately, this pattern runs into a problem when we want to create and use a very generic component, such as a ColorPicker. We would probably want to use the ColorPicker in a variety of places and features within the UI, each needing to use the "result" color value in a different way, so having it dispatch a generic 'COLOR_SELECTED' action won't really suffice. We could include some kind of a callback function within the action, but that's an anti-pattern with Redux, because using non-serializable values in actions or state can break features like time-travel debugging. What we really need is a way to specify behavior specific to a feature, and use that from within the generic component. The answer that I came up with is to have the modal component accept a plain Redux action object as a prop. The component that requested the dialog be shown should specify that action as one of the props to be passed to the modal. When the modal is closed successfully, it should copy the action object, attach its "return value" to the action, and dispatch it. This way, different parts of the UI can use the "return value" of the generic modal in whatever specific functionality they need. Here's how the different pieces look: // In some arbitrary component: const onColorSelected = { type : 'FEATURE_SPECIFIC_ACTION', payload : { someFeatureSpecificData : 42, } }; this.props.dispatch({ type : 'SHOW_MODAL", payload : { modalType : "ColorPicker", modalProps : { initialColor : "red", // Include the pre-configured action object as a prop for the modal onColorSelected } } }); // In the ColorPicker component: handleOkClicked() { if(this.props.onColorSelected) { // If the code that requested this modal included an action object, // clone the action, attach our "return value", and dispatch it const clonedAction = _.clone(this.props.onColorSelected); clonedAction.payload.color = this.state.currentColor; this.props.dispatch(clonedAction); } this.props.hideModal(); } // In some reducer: function handleFeatureSpecificAction(state, action) { const {payload} = action; // Use the data provided by the original requesting code, as well as the // "return value" given to us by the generic modal component const {color, someFeatureSpecificData} = payload; return { ...state, [someFeatureSpecificData] : { ...state[someFeatureSpecificData], color } }; } This technique satisfies all the constraints for our problem. Any part of our application can request that a specific modal component be shown, without needing a nested component to "own" the modal. The display of the modal is driven by our Redux state. And most importantly, we can specify per-feature behavior and use "return values" from generic modals while keeping both our actions and our Redux state plain and serializable, ensuring that features like time-travel debugging still work correctly. About the author Mark Erikson is a software engineer living in southwest Ohio, USA, where he patiently awaits the annual heartbreak from the Reds and the Bengals. Mark is author of the Redux FAQ, maintains the React/Redux Links list and Redux Addons Catalog, and occasionally tweets at @acemarke. He can be usually found in the Reactiflux chat channels, answering questions about React and Redux. He is also slightly disturbed by the number of third-person references he has written in this bio!
Read more
  • 0
  • 0
  • 40946

article-image-introduction-c-and-net
Packt
11 Nov 2016
17 min read
Save for later

Introduction to C# and .NET

Packt
11 Nov 2016
17 min read
In this article by Marino Posadas, the author of the book, Mastering C# and .NET Programming, we will cover the core concepts of C# and .NET, starting from the initial version and principal motivations behind its creation, and covering also the new aspects of the language, that appeared in version 2.0 and 3.0. (For more resources related to this topic, see here.) We'll illustrate all the main concepts with small code snippets, short enough to facilitate its understanding and easy reproduction. We will cover the following topics: C# and its role in the Microsoft Development ecosystem Difference between strongly typed and weakly typed languages The evolution in versions 2.0 and 3.0 Generics Extension methods C#: what's different in the language I had the chance to chat with Hejlsberg a couple of times about the C # language and what the initial purposes and requirements imposed in its creation were and which other languages inspired him or contributed to his ideas. The first time we talked, in Tech-Ed 2001 (at Barcelona, Spain), I asked him about the principles of his language and what makes it different from others. He first said that it was not only him who created the language, but also a group of people, especially Scott Wiltamuth, Peter Golde, Peter Sollich, and Eric Gunnerson. One of the first books ever published on the subject was, A Programmer's Introduction to C#, Gunnerson's.E., APress, 2000). About the principles, he mentioned this: One of the key differences between C# and these other languages, particularly Java, is that we tried to stay much closer to C++ in our design. C# borrows most of its operators, keywords, and statements directly from C++. But beyond these more traditional language issues, one of our key design goals was to make the C# language component-oriented, to add to the language itself all of the concepts that you need when you write components. Concepts such as properties, methods, events, attributes, and documentation are all first-class language constructs. He stated also this: When you write code in C#, you write everything in one place. There is no need for header files, IDL files (Interface Definition Language), GUIDs and complicated interfaces. This means that you can write code that is self-descriptive in this way given that you're dealing with a self-contained unit (let's remember the role of the manifest, optionally embedded in assemblies). In this mode, you can also extend existing technologies in a variety of ways, as we'll see in the examples. Languages: strongly typed, weakly typed, dynamic, and static The C# language is a strongly typed language: this means that any attempt to pass a wrong kind of parameter as an argument, or to assign a value to a variable that is not implicitly convertible, will generate a compilation error. This avoids many errors that only happen at runtime in other languages. In addition, by dynamic, we mean those languages whose rules are applied at runtime, while static languages apply their rules at compile time. JavaScript or PHP are good examples of the former case, and C/C++ of the latter. If we make a graphic representation of this situation, we might come up with something like what is shown in the following figure: In the figure, we can see that C# is clearly strongly typed, but it's much more dynamic than C++ or Scala, to mention a few. Of course, there are several criteria to catalog languages for their typing (weak versus strong) and for their dynamism (dynamic versus static). Note that this has implications in the IDE as well. Editors can tell us which type is expected in every case, and if you use a dynamic declaration such as var, the right side of the equality (if any) will be evaluated, and we will be shown the calculated value for every declaration: Even outside of the .NET world, Visual Studio's IDE is now able to provide strongly typed and Intellisense experiences when using languages such as TypeScript, a superset of JavaScript that transpiles (converts into) pure JavaScript but can be written using the same coding experience as what we would have in C# or any other .NET language. It's available as a separate type of project, if you're curious about it, and the latest up-to-date version is TypeScript 1.8, and it was recently published (you can take a look at a detailed description of its new capabilities at https://blogs.msdn.microsoft.com/typescript/2016/02/22/announcing-typescript-1-8-2/). The main differences So, going back to the title, what made C# different? I'll point out five core points: Everything is an object. Other languages, such as Smalltalk, Lisp, among others, have done this earlier, but due to different reasons, the performance penalty was pretty hard. As you know, it's enough to take a look at the Object Explorer to be able to check where an object comes from. It's a good practice to check the very basic values, such as int or String, which are nothing but aliases of System.Int32 and System.String, and both come from object, as shown in the following screenshot: Using the Boxing and Unboxing techniques, any value type can be converted into an object, and the value of an object can be converted into a simple value type. These conversions are made by simply casting the type to an object (and vice versa) in this manner: // Boxing and Unboxing int y = 3; // this is declared in the stack // Boxing y in a Heap reference z // If we change z, y remains the same. object z = y; // Unboxing y into h (the value of // z is copied to the stack) int h = (int)z; Using Reflection (the technique that allows you to read a component's metadata), an application can call itself or other applications, creating new instances of their containing classes. As a short demo, this simple code launches another instance of a WPF application (a very simple one with just one button, but that doesn't matter): static short counter = 1; private void btnLaunch_Click(object sender, RoutedEventArgs e) { // Establish a reference to this window Type windowType = this.GetType(); // Creates an instance of the Window object objWindow = Activator.CreateInstance(windowType); // cast to a MainWindow type MainWindow aWindow = (MainWindow)objWindow; aWindow.Title = "Reflected Window No: " + (++counter).ToString(); aWindow.Show(); } Now, every time we click on the button, a new instance of the window is created and launched, indicating its creation order in the title's window: You can have access to other components through a technology called Platform Invoke, which means you can call operating systems' functions by importing the existing DLLs using the DllImport attribute: For instance, you can make an external program's window the child of your own window using the SetParent API, which is part of User32.dll, or you can control operating system events, such as trying to shut down the system while our application is still active. Actually, once the permissions are given, your application can call any function located in any of the system's DLL if you need access to native resources. The schema that gives us access to these resources looks like what is shown in the following figure: If you want to try out some of these possibilities, the mandatory resource to keep in mind is http://www.PInvoke.net, where you have most of the useful system APIs, with examples of how to use them in C#. These interoperation capabilities are extended to interactions with applications that admit Automation, such as those in the Microsoft Office Suite, AutoCAD, and so on. Finally, unsafe code allows you to write inline C code with pointers, perform unsafe casts, and even pin down memory in order to avoid accidental garbage collection. However, unsafe does not mean that it is unmanaged. Unsafe code is deeply tied into the security system. There are many situations in which this is very useful. It might be an algorithm that's difficult to implement or a method whose execution is so CPU-intensive that performance penalties become unacceptable. While all this is important, I was surprised by the fact that every event handler in C# (as also in other .NET languages) would have two and only two arguments. So, I asked Anders about it, and his answer was one of the most clear and logical ones that I've ever heard. The evolution in versions 2.0 and 3.0 As we see, even from the very beginning, the Hejlsberg's team started with a complete, flexible, and modern platform, capable to be extended in many ways as technology evolves. This intention became clear since version 2.0. The first actual fundamental change that took place in the language was the incorporation of Generics. Don Syme, who would later on lead the team that created the F# language, was very active and led this team as well, so it was ready for version 2.0 of the .NET Framework (not just in C# but in C++ and VB.NET as well). Generics The purpose of generics was mainly to facilitate the creation of more reusable code (one of the principles of OOP, by the way). The name refers to a set of language features that allow classes, structures, interfaces, methods, and delegates to be declared and defined with unspecified or generic type parameters instead of specific types (see https://msdn.microsoft.com/en-us/library/ms379564(v=vs.80).aspx, for more details). So, you can define members in a sort of abstract definition, and later on, at the time of using it, a real, concrete type will be applied. The basic .NET classes (BCL) were enhanced in the System namespace and a new System.Collections.Generic namespace was created to support this new feature in depth. In addition, new support methods were added to ease the use of this new type, such as Type.IsGenericType (obviously, to check types), Type.GetGenericArguments (self-descriptive), and the very useful Type.MakeGenericType, which can create a generic type of any kind from a previous nonspecified declaration. The following code uses the generic type definition for a Dictionary (Dictionary<,>) and creates an actual (build) type using this technique. The relevant code is the following (the rest, including the output to the console is included in Demo_02_03): // Define a generic Dictionary (the // comma is enough for the compiler to infer number of // parameters, but we didn't decide the types yet. Type generic = typeof(Dictionary<,>); ShowTypeData(generic); // We define an array of types for the Dictionary (Key, Value) // Key is of type string, and Value is of -this- type (Program) // Notice that types could be -in this case- of any kind Type[] typeArgs = { typeof(string), typeof(Program) }; // Now we use MakeGenericType to create a Type representing // the actualType generic type. Type actualType = generic.MakeGenericType(typeArgs); ShowTypeData(actualType); As you see, MakeGenericType expects an array of (concrete) types. Later on (not in the preceding code), we use GetGenericTypeDefinition, IsGenericType, and GetGenericArguments in order to introspect the resulting types and present the following output in the console: So, we have different ways to declare generics with identical results as far as the operations in the code are concerned. Obviously, manipulating already constructed generic types is not the only possibility, since one of the main goals of generics is to avoid casting operations by simplifying the work with collections. Up until version 2.0, collections could only hold basic types: integers, longs, strings, and so on, along with emulating different types of data structures, such as stacks, queues, linked lists, and so on. Besides this, Generics have another big advantage: you can write methods that support working with different types of arguments (and return values) as long as you provide a correct way to handle all possible cases. Once again, the notion of contract will be crucial here. Creating custom generic types and methods Other useful feature is the possibility to use custom generic types. Generic types and the support for optional values through the System.Nullable<T> type were, for many developers, two of the most important features included in version 2.0 of the language. Imagine you have a Customer class, which your application manages. So, in different use cases, you will read collections of customers and perform operations with them. Now, what if you need an operation such as Compare_Customers? What would be the criteria to use in this case? Even worse, what if we would like to use the same criteria with different types of entities, such as Customer and Provider? In these cases, some characteristics of generics come in handy. To start with, we can build a class that has an implementation of the IComparer interface, so we establish out of any uncertainty what the criteria to be used is in order to consider customer C1 bigger or smaller than customer C2. For instance, if the criteria is only Balance, we can start with a basic Customer class, to which we add a static method in order to generate a list of random customers: public class Customer { public string Name { get; set; } public string Country { get; set; } public int Balance { get; set; } public static string[] Countries = { "US", "UK", "India", "Canada", "China" }; public static List<Customer> customersList(int number) { List<Customer> list = new List<Customer>(); Random rnd = new Random(System.DateTime.Now.Millisecond); for (int i = 1; i <= number; i++) { Customer c = new Customer(); c.Name = Path.GetRandomFileName().Replace(".", ""); c.Country = Countries[rnd.Next(0, 4)]; c.Balance = rnd.Next(0, 100000); list.Add(c); } return list; } } Then, we build another CustomerComparer class, which implements the IComparer interface. The difference is that this comparison method is a generic instantiation customized for the Customer objects, so we have the freedom of implementing this scenario just in the way that seems convenient for our logic. In this case, we're using Balance as an ordering criteria, so that we would have the following: public class CustomerComparer : IComparer<Customer> { public int Compare(Customer x, Customer y) { // Implementation of IComparer returns an int // indicating if object x is less than, equal to or // greater than y. if (x.Balance < y.Balance) { return -1; } else if (x.Balance > y.Balance) return 1; else { return 0; } // they're equal } } We can see that the criteria used to compare is just the one we decided for our business logic. Finally, another class, GenericCustomer, which implements an entry point of the application, uses both classes in this manner: public class GenericCustomers { public static void Main() { List<Customer> theList = Customer.customersList(25); CustomerComparer cc = new CustomerComparer(); // Sort now uses our own definition of comparison theList.Sort(cc); Console.WriteLine(" List of customers ordered by Balance"); Console.WriteLine(" " + string.Concat(Enumerable.Repeat("-", 36))); foreach (var item in theList) { Console.WriteLine(" Name: {0}, Country: {1}, t Balance: {2}", item.Name, item.Country, item.Balance); } Console.ReadKey(); } } This produces an output of random customers order by their balance: This is even better: we can change the method so that it supports both customers and providers indistinctly. To do this, we need to abstract a common property of both entities that we can use for comparison. If our implementation of Provider has different or similar fields (but they're not the same), it doesn't matter as long as we have the common factor: a Balance field. So we begin with a simple definition of this common factor, an interface called IPersonBalance: public interface IPersonBalance { int Balance { get; set; } } As long as our Provider class implements this interface, we can later create a common method that's able to compare both objects, so, let's assume our Provider class looks like this: public class Provider : IPersonBalance { public string ProviderName { get; set; } public string ShipCountry { get; set; } public int Balance { get; set; } public static string[] Countries = { "US", "Spain", "India", "France", "Italy" }; public static List<Provider> providersList(int number) { List<Provider> list = new List<Provider>(); Random rnd = new Random(System.DateTime.Now.Millisecond); for (int i = 1; i <= number; i++) { Provider p = new Provider(); p.ProviderName = Path.GetRandomFileName().Replace(".", ""); p.ShipCountry = Countries[rnd.Next(0, 4)]; p.Balance = rnd.Next(0, 100000); list.Add(p); } return list; } } Now, we rewrite the Comparer method to be a GenericComparer class, capable of dealing with both types of entities: public class GenericComparer : IComparer<IPersonBalance> { public int Compare(IPersonBalance x, IPersonBalance y) { if (x.Balance < y.Balance) { return -1; } else if (x.Balance > y.Balance) return 1; else { return 0; } } } Note that in this implementation, IComparer depends on an interface, not on an actual class, and that this interface simply defines the common factor of these entities. Now, our new entry point will put everything together in order to obtain an ordered list of random Provider classes that uses the common comparison method just created: public static void Main() { List<Provider> providerList = Provider.providersList(25); GenericComparer gc = new GenericComparer(); // Sort now uses our own definition of comparison providerList.Sort(gc); Console.WriteLine(" List of providers ordered by Balance"); Console.WriteLine(" " + ("").PadRight(36, '-')); foreach (var item in providerList) { Console.WriteLine(" ProviderName: {0}, S.Country: {1}, t Balance: {2}", item.ProviderName, item.ShipCountry, item.Balance); } Console.ReadKey(); } In this way, we obtain an output like what is shown in the following figure (note that we didn't take much care of formatting in order to focus on the process): The example shows how generics (and interfaces: also generic) come to our rescue in these type of situations, and—as we'll have the opportunity to prove when talking about implementations of design patterns—this is key to facilitating good practices. So far, some of the most critical concepts behind generics have been discussed. However, the real power comes from joining these capabilities with two new features of the language: lambda expressions and the LINQ syntax. Extension methods Finally, we can extend existing classes' functionality. This means extending even the .NET Framework base types, such as int or String. This is a very useful feature, and it's performed in the way it is recommended by the documentation; no violation of basic principles of OOP occur. The procedure is fairly simple. We need to create a new public static top level (not nested) class containing a public static method with an initial argument declaration especially suited for the compiler to assume that the compiled code will be appended to the actual functionality of the type. The procedure can be used with any class, either belonging to the .NET framework or a customized user or class. Once we have the declaration, its usage is fairly simple, as shown in this code: public static class StringExtension { public static string ExtendedString(this string s) { return "{{ " + s + " }}"; } } Note that the first argument, referred with the this keyword, references the string to be used; so, in this example, we will call the method without any extra arguments (although we can pass as many arguments as we need for other extensions). To put it to work, we just have to add something like this: Console.WriteLine("The word " + "evaluate".ExtendedString() + " is extended"); We will get the extended output with the word enclosed in double brackets: Summary So in this article we saw some of the most relevant enhancements made to the C# language in versions 2 and 3. We started by reviewing the main differences between C# and other languages and understanding the meaning of strongly typed, in this case, together with the concepts of static and dynamic. We followed this up with an examination of the generics feature that appeared in version 2.0 of the framework and analyzed some samples to illustrate some typical use cases, including the creation of custom generic methods. Finally, we covered the extension methods. Resources for Article: Further resources on this subject: Debugging Your .NET Application [article] Why we need Design Patterns? [article] Creating a NHibernate session to access database within ASP.NET [article]
Read more
  • 0
  • 0
  • 12375
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-getting-started-flocker
Packt
11 Nov 2016
11 min read
Save for later

Getting Started with Flocker

Packt
11 Nov 2016
11 min read
In this article by Russ McKendrick, the author of the book Docker Data Management with Flocker, we are going to look at what both what problems Flocker has been developed to resolve as well as jumping in at the deep end and perform our first installation of Flocker. (For more resources related to this topic, see here.) By the end of the article, you will have: Configured and used the default Docker volume driver Learned a little about how Flocker came about and who wrote it Installed and configured a Flocker control and storage node Integrated your Flocker installation with Docker Have visualized your volumes using the Volume Hub Interacted with your Flocker installation using the Flocker CLI Have installed and used dvol However, before we start to look at both Docker and Flocker we should talk about compute resource. Compute and Storage Resource In later articles, we will be looking at running Flocker on public cloud services such as Amazon Web Services and Google Cloud so we will not be using those services for the practical examples in our early articles, instead I will be server instances launched in DigitalOcean. DigitalOcean has quickly become the platform of choice for both developers & system administrators who want to experiment with new technologies and services. While their cloud service is simple to use their SSD backed virtual machines offer the performance required to run modern software stacks at a fraction of the price of other public clouds making them perfect for launching prototyping and sandbox environments. For more information on DigitalOcean, please see their website at: https://www.digitalocean.com/. The main reason I choose to use an external cloud provider rather than local virtual machine is that we will need to launch multiple hosts to run Flocker. You do not have to use DigitalOcean, as long as you are able to launch multiple virtual machines and attach some sort of block storage to your instances as a secondary device. Finally, where we are doing manual installations of Flocker and Docker I will give instructions, which cover both CentOS 7.2 and Ubuntu 16.04. Docker If you are reading this article then Docker does not need much of an introduction, it is one of the most talked about technologies of recent years quickly gaining support from pretty much all what I would consider the big players in software. Companies such as Google, Microsoft, Red Hat, VMWare, IBM, Amazon Web Services and Hewlett Packard all offer solutions based on Dockers container technology or have contributed towards its development. Rather than me give you a current state of the union on Docker I would recommend you watch the opening key note from the 2016 Docker Conference, it gives a very rundown of how far Docker has come over the last few years. The video can be found at the following URL https://www.youtube.com/watch?v=vE1iDPx6-Ok. Now you are all caught up with what's going in the world of Docker, let's take a look at what storage options you get out of the box with the core Docker Engine. Installing Docker Engine Let start with a small server instance, install Docker and then run a through a couple of scenarios where we may need storage for our containerized application: Before installing Docker, it is always best to check that you have the latest updates installed, to do this run one of the following commands. On CentOS you need to run: yum -y update and for Ubuntu run: apt-get -y update Both of these commands will check for and automatically install any available updates. Once your server instance is up-to-date, you can go-ahead and install Docker, the simplest way of doing this is to run the installation script provided by Docker by entering the following command: curl -fsSL https://get.docker.com/ | sh This method of installation works on both CentOS and Ubuntu, the script run checks to see which packages need to installed and then installs the latest version of the Docker Engine from Docker's own repositories. If you would prefer to check the contents of the script before running it on your server instance you can go to https://get.docker.com/ in your browser, this will display the full bash script so you can see exactly what the script is doing when executed on your server instance. If everything worked as expected, then you will have latest stable version of Docker installed. You check this by running the following command: docker --version This will return the version and build of the version of Docker which was installed, in my case this is Docker version 1.12.0, build 8eab29e. On both operating systems you can check that the Docker is running by using the following command: systemctl status docker If Docker is stopped then you can use the following command to start it: systemctl start docker The Docker local volume driver Now we have Docker installed and running on our server instance, we can start launching containers. From here, the instructions are the same if you are running CentOS or Ubuntu. We are going to be launching an application called Moby Counter, this was written by Kai Davenport to demonstrate maintaining state between containers. It is a Nodejs application which allows you draw (using Docker logos) in your browser window. The coordinates for your drawing are stored in a Redis key value store, the idea being if you stop, start and even remove the Redis service your drawing should persist: Let start by launching our Redis container, to do this run the following command: docker run -itd --name redis redis:alpine redis-server --appendonly Using a backslash: As we will sometimes have a lot options to pass to the commands we are going to be running, we are going to be using to split the command over multiple lines so that it's easier to follow what is going on, like in the command above. The Docker Hub is the registry service provided by Docker. The service allows you to distribute your container images, either by uploading images directly or by building your Docker files which are stored in either GitHub or BitBicket. We will be using the Docker Hub throughout the article, for more information on the Docker Hub see the official documentation at: https://docs.docker.com/docker-hub/. This will download the official Redis image (https://hub.docker.com/_/redis/) from the Docker Hub and then launch a container called Redis by running the redis-server --appendonly command. Adding the --appendonly flag to the Redis server command means that each time Redis receives a command such as SET it will not only execute the command in the dataset which is held in memory, it will also append it to what's called an AOF file. When Redis is restarted the AOF file is replayed to restore the state of the dataset held in RAM. Note that we are using the Alpine Linux version of the container to keep the download size down. Alpine Linux is a Linux distribution which is designed to be small, in fact the official Docker image based on Alpine Linux is only 5 MB in size! Compared to the sizes other base operating system images such as the official Debian, Ubuntu and CentOS which you can see from the following terminal session It is easy to see why the majority of official Docker images are using Alpine Linux as their base image. Now we have Redis up and running, let's launch our application container by running: docker run -itd --name moby-counter -p 80:80 --link redis:redis russmckendrick/moby-counter You check that your two containers are running using the docker ps command, you should see something like the following terminal session: Now you have the Moby Counter application container running, open your browser and go to the IP address of your server instance, in my case this was http://159.203.185.102 once the page loads you should see a white page which says Click to add logos… in the center. As per the instructions, click around the page to add some Docker logos: Now we have some data stored in the Redis store let's do the equivalent of yanking the power cord out of the container by running the following command: docker rm -f redis This will cause your browser to looking something like the following screen capture: Relaunching the Redis container using the following command: docker run -itd --name redis redis:alpine redis-server --appendonly and refreshing our browser takes us back to the page which says Click to add logos… So what gives, why didn't you see your drawing? It's quite simple, we never told Docker to start the container with any persistent storage. Luckily, the official Redis image is smart and assumes that you should really be using persistent storage to store your data in, even if you don't tell it you want to as no-one likes data loss. Using the docker volume ls command you should be able to see that there are two volumes, both with really long names, in my case the output of the command looked like the following terminal capture: So we now have two Docker Volumes, one which contains our original "master piece" and the second which is blank. Before we remove our Redis container again, let's check to see which of the two Docker Volumes is currently mounted on our Redis container, we can do this by running: docker inspect redis There quite a bit of information shown when the docker inspect command, there information we are after is in the Mounts section, and we need to know the Name: As you can see from the terminal output above, the currently mounted volume in my setup is called b76101d13afc2b33206f5a2bba9a3e9b9176f43ce57f74d5836c824c22c. Now we know the name of the blank volume, we know that the second volume has the data for our drawing. Let's terminate the running Redis container by running: docker rm -f redis and now launch our Redis container again, but this time telling Docker to use the local volume driver and also use volume we know contains the data for our drawing: docker run -itd --name redis --volume-driver=local -v 37cb395253624782836cc39be1aa815682b70f73371abb6d500a:/data redis:alpine redis-server --appendonly yes Make sure you replace the name of volume which immediately follows the -v on the fourth line with the name of the volume you know contains your own data. After a few moments, refresh your browser and you should see your original drawing. Our volume was automatically created by the Redis container as it launched which is why it has a Unique Identification Number (UID) as its name, if we wanted to we could give it more friendly name by running the following command to launch our Redis container: docker run -itd --name redis --volume-driver=local -v redis_data:/data redis:alpine redis-server --appendonly yes As you can when running the docker volume ls command again, this is a lot more identifiable: As we have seen with the simple example above, it is easy to create volumes to persist your data on using Docker Engine, however there is one drawback to using the default volume driver and that is your volumes are only available on a single Docker host. For most people just setting out on their journey into using containers this is fine, however for people who want to host their applications in production or would like a more resilient persistent data store using the local driver quickly becomes a single point of failure. This is where Flocker comes in. Summary In this article we have worked through the most basic Flocker installation possible, we have integrated the installation with Docker, the Volume Hub and also the Flocker command to interact with the volumes created and managed by Flocker. In the next article we are going to look at how Flocker can be used to provide more resilient storage for a cluster of Docker hosts rather than just the single host we have been using in this article. Resources for Article: Further resources on this subject: Docker Hosts [article] Hands On with Docker Swarm [article] Understanding Docker [article]
Read more
  • 0
  • 0
  • 2962

article-image-algorithm-analysis
Packt
11 Nov 2016
12 min read
Save for later

Algorithm Analysis

Packt
11 Nov 2016
12 min read
In this article by Prakash and Achyutuni Sri Krishna Rao, authors of the book R Data Structures and Algorithms, we will discuss how an algorithm can be defined as a set of step-by-step instructions which govern the outline of a program that needs to be executed using computational resources. The execution can be in any programming language such as R, Python, and Java. Data is an intricate component of any program, and depending on how data is organized (data structure), your execution time can vary drastically. That’s why data structure is such a critical component of any good algorithm implementation. (For more resources related to this topic, see here.) The sorting algorithm, which acts as a connecter between the user-defined input and user-desired output, can be approached in multiple ways: Bubble sort and Shell sort, which are simple variants of sorting, but are highly inefficient Insertion sort and Selection sort, primarily used for sorting small datasets Merge sort, Heap sort, and Quick sort, which are efficient ways of sorting based on the complexities involved in an average system runtime Distributed sorts such as counting sort, bucket sort, and radix sort, which can handle both runtime and memory usage Each of these options can, in turn, handle a particular set of instances more effectively. This essentially deduces the concept of a “good algorithm”. An algorithm can be termed as “good” if it possesses attributes such as the following among many others: Shorter running time Lesser memory utilization Simplicity in reading the code Generality in accepting inputs This book will concentrate primarily on running time or time complexity, partly on memory utilization, and their relationship during program execution. Introduction A problem can be approached using multiple algorithms, and each algorithm can be assessed based on certain parameters such as: System runtime Memory requirement However, these parameters are generally affected by external environmental factors such as: Handling of data structures System software and hardware configurations Style of writing and compiling codes Programming language As it is highly impossible to control all external parameters, it becomes difficult to estimate the system runtime of multiple algorithms for performance comparison (ideal scenario analysis). Asymptotic analysis is one such technique which can be used to assess an algorithm’s efficiency without actually coding and compiling the entire program. It is a functional form representing a pseudo system runtime based on the size of input data and the number of operations. It is based on the principle that the growth rate of input data is directly proportional to the system runtime. For example, in the case of insertion sorting, the size represents the length of the input vector, and the number of operations represents the complexity of sort operations. This analysis can only be used to gauge the consideration of implementing the algorithm rather than evaluating the merits and demerits of algorithms in comparison. The following table represents the most widely used growth rate functional forms. The most widely used functional forms of growth rates are based on the size of input data, which are used to analyze the performance of algorithms. These are also considered as pseudo-functional forms to evaluate an algorithm’s system runtime. Memory management in R Memory management primarily deals with the administration of available memory and prediction of additional memory required for smoother and faster execution of functions. The current section will cover the concept of memory allocation, which deals with storage of an object in the R environment. Memory Allocation: R allocates memory differently to different objects in its environment. Memory allocation can be determined using the object_size function from the pryr package. The pryr package can be installed from the CRAN repository using install.packages(“pryr”). The object_size function in pryr is similar to the object.size function in the base package. However, it is more accurate as it: Takes into account the environment size associated with the current object Takes into account the shared elements within a given object under consideration The following are examples of using the object_size function in R to evaluate memory allocation: > object_size(1) ## Memory allocated for a single numeric vector 48 B > object_size(“R”) ## Memory allocated for a single character vector 96 B > object_size(TRUE) ## Memory allocated for a single logical vector 48 B > object_size(1i) ## Memory allocated for a single complex vector 56 B The storage required by an object can be attributed to the following parameters: Metadata: Metadata of an object is defined by the type of object used such as character, integers, logical, and so on. The type can also usually be helpful during debugging. Node pointer: The node pointer maintains the link between the different nodes, and depending on the number of node pointers used, memory requirement changes. For example, a doubly linked list requires more memory than a singly linked list, as it uses two node pointers to connect to the previous and next nodes. Attribute pointer: Pointer to keep reference for attributes; this helps to reduce memory allocation, especially the data stored by a variable. Memory allocation: Length of the vector representing the currently used space Size: Size represents the true allocated space length of the vector. Memory Padding: Padding applied to a component, for example, each element begins after an 8-byte boundary. The Object_size() command is also used to see the inherent memory allocation as shown in the following table: The preceding table shows inherent memory allocated by each data structure/type. Let’s simulate scenarios with varying lengths of a vector with different data types such as integer, character, Boolean, and complex. The simulation is performed by simulating a vector length from 0 to 60 as follows: > vec_length <- 0:60 > num_vec_size <- sapply(vec_length, function(x) object_size(seq(x))) > char_vec_size <- sapply(vec_length, function(x) object_size(rep(“a”,x))) > log_vec_size <- sapply(vec_length, function(x) object_size(rep(TRUE,x))) > comp_vec_size <- sapply(vec_length, function(x) object_size(rep(“2i”,x))) Num_vec_size computes the memory requirement for each numeric vector from zero to 60 number of elements. These elements are integers increasing sequentially, as stated in the function. Similarly, incremental memory requirements are calculated for character (char_vec_size), logical (log_vec_size), and complex (comp_vec_size) vectors. The result obtained from the simulation can be plotted using code. > par(mfrow=c(2,2)) > plot(num_vec_size ~ vec_length, xlab = “Numeric seq vector”, ylab = “Memory allocated (in bytes)”, + type = “n”) > abline(h = (c(0,8,16,32,48,64,128)+40), col = “grey”) > lines(num_vec_size, type = “S”) The result obtained on running the preceding code is shown in following figure. From the following figure, it can be observed that memory allocated to a vector is a function of its length and the object type used. However, the relationship does not seem to be linear—rather, it seems to increase in step. This is due to the fact that for better and consistent performance, R initially assigns big blocks of memory from RAM and handles them internally. These memory blocks are individually assigned to vectors based on the type and the number of elements within. Initially, memory blocks seem to be irregular towards a particular level (128 bytes for numeric/logical vector, and 176 bytes for character/complex vectors), and later become stable with small increments of 8 bytes as can be seen in the plots: Memory allocation based on length of vector Due to initial memory allocation differences, numeric and logical vectors show similar memory allocation patterns, and complex vectors behave similar to the character vectors. Memory management helps to efficiently run an algorithm. However, before the execution of any program, we should evaluate it based on its runtime. In the next sub-section, we will discuss the basic concepts involved in obtaining the runtime of any function, and its comparison with similar functions. System runtime in R System runtime is very essential for benchmarking different algorithms. The process helps us in comparing different options, and pick the best algorithm. The CRAN package microbenchmark is used to evaluate the runtime of any expression/function/code at an accuracy of a sub-millisecond. It is an accurate replacement to the system.time() function. Also, all the evaluations are performed in C code to minimize any overhead. The following methods are used to measure the time elapsed: The QueryPerformanceCounter interface on Windows OS The clock_gettime API on Linux OS The mach_absolute_time function on MAC OS The gethrtime function on Solaris OS In our current example, we shall be using the mtcars data, which is in the package datasets. This data is obtained from 1974 Motor Trend US magazine, which comprises of fuel consumption comparison along with 10 automobile designs and the performance of 32 automobiles (1973-74 models). Now, we would like to perform an operation in which a specific numeric attribute (mpg means miles per gallon) needs to be averaged to the corresponding unique values in an integer attribute (carb means no of carburetors). This can be performed using multiple ways such as aggregate, group_by, by, split, ddply(plyr), tapply, data.table, dplyr, sqldf, dplyr and so on. In our current scenario, we have used the following four ways: aggregate function aggregate(mpg~carb,data=mtcars,mean) ddply from plyr package ddply( mtcars, .(carb),function(x) mean(x$mpg)) data.table format mtcars_tb[,mean(mpg),by=carb] group_by function summarize(group_by(mtcars, carb), mean(mpg)) Then, microbenchmark is used to determine the performance of each of the four ways mentioned in the preceding list. Here, we will be evaluating each expression 100 times. > library(microbenchmark) > MB_res <- microbenchmark( + Aggregate_func=aggregate(mpg~carb,data=mtcars,mean), + Ddply_func=ddply( mtcars, .(carb),function(x) mean(x$mpg)), + Data_table_func = mtcars_tb[,mean(mpg),by=carb], + Group_by_func = summarize(group_by(mtcars, carb), mean(mpg)), + times=1000 + ) The output table is as follows: > MB_res Unit: microseconds expr min lq mean median uq max neval Aggregate_func 851.489 913.8015 1001.9007 944.775 1000.4905 6094.209 1000 Ddply_func 1370.519 1475.1685 1579.6123 1517.322 1575.7855 6598.578 1000 Data_table_func 493.739 552.7540 610.7791 577.495 621.6635 3125.179 1000 Group_by_func 932.129 1008.5540 1095.4193 1033.113 1076.1825 4279.435 1000 The output plot is as follows: > library(ggplot2) > autoplot(MB_res) Distribution of time (microseconds) for 1000 iterations in each type of aggregate operation Among these four expressions and for the given dataset, data.table has performed effectively in the least possible time as compared to the others. However, expressions need to be tested under scenarios with a high number of observations, high number of attributes, and both prior to finalizing the best operator. Best, worst, and average Cases Based on the performance in terms of system runtime, a code can be classified under best, worst or average category for a particular algorithm. Let’s consider a sorting algorithm to understand in detail. A sorting algorithm is used to arrange a numeric vector in an ascending order, wherein the output vector should have the smallest number as its first element and largest number as its last element with intermediate elements in subsequent increasing order. In insertion sorting algorithm, the elements within a vector are arranged based on moving positions. In our scenario, we will be inserting each element at a time into a previously sorted vector, with a smaller set of elements moving towards the end. Now, let’s define best, worst and average-case scenarios for an insertion sorting algorithm. Best Case: A best case is one which requires the least running time. For example: a vector with all elements arranged in increasing order requires least amount of time for sorting. Worst Case: A worst case is one which requires the maximum possible runtime to complete sorting a vector. For example: a vector with all the elements sorted in decreasing order requires most amount of time for sorting. Average Case: An average case is one which requires intermediate time to complete sorting a vector. For example: a vector with half elements sorted in increasing order and the remaining in decreasing order. An average case is assessed using multiple vectors of differently arranged elements. Generally, the best-case scenarios are not considered to benchmark an algorithm, since they evaluate an algorithm most optimistically. However, if the probability of occurrence of best case is high, then algorithms can be compared using the best-case scenarios. Similar to best case, worst-case scenarios evaluate the algorithm most pessimistically. It is only used to benchmark algorithms which are used in real-time applications, such as railway network controls, air traffic controls, and the like. Sometimes, when we are not aware of input data distributions, it is safe to assess the performance of the algorithm based on the worst-case scenario. Most of the times, average-case scenario is used as a representative measure of an algorithm’s performance; however, this is valid only when we are aware of the input data distribution. Average-case scenarios may not evaluate the algorithm properly if the distribution of input data is skewed. In the case of sorting, if most of the input vectors are arranged in descending order, the average-case scenario may not be the best form of evaluating the algorithm. In a nutshell, real-time application scenarios, along with input data distribution, are major criterions to analyze the algorithms based on best, worst, and average cases. Summary This article summarizes the basic concepts and nuances of evaluating algorithms in R. We covered the conceptual theory of memory management and system runtime in R. We discussed the best, worst, and average-case scenarios to evaluate the performance of algorithms. Resources for Article: Further resources on this subject: Reconstructing 3D Scenes [article] Raster Calculations [article] Remote Sensing and Histogram [article]
Read more
  • 0
  • 0
  • 21409

article-image-introduction-practical-business-intelligence
Packt
10 Nov 2016
20 min read
Save for later

Introduction to Practical Business Intelligence

Packt
10 Nov 2016
20 min read
In this article by Ahmed Sherif, author of the book Practical Business Intelligence, is going to explain what is business intelligence? Before answering this question, I want to pose and answer another question. What isn't business intelligence? It is not spreadsheet analysis done with transactional data with thousands of rows. One of the goals of Business Intelligence or BI is to shield the users of the data from the intelligent logic lurking behind the scenes of the application that is delivering the data to them. If the integrity of the data is compromised in any way by an individual not intimately familiar with the data source, then there cannot, by definition, be intelligence in the business decisions made within that same data. The single source of truth is the key for any Business Intelligence operation whether it is a mom and pop soda shop or a Fortune 500 company. Any report, dashboard, or analytical application that is delivering information to a user through a BI tool but the numbers cannot be tied back to the original source will break the trust between the user and the data and will defeat the purpose of Business Intelligence. (For more resources related to this topic, see here.) In my opinion, the most successful tools used for business intelligence directly shield the business user from the query logic used for displaying that same data in some form of visual manner. Business Intelligence has taken many forms in terms of labels over the years. Business Intelligence is the process of delivering actionable business decisions from analytical manipulation and presentation of data within the confines of a business environment. The delivery process mentioned in the definition will focus its attention on. The beauty of BI is that it is not owned by any one particular tool that is proprietary to a specific industry or company. Business Intelligence can be delivered using many different tools, some even that were not originally intended to be used for BI. The tool itself should not be the source where the query logic is applied to generate the business logic of the data. The tool should primarily serve as the delivery mechanism of the query that is generated by the data warehouse that houses both the data, as well as the logic. In this chapter we will cover the following topics: Understanding the Kimball method Understanding business intelligence Data and SQL Working with data and SQL Working with business intelligence tools Downloading and installing MS SQL Server 2014 Downloading and installing AdventureWorks Understanding the Kimball method As we discuss the data warehouse where our data is being housed, we will be remised not to bring up Ralph Kimball, one of the original architects of the data warehouse.  Kimball's methodology incorporated dimensional modeling, which has become the standard for modeling a data warehouse for Business Intelligence purposes. Dimensional modeling incorporates joining tables that have detail data and tables that have lookup data. A detail table is known as a fact table in dimensional modeling. An example of a fact table would be a table holding thousands of rows of transactional sales from a retail store.  The table will house several ID's affiliated with the product, the sales person, the purchase date, and the purchaser just to name a few. Additionally, the fact table will store numeric data for each individual transaction, such as sales quantity for sales amount to name a few examples. These numeric values will be referred to as measures. While there is usually one fact table, there will also be several lookup or dimensional tables that will have one table for each ID that is used in a fact table. So, for example,  there would be one dimensional table for the product name affiliated with a product ID. There would be one dimensional table for the month, week, day, and year of the id affiliated with the date. These dimensional tables are also referred to as Lookup Tables, because they kind of look up what the name of a dimension ID is affiliated with. Usually, you would find as many dimensional tables as there are ID's in the fact table. The dimensional tables would all be joined to the one fact table creating something of a 'star' look. Hence, the name for this type of table join is known as a star schema which is represented diagrammatically in the following figure. It is customary that the fact table will be the largest table in a data warehouse while the lookup tables will all be quite small in rows, some as small as one row. The tables are joined by ID's, also known as surrogate keys. Surrogate keys allow for the most efficient join between a fact table and a dimensional table as they are usually a data type of integer. As more and more detail is added to a dimensional table, that new dimension is just given the next number in line, usually starting with 1. Query performance between tables joins suffers when we introduce non-numeric characters into the join or worse, symbols (although most databases will not allow that). Understanding business intelligence architecture I will continuously hammer home the point that the various tools utilized to deliver the visual and graphical BI components should not house any internal logic to filter data out of the tool nor should it be the source of any built in calculations. The tools themselves should not house this logic as they will be utilized by many different users. If each user who develops a BI app off of the tool incorporates different internal filters without the tool, the single source of truth tying back to the data warehouse will become multiple sources of truths.  Any logic applied to the data to filter out a specific dimension or to calculate a specific measure should be applied in the data warehouse and then pulled into the tool. For example, if the requirement for a BI dashboard was to show current year and prior year sales for US regions only, the filter for region code would be ideally applied in the data warehouse as opposed to inside of the tool. The following is a query written in SQL joining two tables from the AdventureWorks database that highlights the difference between dimenions and measures.  The 'region' column is a dimension column and the 'SalesYTD' and 'SalesPY' are measure columns. In this example, the 'TerritoryID' is serving as the key join between 'SalesTerritory' and 'SalesPerson'. Since the measures are coming from the 'SalesPerson' table, that table will serve as the fact table and 'SalesPerson.TerritoryID' will serve as the fact ID. Since the Region column is dimensional and coming from the 'SalesTerritory' table, that table will serve as the dimensional or lookup table and 'SalesTerritory.TerritoryID' will serve as the dimension ID. In a finely-tuned data warehouse both the fact ID and dimension ID would be indexed to allow for efficient query performance. This performance is obtained by sorting the ID's numerically so that a row from one table that is being joined to another table does not have to be searched through the entire table but only a subset of that table. When the table is only a few hundred rows, it may not seem necessary to index columns, but when the table grows to a few hundred million rows, it may become necessary. Select region.Name as Region ,round(sum(sales.SalesYTD),2) as SalesYTD ,round(sum(sales.SalesLastYear),2) as SalesPY FROM [AdventureWorks2014].[Sales].[SalesTerritory] region left outer join [AdventureWorks2014].[Sales].[SalesPerson] sales on sales.TerritoryID = region.TerritoryID where region.CountryRegionCode = 'US' Group by region.Name order by region.Name asc There are several reasons why applying the logic at the database level is considered a best practice. Most of the time, these requests for filtering data or manipulating calculations are done at the BI tool level because it is easier for the developer than to go to the source. However, if these filters are being performed due to data quality issues then applying logic at the reporting level is only masking an underlying data issue that needs to be addressed across the entire data warehouse. You would be doing yourself a disservice in the long run as you will be establishing a precedence that the data quality would be handled by the report developer as opposed to the database administrator. You are just adding additional work onto your plate. Ideal BI tools will quickly connect to the data source and then allow for slicing and dicing of your dimensions and measures in a manner that will quickly inform the business of useful and practical information. Ultimately, the choice of a BI tool by an individual or an organization will come down to the ease of use of the tool as well as the flexibility to showcase the data through various components such as graphs, charts, widgets, and infographics. Management If you are a Business Intelligence manager looking to establish a department with a variety of tools to help flesh out your requirements, could serve as a good source for interview questions to weed out unqualified candidates. A manager could use to distinguish some of the nuances between these different skillsets and prioritize hiring based on immediate needs. Data Scientist The term Data Scientist has been misused in the BI industry, in my humble opinion. It has been lumped in with Data Analyst as well as BI Developer. Unfortunately, these three positions have separate skillsets and you will do yourself a disservice by assuming one person can do multiple positions successfully. A Data Scientist will be able to apply statistical algorithms behind the data that is being extracted from the BI tools and make predictions about what will happen in the future with that same data set. Due to this skillset, a Data Scientist may find the chapters focusing on R and Python to be of particular importance because of their abilities to leverage predictive capabilities within their BI delivery mechanisms. Data Analyst A Data Analyst is probably the second most misused position behind a Data Scientist. Typically, a Data Analyst should be analyzing the data that is coming out of the BI tools that are connected to the data warehouse. Most Data Analysts are comfortable working with Microsoft Excel. Often times they are asked to take on additional roles in developing dashboards that require additional programming skills.  This is where they would find some comfort using a tool like Power BI, Tableau, or QlikView. These tools would allow for a Data Analyst to quickly develop a storyboard or visualization that would allow for quick analysis with minimal programming skills. Visualization Developer A 'dataviz' developer is someone who can create complex visualizations out of data and showcase interesting interactions between different measures inside of a dataset that cannot necessarily be seen with a traditional chart or graph. More often than not these developers possess some programming background such as JavaScript, HTML, or CSS. These developers are also used to developing applications directly for the web and therefore would find D3.js a comfortable environment to program in. Working with Data and SQL The examples and exercises that will come from the AdventureWorks database.  The AdventureWorks database has a comprehensive list of tables that mimics an actual bicycle retailor. The examples will draw on different tables from the database to highlight BI reporting from the various segments appropriate for the AdventureWorks Company. These segments include Human Resources, Manufacturing, Sales, Purchasing, and Contact Management. A different segment of the data will be highlighted in each chapter utilizing a specific set of tools. A cursory understanding of SQL (structured query language) will be helpful to get a grasp of how data is being aggregated with dimensions and measures. Additionally, an understanding of the SQL statements used will help with the validation process to ensure a single source of truth between the source data and the output inside of the BI tool of choice. For more information about learning SQL, visit the following website: www.sqlfordummies.com Working with business intelligence tools Over the course of the last 20 years, there have been a growing number of software products released that were geared towards Business Intelligence. In addition, there have been a number of software products and programming languages that were not initially built for BI but later on became a staple for the industry. The tools used were chosen based on the fact that they were either built off of open source technology or they were products from companies that provided free versions of their software for development purposes. Many companies from the big enterprise firms have their own BI tools and they are quite popular. However, unless you have a license with them, it is unlikely that you will be able to use their tool without having to shell out a small fortune. Power BI and Excel Power BI is one of the more relatively newer BI tools from Microsoft.  It is known as a self-service solution and integrates seamlessly with other data sources such as Microsoft Excel and Microsoft SQL Server.  Our primary purpose in using Power BI will be to generate interactive dashboards, reports, and datasets for users. In addition to using Power BI we will also focus on utilizing Microsoft Excel to assist with some data analysis and validation of results that are being pulled from our data warehouse.  Pivot tables are very popular within MS Excel and will be used to validate aggregation done inside of the data warehouse. D3.js D3.js, also known as data-driven documents, is a JavaScript library known for delivery beautiful visualizations by manipulating documents based on data. Since D3 is rooted in JavaScript, all visualizations make a seamless transition to the web. D3 allows for major customization to any part of visualization and because of this flexibility, it will require a steeper learning curve that probably any other software program. D3 can consume data easily as a .json or a .csv file.  Additionally, the data can also be imbedded directly within the JavaScript code that renders the visualization on the web. R R is a free and open source statistical programming language that produces beautiful graphics. The R language has been widely used among the statistical community and more recently in the data science and machine learning community as well. Due to this fact, it has picked up steam in recent years as a platform for displaying and delivering effective and practical BI. In addition to visualizing BI, R has the ability to also visualize predictive analysis with algorithms and forecasts. While R is a bit raw in its interface, there have been some IDE's (Integrated Development Environment) that have been developed to ease the user experience. RStudio will be used to deliver the visualisations developed within R. Python Python is considered the most traditional programming language of all the different languages. It is a widely used general purpose programming language with several modules that are very powerful in analysing and visualizing data. Similar to R, Python is a bit raw in its own form for delivering beautiful graphics as a BI tool; however, with the incorporation of an IDE the user interface becomes much more of a pleasurable development experience. PyCharm will be the IDE used to develop BI with Python. PyCharm is free to use and allows creation of the iPython notebook which delivers seamless integration between Python and the powerful modules that will assist with BI. As a note, all code in Python will be developed using the Python 3 syntax. QlikView QlikView is a software company specializing in delivering business intelligence solutions using their desktop tool. QlikView is one of the leaders in delivering quick visualizations based on data and queries through their desktop application. They advertise themselves to be self-service BI for business users. While they do offer solutions that target more enterprise organizations, they also offer a free version of their tool for personal use. Tableau is probably the closest competitor in terms of delivering similar BI solutions. Tableau Tableau is a software company specializing in delivering business intelligence solutions using their desktop tool. If this sounds familiar to QlikView, it's probably because it's true. Both are leaders in the field of establishing a delivery mechanism with easy installation, setup, and connectivity to the available data. Tableau has a free version of their desktop tool. Again, Tableau excels at delivering both beautiful visualizations quickly as well as self-service data discovery to more advanced business users. Microsoft SQL Server Microsoft SQL will serve as the data warehouse for the examples that we will with the BI Tools. Microsoft SQL Server is relatively simple to install and set up as well it is free to download. Additionally, there are example databases that configure seamlessly with it, such as the AdventureWorks database. Downloading and Installing MS SQL Server 2014 First things first. We will need to get started with getting our database and data warehouse up and running so that we can begin to develop our BI environment. We will visit the Microsoft website below to start the download selection process. https://www.microsoft.com/en-us/download/details.aspx?id=42299 Select the specified language that is applicable to you and also select the MS SQL Server Express version with Advanced features that is 64-bit edition as shown in the following screenshot. Ideally you'll want to be working in a 64-bit edition when dealing with servers. After selecting the file, the download process should begin. Depending on your connection speed it could take some time as the file is slightly larger than 1 GB. The next step in the process is selecting a new stand-alone instance of SQL Server 2014 unless you already have a version and wish to upgrade instead as shown in the following screenshot.. After accepting the license terms, continue through the steps in the Global Rules as well as the Product Updates to get to the setup installation files. For the feature selection tab, make sure the following features are selected for your installation as shown in the following screenshot. Our preference is to label a named instance of this database to something related to the work we are doing.  Since this will be used for Business Intelligence, I went ahead and name this instance 'SQLBI' as shown in the following screenshot: The default Server Configuration settings are sufficient for now, there is no need to change anything under that section as shown in the following screenshot. Unless you are required to do so within your company or organization, for personal use it is sufficient to just go with Windows Authentication mode for sign-on as shown in the following screenshot. We will not need to do any configuring of reporting services, so it is sufficient for our purposes to just with installing Reporting Services Native mode without any need for configuration at this time. At this point the installation will proceed and may take anywhere between 20-30 minutes depending on the cpu resources. If you continue to have issues with your installation, you can visit the following website from Microsoft for additional help. http://social.technet.microsoft.com/wiki/contents/articles/23878.installing-sql-server-2014-step-by-step-tutorial.aspx Ultimately, if everything with the installation is successful, you'll want to see all portions of the installation have a green check mark next to their name and be labeled 'Successful' as shown in the following screenshot. Downloading and Installing AdventureWorks We are almost finished with getting our business intelligence data warehouse complete. We are now at the stage where we will extract and load data into our data warehouse. The last part is to download and install the AdventureWorks database from Microsoft. The zipped file for AdventureWorks 2014 is located in the following website from Microsoft: https://msftdbprodsamples.codeplex.com/downloads/get/880661 Once the file is downloaded and unzipped, you will find a file named the following: AdventureWorks2014.bak Copy that file and paste it in the following folder where it will be incorporated with your Microsoft SQL Server 2014 Express Edition. C:Program FilesMicrosoft SQL ServerMSSQL12.SQLBIMSSQLBackup Also note that the MSSQL12.SQLBI subfolder will vary user by user depending on what you named your SQL instance when you were installing MS SQL Server 2014. Once that has been copied over, we can fire up Management Studio for SQL Server 2014 and start up a blank new query by going to File New Query with Current Connection Once you have a blank query set up, copy and paste the following code in the and execute it: use [master] Restore database AdventureWorks2014 from disk = 'C:Program FilesMicrosoft SQL ServerMSSQL12.SQLBIMSSQLBackupAdventureWorks2014.bak' with move 'AdventureWorks2014_data' to 'C:Program FilesMicrosoft SQL ServerMSSQL12.SQLBIMSSQLDATAAdventureWorks2014.mdf', Move 'AdventureWorks2014_log' to 'C:Program FilesMicrosoft SQL ServerMSSQL12.SQLBIMSSQLDATAAdventureWorks2014.ldf' , replace Once again, please note that the MSSQL12.SQLBI subfolder will vary user by user depending on what you named your SQL instance when you were installing MS SQL Server 2014. At this point in time within the database you should have received a message saying that Microsoft SQL Server has processed 24248 pages for database 'AdventureWorks2014'. Once you have refreshed your database tab on the upper left hand corner of SQL Server, the AdventureWorks database will become visible as well as all of the appropriate tables as shown in the following screenshot: One final step that we will need to verify just to make sure that your login account has all of the appropriate server settings. When you right-click on the SQL Server name on the upper left hand portion of Management Studio, select the properties.  Select Permissions inside Properties. Find your username and check all of the rights under the Grant column as shown in the following screenshot: Finally, we need to also ensure that the folder that houses Microsoft SQL Server 2014 also has the appropriate rights enabled for your current user.  That specific folder is located under C:Program FilesMicrosoft SQL Server. For purposes of our exercises, we will assign all rights for the SQL Server user to the following folder as shown in the following screenshot: We are now ready to begin connecting our BI tools to our data! Summary The emphasis will be placed on implementing Business Intelligence best practices within the various tools that will be used based on the different levels of data that is provided within the AdventureWorks database. In the next chapter we will cover extracting additional data from the web that will be joined to the AdventureWorks database. This process is known as web scraping and can be performed with great success using tools such as Python and R. In addition to collecting the data, we will focus on transforming the collected data for optimal query performance. Resources for Article: Further resources on this subject: LabVIEW Basics [article] Thinking Probabilistically [article] Clustering Methods [article]
Read more
  • 0
  • 1
  • 4035

article-image-testing-components-service-dependencies
Victor Mejia
10 Nov 2016
5 min read
Save for later

Testing Components with Service Dependencies

Victor Mejia
10 Nov 2016
5 min read
It is very common for your Angular 2 components to depend on a service that performs actions, such as fetching data. In this post we will look at testing components with service dependencies, and at testing asynchronous actions. We will be using Jasmine for our tests. If you have not read Getting Started Testing Angular 2 Components, I strongly suggest you do so before continuing. Angular 2 Component with a Service Dependency Continuing with our contact manager application, we need to have a ContactService that fetches data from a server: import { Injectable } from '@angular/core'; import { Http } from '@angular/http'; import 'rxjs/add/operator/map'; @Injectable() export class ContactService { constructor(private http: Http){ } getContacts() { return this.http.get('/contacts.json') .map(res => res.json()); } } The Http service is injected here, and TypeScript will automatically assign the injected service to this.http. With this service ready to use, we are now ready to inject it into our ContactsComponent : import { Component, OnInit } from '@angular/core'; import { ContactService } from '../shared/contact.service'; @Component({ selector: 'contacts', template: ` <button (click)="getContacts()">Get Contacts</button> <profile *ngFor="let profile of contacts" [info]="profile"></profile> ` }) export class ContactsComponent implements OnInit { contacts: Array<any>; constructor(private contactService: ContactService) { } ngOnInit() { } getContacts() { this.contactService.getContacts() .subscribe(data => { this.contacts = data; }); } } We have an action set up, so when we click on the button, we make a call to our ContactService to fetch the data and assign the result. Once the call is resolved, the data will display. Setting up your unit test What we have to keep in mind is that we want to test our components in isolation. What this means is that instead of using the actual ContactService implementation, we create a MockContactService that returns mock data (array of Profile s). let mockData = [ { name: 'Victor Mejia', email: 'victor.mejia@example.com', phone: '123-456-7890' } ]; class MockContactService { getContacts(url) { return Observable.create((observer: Observer<Array<Profile>>) => { observer.next(mockData); }); } } When configuring our testing module, we add a new property,providers, where we specify the usage of our mock service: TestBed.configureTestingModule({ declarations: [ContactsComponent], providers: [ { provide: ContactService, useClass: MockContactService } ] }); We can now go ahead and get handles on fixture, component, and element : import { TestBed, async, ComponentFixture } from '@angular/core/testing'; import { ContactsComponent } from './contacts.component'; import { ContactService } from '../shared/contact.service'; import { Profile } from '../shared/profile.model'; import { Observable } from 'rxjs/Observable'; import { Observer } from 'rxjs/Observer'; let mockData = [ { name: 'Victor Mejia', email: 'victor.mejia@example.com', phone: '123-456-7890' } ]; class MockContactService { getContacts(url) { return Observable.create((observer: Observer<Array<Profile>>) => { observer.next(mockData); }); } } let fixture: ComponentFixture<ContactsComponent>; let component: ContactsComponent; let element: HTMLElement; describe('Component: Contacts', () => { beforeEach(async(() => { TestBed.configureTestingModule({ declarations: [ContactsComponent], providers: [ { provide: ContactService, useClass: MockContactService } ] }); TestBed.compileComponents() .then(() => { fixture = TestBed.createComponent(ContactsComponent); component = fixture.debugElement.componentInstance; element = fixture.debugElement.nativeElement }); })); }); Ensuring calls to our service A good test to always perform is to ensure that your actions are making the correct calls to your service. To do so, we can spy on the getContacts() function on the service, calling the component action and then ensuring that the function was indeed called: describe('getContacts', () => { it('should make a call to contactService.getContacts()', () => { spyOn(component.contactService, 'getContacts').and.callThrough(); component.getContacts(); expect(component.contactService.getContacts).toHaveBeenCalled(); }); }); Ensuring data is set A follow-up test to be performed is to ensure that the data is being set on the component after the call to the API is resolved. Since our call to getContacts() is performing an asynchronous action, we should use the async function in the it: it('should set the contacts property after fetching data', async(() => { ... })); It wraps the test function in an asynchronous “test zone”. Basically, it automatically completes when the asynchronous actions are complete. Next, we can make a call to component.getContacts() . However, we don’t want to run our specs until after that call has been resolved. There is a useful function we can use in our fixture, fixture.whenStable(). This returns a promise that resolves after asynchronous activity. Our test should now look as follows: it('should set the contacts property after fetching data', async(() => { component.getContacts(); fixture.whenStable().then(() => { expect(component.contacts).toEqual(mockData); }); })); We simply run a check to ensure that the contacts property is set to what the API call returns. Finer Async Control There are times when you want finer control, such as dealing with time intervals, and so on. To do so, we can simply use the fakeAsync in conjunction with the tick() function to simulate the passage of time. it('asynchronous timed test...', fakeAsync(() => { component.asyncActionWithTime(); tick(2000); // "advance" 2 seconds expect(...).toBe(...); })); Conclusion Angular 2 has wonderful APIs that make it really easy to test your components. We have seen how to test components with service dependencies, along with asynchronous actions. Time to start writing tests!
Read more
  • 0
  • 0
  • 9531
article-image-introduction-javascript
Packt
10 Nov 2016
15 min read
Save for later

Introduction to JavaScript

Packt
10 Nov 2016
15 min read
In this article by Simon Timms, author of the book, Mastering JavaScript Design Patterns - Second Edition, we will explore the history of JavaScript and how it came to be the important language that it is today (For more resources related to this topic, see here.) JavaScript is an evolving language that has come a long way from its inception. Possibly more than any other programming language, it has grown and changed with the growth of the World Wide Web. As JavaScript has evolved and grown in importance, the need to apply rigorous methods to its construction has also grown. The road to JavaScript We'll never know how language first came into being. Did it slowly evolve from a series of grunts and guttural sounds made during grooming rituals? Perhaps it developed to allow mothers and their offspring to communicate. Both of these are theories, all but impossible to prove. Nobody was around to observe our ancestors during that important period. In fact, the general lack of empirical evidence lead the Linguistic Society of Paris to ban further discussions on the topic, seeing it as unsuitable for serious study. The early days Fortunately, programming languages have developed in recent history and we've been able to watch them grow and change. JavaScript has one of the more interesting histories of modern programming languages. During what must have been an absolutely frantic 10 days in May of 1995, a programmer at Netscape wrote the foundation for what would grow up to be modern JavaScript. At the time, Netscape was involved in the first of the browser wars with Microsoft. The vision for Netscape was far grander than simply developing a browser. They wanted to create an entire distributed operating system making use of Sun Microsystems' recently-released Java programming language. Java was a much more modern alternative to the C++ Microsoft was pushing. However, Netscape didn't have an answer to Visual Basic. Visual Basic was an easier to use programming language, which was targeted at developers with less experience. It avoided some of the difficulties around memory management that make C and C++ notoriously difficult to program. Visual Basic also avoided strict typing and overall allowed more leeway: Brendan Eich was tasked with developing Netscape repartee to VB. The project was initially codenamed Mocha, but was renamed LiveScript before Netscape 2.0 beta was released. By the time the full release was available, Mocha/LiveScript had been renamed JavaScript to tie it into the Java applet integration. Java Applets were small applications which ran in the browser. They had a different security model from the browser itself and so were limited in how they could interact with both the browser and the local system. It is quite rare to see applets these days, as much of their functionality has become part of the browser. Java was riding a popular wave at the time and any relationship to it was played up. The name has caused much confusion over the years. JavaScript is a very different language from Java. JavaScript is an interpreted language with loose typing, which runs primarily on the browser. Java is a language that is compiled to bytecode, which is then executed on the Java Virtual Machine. It has applicability in numerous scenarios, from the browser (through the use of Java applets), to the server (Tomcat, JBoss, and so on), to full desktop applications (Eclipse, OpenOffice, and so on). In most laypersons' minds, the confusion remains. JavaScript turned out to be really quite useful for interacting with the web browser. It was not long until Microsoft had also adopted JavaScript into their Internet Explorer to complement VBScript. The Microsoft implementation was known as JScript. By late 1996, it was clear that JavaScript was going to be the winning web language for the near future. In order to limit the amount of language deviation between implementations, Sun and Netscape began working with the European Computer Manufacturers Association (ECMA) to develop a standard to which future versions of JavaScript would need to comply. The standard was released very quickly (very quickly in terms of how rapidly standards organizations move), in July of 1997. On the off chance that you have not seen enough names yet for JavaScript, the standard version was called ECMAScript, a name which still persists in some circles. Unfortunately, the standard only specified the very core parts of JavaScript. With the browser wars raging, it was apparent that any vendor that stuck with only the basic implementation of JavaScript would quickly be left behind. At the same time, there was much work going on to establish a standard Document Object Model (DOM) for browsers. The DOM was, in effect, an API for a web page that could be manipulated using JavaScript. For many years, every JavaScript script would start by attempting to determine the browser on which it was running. This would dictate how to address elements in the DOM, as there were dramatic deviations between each browser. The spaghetti of code that was required to perform simple actions was legendary. I remember reading a year-long 20-part series on developing a Dynamic HTML (DHTML) drop down menu such that it would work on both Internet Explorer and Netscape Navigator. The same functionally can now be achieved with pure CSS without even having to resort to JavaScript. DHTML was a popular term in the late 1990s and early 2000s. It really referred to any web page that had some sort of dynamic content that was executed on the client side. It has fallen out of use, as the popularity of JavaScript has made almost every page a dynamic one. Fortunately, the efforts to standardize JavaScript continued behind the scenes. Versions 2 and 3 of ECMAScript were released in 1998 and 1999. It looked like there might finally be some agreement between the various parties interested in JavaScript. Work began in early 2000 on ECMAScript 4, which was to be a major new release. A pause Then, disaster struck. The various groups involved in the ECMAScript effort had major disagreements about the direction JavaScript was to take. Microsoft seemed to have lost interest in the standardization effort. It was somewhat understandable, as it was around that time that Netscape self-destructed and Internet Explorer became the de-facto standard. Microsoft implemented parts of ECMAScript 4 but not all of it. Others implemented more fully-featured support, but without the market leader on-board, developers didn't bother using them. Years passed without consensus and without a new release of ECMAScript. However, as frequently happens, the evolution of the Internet could not be stopped by a lack of agreement between major players. Libraries such as jQuery, Prototype, Dojo, and Mootools, papered over the major differences in browsers, making cross-browser development far easier. At the same time, the amount of JavaScript used in applications increased dramatically. The way of GMail The turning point was, perhaps, the release of Google's GMail application in 2004. Although XMLHTTPRequest, the technology behind Asynchronous JavaScript and XML (AJAX), had been around for about five years when GMail was released, it had not been well-used. When GMail was released, I was totally knocked off my feet by how smooth it was. We've grown used to applications that avoid full reloads, but at the time, it was a revolution. To make applications like that work, a great deal of JavaScript is needed. AJAX is a method by which small chunks of data are retrieved from the server by a client instead of refreshing the entire page. The technology allows for more interactive pages that avoid the jolt of full page reloads. The popularity of GMail was the trigger for a change that had been brewing for a while. Increasing JavaScript acceptance and standardization pushed us past the tipping point for the acceptance of JavaScript as a proper language. Up until that point, much of the use of JavaScript was for performing minor changes to the page and for validating form input. I joke with people that in the early days of JavaScript, the only function name which was used was Validate(). Applications such as GMail that have a heavy reliance on AJAX and avoid full page reloads are known as Single Page Applications or SPAs. By minimizing the changes to the page contents, users have a more fluid experience. By transferring only JavaScript Object Notation (JSON) payload instead of HTML, the amount of bandwidth required is also minimized. This makes applications appear to be snappier. In recent years, there have been great advances in frameworks that ease the creation of SPAs. AngularJS, backbone.js, and ember are all Model View Controller style frameworks. They have gained great popularity in the past two to three years and provide some interesting use of patterns. These frameworks are the evolution of years of experimentation with JavaScript best practices by some very smart people. JSON is a human-readable serialization format for JavaScript. It has become very popular in recent years, as it is easier and less cumbersome than previously popular formats such as XML. It lacks many of the companion technologies and strict grammatical rules of XML, but makes up for it in simplicity. At the same time as the frameworks using JavaScript are evolving, the language is too. 2015 saw the release of a much-vaunted new version of JavaScript that had been under development for some years. Initially called ECMAScript 6, the final name ended up being ECMAScript-2015. It brought with it some great improvements to the ecosystem. Browser vendors are rushing to adopt the standard. Because of the complexity of adding new language features to the code base, coupled with the fact that not everybody is on the cutting edge of browsers, a number of other languages that transcompile to JavaScript are gaining popularity. CoffeeScript is a Python-like language that strives to improve the readability and brevity of JavaScript. Developed by Google, Dart is being pushed by Google as an eventual replacement for JavaScript. Its construction addresses some of the optimizations that are impossible in traditional JavaScript. Until a Dart runtime is sufficiently popular, Google provides a Dart to the JavaScript transcompiler. TypeScript is a Microsoft project that adds some ECMAScript-2015 and even some ECMAScript-201X syntax, as well as an interesting typing system, to JavaScript. It aims to address some of the issues that large JavaScript projects present. The point of this discussion about the history of JavaScript is twofold: first, it is important to remember that languages do not develop in a vacuum. Both human languages and computer programming languages mutate based on the environments in which they are used. It is a popularly held belief that the Inuit people have a great number of words for "snow", as it was so prevalent in their environment. This may or may not be true, depending on your definition for the word and exactly who makes up the Inuit people. There are, however, a great number of examples of domain-specific lexicons evolving to meet the requirements for exact definitions in narrow fields. One need look no further than a specialty cooking store to see the great number of variants of items which a layperson such as myself would call a pan. The Sapir–Whorf hypothesis is a hypothesis within the linguistics domain, which suggests that not only is language influenced by the environment in which it is used, but also that language influences its environment. Also known as linguistic relativity, the theory is that one's cognitive processes differ based on how the language is constructed. Cognitive psychologist Keith Chen has proposed a fascinating example of this. In a very highly-viewed TED talk, Dr. Chen suggested that there is a strong positive correlation between languages that lack a future tense and those that have high savings rates (https://www.ted.com/talks/keith_chen_could_your_language_affect_your_ability_to_save_money/transcript). The hypothesis at which Dr. Chen arrived is that when your language does not have a strong sense of connection between the present and the future, this leads to more reckless behavior in the present. Thus, understanding the history of JavaScript puts one in a better position to understand how and where to make use of JavaScript. The second reason I explored the history of JavaScript is because it is absolutely fascinating to see how quickly such a popular tool has evolved. At the time of writing, it has been about 20 years since JavaScript was first built and its rise to popularity has been explosive. What more exciting thing is there than to work in an ever-evolving language? JavaScript everywhere Since the GMail revolution, JavaScript has grown immensely. The renewed browser wars, which pit Internet Explorer and Edge against Chrome, against Firefox, have lead to building a number of very fast JavaScript interpreters. Brand new optimization techniques have been deployed and it is not unusual to see JavaScript compiled to machine-native code for the added performance it gains. However, as the speed of JavaScript has increased, so has the complexity of the applications built using it. JavaScript is no longer simply a language for manipulating the browser, either. The JavaScript engine behind the popular Chrome browser has been extracted and is now at the heart of a number of interesting projects such as Node.js. Node.js started off as a highly asynchronous method of writing server-side applications. It has grown greatly and has a very active community supporting it. A wide variety of applications have been built using the Node.js runtime. Everything from build tools to editors have been built on the base of Node.js. Recently, the JavaScript engine for Microsoft Edge, ChakraCore, was also open sourced and can be embedded in NodeJS as an alternative to Google's V8. SpiderMonkey, the FireFox equivalent, is also open source and is making its way into more tools. JavaScript can even be used to control microcontrollers. The Johnny-Five framework is a programming framework for the very popular Arduino. It brings a much simpler approach to programming devices than the traditional low-level languages used for programming these devices. Using JavaScript and Arduino opens up a world of possibilities, from building robots to interacting with real-world sensors. All of the major smartphone platforms (iOS, Android, and Windows Phone) have an option to build applications using JavaScript. The tablet space is much the same with tablets supporting programming using JavaScript. Even the latest version of Windows provides a mechanism for building applications using JavaScript: JavaScript is becoming one of the most important languages in the world. Although language usage statistics are notoriously difficult to calculate, every single source which attempts to develop a ranking puts JavaScript in the top 10: Language index Rank of JavaScript Langpop.com 4 Statisticbrain.com 4 Codeval.com 6 TIOBE 8 What is more interesting is that most of of these rankings suggest that the usage of JavaScript is on the rise. The long and short of it is that JavaScript is going to be a major language in the next few years. More and more applications are being written in JavaScript and it is the lingua franca for any sort of web development. Developer of the popular Stack Overflow website Jeff Atwood created Atwood's Law regarding the wide adoption of JavaScript: "Any application that can be written in JavaScript, will eventually be written in JavaScript" – Atwood's Law, Jeff Atwood This insight has been proven to be correct time and time again. There are now compilers, spreadsheets, word processors—you name it—all written in JavaScript. As the applications which make use of JavaScript increase in complexity, the developer may stumble upon many of the same issues as have been encountered in traditional programming languages: how can we write this application to be adaptable to change? This brings us to the need for properly designing applications. No longer can we simply throw a bunch of JavaScript into a file and hope that it works properly. Nor can we rely on libraries such as jQuery to save ourselves. Libraries can only provide additional functionality and contribute nothing to the structure of an application. At least some attention must now be paid to how to construct the application to be extensible and adaptable. The real world is ever-changing, and any application that is unable to change to suit the changing world is likely to be left in the dust. Design patterns provide some guidance in building adaptable applications, which can shift with changing business needs. Summary JavaScript has an interesting history and is really coming of age. With server-side JavaScript taking off and large JavaScript applications becoming common, there is a need for more diligence in building JavaScript applications. For more information on JavaScript, you can check other books by Packt mentioned as follows: Mastering JavaScript Promises: https://www.packtpub.com/application-development/mastering-javascript-promises Mastering JavaScript High Performance: https://www.packtpub.com/web-development/mastering-javascript-high-performance JavaScript : Functional Programming for JavaScript Developers: https://www.packtpub.com/web-development/javascript-functional-programming-javascript-developers Resources for Article: Further resources on this subject: API with MongoDB and Node.js [article] Tips & Tricks for Ext JS 3.x [article] Saying Hello! [article]
Read more
  • 0
  • 0
  • 12306

article-image-managing-users-and-groups
Packt
10 Nov 2016
7 min read
Save for later

Managing Users and Groups

Packt
10 Nov 2016
7 min read
In this article, we will cover the following recipes: Creating user account Creating user accounts in batch mode Creating a group Introduction In this article by Uday Sawant, the author of the book Ubuntu Server Cookbook, you will see how to add new users to the Ubuntu server, update existing users. You will get to know the default setting for new users and how to change them. (For more resources related to this topic, see here.) Creating user account While installing Ubuntu, we add a primary user account on the server; if you are using the cloud image, it comes preinstalled with the default user. This single user is enough to get all tasks done in Ubuntu. There are times when you need to create more restrictive user accounts. This recipe shows how to add a new user to the Ubuntu server. Getting ready You will need super user or root privileges to add a new user to the Ubuntu server. How to do it… Follow these steps to create the new user account: To add a new user in Ubuntu, enter following command in your shell: $ sudo adduser bob Enter your password to complete the command with sudo privileges: Now enter a password for the new user: Confirm the password for the new user: Enter the full name and other information about new user; you can skip this part by pressing the Enter key. Enter Y to confirm that information is correct: This should have added new user to the system. You can confirm this by viewing the file /etc/passwd: How it works… In Linux systems, the adduser command is higher level command to quickly add a new user to the system. Since adduser requires root privileges, we need to use sudo along with the command, adduser completes following operations: Adds a new user Adds a new default group with the same name as the user Chooses UID (user ID) and GID (group ID) conforming to the Debian policy Creates a home directory with skeletal configuration (template) from /etc/skel Creates a password for the new user Runs the user script, if any If you want to skip the password prompt and finger information while adding the new user, use the following command: $ sudo adduser --disabled-password --gecos "" username Alternatively, you can use the useradd command as follows: $ sudo useradd -s <SHELL> -m -d <HomeDir> -g <Group> UserName Where: -s specifies default login shell for the user -d sets the home directory for the user -m creates a home directory if one does not already exist -g specifies the default group name for the user Creating a user with the command useradd does not set password for the user account. You can set or change the user password with the following command: $sudo passwd bob This will change the password for the user account bob. Note that if you skip the username part from the preceding command you will end up changing the password of root account. There's more… With adduser, you can do five different tasks: Add a normal user Add a system user with system option Add user group with the--group option and without the--system option Add a system group when called with the --system option Add an existing user to existing group when called with two non-option arguments Check out the manual page man adduser to get more details. You can also configure various default settings for the adduser command. A configuration file /etc/adduser.conf can be used to set the default values to be used by the adduser, addgroup, and deluser commands. A key value pair of configuration can set various default values, including the home directory location, directory structure skel to be used, default groups for new users, and so on. Check the manual page for more details on adduser.conf with following command: $ man adduser.conf See also Check out the command useradd, a low level command to add new user to system Check out the command usermod, a command to modify a user account See why every user has his own group at: http://unix.stackexchange.com/questions/153390/why-does-every-user-have-his-own-group Creating user accounts in batch mode In this recipe, we will see how to create multiple user accounts in batch mode without using any external tool. Getting ready You will need a user account with root or root privileges. How to do it... Follow these steps to create a user account in batch mode: Create a new text file users.txt with the following command: $ touch users.txt Change file permissions with the following command: $ chmod 600 users.txt Open users.txt with GNU nano and add user accounts details: $ nano users.txt Press Ctrl + O to save the changes. Press Ctrl + X to exit GNU nano. Enter $ sudo newusers users.txt to import all users listed in users.txt file. Check /etc/passwd to confirm that users are created: How it works… We created a database of user details listed in format as the passwd file. The default format for each row is as follows: username:passwd:uid:gid:full name:home_dir:shell Where: username: This is the login name of the user. If a user exists, information for user will be changed; otherwise, a new user will be created. password: This is the password of the user. uid: This is the uid of the user. If empty, a new uid will be assigned to this user. gid: This is the gid for the default group of user. If empty, a new group will be created with the same name as the username. full name: This information will be copied to the gecos field. home_dir: This defines the home directory of the user. If empty, a new home directory will be created with ownership set to new or existing user. shell: This is the default login shell for the user. The new user command reads each row and updates the user information if user already exists, or it creates a new user. We made the users.txt file accessible to owner only. This is to protect this file, as it contains the user's login name and password in unencrypted format. Creating a group Group is a way to organize and administer user accounts in Linux. Groups are used to collectively assign rights and permissions to multiple user accounts. Getting ready You will need super user or root privileges to add a group to the Ubuntu server. How to do it... Enter the following command to add a new group: $ sudo addgroup guest Enter your password to complete addgroup with root privileges. How it works… Here, we are simply adding a new group guest to the server. As addgroup needs root privileges, we need to use sudo along with the command. After creating a new group, addgroup displays the GID of the new group. There's more… Similar to adduser, you can use addgroup in different modes: Add a normal group when used without any options Add a system group with the--system option Add an existing user to existing group when called with two non-option arguments Check out groupadd, a low level utility to add new group to the server See also Check out groupadd, a low level utility to add new group to the server Summary In this article, we have discussed how to create user account, how to create a group and also about how to create user accounts in batch mode. Resources for Article: Further resources on this subject: Directory Services [article] Getting Started with Ansible [article] Lync 2013 Hybrid and Lync Online [article]
Read more
  • 0
  • 0
  • 25543

article-image-how-build-search-engine
JanuVerma
10 Nov 2016
6 min read
Save for later

How to Build a Search Engine

JanuVerma
10 Nov 2016
6 min read
In this post, I will discuss how you can build a search engine like Google. We will start with a definition of a search engine, and I will go on to present the core concepts (mathematics and implementation). A search engine is an information retrieval system that returns documents relevant to a user submitted query, ranked by their relevance to the query. There are many subtle issues in the last statement;for example, what does relevance mean in this context? We will make all of these notions clear as we go further. In this post, by documents I mean textual documents or webpages. There are surely search engines for images, videos or any type of documents, the discussion of which is beyond the scope of this post. Information Retrieval System Consider a corpus of textual documents, and we want to build a system that searches for a query, and returns the documents relevant to the query. The easiest way to accomplish this is to return all of the documents that contain the query. Such a model is called Boolean model of information retrieval (BIR). Mathematically, we can represent a document (and query) as a set of the words it contains. The BIR model retrieves documents that have non-zero intersection with the query. Notice that the retrieved documents are all indistinguishable; there is no inherent ranking method. Another class of models is the vector space model, and all modern search engines use such a model as their base system. In these models, a document is represented as a vector in a high-dimensional vector space. The dimension of the vector is equal to the total number of words in the corpus, and there is an entry corresponding to each word. The documents are retrieved based on the similarity of the corresponding vectors. For example, doc = (v1, v2, ... , vk) is the vector corresponding to some document, and q = (q1, q2, ..., qk) is the vector corresponding to the query, then the similarity of the document and the query can be computed as the vector similarity (distance). There are many choices for such similarity,for example,Euclidean distance, Jaccard distance, Manhattan distance, and so on. In most modern search engines cosine similarity is used: cos(doc,q) = (doc . q) / ||doc|| ||q|| where . is the dot product of the vectors, and || is the norm of the vector. There are difference schemes for choosing the vectors as well, in binary scheme, so the vector entry corresponding to a word is 1 if the word is present in the document, and 0 otherwise. The most popular scheme is the so-called term frequence- inverse document frequency (tf-idf), which assigns higher score of words that are characteristic to the current document. Sothe tf-idf of word w in document d can be computed as: tf-idf(w, d) = tf(w,d) * log(N/n) Where tf(w,d) is the frequency of word w in document d, N is the total number of documents, and n is the number of documents that contain w. Thus tf-idf taxes words that are present in a large number of documents, are not good as features in a model. The retrieved documents are ranked according to the similarity value. There are also probablistic models for retrieving documents, where we estimate the probability that the document d is relevant to a query q, and the documents are then ranked by this probability estimate. For implementation of these models and more, check out InfoR. Computational Complexities The process of retrieval based on vector space model is incredibly expensive. For each search query, we need to compute its similarity with all the web documents, even though most of the documents don't even contain that word. Going by the standard method, it's impossible to achieve the results at the speed we experience while using Google. The trick is to store documents in a data structure, which facilitates fast computation. The most commonly used data structure is the inverted index. A forward index stores the lists of words for each document, an invereted index stores documents for each word in a hashmap-style structure: Inverted index = {word1: [doc1, doc24, doc77], word2: [doc34, doc97], .... wordk: [doc11, doc22, doc1020, doc23]} Given such a structure, one can just jump to the words in the query in constant time, and compute the similarity with a very small number of documents, ones containing the words in the query. Now building such an index also presents a serious computational problem. This problem was the reason behind the development of the MapReduce paradigm, which is the distributed, massively parallel system, discovered by Google. The use of MapReduce system has extended to all major big data problems. If you are curious refer to the original paper. Web Search Engines The working of search engine can be described in the following steps: Crawling: First of all, we need to crawl the web to extract the HTML documents. There are many open source services that provide the crawling capabilities, for example, Nutch, Scrappy,and so on. The documents are then processed to extract the text, the hyperlinks, geo-location, and so on. based on the complexity of the retrieval model. Indexing: We want the documents to be stored in such a way that facilitates a quick search. As discussed earlier, building and maintaining an inverted index is a crucial step in building a search engine. Retrieval: We have discussed retrieval models based on vector space and that probability above. Such a model is used as the base for all major web engines. In addition to the cosine similarity, all web search engines use PageRank for ranking the retrieved documents. This algorithm, discovered by founders of Google, uses the quality of hyperlinks in a webpage for ranking. Mathematically, pagerank assumes that the web is a network graph where documents are linked via hyperlinks. A document receives higher pagerank if there are many 'high-quality' links in it. The PageRank can also be interpreted as the probability that a person randomly clicking on links will arrive at this particular page. For the more mathematically inclined, this translates to computing the principal eigenvector of the transition matrix of random jumps. For more details, refer to this article on American Mathematical Society website. Modern web search engines like Google use more than 200 heuristics on top of the standard retrieval model! Conclusion In this post, we discussed the central ideas of information retrieval and challenges in building a web search engine. It's not an easy problem. Even today, 18 years after the origin of Google, significant research, both at universities and in industry, is being pursued in this direction. About the author Janu Verma is a researcher in the IBM T.J. Watson Research Center, New York. His research interests are in mathematics, machine learning, information visualization, computational health and biology. He has held research positions at Cornell University, Kansas State University, Tata Institute of Fundamental Research, Indian Institute of Science, and Indian Statistical Institute. He has also written papers for IEEE Vis, KDD, International Conference on HealthCare Informatics, Computer Graphics and Applications, Nature Genetics, IEEE Sensors Journals,and so on. His current focus is on the development of visual analytics systems for prediction and understanding. Check out his personal website.
Read more
  • 0
  • 0
  • 10832
article-image-connecting-react-redux-and-firebase-part-2
AJ Webb
10 Nov 2016
11 min read
Save for later

Connecting React to Redux and Firebase – Part 2

AJ Webb
10 Nov 2016
11 min read
This is the second part in a series on using Redux and Firebase with React. If you haven't read through the first part, then you should go back and do so, since this post will build on the other. If you have gone through the first part, then you're in the right place. In this final part of this two-part series, you will be creating actions in your app to update the store. Then you will create a Firebase app and set up async actions that will subscribe to Firebase. Whenever the data on Firebase changes, the store will automatically update and your app will receive the latest state. After all of that is wired up, you'll create a quick user interface. Lets get started. Creating Actions in Redux For your actions, you are going to create a new file inside of src/: [~/code/yak]$ touch src/actions.js Inside of actions.js, you will create your first action; an action in Redux is simply an object that contains a type and a payload. The type allows you to catch the action inside of your reducer and then use the payload to update state. The type can be a string literal or a constant. I recommend using constants for the same reasons given by the Redux team. In this app you will set up some constants to follow this practice. [~/code/yak]$ touch src/constants.js Create your first constant in src/constants.js: export const SEND_MESSAGE = 'CHATS:SEND_MESSAGE'; It is good practice to namespace your constants, like the constant above the CHATS. Namespace can be specific to the CHATS portion of the application and helps document your constants. Now go ahead and use that constant to create an action. In src/actions.js, add: import { SEND_MESSAGE } from './constants'; export function sendMessage(message) { return { type: SEND_MESSAGE, payload: message }; } You now have your first action! You just need to add a way to handle that action in your reducer. So open up src/reducers.js and do just that. First import the constant. import { SEND_MESSAGE } from './constants'; Then inside of the yakApp function, you'll handle different actions: export function yakApp(state = initialState, action) { switch(action.type) { case SEND_MESSAGE: return Object.assign({}, state, { messages: state.messages.concat([action.payload]) }); default: return state; } } A couple of things are happening here, you'll notice that there is a default case that returns state, your reducer must return some sort of state. If you don't return state your app will stop working. Redux does a good job of letting you know what's happening, but it is still good to know. Another thing to notice is that the SEND_MESSAGE case does not mutate the state. Instead it creates a new state and returns it. Mutating the state can result in bad side effects that are hard to debug; do your very best to never mutate the state. You should also avoid mutating the arguments passed to a reducer, performing side effects and calling any non-pure functions within your reducer. For the most part, your reducer is set up to take the new state and return it in conjunction with the old state. Now that you have an action and a reducer that is handling that action, you are ready for your component to interact with them. In src/App.js add an input and a button: <p className="App-intro"> <input type="text" />{' '} <button>Send Message</button> </p> Now that you have the input and button in, you're going to add two things. The first is a function that runs when the button is clicked. The second is a reference on the input so that the function can easily find the value of your input. js <p className="App-intro"> <input type="text" ref={(i) => this.message = i}/>{' '} <button onClick={this.handleSendMessage.bind(this)}>Send Message</button> </p> Now that you have those set, you will need to add the sendMessage function, but you'll also need to be able to dispatch an action. So you'll want to map your actions to props, similar to how you mapped state to props in the previous guide. At the end of the file, under the mapStateToProps function, add the following function: function mapDispatchToProps(dispatch) { return { sendMessage: (msg) => dispatch(sendMessage(msg)) }; } Then you'll need to add the mapDispatchToProps function to the export statement: export default connect(mapStateToProps, mapDispatchToProps)(App); And you'll need to import the sendMessage action into src/App.js: import { sendMessage } from './actions'; Finally you'll need to create the handleSendMessage method inside of your App class, just above the render method: handleSendMessage() { const { sendMessage } = this.props; const { value } = this.message; if (!value) { return false; } sendMessage(value); this.message.value = ''; } If you still have a console.log statement inside of your render method, from the last guide, you should see your message being added to the array each time you click on the button. The final piece of UI that you need to create is a list of all the messages you've received. Add the following to the render method in src/App.js, just below the <p/> that contains your input: {messages.map((message, i) => ( <p key={i} style={{textAlign: 'left', padding: '0 10px'}}>{message}</p> ))} Now you have a crude chat interface; you can improve it later. Set up Firebase If you've never used Firebase before, you'll first need a Google account. If you don't have a Google account, sign up for one; then you'll be able to create a new Firebase project. Head over to Firebase and create a new project. If you have trouble naming it, try YakApp_[YourName]. After you have created your Firebase project, you'll be taken to the project. There is a button that says "Add Firebase to your web app"; click on it and you'll be able to get the configuration information that you will need for your app to be able to work with Firebase. A dialog will open and you should see something like this: You will only be using the database for this project, so you'll only need a portion of the config. Keep the config handy while you prepare the app to use Firebase. First add the Firebase package to your app: [~/code/yak]$ npm install --save firebase Open src/actions.js and import firebase: import firebase from 'firebase' You will only be using Firebase in your actions; that is the reason you are importing it here. Once imported, you'll need to initialize Firebase. After the import section, add your Firebase config (the one from the image above), initialize Firebase and create a reference to messages: const firebaseConfig = { apiKey: '[YOUR API KEY]', databaseURL: '[YOUR DATABASE URL]' } firebase.initializeApp(firebaseConfig); const messages = firebase.database().ref('messages'); You'll notice that you only need the apiKey and databaseUrl for now. Eventually you might want to add auth and storage, and other facets of Firebase, but for now you only need database. Now that Firebase is set up and initialized, you can use it to store the chat history. One last thing before you leave the Firebase console: Firebase automatically sets the database rules to require users to be logged in. That is a great idea, but authentication is outside of the scope of this post. So you'll need to turn that off in the rules. In the Firebase console, click on the "Database" navigation item on the left side of the page; now there should be a tab bar with a "Rules" option. Click on "Rules" and replace what is in the textbox with: { "rules": { ".read": true, ".write": true } } Create subscriptions In order to subscribe to Firebase, you will need a way to send async actions. So you'll need another package to help with that. Go ahead and install redux-thunk. [~/code/yak]$ npm install --save redux-thunk After it's finished installing, you'll need to add it as middleware to your store. That means it's time to head back to src/index.js and add the extra parameters to the createStore function. First import redux-thunk into src/index.js and import another method from redux itself. import { createStore, applyMiddleware } from 'redux';import thunk from 'redux-thunk'; Now update the createStore call to be: const store = createStore(yakApp, undefined, applyMiddleware(thunk)); You are now ready to create async actions. In order to create more actions, you're going to need more constants. Head over to src/constants.js and add the following constant. export const RECEIVE_MESSAGE = 'CHATS:RECEIVE_MESSAGE'; This is the only constant you will need for now; after this guide, you should create edit and delete actions. At that point, you'll need more constants for those actions. For now, only concern yourself with adding and subscribing. You'll need to refactor your actions and reducer to handle these new features. In src/actions.js, refactor the sendMessage action to be an async action. The way redux-thunk works is that it intercepts any action that is dispatched and inspects it. If the action returns a function instead of an object, redux-thunk stops it from reaching the reducer and runs the function. If the action returns an object, redux-thunk ignores it and allows it to continue to the reducer. Change sendMessage to look like the following: export function sendMessage(message) { return function() { messages.push(message); }; } Now when you type a message in and hit the "Submit Message" button, the message will get stored in Firebase. Try it out! UH OH! There's a problem! You are adding messages to Firebase but they aren't showing up in your app anymore! That's ok. You can fix that! You'll need to create a few more actions though, and refactor your reducer. First off, you can delete one of your constants. You no longer need the constant from earlier: export const SEND_MESSAGE = 'CHATS:SEND_MESSAGE'; Go ahead and remove it; that leaves you with only one constant. Change your constant import in both src/actions.js and src/reducer.js: import { RECEIVE_MESSAGE } from './constants'; Now in actions, add the following action: function receiveMessage(message) { return { type: RECEIVE_MESSAGE, payload: message }; } That should look familiar; it's almost identical to the original sendMessage action. You'll also need to rename the action that your reducer is looking for. So now your reducer function should look like this: export function yakApp(state = initialState, action) { switch(action.type) { case RECEIVE_MESSAGE: return Object.assign({}, state, { messages: state.messages.concat([action.payload]) }); default: return state; } } Before the list starts to show up again, you'll need to create a subscription to Firebase. Sounds more complicated than it really is. Add the following action to src/actions.js: export function subscribeToMessages() { return function(dispatch) { messages.on('child_added', data => dispatch(receiveMessage(data.val()))); } } Now, in src/App.js you'll need to dispatch that action and set up the subscription. First change your import statement from ./actions to include subscribeToMessages: import { sendMessage, subscribeToMessages } from './actions'; Now, in mapDispatchToProps you need to map subscribeToMessages: function mapDispatchToProps(dispatch) { return { sendMessage: (msg) => dispatch(sendMessage(msg)), subscribeToMessages: () => dispatch(subscribeToMessages()), }; } Finally, inside of the App class, add the componentWillMount life cycle method just above the handleSendMessage method, and call subscribeToMessages: componentWillMount() { this.props.subscribeToMessages(); } Once you save the file, you should see that your app is subscribed to the Firebase database, which is automatically updating your Redux state, and your UI is displaying all the messages stored in Firebase. Have some fun and open another browser window! Conclusion You have now created a React app from scratch, added Redux to it, refactored it to connect to Firebase and, via subscriptions, updated your entire app state! What will you do now? Of course the app could use a better UI; you could redesign it! Maybe make it a little more usable. Try learning how to create users and show who yakked about what! The sky is the limit, now that you know how to put all the pieces together! Feel free to share what you've done with the app in the comments! About the author AJ Webb is team lead and frontend engineer for @tannerlabs and a co-creator of Payba.cc.
Read more
  • 0
  • 0
  • 17251

article-image-why-we-need-design-patterns
Packt
10 Nov 2016
16 min read
Save for later

Why we need Design Patterns?

Packt
10 Nov 2016
16 min read
In this article by Praseed Pai, and Shine Xavier, authors of the book .NET Design Patterns, we will try to understand the necessity of choosing a pattern-based approach to software development. We start with some principles of software development, which one might find useful while undertaking large projects. The working example in the article starts with a requirements specification and progresses towards a preliminary implementation. We will then try to iteratively improve the solution using patterns and idioms, and come up with a good design that supports a well-defined programming Interface. In this process, we will learn about some software development principles (listed below) one can adhere to, including the following: SOLID principles for OOP Three key uses of design patterns Arlow/Nuestadt archetype patterns Entity, value, and data transfer objects Leveraging the .NET Reflection API for plug-in architecture (For more resources related to this topic, see here.) Some principles of software development Writing quality production code consistently is not easy without some foundational principles under your belt. The purpose of this section is to whet the developer's appetite, and towards the end, some references are given for detailed study. Detailed coverage of these principles warrants a separate book on its own scale. The authors have tried to assimilate the following key principles of software development which would help one write quality code: KISS: Keep it simple, Stupid DRY: Don't repeat yourself YAGNI: You aren't gonna need it Low coupling: Minimize coupling between classes SOLID principles: Principles for better OOP William of Ockham had framed the maxim Keep it simple, Stupid (KISS). It is also called law of parsimony. In programming terms, it can be translated as "writing code in a straightforward manner, focusing on a particular solution that solves the problem at hand". This maxim is important because, most often, developers fall into the trap of writing code in a generic manner for unwarranted extensibility. Even though it initially looks attractive, things slowly go out of bounds. The accidental complexity introduced in the code base for catering to improbable scenarios, often reduces readability and maintainability. The KISS principle can be applied to every human endeavor. Learn more about KISS principle by consulting the Web. Don't repeat yourself (DRY), a maxim which most programmers often forget while implementing their domain logic. Most often, in a collaborative development scenario, code gets duplicated inadvertently due to lack of communication and proper design specifications. This bloats the code base, induces subtle bugs, and make things really difficult to change. By following the DRY maxim at all stages of development, we can avoid additional effort and make the code consistent. The opposite of DRY is write everything twice (WET). You aren't gonna need it (YAGNI), a principle that compliments the KISS axiom. It serves as a warning for people who try to write code in the most general manner, anticipating changes right from the word go,. Too often, in practice, most of this code is not used to make potential code smells. While writing code, one should try to make sure that there are no hard-coded references to concrete classes. It is advisable to program to an interface as opposed to an implementation. This is a key principle which many patterns use to provide behavior acquisition at runtime. A dependency injection framework could be used to reduce coupling between classes. SOLID principles are a set of guidelines for writing better object-oriented software. It is a mnemonic acronym that embodies the following five principles: 1 Single Responsibility Principle (SRP) A class should have only one responsibility. If it is doing more than one unrelated thing, we need to split the class. 2 Open Close Principle (OCP) A class should be open for extension, closed for modification. 3 Liskov Substitution Principle (LSP) Named after Barbara Liskov, a Turing Award laureate, who postulated that a sub-class (derived class) could substitute any super class (base class) references without affecting the functionality. Even though it looks like stating the obvious, most implementations have quirks which violate this principle. 4 Interface segregation principle (ISP) It is more desirable to have multiple interfaces for a class (such classes can also be called components) than having one Uber interface that forces implementation of all methods (both relevant and non-relevant to the solution context). 5 Dependency Inversion (DI) This is a principle which is very useful for Framework design. In the case of Frameworks, the client code will be invoked by server code, as opposed to the usual process of client invoking the server. The main principle here is that abstraction should not depend upon details, rather, details should depend upon abstraction. This is also called the "Hollywood Principle" (Do not call us, we will call you back). The authors consider the preceding five principles primarily as a verification mechanism. This will be demonstrated by verifying the ensuing case study implementations for violation of these principles. Karl Seguin has written an e-book titled Foundations of Programming – Building Better Software, which covers most of what has been outlined here. Read his book to gain an in-depth understanding of most of these topics. The SOLID principles are well covered in the Wikipedia page on the subject, which can be retrieved from https://en.wikipedia.org/wiki/SOLID_(object-oriented_design). Robert Martin's Agile Principles, Patterns, and Practices in C# is a definitive book on learning about SOLID, as Robert Martin itself is the creator of these principles, even though Michael Feathers coined the acronym. Why patterns are required? According to the authors, the three key advantages of pattern-oriented software development that stand out are as follows: A language/platform-agnostic way to communicate about software artifacts A tool for refactoring initiatives (targets for refactoring) Better API design With the advent of the pattern movement, the software development community got a canonical language to communicate about software design, architecture, and implementation. Software development is a craft which has got trade-offs attached to each strategy, and there are multiple ways to develop software. The various pattern catalogues brought some conceptual unification for this cacophony in software development. Most developers around the world today who are worth their salt, can understand and speak this language. We believe you will be able to do the same at the end of the article. Fancy yourself stating the following about your recent implementation: For our tax computation example, we have used command pattern to handle the computation logic. The commands (handlers) are configured using an XML file, and a factory method takes care of the instantiation of classes on the fly using Lazy loading. We cache the commands, and avoid instantiation of more objects by imposing singleton constraints on the invocation. We support prototype pattern where command objects can be cloned. The command objects have a base implementation, where concrete command objects use the template method pattern to override methods which are necessary. The command objects are implemented using the design by contracts idiom. The whole mechanism is encapsulated using a Façade class, which acts as an API layer for the application logic. The application logic uses entity objects (reference) to store the taxable entities, attributes like tax parameters are stored as value objects. We use data transfer object (DTO) to transfer the data from the application layer to the computational layer. Arlow/Nuestadt-based archetype pattern is the unit of structuring the tax computation logic. For some developers, the preceding language/platform-independent description of the software being developed is enough to understand the approach taken. This will boos developer productivity (during all phases of SDLC, including development, maintenance, and support) as the developers will be able to get a good mental model of the code base. Without Pattern catalogs, such succinct descriptions of the design or implementation would have been impossible. In an Agile software development scenario, we develop software in an iterative fashion. Once we reach a certain maturity in a module, developers refactor their code. While refactoring a module, patterns do help in organizing the logic. The case study given next will help you to understand the rationale behind "Patterns as refactoring targets". APIs based on well-defined patterns are easy to use and impose less cognitive load on programmers. The success of the ASP.NET MVC framework, NHibernate, and API's for writing HTTP modules and handlers in the ASP.NET pipeline are a few testimonies to the process. Personal income tax computation - A case study Rather than explaining the advantages of patterns, the following example will help us to see things in action. Computation of the annual income tax is a well-known problem domain across the globe. We have chosen an application domain which is well known to focus on the software development issues. The application should receive inputs regarding the demographic profile (UID, Name, Age, Sex, Location) of a citizen and the income details (Basic, DA, HRA, CESS, Deductions) to compute his tax liability. The System should have discriminants based on the demographic profile, and have a separate logic for senior citizens, juveniles, disabled people, old females, and others. By discriminant we mean demographic that parameters like age, sex and location should determine the category to which a person belongs and apply category-specific computation for that individual. As a first iteration, we will implement logic for the senior citizen and ordinary citizen category. After preliminary discussion, our developer created a prototype screen as shown in the following image: Archetypes and business archetype pattern The legendary Swiss psychologist, Carl Gustav Jung, created the concept of archetypes to explain fundamental entities which arise from a common repository of human experiences. The concept of archetypes percolated to the software industry from psychology. The Arlow/Nuestadt patterns describe business archetype patterns like Party, Customer Call, Product, Money, Unit, Inventory, and so on. An Example is the Apache Maven archetype, which helps us to generate projects of different natures like J2EE apps, Eclipse plugins, OSGI projects, and so on. The Microsoft patterns and practices describes archetypes for targeting builds like Web applications, rich client application, mobile applications, and services applications. Various domain-specific archetypes can exist in respective contexts as organizing and structuring mechanisms. In our case, we will define some archetypes which are common in the taxation domain. Some of the key archetypes in this domain are: Sr.no Archetype Description 1 SeniorCitizenFemale Tax payers who are female, and above the age of 60 years 2 SeniorCitizen Tax payers who are male, and above the age of 60 years 3 OrdinaryCitizen Tax payers who are Male/Female, and above 18 years of age 3 DisabledCitizen Tax payers who have any disability 4 MilitaryPersonnel Tax payers who are military personnel 5 Juveniles Tax payers whose age is less than 18 years We will use demographic parameters as discriminant to find the archetype which corresponds to the entity. The whole idea of inducing archetypes is to organize the tax computation logic around them. Once we are able to resolve the archetypes, it is easy to locate and delegate the computations corresponding to the archetypes. Entity, value, and data transfer objects We are going to create a class which represents a citizen. Since citizen needs to be uniquely identified, we are going to create an entity object, which is also called reference object (from DDD catalog). The universal identifier (UID) of an entity object is the handle which an application refers. Entity objects are not identified by their attributes, as there can be two people with the same name. The ID uniquely identifies an entity object. The definition of an entity object is given as follows: public class TaxableEntity { public int Id { get; set; } public string Name { get; set; } public int Age { get; set; } public char Sex { get; set; } public string Location { get; set; } public TaxParamVO taxparams { get; set; } } In the preceding class definition, Id uniquely identifies the entity object. TaxParams is a value object (from DDD catalog) associated with the entity object. Value objects do not have a conceptual identity. They describe some attributes of things (entities). The definition of TaxParams is given as follows: public class TaxParamVO { public double Basic {get;set;} public double DA { get; set; } public double HRA { get; set; } public double Allowance { get; set; } public double Deductions { get; set; } public double Cess { get; set; } public double TaxLiability { get; set; } public bool Computed { get; set; } } While writing applications ever since Smalltalk, Model-view-controller (MVC) is the most dominant paradigm for structuring applications. The application is split into a model layer ( which mostly deals with data), view layer (which acts as a display layer), and a controller (to mediate between the two). In the Web development scenario, they are physically partitioned across machines. To transfer data between layers, the J2EE pattern catalog identified the DTO to transfer data between layers. The DTO object is defined as follows: public class TaxDTO { public int id { } public TaxParamVO taxparams { } } If the layering exists within the same process, we can transfer these objects as-is. If layers are partitioned across processes or systems, we can use XML or JSON serialization to transfer objects between the layers. A computation engine We need to separate UI processing, input validation, and computation to create a solution which can be extended to handle additional requirements. The computation engine will execute different logic depending upon the command received. The GoF command pattern is leveraged for executing the logic based on the command received. The command pattern consists of four constituents. They are: Command object Parameters Command Dispatcher Client The command object's interface has an Execute method. The parameters to the command objects are passed through a bag. The client invokes the command object by passing the parameters through a bag to be consumed by the Command Dispatcher. The Parameters are passed to the command object through the following data structure: public class COMPUTATION_CONTEXT { private Dictionary<String, Object> symbols = new Dictionary<String, Object>(); public void Put(string k, Object value) { symbols.Add(k, value); } public Object Get(string k) { return symbols[k]; } } The ComputationCommand interface, which all the command objects implement, has only one Execute method, which is shown next. The Execute method takes a bag as parameter. The COMPUTATION_CONTEXT data structure acts as the bag here. Interface ComputationCommand { bool Execute(COMPUTATION_CONTEXT ctx); } Since we have already implemented a command interface and bag to transfer the parameters, it is time that we implement a command object. For the sake of simplicity, we will implement two commands where we hardcode the tax liability. public class SeniorCitizenCommand : ComputationCommand { public bool Execute(COMPUTATION_CONTEXT ctx) { TaxDTO td = (TaxDTO)ctx.Get("tax_cargo"); //---- Instead of computation, we are assigning //---- constant tax for each arcetypes td.taxparams.TaxLiability = 1000; td.taxparams.Computed = true; return true; } } public class OrdinaryCitizenCommand : ComputationCommand { public bool Execute(COMPUTATION_CONTEXT ctx) { TaxDTO td = (TaxDTO)ctx.Get("tax_cargo"); //---- Instead of computation, we are assigning //---- constant tax for each arcetypes td.taxparams.TaxLiability = 1500; td.taxparams.Computed = true; return true; } } The commands will be invoked by a CommandDispatcher Object, which takes an archetype string and a COMPUTATION_CONTEXT object. The CommandDispatcher acts as an API layer for the application. class CommandDispatcher { public static bool Dispatch(string archetype, COMPUTATION_CONTEXT ctx) { if (archetype == "SeniorCitizen") { SeniorCitizenCommand cmd = new SeniorCitizenCommand(); return cmd.Execute(ctx); } else if (archetype == "OrdinaryCitizen") { OrdinaryCitizenCommand cmd = new OrdinaryCitizenCommand(); return cmd.Execute(ctx); } else { return false; } } } The application to engine communication The data from the application UI, be it Web or Desktop, has to flow to the computation engine. The following ViewHandler routine shows how data, retrieved from the application UI, is passed to the engine, via the Command Dispatcher, by a client: public static void ViewHandler(TaxCalcForm tf) { TaxableEntity te = GetEntityFromUI(tf); if (te == null){ ShowError(); return; } string archetype = ComputeArchetype(te); COMPUTATION_CONTEXT ctx = new COMPUTATION_CONTEXT(); TaxDTO td = new TaxDTO { id = te.id, taxparams = te.taxparams}; ctx.Put("tax_cargo",td); bool rs = CommandDispatcher.Dispatch(archetype, ctx); if ( rs ) { TaxDTO temp = (TaxDTO)ctx.Get("tax_cargo"); tf.Liabilitytxt.Text = Convert.ToString(temp.taxparams.TaxLiability); tf.Refresh(); } } At this point, imagine that a change in requirement has been received from the stakeholders. Now, we need to support tax computation for new categories. Initially, we had different computations for senior citizen and ordinary citizen. Now we need to add new Archetypes. At the same time, to make the software extensible (loosely coupled) and maintainable, it would be ideal if we provide the capability to support new Archetypes in a configurable manner as opposed to recompiling the application for every new archetype owing to concrete references. The Command Dispatcher object does not scale well to handle additional archetypes. We need to change the assembly whenever a new archetype is included, as the tax computation logic varies for each archetype. We need to create a pluggable architecture to add or remove archetypes at will. The plugin system to make system extensible Writing system logic without impacting the application warrants a mechanism—that of loading a class on the fly. Luckily, the .NET Reflection API provides a mechanism for one to load a class during runtime, and invoke methods within it. A developer worth his salt should learn the Reflection API to write systems which change dynamically. In fact, most of the technologies like ASP.NET, Entity framework, .NET Remoting, and WCF work because of the availability of Reflection API in the .NET stack. Henceforth, we will be using an XML configuration file to specify our tax computation logic. A sample XML file is given next: <?xml version="1.0"?> <plugins> <plugin archetype ="OrindaryCitizen" command="TaxEngine.OrdinaryCitizenCommand"/> <plugin archetype="SeniorCitizen" command="TaxEngine.SeniorCitizenCommand"/> </plugins> The contents of the XML file can be read very easily using LINQ to XML. We will be generating a Dictionary object by the following code snippet: private Dictionary<string,string> LoadData(string xmlfile) { return XDocument.Load(xmlfile) .Descendants("plugins") .Descendants("plugin") .ToDictionary(p => p.Attribute("archetype").Value, p => p.Attribute("command").Value); } Summary In this article, we have covered quite a lot of ground in understanding why pattern-oriented software development is a good way to develop modern software. We started the article citing some key principles. We progressed further to demonstrate the applicability of these key principles by iteratively skinning an application which is extensible and resilient to changes. Resources for Article: Further resources on this subject: Debugging Your .NET Application [article] JSON with JSON.Net [article] Using ASP.NET Controls in SharePoint [article]
Read more
  • 0
  • 0
  • 37642
Modal Close icon
Modal Close icon