Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7014 Articles
article-image-introduction-sdn-transformation-legacy-sdn
Packt
07 Jun 2017
23 min read
Save for later

Introduction to SDN - Transformation from legacy to SDN

Packt
07 Jun 2017
23 min read
In this article, by Reza Toghraee, the author of the book, Learning OpenDayLight, we will: What is and what is not SDN Components of a SDN Difference between SDN and overlay The SDN controllers (For more resources related to this topic, see here.) You might have heard about Software-Defined Networking (SDN). If you are in networking industry this is a topic which probably you have studied initially when first time you heard about the SDN.To understand the importance of SDN and SDN controller, let's look at Google. Google silently built its own networking switches and controller called Jupiter. A home grown project which is mostly software driven and supports such massive scale of Google. The SDN base is There is a controller who knows the whole network. OpenDaylight (ODL), is a SDN controller. In other words, it's the central brain of the network. Why we are going towards SDN Everyone who is hearing about SDN, should ask this question that why are we talking about SDN. What problem is it trying to solve? If we look at traditional networking (layer 2, layer 3 with routing protocols such as BGP, OSPF) we are completely dominated by what is so called protocols. These protocols in fact have been very helpful to the industry. They are mostly standard. Different vendor and products can communicate using standard protocols with each other. A Cisco router can establish a BGP session with a Huawei switch or an open source Quagga router can exchange OSPF routes with a Juniper firewall. Routing protocol is a constant standard with solid bases. If you need to override something in your network routing, you have to find a trick to use protocols, even by using a static route. SDN can help us to come out of routing protocol cage, look at different ways to forward traffic. SDN can directly program each switch or even override a route which is installed by routing protocol. There are high-level benefits of using SDN which we explain few of them as follows: An integrated network: We used to have a standalone concept in traditional network. Each switch was managed separately, each switch was running its own routing protocol instance and was processing routing information messages from other neighbors. In SDN, we are migrating to a centralized model, where the SDN controller becomes the single point of configuration of the network, where you will apply the policies and configuration. Scalable layer 2 across layer 3:Having a layer 2 network across multiple layer 3 network is something which all network architects are interested and till date we have been using proprietary methods such as OTV or by using a service provider VPLS service. With SDN, we can create layer 2 networks across multiple switches or layer 3 domains (using VXLAN) and expand the layer 2 networks. In many cloud environment, where the virtual machines are distributed across different hosts in different datacenters, this is a major requirement. Third-party application programmability: This is a very generic term, isn't it? But what I'm referring to is to let other applications communicate with your network. For example,In many new distributed IP storage systems, the IP storage controller has ability to talk to network to provide the best, shortest path to the storage node. With SDN we are letting other applications to control the network. Of course this control has limitation and SDN doesn't allow an application to scrap the whole network. Flexible application based network:In SDN, everything is an application. L2/L3, BGP, VMware Integration, and so on all are applications running in SDN controller. Service chaining:On the fly you add a firewall in the path or a load balancer. This is service insertion. Unified wired and wireless: This is an ideal benefit, to have a controller which supports both wired and wireless network. OpenDaylight is the only controller which supports CAPWAP protocols which allows integration with wireless access points. Components of a SDN A software defined network infrastructure has two main key components: The SDN Controller (only one, could be deployed in a highly available cluster) The SDN enabled switches(multiple switches, mostly in a Clos topology in a datacenter):   SDN controller is the single brain of the SDN domain. In fact, an SDN domain is very similar to a chassis based switch. You can imagine supervisor or management module of a chassis based switch as a SDN controller and rest of the line card and I/O cards as SDN switches. The main difference between a SDN network and a chassis based switch is that you can scale out the SDN with multiple switches, where in a chassis based switch you are limited to the number of slots in that chassis: Controlling the fabric It is very important that you understand the main technologies involved in SDN. These methods are used by SDN controllers to manage and control the SDN network. In general, there are two methods available for controlling the fabric: Direct fabric programming: In this method, SDN controller directly communicates with SDN enabled switches via southbound protocols such as OpenFlow, NETCONF and OVSDB. SDN controller programs each switch member with related information about fabric, and how to forward the traffic. Direct fabric programming is the method used by OpenDaylight. Overlay:In overlay method, SDN controller doesn't rely on programing the network switches and routers. Instead it builds an virtual overlay network on top of existing underlay network. The underlay network can be a L2 or L3 network with traditional network switches and router, just providing IP connectivity. SDN controller uses this platform to build the overlay using encapsulation protocols such as VXLAN and NVGRE. VMware NSX uses overlay technology to build and control the virtual fabric. SDN controllers One of the key fundamentals of SDN is disaggregation. Disaggregation of software and hardware in a network and also disaggregation of control and forwarding planes. SDN controller is the main brain and controller of an SDN environment, it's a strategic control point within the network and responsible for communicating information to: Routers and switches and other network devices behind them. SDN controllers uses APIs or protocols (such as OpenFlow or NETCONF) to communicate with these devices. This communication is known as southbound Upstream switches, routers or applications and the aforementioned business logic (via APIs or protocols). This communication is known as northbound. An example for a northbound communication is a BGP session between a legacy router and SDN controller. If you are familiar with chassis based switches like Cisco Catalyst 6500 or Nexus 7k chassis, you can imagine a SDN network as a chassis, with switches and routers as its I/O line cards and SDN controller as its supervisor or management module. Infact SDN is similar to a very scalable chassis where you don't have any limitation on number of physical slots. SDN controller is similar to role of management module of a chassis based switch and it controls all switches via its southbound protocols and APIs. The following table compares the SDN controller and a chassis based switch:  SDN Controller Chassis based switch Supports any switch hardware Supports only specific switch line cards Can scale out, unlimited number of switches Limited to number of physical slots in the chassis Supports high redundancy by multiple controllers in a cluster Supports dual management redundancy, active standby Communicates with switches via southbound protocols such as OpenFlow, NETCONF, BGP PCEP Use proprietary protocols between management module and line cards Communicates with routers, switches and applications outside of SDN via northbound protocols such as BGP, OSPF and direct API Communicates with other routers and switches outside of chassis via standard protocols such as BGP, OSPF or APIs. The first protocol that popularized the concept behind SDN was OpenFlow. When conceptualized by networking researchers at Stanford back in 2008, it was meant to manipulate the data plane to optimize traffic flows and make adjustments, so the network could quickly adapt to changing requirements. Version 1.0 of the OpenFlow specification was released in December of 2009; it continues to be enhanced under the management of the Open Networking Foundation, which is a user-led organization focused on advancing the development of open standards and the adoption of SDN technologies. OpenFlow protocol was the first protocol that helped in popularizing SDN. OpenFlow is a protocol designed to update the flow tables in a switch. Allowing the SDN controller to access the forwarding table of each member switch or in other words to connect control plane and data plane in SDN world. Back in 2008, OpenFlow conceptualized by networking researchers at Stanford University, the initial use of OpenFlow was to alter the switch forwarding tables to optimize traffic flows and make adjustments, so the network could quickly adapt to changing requirements. After introduction of OpenFlow, NOX introduced as original OpenFlow controller (still there wasn't concept of SDN controller). NOX was providing a high-level API capable of managing and also developing network control applications. Separate applications were required to run on top of NOX to manage the network.NOX was initially developed by Nicira networks (which acquired by VMware, and finally became part of VMware NSX). NOX introduced along with OpenFlow in 2009. NOX was a closed source product but ultimately it was donated to SDN community which led to multiple forks and sub projects out of original NOX. For example, POX is a sub project of NOX which provides Python support. Both NOX and POX were early controllers. NOX appears an inactive development, however POX is still in use by the research community as it is a Python based project and can be easily deployed. POX is hosted at http://github.com/noxrepo/pox NOX apart from being the first OpenFlow or SDN controller also established a programing model which inherited by other subsequent controllers. The model was based on processing of OpenFlow messages, with each incoming OpenFlow message trigger an event that had to be processed individually. This model was simple to implement but not efficient and robust and couldn't scale. Nicira along with NTT and Google started developing ONIX, which was meant to be a more abstract and scalable for large deployments. ONIX became the base for Nicira (the core of VMware NSX or network virtualization platform) also there are rumors that it is also the base for Google WAN controller. ONIX was planned to become open source and donated to community but for some reasons the main contributors decided to not to do it which forced the SDN community to focus on developing other platforms. Started in 2010, a new controller introduced,the Beacon controller and it became one of the most popular controllers. It born with contribution of developers from Stanford University. Beacon is a Java-based open source OpenFlow controller created in 2010. It has been widely used for teaching, research, and as the basis of Floodlight. Beacon had the first built-in web user-interface which was a huge step forward in the market of SDN controllers. Also it provided a easier method to deploy and run compared to NOX. Beacon was an influence for design of later controllers after it, however it was only supporting star topologies which was one of the limitations on this controller. Floodlight was a successful SDN controller which was built as a fork of Beacon. BigSwitch networks is developing Floodlight along with other developers. In 2013, Beacon popularity started to shrink down and Floodlight started to gain popularity. Floodlight had fixed many issues of Beacon and added lots of additional features which made it one of the most feature rich controllers available. It also had a web interface, a Java-based GUI and also could get integrated with OpenStack using quantum plugin. Integration with OpenStack was a big step forward as it could be used to provide networking to a large pool of virtual machines, compute and storage. Floodlight adoption increased by evolution of OpenStack and OpenStack adopters. This gave Floodlight greater popularity and applicability than other controllers that came before. Most of controllers came after Floodlight also supported OpenStack integration. Floodlight is still supported and developed by community and BigSwitch networks, and is a base for BigCloud Fabric (the BigSwitch's commercial SDN controller). There are other open source SDN controllers which introduced such as Trema (ruby-based from NEC), Ryu (supported by NTT), FlowER, LOOM and the recent OpenMUL. The following table shows the current open source SDN controllers:  Active open source SDNcontroller Non-active open source SDN controllers Floodlight Beacon OpenContrail FlowER OpenDaylight NOX LOOM NodeFlow OpenMUL   ONOS   POX   Ryu   Trema     OpenDaylight OpenDaylight started in early 2013, and was originally led by IBM and Cisco. It was a new collaborative open source project. OpenDaylight hosted under Linux Foundation and draw support and interest from many developers and adopters. OpenDaylight is a platform to provide common foundations and a robust array of services for SDN environments. OpenDaylight uses a controller model which supports OpenFlow as well as other southbound protocols. It is the first open source controller capable of employing non-OpenFlow proprietary control protocols which eventually lets OpenDaylight to integrate with modern and multi-vendor networks. The first release of OpenDaylight in February 2014 with code name of Hydrogen, followed by Helium in September 2014. The Helium release was significant because it marked a change in direction for the platform that has influenced the way subsequent controllers have been architected. The main change was in the service abstraction layer, which is the part of the controller platform that resides just above the southbound protocols, such as OpenFlow, isolating them from the northbound side and where the applications reside. Hydrogen used an API-driven Service Abstraction Layer (AD-SAL), which had limitations specifically, it meant the controller needed to know about every type of device in the network AND have an inventory of drivers to support them. Helium introduced a Model-driven service abstraction layer (MD-SAL), which meant the controller didn't have to account for all the types of equipment installed in the network, allowing it to manage a wide range of hardware and southbound protocols. Helium release made the framework much more agile and adaptable to changes in the applications; an application could now request changes to the model, which would be received by the abstraction layer and forwarded to the network devices. The OpenDaylight platform built on this advancement in its third release, Lithium, which was introduced in June of 2015. This release focused on broadening the programmability of the network, enabling organizations to create their own service architectures to deliver dynamic network services in a cloud environment and craft intent-based policies. Lithium release was worked on by more than 400 individuals, and contributions from Big Switch Networks, Cisco, Ericsson, HP, NEC, and so on, making it one of the fastest growing open source projects ever. The fourth release, Beryllium come out in February of 2016 and the most recent fifth release, Boron released in September 2016. Many vendors have built and developed commercial SDN controller solutions based on OpenDaylight. Each product has enhanced or added features to OpenDaylight to have some differentiating factor. The use of OpenDaylight in different vendor products are: A base, but sell a commercial version with additional proprietary functionality—for example: Brocade, Ericsson, Ciena, and so on. Part of their infrastructure in their Network as a Service (or XaaS) offerings—for example: Telstra, IBM, and so on. Elements for use in their solution—for example: ConteXtream (now part of HP) Open Networking Operating System (ONOS), which was open sourced in December 2014 is focused on serving the needs of service providers. It is not as widely adopted as OpenDaylight, ONOS has been finding success and gaining momentum around WAN use cases. ONOS is backed by numerous organizations including AT&T, Cisco, Fujitsu, Ericsson, Ciena, Huawei, NTT, SK Telecom, NEC, and Intel, many of whom are also participants in and supporters of OpenDaylight. Apart from open source SDN controllers, there are many commercial, proprietary controllers available in the market. Products such as VMware NSX, Cisco APIC, BigSwitch Big Cloud Fabric, HP VAN and NEC ProgrammableFlow are example commercial and proprietary products. The following table lists the commercially available controllers and their relationship to OpenDaylight:  ODL-based ODL-friendly Non-ODL based Avaya Cyan (acquired by Ciena) BigSwitch Brocade HP Juniper Ciena NEC Cisco ConteXtream (HP) Nuage Plexxi Coriant   PLUMgrid Ericsson Pluribus Extreme Sonus Huawei (also ships non-ODL controller) VMware NSX Core features of SDN Regardless of an open source or a proprietary SDN platform, there are core features and capabilities which requires the SDN platform to support. These capabilities include: Fabric programmability:Providing the ability to redirect traffic, apply filters to packets (dynamically), and leverage templates to streamline the creation of custom applications. Ensuring northbound APIs allow the control information centralized in the controller available to be changed by SDN applications. This will ensure the controller can dynamically adjust the underlying network to optimize traffic flows to use the least expensive path, take into consideration varying bandwidth constraints, meet quality of service (QoS) requirements. Southbound protocol support:Enabling the controller to communicate to switches and routers and manipulate and optimize how they manage the flow of traffic. Currently OpenFlow is the most standard protocol used between different networking vendors, while there are other southbound protocols that can be used. A SDN platform should support different versions of OpenFlow in order to provide compatibility with different switching equipments. External API support:Ensuring the controller can be used within the varied orchestration and cloud environments such as VMware vSphere, OpenStack, and so on. Using APIs the orchestration platform can communicate with SDN platform in order to publish network policies. For example VMware vSphere shall talk to SDN platform to extend the virtual distributed switches(VDS) from virtual environment to the physical underlay network without any requirement form an network engineer to configure the network. Centralized monitoring and visualization:Since SDN controller has a full visibility over the network, it can offer end-to-end visibility of the network and centralized management to improve overall performance, simplify the identification of issues and accelerate troubleshooting. The SDN controller will be able to discover and present a logical abstraction of all the physical links in the network, also it can discover and present a map of connected devices (MAC addresses) which are related to virtual or physical devices connected to the network. The SDN controller support monitoring protocols, such as syslog, snmp and APIs in order to integrate with third-party management and monitoring systems. Performance: Performance in a SDN environment mainly depends on how fast SDN controller fills the flow tables of SDN enabled switches. Most of SDN controllers pre-populate the flow tables on switches to minimize the delay. When a SDN enabled switch receives a packet which doesn't find a matching entry in its flow table, it sends the packet to the SDN controller in order to find where the packet needs to get forwarded to. A robust SDN solution should ensure that the number of requests form switches are minimum and SDN controller doesn't become a bottleneck in the network. High availability and scalability: Controllers must support high availability clusters to ensure reliability and service continuity in case of failure of a controller. Clustering in SDN controller expands to scalability. A modern SDN platform should support scalability in order to add more controller nodes with load balancing in order to increase the performance and availability. Modern SDN controllers support clustering across multiple different geographical locations. Security:Since all switches communicate with SDN controller, the communication channel needs to be secured to ensure unauthorized devices doesn't compromise the network. SDN controller should secure the southbound channels, use encrypted messaging and mutual authentication to provide access control. Apart from that the SDN controller must implement preventive mechanisms to prevent from denial of services attacks. Also deployment of authorization levels and level controls for multi-tenant SDN platforms is a key requirement. Apart from the aforementioned features SDN controllers are likely to expand their function in future. They may become a network operating system and change the way we used to build networks with hardware, switches, SFPs and gigs of bandwidth. The future will look more software defined, as the silicon and hardware industry has already delivered their promises for high performance networking chips of 40G, 100G. Industry needs more time to digest the new hardware and silicons and refresh the equipment with new gears supporting 10 times the current performance. Current SDN controllers In this section, I'm putting the different SDN controllers in a table. This will help you to understand the current market players in SDN and how OpenDaylight relates to them:  Vendors/product Based on OpenDaylight? Commercial/open source Description Brocade SDN controller Yes Commercial It's a commercial version of OpenDaylight, fully supported and with extra reliable modules. Cisco APIC No Commercial Cisco Application Policy Infrastructure Controller (APIC) is the unifying automation and management point for the Application Centric Infrastructure (ACI) data center fabric. Cisco uses APIC controller and Nexus 9k switches to build the fabric. Cisco uses OpFlex as main southbound protocol. Erricson SDN controller Yes Commercial Ericsson's SDN controller is a commercial (hardened) version OpenDaylight SDN controller. Domain specific control applications that use the SDN controller as platform form the basis of the three commercial products in our SDN controller portfolio. Juniper OpenContrial /Contrail No Both OpenContrail is opensource, and Contrail itself is a commercial product. Juniper Contrail Networking is an open SDN solution that consists of Contrail controller, Contrail vRouter, an analytics engine, and published northbound APIs for cloud and NFV. OpenContrail is also available for free from Juniper. Contrail promotes and use MPLS in datacenter. NEC Programmable Flow No Commercial NEC provides its own SDN controller and switches. NEC SDN platform is one of choices of enterprises and has lots of traction and some case studies.   Avaya SDN Fx controller Yes Commercial Based on OpenDaylight, bundled as a solution package.   Big Cloud Fabric No Commercial BigSwitch networks solution is based on Floodlight opensource project. BigCloud Fabric is a robust, clean SDN controller and works with bare metal whitebox switches. BigCloud Fabric includes SwitchLightOS which is a switch operating system can be loaded on whitebox switches with Broadcom Trident 2 or Tomahawk silicons. The benefit of BigCloud Fabric is that you are not bound to any hardware and you can use baremetal switches from any vendor.   Ciena's Agility Yes Commercial Ciena's Agility multilayer WAN controller is built atop the open-source baseline of the OpenDaylight Project—an open, modular framework created by a vendor-neutral ecosystem (rather than a vendor-centric ego-system) that will enable network operators to source network services and applications from both Ciena's Agility and others. HP VAN (Virtual Application Network) No Commercial The building block of the HP open SDN ecosystem, the controller allows third-party developers to deliver innovative SDN solutions. Huawei Agile controller Yes and No (based on editions) Commercial Huawei's SDN controller which integrates as a solution with Huawei enterprise switches Nuage No Commercial Nuage Networks VSP provides SDN capabilities for clouds of all sizes. It is implemented as a non-disruptive overlay for all existing virtualized and non-virtualized server and network resources. Pluribus Netvisor No Commercial Netvisor Premium and Open Netvisor Linux are distributed network operating systems. Open Netvisor integrates atraditional, interoperable networking stack (L2/L3/VXLAN) with an SDN distributed controller that runs in everyswitch of the fabric. VMware NSX No Commercial VMware NSX is an Overlay type of SDN, which currently works with VMware vSphere. The plan is to support OpenStack in future. VMware NSX also has built-in firewall, router and L4 load balancers allowing micro segmentation. OpenDaylight as an SDN controller Previously, we went through the role of SDN controller, and a brief history of ODL.ODL is a modular open SDN platform which allows developers to build any network or business application on top of it to drive the network in the way they want. Currently OpenDaylight has reached to its fifth release (Boron, which is the fifth element in periodic table). ODL releases are named based on periodic table elements, started from first release the Hydrogen. ODL has a 6 month release period, with many developers working on expanding the ODL, 2 releases per year is expected from community. For technical readers to understand it more clearly, the following diagram will help: ODL platform has a broad set of use cases for multivendor, brown field, green fields, service providers and enterprises. ODL is a foundation for networks of the future. Service providers are using ODL to migrate their services to a software enabled level with automatic service delivery and coming out of circuit-based mindset of service delivery. Also they work on providing a virtualized CPE with NFV support in order to provide flexible offerings. Enterprises use ODL for many use cases, from datacenter networking, Cloud and NFS, network automation and resource optimization, visibility, control to deploying a fully SDN campus network. ODL uses a MD-SAL which makes it very scalable and lets it incorporate new applications and protocols faster. ODL supports multiple standard and proprietary southbound protocols, for example with full support of OpenFlow and OVSDB, ODL can communicate with any standard hardware (or even the virtual switches such as Open vSwitch(OVS) supporting such protocols). With such support, ODL can be deployed and used in multivendor environments and control hardware from different vendors from a single console no matter what vendor and what device it is, as long as they support standard southbound protocols. ODL uses a micro service architecture model which allows users to control applications, protocols and plugins while deploying SDN applications. Also ODL is able to manage the connection between external consumers and providers. The followingdiagram explains the ODL footprint and different components and projects within the ODL: Micro servicesarchitecture ODL stores its YANG data structure in a common data store and uses messaging infrastructure between different components to enable a model-driven approach to describe the network and functions. In ODL MD-SAL, any SDN application can be integrated as a service and then loaded into the SDN controller. These services (apps) can be chained together in any number and ways to match the application needs. This concept allows users to only install and enable the protocols and services they need which makes the system light and efficient. Also services and applications created by users can be shared among others in the ecosystem since the SDN application deployment for ODL follows a modular design. ODL supports multiple southbound protocols. OpenFlow and OpenFlow extension such as Table Type Patterns (TTP), as well as other protocols including NETCONF, BGP/PCEP, CAPWAP and OVSDB. Also ODL supports Cisco OpFlex protocol: ODL platform provides a framework for authentication, authorization and accounting (AAA), as well as automatic discovery and securing of network devices and controllers. Another key area in security is to use encrypted and authenticated communication trough southbound protocols with switches and routers within the SDN domain. Most of southbound protocols support security encryption mechanisms. Summary In this article we learned about SDN, and why it is important. We reviewed the SDN controller products, the ODL history as well as core features of SDN controllers and market leader controllers. We managed to dive in some details about SDN . Resources for Article: Further resources on this subject: Managing Network Devices [article] Setting Up a Network Backup Server with Bacula [article] Point-to-Point Networks [article]
Read more
  • 0
  • 0
  • 28980

article-image-erasure-coding-cold-storage
Packt
07 Jun 2017
20 min read
Save for later

Erasure coding for cold storage

Packt
07 Jun 2017
20 min read
In this article by Nick Frisk, author of the book Mastering Ceph, we will get acquainted with erasure coding. Ceph's default replication level provides excellent protection against data loss by storing three copies of your data on different OSD's. The chance of losing all three disks that contain the same objects within the period that it takes Ceph to rebuild from a failed disk, is verging on the extreme edge of probability. However, storing 3 copies of data vastly increases both the purchase cost of the hardware but also associated operational costs such as power and cooling. Furthermore, storing copies also means that for every client write, the backend storage must write three times the amount of data. In some scenarios, either of these drawbacks may mean that Ceph is not a viable option. Erasure codes are designed to offer a solution. Much like how RAID 5 and 6 offer increased usable storage capacity over RAID1, erasure coding allows Ceph to provide more usable storage from the same raw capacity. However also like the parity based RAID levels, erasure coding brings its own set of disadvantages. (For more resources related to this topic, see here.) In this article you will learn: What is erasure coding and how does it work Details around Ceph's implementation of erasure coding How to create and tune an erasure coded RADOS pool A look into the future features of erasure coding with Ceph Kraken release What is erasure coding Erasure coding allows Ceph to achieve either greater usable storage capacity or increase resilience to disk failure for the same number of disks versus the standard replica method. Erasure coding achieves this by splitting up the object into a number of parts and then also calculating a type of Cyclic Redundancy Check, the Erasure code, and then storing the results in one or more extra parts. Each part is then stored on a separate OSD. These parts are referred to as k and m chunks, where k refers to the number of data shards and m refers to the number of erasure code shards. As in RAID, these can often be expressed in the form k+m or 4+2 for example. In the event of an OSD failure which contains an objects shard which isone of the calculated erasure codes, data is read from the remaining OSD's that store data with no impact. However, in the event of an OSD failure which contains the data shards of an object, Ceph can use the erasure codes to mathematically recreate the data from a combination of the remaining data and erasure code shards. k+m The more erasure code shards you have, the more OSD failure's you can tolerate and still successfully read data. Likewise the ratio of k to m shards each object is split into, has a direct effect on the percentage of raw storage that is required for each object. A 3+1 configuration will give you 75% usable capacity, but only allows for a single OSD failure and so would not be recommended. In comparison a three way replica pool, only gives you 33% usable capacity. 4+2 configurations would give you 66% usable capacity and allows for 2 OSD failures. This is probably a good configuration for most people to use. At the other end of the scale a 18+2 would give you 90% usable capacity and still allows for 2 OSD failures. On the surface this sounds like an ideal option, but the greater total number of shards comes at a cost. The higher the number of total shards has a negative impact on performance and also an increased CPU demand. The same 4MB object that would be stored as a whole single object in a replicated pool, is now split into 20 x 200KB chunks, which have to be tracked and written to 20 different OSD's. Spinning disks will exhibit faster bandwidth, measured in MB/s with larger IO sizes, but bandwidth drastically tails off at smaller IO sizes. These smaller shards will generate a large amount of small IO and cause additional load on some clusters. Also its important not to forget that these shards need to be spread across different hosts according to the CRUSH map rules, no shard belonging to the same object can be stored on the same host as another shard from the same object. Some clusters may not have a sufficient number hosts to satisfy this requirement. Reading back from these high chunk pools is also a problem. Unlike in a replica pool where Ceph can read just the requested data from any offset in an object, in an Erasure pool, all shards from all OSD's have to be read before the read request can be satisfied. In the 18+2 example this can massively amplify the amount of required disk read ops and average latency will increase as a result. This behavior is a side effect which tends to only cause a performance impact with pools that use large number of shards. A 4+2 configuration in some instances will get a performance gain compared to a replica pool, from the result of splitting an object into shards.As the data is effectively striped over a number of OSD's, each OSD is having to write less data and there is no secondary and tertiary replica's to write. How does erasure coding work in Ceph As with Replication, Ceph has a concept of a primary OSD, which also exists when using erasure coded pools. The primary OSD has the responsibility of communicating with the client, calculating the erasure shards and sending them out to the remaining OSD's in the Placement Group (PG) set. This is illustrated in the diagram below: If an OSD in the set is down, the primary OSD, can use the remaining data and erasure shards to reconstruct the data, before sending it back to the client. During read operations the primary OSD requests all OSD's in the PG set to send their shards. The primary OSD uses data from the data shards to construct the requested data, the erasure shards are discarded. There is a fast read option that can be enabled on erasure pools, which allows the primary OSD to reconstruct the data from erasure shards if they return quicker than data shards. This can help to lower average latency at the cost of slightly higher CPU usage. The diagram below shows how Ceph reads from an erasure coded pool: The next diagram shows how Ceph reads from an erasure pool, when one of the data shards is unavailable. Data is reconstructed by reversing the erasure algorithm using the remaining data and erasure shards. Algorithms and profiles There are a number of different Erasure plugins you can use to create your erasure coded pool. Jerasure The default erasure plugin in Ceph is the Jerasure plugin, which is a highly optimized open source erasure coding library. The library has a number of different techniques that can be used to calculate the erasure codes. The default is Reed Solomon and provides good performance on modern processors which can accelerate the instructions that the technique uses. Cauchy is another technique in the library, it is a good alternative to Reed Solomon and tends to perform slightly better. As always benchmarks should be conducted before storing any production data on an erasure coded pool to identify which technique best suits your workload. There are also a number of other techniques that can be used, which all have a fixed number of m shards. If you are intending on only having 2 m shards, then they can be a good candidate, as there fixed size means that optimization's are possible lending to increased performance. In general the jerasure profile should be prefer in most cases unless another profile has a major advantage, as it offers well balanced performance and is well tested. ISA The ISA library is designed to work with Intel processors and offers enhanced performance. It too supports both Reed Solomon and Cauchy techniques. LRC One of the disadvantages of using erasure coding in a distributed storage system is that recovery can be very intensive on networking between hosts. As each shard is stored on a separate host, recovery operations require multiple hosts to participate in the process. When the crush topology spans multiple racks, this can put pressure on the inter rack networking links. The LRC erasure plugin, which stands for Local Recovery Codes, adds an additional parity shard which is local to each OSD node. This allows recovery operations to remain local to the node where a OSD has failed and remove the need for nodes to receive data from all other remaining shard holding nodes. However the addition of these local recovery codes does impact the amount of usable storage for a given number of disks. In the event of multiple disk failures, the LRC plugin has to resort to using global recovery as would happen with the jerasure plugin. SHingled Erasure Coding The SHingled Erasure Coding (SHEC) profile is designed with similar goals to the LRC plugin, in that it reduces the networking requirements during recovery. However instead of creating extra parity shards on each node, SHEC shingles the shards across OSD's in an overlapping fashion. The shingle part of the plugin name represents the way the data distribution resembles shingled tiles on a roof of a house. By overlapping the parity shards across OSD's, the SHEC plugin reduces recovery resource requirements for both single and multiple disk failures. Where can I use erasure coding Since the Firefly release of Ceph in 2014, there has been the ability to create a RADOS pool using erasure coding. There is one major thing that you should be aware of, the erasure coding support in RADOS does not allow an object to be partially updated. You can write to an object in an erasure pool, read it back and even overwrite it whole, but you cannot update a partial section of it. This means that erasure coded pools can't be used for RBD and CephFS workloads and is limited to providing pure object storage either via the Rados Gateway or applications written to use librados. The solution at the time was to use the cache tiering ability which was released around the same time, to act as a layer above an erasure coded pools that RBD could be used. In theory this was a great idea, in practice, performance was extremely poor. Every time an object was required to be written to, the whole object first had to be promoted into the cache tier. This act of promotion probably also meant that another object somewhere in the cache pool was evicted. Finally the object now in the cache tier could be written to. This whole process of constantly reading and writing data between the two pools meant that performance was unacceptable unless a very high percentage of the data was idle. During the development cycle of the Kraken release, an initial implementation for support for direct overwrites on n erasure coded pool was introduced. As of the final Kraken release, support is marked as experimental and is expected to be marked as stable in the following release. Testing of this feature will be covered later in this article. Creating an erasure coded pool Let's bring our test cluster up again and switch into SU mode in Linux so we don't have to keep prepending sudo to the front of our commands Erasure coded pools are controlled by the use of erasure profiles, these control how many shards each object is broken up into including the split between data and erasure shards. The profiles also include configuration to determine what erasure code plugin is used to calculate the hashes. The following plugins are available to use <list of plugins> To see a list of the erasure profiles run # cephosd erasure-code-profile ls You can see there is a default profile in a fresh installation of Ceph. Lets see what configuration options it contains # cephosd erasure-code-profile get default The default specifies that it will use the jerasure plugin with the Reed Solomon error correcting codes and will split objects into 2 data shards and 1 erasure shard. This is almost perfect for our test cluster, however for the purpose of this exercise we will create a new profile. # cephosd erasure-code-profile set example_profile k=2 m=1 plugin=jerasure technique=reed_sol_van # cephosd erasure-code-profile ls You can see our new example_profile has been created. Now lets create our erasure coded pool with this profile: # cephosd pool create ecpool 128 128 erasure example_profile The above command instructs Ceph to create a new pool called ecpool with a 128 PG's. It should be an erasure coded pool and should use our "example_profile" we previously created. Lets create an object with a small text string inside it and the prove the data has been stored by reading it back: # echo "I am test data for a test object" | rados --pool ecpool put Test1 – # rados --pool ecpool get Test1 - That proves that the erasure coded pool is working, but it's hardly the most exciting of discoveries. Lets have a look to see if we can see what's happening at a lower level. First, find out what PG is holding the object we just created # cephosd map ecpoolTest1 The result of the above command tells us that the object is stored in PG 3.40 on OSD's1, 2 and 0. In this example Ceph cluster that's pretty obvious as we only have 3 OSD's, but in larger clusters that is a very useful piece of information. We can now look at the folder structure of the OSD's and see how the object has been split. The PG's will likely be different on your test cluster, so make sure the PG folder structure matches the output of the "cephosd map" command above. ls -l /var/lib/ceph/osd/ceph-2/current/1.40s0_head/ # ls -l /var/lib/ceph/osd/ceph-1/current/1.40s1_head/ # ls -l /var/lib/ceph/osd/ceph-0/current/1.40s2_head/                 total 4 Notice how the PG directory names have been appended with the shard number, replicated pools just have the PG number as their directory name. If you examine the contents of the object files, you will see our text string that we entered into the object when we created it. However due to the small size of the text string, Ceph has padded out the 2nd shard with null characters and the erasure shard hence will contain the same as the first. You can repeat this example with a new object containing larger amounts of text to see how Ceph splits the text into the shards and calculates the erasure code. Overwrites on erasure code pools with Kraken Introduced for the first time in the Kraken release of Cephas an experimental feature, was the ability to allow partial overwrites on erasure coded pools. Partial overwrite support allows RBD volumes to be created on erasure coded pools, making better use of raw capacity of the Ceph cluster. In parity RAID, where a write request doesn't span the entire stripe, a read modify write operation is required. This is needed as the modified data chunks will mean the parity chunk is now incorrect. The RAID controller has to read all the current chunks in the stripe, modify them in memory, calculate the new parity chunk and finally write this back out to the disk. Ceph is also required to perform this read modify write operation, however the distributed model of Ceph increases the complexity of this operation.When the primary OSD for a PG receives a write request that will partially overwrite an existing object, it first works out which shards will be not be fully modified by the request and contacts the relevant OSD's to request a copy of these shards. The primary OSD then combines these received shards with the new data and calculates the erasure shards. Finally the modified shards are sent out to the respective OSD's to be committed. This entire operation needs to conform the other consistency requirements Ceph enforces, this entails the use of temporary objects on the OSD, should a condition arise that Ceph needs to roll back a write operation. This partial overwrite operation, as can be expected, has a performance impact. In general the smaller the write IO's, the greater the apparent impact. The performance impact is a result of the IO path now being longer, requiring more disk IO's and extra network hops. However, it should be noted that due to the striping effect of erasure coded pools, in the scenario where full stripe writes occur, performance will normally exceed that of a replication based pool. This is simply down to there being less write amplification due to the effect of striping. If performance of an Erasure pool is not suitable, consider placing it behind a cache tier made up of a replicated pool. Despite partial overwrite support coming to erasure coded pools in Ceph, not every operation is supported. In order to store RBD data on an erasure coded pool, a replicated pool is still required to hold key metadata about the RBD. This configuration is enabled by using the –data-pool option with the rbd utility. Partial overwrite is also not recommended to be used with Filestore. Filestore lacks several features that partial overwrites on erasure coded pools uses, without these features extremely poor performance is experienced. Demonstration This feature requires the Kraken release or newer of Ceph. If you have deployed your test cluster with the Ansible and the configuration provided, you will be running Ceph Jewel release. The following steps show how to use Ansible to perform a rolling upgrade of your cluster to the Kraken release. We will also enable options to enable experimental options such as bluestore and support for partial overwrites on erasure coded pools. Edit your group_vars/ceph variable file and change the release version from Jewel to Kraken. Also add: ceph_conf_overrides: global: enable_experimental_unrecoverable_data_corrupting_features: "debug_white_box_testing_ec_overwrites bluestore" And to correct a small bug when using Ansible to deploy Ceph Kraken, add: debian_ceph_packages: - ceph - ceph-common - ceph-fuse To the bottom of the file run the following Ansible playbook: ansible-playbook -K infrastructure-playbooks/rolling_update.yml Ansible will prompt you to make sure that you want to carry out the upgrade, once you confirm by entering yes the upgrade process will begin. Once Ansible has finished, all the stages should be successful as shown below: Your cluster has now been upgraded to Kraken and can be confirmed by running ceph -v on one of yours VM's running Ceph. As a result of enabling the experimental options in the configuration file, every time you now run a Ceph command, you will be presented with the following warning. This is designed as a safety warning to stop you running these options in a live environment, as they may cause irreversible data loss. As we are doing this on a test cluster, that is fine to ignore, but should be a stark warning not to run this anywhere near live data. The next command that is required to be run is to enable the experimental flag which allows partial overwrites on erasure coded pools. DO NOT RUN THIS ON PRODUCTION CLUSTERS cephosd pool get ecpooldebug_white_box_testing_ec_overwrites true Double check you still have your erasure pool called ecpool and the default RBD pool # cephosdlspools 0 rbd,1ecpool, And now create the rbd. Notice that the actual RBD header object still has to live on a replica pool, but by providing an additional parameter we can tell Ceph to store data for this RBD on an erasure coded pool. rbd create Test_On_EC --data-pool=ecpool --size=1G The command should return without error and you now have an erasure coded backed RBD image. You should now be able to use this image with any librbd application. Note: Partial overwrites on Erasure pools require Bluestore to operate efficiently. Whilst Filestore will work, performance will be extremely poor. Troubleshooting the 2147483647 error An example of this error is shown below when running the ceph health detail command. If you see 2147483647 listed as one of the OSD's for an erasure coded pool, this normally means that CRUSH was unable to find a sufficient number of OSD's to complete the PG peering process. This is normally due to the number of k+m shards being larger than the number of hosts in the CRUSH topology. However, in some cases this error can still occur even when the number of hosts is equal or greater to the number of shards. In this scenario it's important to understand how CRUSH picks OSD's as candidates for data placement. When CRUSH is used to find a candidate OSD for a PG, it applies the crushmap to find an appropriate location in the crush topology. If the result comes back as the same as a previous selected OSD, Ceph will retry to generate another mapping by passing slightly different values into the crush algorithm. In some cases if there is a similar number of hosts to the number of erasure shards, CRUSH may run out of attempts before it can suitably find correct OSD mappings for all the shards. Newer versions of Ceph has mostly fixed these problems by increasing the CRUSH tunable choose_total_tries. Reproducing the problem In order to aid understanding of the problem in more detail, the following steps will demonstrate how to create an erasure coded profile that will require more shards than our 3 node cluster can support. Firstly, like earlier in the articlecreate a new erasure profile, but modify the k/m parameters to be k=3 m=1: $ cephosd erasure-code-profile set broken_profile k=3 m=1 plugin=jerasure technique=reed_sol_van And now create a pool with it: $ cephosd pool create broken_ecpool 128 128 erasure broken_profile If we look at the output from ceph -s, we will see that the PG's for this new pool are stuck in the creating state. The output of ceph health detail, shows the reason why and we see the 2147483647 error. If you encounter this error and it is a result of your erasure profile being larger than your number of hosts or racks, depending on how you have designed your crushmap. Then the only real solution is to either drop the number of shards, or increase number of hosts. Summary In this article you have learnt what erasure coding is and how it is implemented in Ceph. You should also have an understanding of the different configuration options possible when creating erasure coded pools and their suitability for different types of scenarios and workloads. Resources for Article: Further resources on this subject: Ceph Instant Deployment [article] Working with Ceph Block Device [article] GNU Octave: Data Analysis Examples [article]
Read more
  • 0
  • 0
  • 17450

article-image-microsoft-azure-stack-architecture
Packt
07 Jun 2017
13 min read
Save for later

The Microsoft Azure Stack Architecture

Packt
07 Jun 2017
13 min read
In this article by Markus Klein and Susan Roesner, authors of the book Azure Stack for Datacenters, we will help you to plan, build, run, and develop your own Azure-based datacenter running Azure Stack technology. The goal is that the technology in your datacenter will be a 100 percent consistent using Azure, which provides flexibility and elasticity to your IT infrastructure. We will learn about: Cloud basics The Microsoft Azure Stack Core management services Using Azure Stack Migrating services to Azure Stack (For more resources related to this topic, see here.) Cloud as the new IT infrastructure Regarding the technical requirements of today's IT, the cloud is always a part of the general IT strategy. It does not depend upon the region in which the company is working in, nor does it depend upon the part of the economy—99.9 percent of all companies have cloud technology already in their environment. The good question for a lot of CIOs is general: "To what extent do we allow cloud services, and what does that mean to our infrastructure?" So it's a matter of compliance, allowance, and willingness. The top 10most important questions for a CIO to prepare for the cloud are as follows: Are we allowed to save our data in the cloud? What classification of data can be saved in the cloud? How flexible are we regarding the cloud? Do we have the knowledge to work with cloud technology? How does our current IT setup and infrastructure fit into the cloud's requirements? Is our current infrastructure already prepared for the cloud? Are we already working with a cloud-ready infrastructure? Is our Internet bandwidth good enough? What does the cloud mean to my employees? Which technology should we choose? Cloud Terminology The definition of the term "cloud" is not simple, but we need to differentiate between the following: Private cloud: This is a highly dynamic IT infrastructure based on virtualization technology that is flexible and scalable. The resources are saved in a privately owned datacenter either in your company or a service provider of your choice. Public cloud: This is a shared offering of IT infrastructure services that are provided via the Internet. Hybrid cloud: This is a mixture of a private and public cloud. Depending on compliance or other security regulations, the services that could be run in a public datacenter are already deployed there, but the services that need to be stored inside the company are running there. The goal is to run these services on the same technology to provide the agility, flexibility, and scalability to move services between public and private datacenters. In general, there are some big players within the cloud market (for example, Amazon Web Services, Google, Azure, and even Alibaba). If a company is quite Microsoft-minded from the infrastructure point of view, they should have a look at Microsoft Azure datacenters. Microsoft started in 2008 with their first datacenter, and today, they invest a billion dollars every month in Azure. As of today, there are about 34 official datacenters around the world that form Microsoft Azure, besides some that Microsoft does not talk about (for example, USGovernment Azure). There are some dedicated datacenters, such as the German Azure cloud, that do not have connectivity to Azure worldwide. Due to compliance requirements, these frontiers need to exist, but the technology of each Azure datacenter is the same although the services offered may vary. The following map gives an overview of the locations (so-called regions) in Azure as of today and provide an idea of which ones will be coming soon: The Microsoft cloud story When Microsoft started their public cloud, they decided that there must be a private cloud stack too, especially, to prepare their infrastructure to run in Azure sometime in the future. The first private cloud solution was the System Center suite, with System Center Orchestrator and Service Provider Foundation (SPF) and Service Manager as the self-service portal solution. Later on, Microsoft launched Windows Azure Pack for Windows Server. Today, Windows Azure Pack is available as a product focused on the private cloud and provides a self-service portal (the well-known old Azure Portal, code name red dog frontend), and it uses the System Center suite as its underlying technology: Microsoft Azure Stack In May2015, Microsoft formally announced a new solution that brings Azure to your datacenter. This solution was named Microsoft Azure Stack. To put it in one sentence: Azure Stack is the same technology with the same APIs and portal as Public Azure, but you could run it in your datacenter or in that of your service provider. With Azure Stack, System Center is completely gone because everything is the way it is in Azure now, and in Azure, there is no System Center at all. This is what the primary focus of this article is. The following diagram gives a current overview of the technical design of Azure Stack compared with Azure: The one and only difference between Azure Stack and Azure is the cloud infrastructure. In Azure, there are thousands of servers that are part of the solution; with Azure Stack, the number is slightly smaller. That's why there is the cloud-inspired infrastructure based on Windows Server, Hyper-V, and Azure technologies as the underlying technology stack. There is no System Center product in this stack anymore. This does not mean that it cannot be there (for example, SCOM for on-premise monitoring), but Azure Stack itself provides all functionality with the solution itself. For stability and functionality, Microsoft decided to provide Azure Stack as a so-called integrated system, so it will come to your door with the hardware stack included. The customer buys Azure Stack as a complete technology stack. At general availability(GA), the hardware OEMs are HPE, DellEMC, and Lenovo. In addition to this, there will be a one-host PoC deployment available for download that could be run as a proof of concept solution on every type of hardware, as soon as it meets the hardware requirements. Technical design Looking at the technical design a bit more in depth, there are some components that we need to dive deeper into: The general basis of Azure Stack is Windows Server 2016 technology, which builds the cloud-inspired infrastructure: Storage Spaces Direct (S2D) VXLAN Nano Server Azure Resource Manager (ARM) Storage Spaces Direct (S2D) Storage Spaces and Scale-Out File Server were technologies that came with Windows Server 2012. The lack of stability in the initial versions and the issues with the underlying hardware was a bad phase. The general concept was a shared storage setup using JBODs controlled from Windows Server 2012 Storage Spaces servers and a magic Scale-Out File Server cluster that acted as the single point of contact for storage: With Windows Server 2016, the design is quite different and the concept relies on a shared-nothing model, even with local attached storage: This is the storage design Azure Stack is coming up with as one of its main pillars. VXLANnetworking technology With Windows Server 2012, Microsoft introduced Software-Defined Networking(SDN)and the NVGRE technology. Hyper-V Network Virtualization supports Network Virtualization using Generic Routing Encapsulation (NVGRE) as the mechanism to virtualize IP addresses. In NVGRE, the virtual machine's packet is encapsulated inside another packet: Hyper-V Network Virtualization supports NVGRE as the mechanism to virtualize IP addresses. In NVGRE, the virtual machine's packet is encapsulated inside another packet. VXLAN comes as the new SDNv2protocol, is RFC compliant, and is supported by most network hardware vendors by default. The Virtual eXtensible Local Area Network (VXLAN) RFC 7348 protocol has been widely adopted in the marketplace, with support from vendors such as Cisco, Brocade, Arista, Dell, and HP. The VXLAN protocol uses UDP as the transport: Nano Server Nano Server offers a minimal-footprint headless version of Windows Server 2016. It completely excludes the graphical user interface, which means that it is quite small, headless, and easy to handle regarding updates and security fixes, but it doesn't provide the GUI expected by customers of Windows Server. Azure Resource Manager (ARM) The “magical” Azure Resource Manager is a 1-1 bit share with ARM from Azure, so it has the same update frequency and features that are available in Azure, too. ARM is a consistent management layer that saves resources, dependencies, inputs, and outputs as an idempotent deployment as a JSON file called an ARM template. This template defines the tastes of a deployment, whether it be VMs, databases, websites, or anything else. The goal is that once a template is designed, it can be run on each Azure-based cloud platform, including Azure Stack. ARM provides cloud consistency with the finest granularity, and the only difference between the clouds is the region the template is being deployed to and the corresponding REST endpoints. ARM not only provides a template for a logical combination of resources within Azure, it manages subscriptions and role-based access control(RBAC) and defines the gallery, metric, and usage data, too. This means quite simply that everything that needs to be done with Azure resources should be done with ARM. Not only does Azure Resource Manager design one virtual machine, it is responsible for setting up one to a bunch of resources that fit together for a specific service. Even ARM templates can be nested; this means they can depend on each other. When working with ARM, you should know the following vocabulary: Resource: Are source is a manageable item available in Azure Resource group: A resource group is the container of resources that fit together within a service Resource provider: A resource provider is a service that can be consumed within Azure Resource manager template: A resource manager template is the definition of a specific service Declarative syntax: Declarative syntax means that the template does not define the way to set up a resource; it just defines how the result and the resource itself has the feature to set up and configure itself to fulfill the syntax to create your own ARM templates, you need to fulfill the following minimum requirements: A test editor of your choice Visual Studio Community Edition Azure SDK Visual Studio Community Edition is available for free from the Internet. After setting these things up, you could start it and define your own templates: Setting up a simple blank template looks like this: There are different ways to get a template so that you can work on and modify it to fit your needs: Visual Studio templates Quick-start templates on GitHub Azure ARM templates You could export the ARM template directly from Azure Portal if the resource has been deployed: After clicking on View template, the following opens up: For further reading on ARM basics, the Getting started with Azure Resource Managerdocument is a good place to begin: http://aka.ms/GettingStartedWithARM. PowerShell Desired State Configuration We talked about ARM and ARM templates that define resources, but they are unable to design the waya VM looks inside, specify which software needs to be installed, and how the deployment should be done. This is why we need to have a look at VMextensions.VMextensions define what should be done after ARM deployment has finished. In general, the extension could be anything that's a script. The best practice is to use PowerShell and its add-on called Desired State Configuration (DSC). DSC defines—quite similarly to ARM—how the software needs to be installed and configured. The great concept is that it also monitors whether the desired state of a virtual machine is changing (for example, because an administrator uninstalls or reconfigures a machine). If it does, it makes sure within minutes whether the original state will be fulfilled again and rolls back the actions to the desired state: Migrating services to Azure Stack If you are running virtual machines today, you're already using a cloud-based technology, although we do not call it cloud today. Basically, this is the idea of a private cloud. If you are running Azure Pack today, you are quite near Azure Stack from the processes point of view but not the technology part. There is a solution called connectors for Azure Pack that lets you have one portal UI for both cloud solutions. This means that the customer can manage everything out of the Azure Stack Portal, although services run in Azure Pack as a legacy solution. Basically, there is no real migration path within Azure Stack. But the way to solve this is quite easy, because you could use every tool that you can use to migrate services to Azure. Azure website migration assistant The Azure website migration assistant will provide a high-level readiness assessment for existing websites. This report outlines sites that are ready to move and elements that may need changes, and it highlights unsupported features. If everything is prepared properly, the tool creates any website and associated database automatically and synchronizes the content. You can learn more about it at https://azure.microsoft.com/en-us/downloads/migration-assistant/: For virtual machines, there are two tools available: Virtual Machines Readiness Assessment Virtual Machines Optimization Assessment Virtual Machines Readiness Assessment The Virtual Machines Readiness Assessment tool will automatically inspect your environment and provide you with a checklist and detailed report on steps for migrating the environment to the cloud. The download location is https://azure.microsoft.com/en-us/downloads/vm-readiness-assessment/. If you run the tool, you will get an output like this: Virtual Machines Optimization Assessment The Virtual Machine Optimization Assessment tool will at first start with a questionnaire and ask several questions about your deployment. Then, it will create an automated data collection and analysis of your Azure VMs. It generates a custom report with tenprioritized recommendations across six focus areas. These areas are security and compliance, performance and scalability, and availability and business continuity. The download location ishttps://azure.microsoft.com/en-us/downloads/vm-optimization-assessment/. Summary Azure Stack provides a real Azure experience in your datacenter. The UI, administrative tools, and even third-party solutions should work properly. The design of Azure Stack is a very small instance of Azure with some technical design modifications, especially regarding the compute, storage, and network resource providers. These modifications give you a means to start small, think big, and deploy large when migrating services directly to public Azure sometime in the future, if needed. The most important tool for planning, describing, defining, and deploying Azure Stack services is Azure Resource Manager, just like in Azure. This provides you a way to create your services just once but deploy them many times. From the business perspective, this means you have better TCO and lower administrative costs. Resources for Article: Further resources on this subject: Deploying and Synchronizing Azure Active Directory [article] What is Azure API Management? [article] Installing and Configuring Windows Azure Pack [article]
Read more
  • 0
  • 0
  • 29512

article-image-web-application-information-gathering
Packt
05 Jun 2017
4 min read
Save for later

Web Application Information Gathering

Packt
05 Jun 2017
4 min read
In this article by Ishan Girdhar, author of the book, Kali Linux Intrusion and Exploitation Cookbook, we will cover the following recipes: Setup API keys for the recon-ng framework Use recon-ng for reconnaissance (For more resources related to this topic, see here.) Setting up API keys for recon-ng framework In this recipe, we will see how we need to set up API keys before we start using recon-ng. Recon-ng is one of the most powerful information gathering tools, if used appropriately, it can help pentesters locating good amount of information from public sources. With the latest version available, recon-ng provides the flexibility to set it up as your own app/client in various social networking websites. Getting ready For this recipe, you require an Internet connection and web browser. How to do it... To set up recon-ng API keys, open the terminal and launch recon-ng and type the commands shown in the following screenshot: Next, type keys list as shown in the following screenshot: Let's start by adding twitter_API & twitter_secret. Log in to Twitter, go to https://apps.twitter.com/, and create a new application as shown in the following screenshot: Click on Create Application once the application is created, navigate to Keys & Access tokens tabs, and copy the secret key and API key as shown in the following screenshot: Copy the API key and reopen the terminal window again run the following command to add the key: Keys add twitter_api <your-copied-api-key> Now, enter the following command to enter the twitter_secret name in recon-ng: keys add twitter_secret <you_twitter_secret> Once you added the keys, you can see the keys added in the recon-ng tool by entering the following command: keys list How it works... In this recipe, you learned how to add API keys to the recon-ng tool. To demonstrate the same, we have created a Twitter application and used Twitter_API and Twitter_Secret and added them to the recon-ng tool. The result is as shown in the following screenshot: Similarly, you will need to include all the API keys here in the recon-ng if you want to gather information from these sources. In next recipe, you will learn how to use recon-ng for information gathering. Use recon-ng for reconnaissance In this recipe, you will learn to use recon-ng for reconnaissance. Recon-ng is a full-featured Web Reconnaissance framework written in Python. Complete with independent modules, database interaction, built-in convenience functions, interactive help, and command completion, Recon-ng provides a powerful environment in which open source web-based reconnaissance can be conducted quickly and thoroughly. Getting ready To install Kali Linux, you will require an Internet connection. How to do it... Open a terminal and start the recon-ng framework, as shown in the following screenshot: Recon-ng has the look and feel like that of Metasploit. To see all the available modules, enter the following command: show modules Recon-ng will list all available modules, as shown in the following screenshot: Let's go ahead and use our first module for information gathering. Enter the following command: use recon/domains-vulnerabilities/punkspider Now, enter the commands shown in the following screenshot: As you can see, there are some vulnerabilities discovered and are available publically. Let's use another module that fetches any known and reported vulnerabilities from xssed.com. The XSSed project was created in early February 2007 by KF and DP. It provides information on all things related to cross-site scripting vulnerabilities and is the largest online archive of XSS vulnerable websites. It's a good repository of XSS to gather information. To begin with, enter the following command: Show module use recon/domains-vulnerabilities/xssed Show Options Set source Microsoft.com Show Options RUN You will see the following output: As you can see, recon-ng has aggregated the publically available vulnerabilities from XSSed, as shown in the following screenshot: Similarly, you can keep using the different modules until and unless you get your required information regarding your target. Summary In this article, you learned how to add API keys to the recon-ng tool. To demonstrate the same, we have created a Twitter application and used Twitter_API and Twitter_Secret and added them to the recon-ng tool. You also learned how to use recon-ng for reconnaissance. Resources for Article: Further resources on this subject: Getting Started with Metasploitable2 and Kali Linux [article] Wireless Attacks in Kali Linux [article] What is Kali Linux [article]
Read more
  • 0
  • 0
  • 24882

article-image-booting-android-system-using-pxenfs
Packt
29 May 2017
31 min read
Save for later

Booting Up Android System Using PXE/NFS

Packt
29 May 2017
31 min read
In this article by Roger Ye, the author of the book Android System Programming, introduces two challenges present in most of the embedded Linux system programming that you need to resolve before you can boot up your system. These two challenges are: How to load your kernel and ramdisk image? Where do you store your filesystem? (For more resources related to this topic, see here.) This is true for Android systems as well. After you got a development board, you have to build the bootloader first and flash it to the storage on the board before you can move to the next step. After that, you have to build the kernel, ramdisk, and filesystem. You have to repeat this tedious build, flash, and test process again and again. In this process, you need to use special tools to flash various images for the development board. Many embedded system developers want to get rid of the process of flashing images so that they can concentrate on the development work itself. Usually, they use two techniques PXE boot and NFS filesystem. If you search "Android NFS" on the Internet, you can find many articles or discussions about this topic. I don't have a development board on my hand, so I will use VirtualBox as a virtual hardware board to demonstrate how to boot a system using PXE bootloader and NFS as filesystem. To repeat the same process in this article, you need to have the following hardware and software environment. A computer running Ubuntu 14.04 as the host environment VirtualBox Version 5.1.2 or above A virtual machine running Android x86vbox A virtual machine running Ubuntu 14.04 as PXE server (optional) Android x86vbox is a ROM that I developed in the book Android System Programming. You can download the ROM image at the following URL: https://sourceforge.net/projects/android-system-programming/files/android-7/ch14/ch14.zip/download After you download the preceding ZIP file, you can find a list of files here: initrd.img: This is the modified ramdisk image from open source project android-x86 kernel: NFS-enabled Android kernel for device x86vbox ramdisk.img: ramdisk for the Android boot ramdisk-recovery.img: ramdisk for the recovery boot update-android-7.1.1_r4_x86vbox_ch14_r1.zip: OTA update image of x86vbox, you can install this image using recovery Setting up a PXE boot environment What is PXE? PXE means Preboot eXecution Environment. Before we can boot a Linux environment, what we need is to find a way to load kernel and ramdisk to the system memory. This is one of the major tasks performed by most of Linux bootloader. The bootloader usually fetches kernel and ramdisk from a kind of storage device, such as flash storage, harddisk, or USB. It can also be retrieved from a network connection. PXE is a method that we can boot a device with LAN connection and a PXE-capable network interface controller (NIC). As shown in the following diagram, PXE uses DHCP and TFTP protocols to complete the boot process. In a simplest environment, a PXE server is setup as both DHCP and TFTP server. The client NIC obtains the IP address from DHCP server and uses TFTP protocol to get the kernel and ramdisk images to start the boot process. I will demonstrate how to prepare a PXE-capable ROM for VirtualBox virtio network adapter so we can use this ROM to boot the system via PXE. You will also learn how to set up a PXE server which is the key element in the PXE boot setup. In VirtualBox, it also includes a built-in PXE server. We will explore this option as well. Preparing PXE Boot ROM Even though PXE boot is supported by VirtualBox, but the setup is not consistent on different host platforms. You may get error message like PXE-E3C - TFTP Error - Access Violation during the boot. This is because the PXE boot depends on LAN boot ROM. When you choose different network adapters, you may get different test results. To get a consistent test result, you can use the LAN boot ROM from Etherboot/gPXE project. gPXE is an open source (GPL) network bootloader. It provides a direct replacement for proprietary PXE ROMs, with many extra features such as DNS, HTTP, iSCSI, and so on. There is a page at gPXE project website about how to set up LAN boot ROM for VirtualBox: http://www.etherboot.org/wiki/romburning/vbox The following table is a list of network adapters supported by VirtualBox. VirtualBox adapters PCI vendor ID PCI device ID Mfr name Device name Am79C970A 1022h 2000h AMD PCnet-PCI II (AM79C970A) Am79C973 1022h 2000h AMD PCnet-PCI III (AM79C973) 82540EM 8086h 100Eh Intel Intel PRO/1000 MT Desktop (82540EM) 82543GC 8086h 1004h Intel Intel PRO/1000 T Server (82543GC) 82545EM 8086h 100Fh Intel Intel PRO/1000 MT Server (82545EM) virtio 1AF4h 1000h   Paravirtualized Network (virtio-net) Since paravirtualized network has better performance in most of the situation, we will explore how to support PXE boot using virtio-net network adapter. Downloading and building the LAN boot ROM There may be LAN boot ROM binary image available on the Internet, but it is not provided at gPXE project. We have to build from source code according to the instructions from gPXE project website. Let's download and build the source code using the following commands. $ git clone git://git.etherboot.org/scm/gpxe.git $ cd gpxe/src $ make bin/1af41000.rom # for virtio 1af4:1000 Fixing up the ROM image Before the ROM image can be used, the ROM image has to be updated due to VirtualBox have the following requirements on ROM image size: Size must be 4K aligned (that is, a multiple of 4096) Size must not be greater than 64K Let's check the image size first and make sure it is not larger than 65536 bytes (64K): $ ls -l bin/1af41000.rom | awk '{print $5}' 62464 We can see that it is less than 64K. Now, we have to pad the image file to a 4K boundary. We can do this using the following commands. $ python >>> 65536 - 62464 # Calculate padding size 3072 >>> f = open('bin/1af41000.rom', 'a') >>> f.write(' ' * 3072) # Pad with zeroes We can check the image file size again. $ ls -l 1af41000.rom | awk '{print $5}' 65536 As we can see, the size is 64K now. Configuring the virtual machine to use the LAN boot ROM To use this LAN boot ROM, we can use command VBoxManage to update VirtualBox settings. We use the following command to set the LanBootRom path: $ VBoxManage setextradata $VM_NAME VBoxInternal/Devices/pcbios/0/Config/LanBootRom /path/to/1af41000.rom Replace $VM_NAME with your VM's name. If you use global as $VM_NAME then all VMs will use the gPXE LAN boot ROM. To remove the above configuration, you just have to reset the path value as below. $ VBoxManage setextradata $VM_NAME VBoxInternal/Devices/pcbios/0/Config/LanBootRom You can also check the current configuration using the below command: $ VBoxManage getextradata $VM_NAME VBoxInternal/Devices/pcbios/0/Config/LanBootRom Value: /path/to/1af41000.rom If you don't want to build LAN boot ROM yourself, you can use the one that I posted at: https://sourceforge.net/projects/android-system-programming/files/android-7/ch14/1af41000.rom/download Setting up PXE boot environment With a proper PXE ROM installed, we can set up the PXE on the host now. Before we setup a PXE server, we need to think about the network connections. There are three ways a virtual machine in VirtualBox can connect to the network. Bridged network: Connect to the same physical network as the host. It looks like the virtual machine connects to the same LAN connection as the host. Host only network: Connect to a virtual network, which is only visible by the virtual machine and the host. In this configuration, the virtual machine cannot connect to outside network, such as Internet. NAT network: Connect to the host network through NAT. This is the most common choice. In this configuration, the virtual machine can access to the external network, but the external network cannot connect to the virtual machine directly. For an example, if you set up a FTP service on the virtual machine, the computers on the LAN of the host cannot access this FTP service. If you want to publish this service, you have to use port forwarding setting to do this. With these concepts in mind, if you want to use a dedicated machine as the PXE server, you can use bridged network in your environment. However, you must be very careful using this kind of setup. This is usually done by the IT group in your organization, since you cannot setup a DHCP server on the LAN without affecting others. We won't use this option here. The host only network is actually a good choice for this case, because this kind of network is an isolated network configuration. The network connection only exists between the host and the virtual machine. The problem is we cannot access to the outside network. We will configure two network interfaces to our virtual machine instance one host only network for the PXE boot and one NAT network to access Internet. We will see this configuration later. In VirtualBox, it also has a built-in PXE server in NAT network. With this option, we don't need to setup PXE server by ourselves. We will explain how to set up our own PXE boot environment first and then explain how to use the built-in PXE server of VirtualBox. As we can see in the following figure, we have two virtual machines pxeAndroid and PXE Server in our setup. The upper part PXE Server is optional. If we use the built-in PXE server, both PXE server and NFS server will be on the development host. Let's look at how to set up our own PXE server first. To setup a PXE boot environment, we need to install a TFTP and DHCP server. I assume that you can set up a Linux virtual machine by yourself. I will use Ubuntu as an example here. In your environment, you have to create two virtual machines. A PXE server with a host only network interface A virtual machine to boot Android with a host only network interface and a NAT network interface Setting up TFTP server We can install tftp server in the PXE server using the following command: $ sudo apt-get install tftpd-hpa After the tftp server is installed, we need to set up PXE boot configuration in the folder /var/lib/tftpboot. We can use the following command to start tftp server. $ sudo service tftpd-hpa restart Configuring DHCP server Once tftp server is installed, we need to install a DHCP server. We can install DHCP server using the following command. $ sudo apt-get install isc-dhcp-server After install the DHCP server, we have to add the following lines into the DHCP server configuration file at /etc/dhcp/dhcpd.conf. subnet 192.168.56.0 netmask 255.255.255.0 { range 192.168.56.10 192.168.56.99; filename "pxelinux.0"; } We use the IP address range 192.168.56.x for the host only subnet, since this is the default range after we create a host-only network in VirtualBox. There may be more than one host only network configured in your VirtualBox environment. You may want to check the right host only network configuration that you want to use and set the above configuration file according to the host only network setup. Configuring and testing the PXE boot After we set up the PXE server, we can create a virtual machine instance to test the environment. We will demonstrate this using Ubuntu 14.04 as the host environment. The same setup can be duplicated to Windows or OS X environment as well. If you use a Windows environment, you have to set up the NFS server inside the PXE server. The Windows host cannot support NFS. Setting up the Android Virtual Machine Let's create a virtual machine called pxeAndroid in VirtualBox first. After start VirtualBox, we can click on  the button New to create a new virtual machine as shown in the following screenshot: We call it pxeAndroid and choose Linux as the type of virtual machine. We can just follow the wizard to create this virtual machine with suitable configuration. After the virtual machine is created, we need to make a few changes to the settings. The first one need to be changed is the network configuration as I mentioned before we need both NAT and host Only connections. We can click on the name of virtual machine pxeAndroid first and then click on the button Settings to change the settings. Select the option Network in the left-hand side, as we can see from the following screen: We select the Adapter 1 and it is default to NAT network. We need to change the Adapter Type to Paravirtualized Network (virtio-net) since we will use the PXE ROM that we just built. The NAT network can connect to the outside network. It supports port forwarding so that we can access certain services in the virtual machine. The one that we need to set up here is the ADB service. We need to use ADB to debug the pxeAndroid device later. We can set up the port forwarding for ADB as follows: Now, we can select Adapter 2 to set up a host-only network as the following figure: We choose the adapter as Host-only Adapter and Adapter Type as Paravirtualized Network (virtio-net). Next, we can click on the System option to set the boot order so the default boot order is to boot from the network interface as the following figure: Configuring pxelinux.cfg Before we can test the virtual machine we just setup, we need to specify in the configuration file to let the PXE boot to know where to find the kernel and ramdisk images. The PXE boot process is something like this. When the virtual machine pxeAndroid power on, the client will get the IP address through DHCP. After the DHCP configuration is found, the configuration includes the standard information such as IP address, subnet mask, gateway, DNS, and so on. In addition, it also provides the location of TFTP server and the filename of a boot image. The name of boot image is usually called pxelinux.0 as we can see in the previous section when we set up DHCP server. The name of boot image is vmname.pxe for the built-in PXE boot environment. Where the vmname should be the name of virtual machine. For example, it is pxeAndroid.pxe for our virtual machine. The client contacts TFTP server to obtain the boot image. TFTP server sends the boot image (pxelinux.0 or vmname.pxe), and the client executes it. By default, the boot image searches the pxelinux.cfg directory on TFTP server for boot configuration files. The client downloads all the files it needs (kernel, ramdisk, or root filesystem) and then loads them. The target machine pxeAndroid reboots. In the above step 5, the boot image searches the boot configuration files in the following steps: First, it searches for the boot configuration file that is named according to the MAC address represented in lower case hexadecimal digits with dash separators. For example, for the MAC address 08:00:27:90:99:7B, it searches for the file 08-00-27-90-99-7b. Then, it searches for the configuration file using the IP address (of the machine that is being booted) in upper case hexadecimal digits. For example, for the IP address 192.168.56.100, it searches for the file C0A83864. If that file is not found, it removes one hexadecimal digit from the end and tries again. However, if the search is still not successful, it finally looks for a file named default (in lower case). For example, if the boot filename is /var/lib/tftpboot/pxelinux.0, the Ethernet MAC address is 08:00:27:90:99:7B, and the IP address is 192.168.56.100, the boot image looks for file names in the following order: /var/lib/tftpboot/pxelinux.cfg/08-00-27-90-99-7b /var/lib/tftpboot/pxelinux.cfg/C0A83864 /var/lib/tftpboot/pxelinux.cfg/C0A8386 /var/lib/tftpboot/pxelinux.cfg/C0A838 /var/lib/tftpboot/pxelinux.cfg/C0A83 /var/lib/tftpboot/pxelinux.cfg/C0A8 /var/lib/tftpboot/pxelinux.cfg/C0A /var/lib/tftpboot/pxelinux.cfg/C0 /var/lib/tftpboot/pxelinux.cfg/C /var/lib/tftpboot/pxelinux.cfg/default The boot image pxelinux.0 is part of an open source project syslinux. We can get the boot image and the menu user interface from Syslinux project using the following commands: $ sudo apt-get install syslinux After Syslinux is installed, pxelinux.0 can be copied to the TFTP root folder as . $ cp /usr/lib/syslinux/pxelinux.0 /var/lib/tftpboot/pxelinux.0 To have a better user interface, we can copy menu.c32 to the TFTP folder as well. $ cp /usr/lib/syslinux/menu.c32 /var/lib/tftpboot/menu.c32 pxelinux.cfg/default Now, we will look at how to configure the boot configuration file pxelinux.cfg/default. In our setup, it looks like the following code snippet: prompt 1 default menu.c32 timeout 100 label 1. NFS Installation (serial port) - x86vbox menu x86vbox_install_serial kernel x86vbox/kernel append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img root=/dev/nfs rw androidboot.hardware=x86vbox INSTALL=1 DEBUG=2 SRC=/x86vbox ROOT=192.168.56.1:/home/sgye/vol1/android-6/out/target/product qemu=1 qemu.gles=0 label 2. x86vbox (ROOT=/dev/sda1, serial port) menu x86vbox_sda1 kernel x86vbox/kernel append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img androidboot.hardware=x86vbox DEBUG=2 SRC=/android-x86vbox ROOT=/dev/sda1 ... The syntax in the boot configuration file can be found at the following URL from Syslinux project: http://www.syslinux.org/wiki/index.php?title=SYSLINUX In the mentioned configuration file that we use, we can see the following commands and options: prompt: It will let the bootloader know whether it will show a LILO-style "boot:" prompt. With this command line prompt, you can input the option directly. All the boot options define by the command label. default: It defines the default boot option. timeout: If more than one label entry is available, this directive indicates how long to pause at the boot: prompt until booting automatically, in units of 1/10 s. The timeout is cancelled when any key is pressed, the assumption being the user will complete the command line. A timeout of zero will disable the timeout completely. The default is 0. label: A human-readable string that describes a kernel and options. The default label is linux, but you can change this with the DEFAULT keyword. kernel: The kernel file that the boot image will boot. append: The kernel command line, which can be passed to the kernel during the boot. In this configuration file, we show two boot options. In the first option, we can boot to a minimum Linux environment using NFS root filesystem. We can install the x86vbox images from that environment to hard disk. The source location of installation is your AOSP build output folder. In the second option, we can boot x86vbox from disk partition /dev/sda1. After the x86vbox image is installed on the partition /dev/sda1, the Android system can be started using the second option. Using VirtualBox internal PXE booting with NAT VirtualBox provides a built-in support for PXE boot using NAT network. We can also set up PXE boot using this built-in facility. There are a few minor differences between the built-in PXE and the one that we set up in the PXE server. The built-in PXE uses the NAT network connection while the PXE server uses host only network connection. TFTP root is at /var/lib/tftpboot for the normal PXE setup while the built-in TFTP root is at $HOME/.VirtualBox/TFTP on Linux or %USERPROFILE%.VirtualBoxTFTP on Windows. Usually, the default boot image name is pxelinux.0, but it is vmname.pxe for the VirtualBox built-in PXE. For example, if we use pxeAndroid as virtual machine name, we have to make a copy of pxelinux.0 and name it pxeAndroid.pxe under the VirtualBox TFTP root folder. If you choose to use the built-in PXE support, you don't have to create a PXE server by yourself. This is the recommended test environment to simplify the test process. Setting up serial port for debugging The reason that we want to boot Android using PXE and NFS is that we want to use a very simple bootloader and find an easier way to debug the system. In order to see the debug log, we want to redirect the debug output from the video console to a serial port so that we can separate the graphic user interface from the debug output. We need to do two things in order to meet our goals. The Linux kernel debug message can be re-directed to a specific channel using kernel command-line arguments. We specify this in PXE boot configuration with option console=ttyS3,115200. This is defined in pxelinux.cfg/default as follows: label 1. NFS Installation (serial port) - x86vbox menu x86vbox_install_serial kernel x86vbox/kernel append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img root=/dev/nfs rw androidboot.hardware=x86vbox INSTALL=1 DEBUG=2 SRC=/x86vbox ROOT=192.168.56.1:/home/sgye/vol1/android-6/out/target/product qemu=1 qemu.gles=0 We will explain the details about kernel parameters in the option append later. The next thing is that we need to create a virtual serial port so that we can connect to. We configure this in the virtual machine settings page as shown in the following screen: We use a host pipe to simulate the virtual serial port. We can set the path as something like /tmp/pxeAndroid_p. The mapping between COMx to /dev/ttySx can be found here: /dev/ttyS0 - COM1 /dev/ttyS1 - COM2 /dev/ttyS2 - COM3 /dev/ttyS3 - COM4 To connect to the host pipe, we can use a tool like minicom in Linux or putty in Windows. If you don't have minicom installed, you can install and configure minicom as shown in the host environment: $ sudo apt-get install minicom To setup minicom, we can use the following command: $ sudo minicom -s After minicom start, select Serial port setup, and set Serial Device as unix#/tmp/pxeAndroid_p. Once this is done, select Save setup as dfl and Exit from minicom as shown in the following screenshot. Now, we can connect to the virtual serial port using minicom: After we made all the changes for the configuration, we can power on the virtual machine and test it. We should be able to see the following boot up screen: We can see from the preceding screenshot that the virtual machine loads the file pxelinux.cfg/default and wait on the boot prompt. We are ready to boot from PXE ROM now. Build AOSP images To build the x86vbox images in this article, we can retrieve the source code using the following commands: $ repo init https://github.com/shugaoye/manifests -b android-7.1.1_r4_ch14_aosp $ repo sync After the source code is ready for use, we can set the environment and build the system as shown here: $ . build/envsetup.sh $ lunch x86vbox-eng $ make -j4 To build initrd.img, we can run the following command. $ make initrd USE_SQUASHFS=0 We can also build an OTA update image which can use recovery to install it. $ cd device/generic/x86vbox $ make dist NFS filesystem Since I am discussing about Android system programming, I will assume you know how to build Android images from AOSP source code. In our setup, we will use the output from the AOSP build to boot the Android system in VirtualBox. They are not able to be used by VirtualBox directly. For example, the system.img can be used by emulator, but not VirtualBox. VirtualBox can use the standard virtual disk images in VDI, VHD, or VMDK formats, but not the raw disk image as system.img. In some open source projects, such as the android-x86 project, the output is an installation image, such as ISO or USB disk image formats. With an installation image, it can be burnt to CD ROM or USB drive. Then, we can boot VirtualBox from CD ROM or USB to install the system just like how we install Windows on our PC. It is quite tedious and not efficient to use this method, when we are debugging a system. As a developer, we want a simple and quick way that we can start the debugging immediately after we build the system. The method that we will use here is to boot the system using NFS filesystem. The key point is that we will treat the output folder of AOSP build as the root filesystem directly so that we can boot the system using it without any additional work. If you are an embedded system developer, you may be used this method in your work already. When we work on the initial debugging phase of an embedded Linux system, we often use NFS filesystem as a root filesystem. With this method, we can avoid to flash the images to the flash storage every time after the build. Preparing the kernel To support NFS boot, we need a Linux kernel with NFS filesystem support. The default Linux kernel for Android doesn't have NFS boot support. In order to boot Android and mount NFS directory as root filesystem, we have to re-compile Linux kernel with the following options enabled: CONFIG_IP_PNP=y CONFIG_IP_PNP_DHCP=y CONFIG_IP_PNP_BOOTP=y CONFIG_IP_PNP_RARP=y CONFIG_USB_USBNET=y CONFIG_USB_NET_SMSC95XX=y CONFIG_USB=y CONFIG_USB_SUPPORT=y CONFIG_USB_ARCH_HAS_EHCI=y CONFIG_NETWORK_FILESYSTEMS=y CONFIG_NFS_FS=y CONFIG_NFS_V3=y CONFIG_NFS_V3_ACL=y CONFIG_ROOT_NFS=y The kernel source code used in this article is a modified version by me for the book Android System Programming. You can find the source code at the following URL: https://github.com/shugaoye/goldfish We can get the source code using the following command: $ git clone https://github.com/shugaoye/goldfish -b android-7.1.1_r4_x86vbox_ch14_r We can use menuconfig to change the kernel configuration or copy a configuration file with NFS support. To configure kernel build using menuconfig, we can use the following commands: $ . build/envsetup.sh $ lunch x86vbox-eng $ make -C kernel O=$OUT/obj/kernel ARCH=x86 menuconfig We can also use the configuration file with NFS enable from my GitHub directly. We can observe the difference between this configuration file and the default kernel configuration file from android-x86 project as shown here: $ diff kernel/arch/x86/configs/android-x86_defconfig ~/src/android-x86_nfs_defconfig 216a217 > # CONFIG_SYSTEM_TRUSTED_KEYRING is not set 1083a1085 > CONFIG_DNS_RESOLVER=y 1836c1838 < CONFIG_VIRTIO_NET=m --- > CONFIG_VIRTIO_NET=y 1959c1961 < CONFIG_E1000=m --- > CONFIG_E1000=y 5816a5819 > # CONFIG_ECRYPT_FS is not set 5854,5856c5857,5859 < CONFIG_NFS_FS=m < CONFIG_NFS_V2=m < CONFIG_NFS_V3=m --- > CONFIG_NFS_FS=y > CONFIG_NFS_V2=y > CONFIG_NFS_V3=y 5858c5861 < # CONFIG_NFS_V4 is not set --- > CONFIG_NFS_V4=y 5859a5863,5872 > CONFIG_NFS_V4_1=y > CONFIG_NFS_V4_2=y > CONFIG_PNFS_FILE_LAYOUT=y > CONFIG_PNFS_BLOCK=y > CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" > # CONFIG_NFS_V4_1_MIGRATION is not set > CONFIG_NFS_V4_SECURITY_LABEL=y > CONFIG_ROOT_NFS=y > # CONFIG_NFS_USE_LEGACY_DNS is not set > CONFIG_NFS_USE_KERNEL_DNS=y 5861,5862c5874,5875 < CONFIG_GRACE_PERIOD=m < CONFIG_LOCKD=m --- > CONFIG_GRACE_PERIOD=y > CONFIG_LOCKD=y 5865c5878,5880 < CONFIG_SUNRPC=m --- > CONFIG_SUNRPC=y > CONFIG_SUNRPC_GSS=y > CONFIG_SUNRPC_BACKCHANNEL=y 5870a5886 > # CONFIG_CIFS_UPCALL is not set 5873a5890 > # CONFIG_CIFS_DFS_UPCALL is not set 6132c6149,6153 < # CONFIG_KEYS is not set --- > CONFIG_KEYS=y > # CONFIG_PERSISTENT_KEYRINGS is not set > # CONFIG_BIG_KEYS is not set > # CONFIG_ENCRYPTED_KEYS is not set > # CONFIG_KEYS_DEBUG_PROC_KEYS is not set 6142a6164 > # CONFIG_INTEGRITY_SIGNATURE is not set 6270a6293 > # CONFIG_ASYMMETRIC_KEY_TYPE is not set 6339a6363 > CONFIG_ASSOCIATIVE_ARRAY=y 6352a6377 > CONFIG_OID_REGISTRY=y We can copy this configuration file and use it to build Linux kernel as shown here: $ cp ~/src/android-x86_nfs_defconfig out/target/product/x86/obj/kernel/.config $ . build/envsetup.sh $ lunch x86vbox-eng $ make -C kernel O=$OUT/obj/kernel ARCH=x86 After the build, we can copy the kernel and ramdisk files to the TFTP root at /var/lib/tftpboot/x86vbox or $HOME/.VirtualBox/TFTP/x86vbox. Setting up NFS server After we prepare the Android kernel, we need to setup a NFS server on our development host so that we can mount to the NFS folders exported by our NFS server. We can check whether the NFS server is already installed or not using the following command: $ dpkg -l | grep nfs If the NFS server is not installed, we can install it using the following command: $ sudo apt-get install nfs-kernel-server Once we have a NFS server ready, we need to export our root filesystem through NFS. We will use the AOSP build output folder as we mentioned previously. We can add the following line to the configuration file /etc/exports. $AOSP/out/target/product/ *(rw,sync,insecure,no_subtree_check,async) After that, we execute the following command to export the folder $AOSP/out/target/product. You need to replace $AOSP to the absolute path in your setup. $ sudo exportfs -a Configuring PXE boot menu We can use PXE boot ROM to support the boot path like a real Android device. As we know that Android device can boot to three different modes, they are the bootloader mode, the recovery mode and the normal start up. With PXE boot ROM, we can easily support the same and more. By configuring the file pxelinux.cfg/default, we can allow x86vbox to boot in different paths. We will configure multiple boot paths here. Booting to NFS installation We can boot the system to an installation mode so that we can borrow the installation script from android-x86 project to install x86vbox images to the virtual hard disk. label 1. NFS Installation (serial port) - x86vbox menu x86vbox_install_serial kernel x86vbox/kernel append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img root=/dev/nfs rw androidboot.hardware=x86vbox INSTALL=1 DEBUG=2 SRC=/x86vbox ROOT=192.168.56.1:$AOSP/out/target/product In this configuration, we use the NFS-capable kernel from TFTP folder, such as $HOME/.VirtualBox/TFTP/x86vbox/kernel. The ramdisk image initrd.img is also stored in the same folder. Both files under TFTP folder can actually be the symbol links to the AOSP output. In this case, we don't have to copy them after the build. We use the following three options to configure the NFS boot. ip=dhcp: Use DHCP to get IP address from DHCP server. The DHCP server can be the built-in DHCP server of VirtualBox or the one that we set up previously. root=/dev/nfs: Use NFS boot. ROOT=10.0.2.2:$AOSP/out/target/product: The root is the AOSP output folder in the development host. If we use the built-in PXE, the IP address 10.0.2.2 is the default host IP address in the NAT network. It could be changed using the VirtualBox configuration. We want to monitor the debug output so we set the console to the virtual serial port that we configured previously as console=ttyS3,115200. We can use a host pipe to connect to it using minicom. We set three kernel parameters using by the android-x86 init script and installation script. INSTALL=1: Tells the init script that we want to install the system. DEBUG=2: This will bring us to the debug console during the boot process. SRC=/x86vbox: This is the directory for the android root filesystem. Finally, the option androidboot.hardware=x86vbox is passed to the Android init process to tell which init script to run. In this case, the device init script init.x86vbox.rc will be executed. In our PXE boot menu, we can add another configuration for the installation without option console=ttyS3,115200. In this case, all debug output will print on the screen which is the default standard output. Booting to hard disk We can have another option as shown to boot the system from hard disk after we install the system using the previous configuration. label 2. x86vbox (ROOT=/dev/sda1, serial port) menu x86vbox_sda1 kernel x86vbox/kernel append ip=dhcp console=ttyS3,115200 initrd=x86vbox/initrd.img androidboot.hardware=x86vbox DEBUG=2 SRC=/android-x86vbox ROOT=/dev/sda1 In the preceding configuration, we use device /dev/sda1 as root and we don't have the option INSTALL=1. With this configuration, the virtual machine will boot to Android system from hard disk /dev/sda1 and the debug output will print to virtual serial port. We can configure another similar configuration which prints the debug output to the screen. Booting to recovery With PXE boot menu, we can configure the system to boot to recovery as well. We can see the following configuration: label 5. x86vbox recovery (ROOT=/dev/sda2) menu x86vbox_recovery kernel x86vbox/kernel append ip=dhcp console=ttyS3,115200 initrd=x86vbox/ramdisk-recovery.img androidboot.hardware=x86vbox DEBUG=2 SRC=/android-x86vbox ROOT=/dev/sda2 Note the difference here is that we use recovery ramdisk instead of initrd.img. Since recovery is a self-contained environment, we can set variable ROOT to another partition as well. We can use recovery to install an OTA update image. With PXE boot, you can explore many different possibilities to play with various boot methods and images. With all this setup, we can boot to PXE boot menu as the following screenshot: We can select an option from the PXE boot menu above to boot to a debug console as shown here: From the preceding debug output, we can see that the virtual machine obtains the IP address 10.0.2.15 from DHCP server 10.0.2.2. The NFS root is found at IP address 192.168.56.1, which is the development host. It uses a different IP address range is because we use two network interfaces in our configuration. We use a NAT network interface which has the IP address range in 10.0.2.x and a host-only network interface which has the IP address range in 192.168.56.x. The IP address 10.0.2.2 is the IP address of the development host in NAT network while IP address 192.168.56.1 is the IP address of the development host in host only network. In this setup, we use the VirtualBox built-in PXE support so both DHCP and TFTP server are on the NAT network interface. If we use a separate PXE server, both DHCP and TFTP server will be on the host only network interface. It is possible to boot the Android system from the directory $OUT/system using NFS filesystem. In that case, we don't need any installation process at all. However, we need to make changes to netd to disable flushing the routing rules. The changes can be done in the following file in the function flushRules: $AOSP/system/netd/server/RouteController.cpp Without this change, the network connection will be reset after the routing rules are flushed. However, we can still use NFS boot to perform the first stage boot or install the system to hard disk. This alternative already makes our development process much efficient. Summary In this article, you learned a debugging method with the combination of PXE boot and NFS root filesystem. This is a common practice in the embedded Linux development world. We try to use the similar setup for the Android system development. As we can see this setup can make the development and debugging process more efficiently. We can use this setup to remove the dependency of bootloader. We can also reduce the time to flash or provision the build images to the device. Even though we did all the exploration in VirtualBox, you can reuse the same method in your hardware board development as well. Resources for Article: Further resources on this subject: Setting up Development Environment for Android Wear Applications [article] Practical How-To Recipes for Android [article] Optimizing Games for Android [article]
Read more
  • 0
  • 1
  • 40188

article-image-top-10-deep-learning-frameworks
Amey Varangaonkar
25 May 2017
9 min read
Save for later

Top 10 deep learning frameworks

Amey Varangaonkar
25 May 2017
9 min read
Deep learning frameworks are powering the artificial intelligence revolution. Without them, it would be almost impossible for data scientists to deliver the level of sophistication in their deep learning algorithms that advances in computing and processing power have made possible. Put simply, deep learning frameworks make it easier to build deep learning algorithms of considerable complexity. This follows a wider trend that you can see in other fields of programming and software engineering; open source communities are continually are developing new tools that simplify difficult tasks and minimize arduous ones. The deep learning framework you choose to use is ultimately down to what you're trying to do and how you work already. But to get you started here is a list of 10 of the best and most popular deep learning frameworks being used today. What are the best deep learning frameworks? Tensorflow One of the most popular Deep Learning libraries out there, Tensorflow, was developed by the Google Brain team and open-sourced in 2015. Positioned as a ‘second-generation machine learning system’, Tensorflow is a Python-based library capable of running on multiple CPUs and GPUs. It is available on all platforms, desktop, and mobile. It also has support for other languages such as C++ and R and can be used directly to create deep learning models, or by using wrapper libraries (for e.g. Keras) on top of it. In November 2017, Tensorflow announced a developer preview for Tensorflow Lite, a lightweight machine learning solution for mobile and embedded devices. The machine learning paradigm is continuously evolving - and the focus is now slowly shifting towards developing machine learning models that run on mobile and portable devices in order to make the applications smarter and more intelligent. Learn how to build a neural network with TensorFlow. If you're just starting out with deep learning, TensorFlow is THE go-to framework. It’s Python-based, backed by Google, has a very good documentation, and there are tons of tutorials and videos available on the internet to guide you. You can check out Packt’s TensorFlow catalog here. Keras Although TensorFlow is a very good deep learning library, creating models using only Tensorflow can be a challenge, as it is a pretty low-level library and can be quite complex to use for a beginner. To tackle this challenge, Keras was built as a simplified interface for building efficient neural networks in just a few lines of code and it can be configured to work on top of TensorFlow. Written in Python, Keras is very lightweight, easy to use, and pretty straightforward to learn. Because of these reasons, Tensorflow has incorporated Keras as part of its core API. Despite being a relatively new library, Keras has a very good documentation in place. If you want to know more about how Keras solves your deep learning problems, this interview by our best-selling author Sujit Pal should help you. Read now: Why you should use Keras for deep learning [box type="success" align="" class="" width=""]If you have some knowledge of Python programming and want to get started with deep learning, this is one library you definitely want to check out![/box] Caffe Built with expression, speed, and modularity in mind, Caffe is one of the first deep learning libraries developed mainly by Berkeley Vision and Learning Center (BVLC). It is a C++ library which also has a Python interface and finds its primary application in modeling Convolutional Neural Networks. One of the major benefits of using this library is that you can get a number of pre-trained networks directly from the Caffe Model Zoo, available for immediate use. If you’re interested in modeling CNNs or solve your image processing problems, you might want to consider this library. Following the footsteps of Caffe, Facebook also recently open-sourced Caffe2, a new light-weight, modular deep learning framework which offers greater flexibility for building high-performance deep learning models. Torch Torch is a Lua-based deep learning framework and has been used and developed by big players such as Facebook, Twitter and Google. It makes use of the C/C++ libraries as well as CUDA for GPU processing.  Torch was built with an aim to achieve maximum flexibility and make the process of building your models extremely simple. More recently, the Python implementation of Torch, called PyTorch, has found popularity and is gaining rapid adoption. PyTorch PyTorch is a Python package for building deep neural networks and performing complex tensor computations. While Torch uses Lua, PyTorch leverages the rising popularity of Python, to allow anyone with some basic Python programming language to get started with deep learning. PyTorch improves upon Torch’s architectural style and does not have any support for containers - which makes the entire deep modeling process easier and transparent to you. Still wondering how PyTorch and Torch are different from each other? Make sure you check out this interesting post on Quora. Deeplearning4j DeepLearning4j (or DL4J) is a popular deep learning framework developed in Java and supports other JVM languages as well. It is very slick and is very widely used as a commercial, industry-focused distributed deep learning platform. The advantage of using DL4j is that you can bring together the power of the whole Java ecosystem to perform efficient deep learning, as it can be implemented on top of the popular Big Data tools such as Apache Hadoop and Apache Spark. [box type="success" align="" class="" width=""]If Java is your programming language of choice, then you should definitely check out this framework. It is clean, enterprise-ready, and highly effective. If you’re planning to deploy your deep learning models to production, this tool can certainly be of great worth![/box] MXNet MXNet is one of the most languages-supported deep learning frameworks, with support for languages such as R, Python, C++ and Julia. This is helpful because if you know any of these languages, you won’t need to step out of your comfort zone at all, to train your deep learning models. Its backend is written in C++ and cuda, and is able to manage its own memory like Theano. MXNet is also popular because it scales very well and is able to work with multiple GPUs and computers, which makes it very useful for the enterprises. This is also one of the reasons why Amazon made MXNet its reference library for Deep Learning too. In November, AWS announced the availability of ONNX-MXNet, which is an open source Python package to import ONNX (Open Neural Network Exchange) deep learning models into Apache MXNet. Read why MXNet is a versatile deep learning framework here. Microsoft Cognitive Toolkit Microsoft Cognitive Toolkit, previously known by its acronym CNTK, is an open-source deep learning toolkit to train deep learning models. It is highly optimized and has support for languages such as Python and C++. Known for its efficient resource utilization, you can easily implement efficient Reinforcement Learning models or Generative Adversarial Networks (GANs) using the Cognitive Toolkit. It is designed to achieve high scalability and performance and is known to provide high-performance gains when compared to other toolkits like Theano and Tensorflow when running on multiple machines. Here is a fun comparison of TensorFlow versus CNTK, if you would like to know more. deeplearn.js Gone are the days when you required serious hardware to run your complex machine learning models. With deeplearn.js, you can now train neural network models right on your browser! Originally developed by the Google Brain team, deeplearn.js is an open-source, JavaScript-based deep learning library which runs on both WebGL 1.0 and WebGL 2.0. deeplearn.js is being used today for a variety of purposes - from education and research to training high-performance deep learning models. You can also run your pre-trained models on the browser using this library. BigDL BigDL is distributed deep learning library for Apache Spark and is designed to scale very well. With the help of BigDL, you can run your deep learning applications directly on Spark or Hadoop clusters, by writing them as Spark programs. It has a rich deep learning support and uses Intel’s Math Kernel Library (MKL) to ensure high performance. Using BigDL, you can also load your pre-trained Torch or Caffe models into Spark. If you want to add deep learning functionalities to a massive set of data stored on your cluster, this is a very good library to use. [box type="shadow" align="" class="" width=""]Editor's Note: We have removed Theano and Lasagne from the original list due to the Theano retirement announcement. RIP Theano! Before Tensorflow, Caffe or PyTorch came to be, Theano was the most widely used library for deep learning. While it was a low-level library supporting CPU as well as GPU computations, you could wrap it with libraries like Keras to simplify the deep learning process. With the release of version 1.0, it was announced that the future development and support for Theano would be stopped. There would be minimal maintenance to keep it working for the next one year, after which even the support activities on the library would be suspended completely. “Supporting Theano is no longer the best way we can enable the emergence and application of novel research ideas”, said Prof. Yoshua Bengio, one of the main developers of Theano. Thank you Theano, you will be missed! Goodbye Lasagne Lasagne is a high-level deep learning library that runs on top of Theano.  It has been around for quite some time now and was developed with the aim of abstracting the complexities of Theano, and provide a more friendly interface to the users to build and train neural networks. It requires Python and finds many similarities to Keras, which we just saw above. However, if we are to find differences between the two, Keras is faster and has a better documentation in place.[/box] There are many other deep learning libraries and frameworks available for use today – DSSTNE, Apache Singa, Veles are just a few worth an honorable mention. Which deep learning frameworks will best suit your needs? Ultimately, it depends on a number of factors. If you want to get started with deep learning, your safest bet would be to use a Python-based framework like Tensorflow, which are quite popular. For seasoned professionals, the efficiency of the trained model, ease of use, speed and resource utilization are all important considerations for choosing the best deep learning framework.
Read more
  • 0
  • 0
  • 63122
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-introduction-titanic-datasets
Packt
09 May 2017
11 min read
Save for later

Introduction to Titanic Datasets

Packt
09 May 2017
11 min read
In this article by Alexis Perrier, author of the book Effective Amazon Machine Learning says artificial intelligence and big data have become a ubiquitous part of our everyday lives; cloud-based machine learning services are part of a rising billion-dollar industry. Among the several such services currently available on the market, Amazon Machine Learning stands out for its simplicity. Amazon Machine Learning was launched in April 2015 with a clear goal of lowering the barrier to predictive analytics by offering a service accessible to companies without the need for highly skilled technical resources. (For more resources related to this topic, see here.) Working with datasets You cannot do predictive analytics without a dataset. Although we are surrounded by data, finding datasets that are adapted to predictive analytics is not always straightforward. In this section, we present some resources that are freely available. The Titanic datasetis a classic introductory datasets for predictive analytics. Finding open datasets There is a multitude of dataset repositories available online, from local to global public institutions to non-profit and data-focused start-ups. Here’s a small list of open dataset resources that are well suited forpredictive analytics. This, by far, is not an exhaustive list. This thread on Quora points to many other interesting data sources:https://www.quora.com/Where-can-I-find-large-datasets-open-to-the-public.You can also ask for specific datasets on Reddit at https://www.reddit.com/r/datasets/. The UCI Machine Learning Repository is a collection of datasets maintained by UC Irvine since 1987, hosting over 300 datasets related to classification, clustering, regression, and other ML tasks Mldata.org from the University of Berlinor the Stanford Large Network Dataset Collection and other major universities alsooffer great collections of open datasets Kdnuggets.com has an extensive list of open datasets at http://www.kdnuggets.com/datasets Data.gov and other US government agencies;data.UN.org and other UN agencies AWS offers open datasets via partners at https://aws.amazon.com/government-education/open-data/. The following startups are data centered and give open access to rich data repositories: Quandl and quantopian for financial datasets Datahub.io, Enigma.com, and Data.world are dataset-sharing sites Datamarket.com is great for time series datasets Kaggle.com, the data science competition website, hosts over 100 very interesting datasets AWS public datasets:AWS hosts a variety of public datasets,such as the Million Song Dataset, the mapping of the Human Genome, the US Census data as well as many others in Astrology, Biology, Math, Economics, and so on. These datasets are mostly available via EBS snapshots although some are directly accessible on S3. The datasets are large, from a few gigabytes to several terabytes, and are not meant to be downloaded on your local machine; they are only to be accessible via an EC2 instance (take a look at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-public-data-sets.htmlfor further details).AWS public datasets are accessible at https://aws.amazon.com/public-datasets/. Introducing the Titanic dataset We will use the classic Titanic dataset. The dataconsists of demographic and traveling information for1,309 of the Titanic passengers, and the goal isto predict the survival of these passengers. The full Titanic dataset is available from the Department of Biostatistics at the Vanderbilt University School of Medicine (http://biostat.mc.vanderbilt.edu/wiki/pub/Main/DataSets/titanic3.csv)in several formats. The Encyclopedia Titanica website (https://www.encyclopedia-titanica.org/) is the website of reference regarding the Titanic. It contains all the facts, history, and data surrounding the Titanic, including a full list of passengers and crew members. The Titanic datasetis also the subject of the introductory competition on Kaggle.com (https://www.kaggle.com/c/titanic, requires opening an account with Kaggle). You can also find a csv version in GitHub repository at https://github.com/alexperrier/packt-aml/blob/master/ch4. The Titanic data containsa mix of textual, Boolean, continuous, and categorical variables. It exhibits interesting characteristics such as missing values, outliers, and text variables ripe for text mining--a rich database that will allow us to demonstrate data transformations. Here’s a brief summary of the 14attributes: pclass: Passenger class (1 = 1st; 2 = 2nd; 3 = 3rd) survival: A Boolean indicating whether the passenger survived or not (0 = No; 1 = Yes); this is our target name: A field rich in information as it contains title and family names sex: male/female age: Age, asignificant portion of values aremissing sibsp: Number of siblings/spouses aboard parch: Number of parents/children aboard ticket: Ticket number. fare: Passenger fare (British Pound). cabin: Doesthe location of the cabin influence chances of survival? embarked: Port of embarkation (C = Cherbourg; Q = Queenstown; S = Southampton) boat: Lifeboat, many missing values body: Body Identification Number home.dest: Home/destination Take a look at http://campus.lakeforest.edu/frank/FILES/MLFfiles/Bio150/Titanic/TitanicMETA.pdf for more details on these variables. We have 1,309 records and 14 attributes, three of which we will discard. The home.dest attribute hastoo few existing values, the boat attribute is only present for passengers who have survived, and thebody attributeis only for passengers who have not survived. We will discard these three columnslater on while using the data schema. Preparing the data Now that we have the initial raw dataset, we are going to shuffle it, split it into a training and a held-out subset, and load it to an S3 bucket. Splitting the data In order to build and select the best model, we need to split the dataset into three parts: training, validation, and test, with the usual ratios being 60%, 20%, and 20%. The training and validation sets are used to build several models and select the best one while the test or held-out set, is used for the final performance evaluation on previously unseen data. Since Amazon ML does the job of splitting the dataset used for model training and model evaluation into a training and a validation subsets, we only need to split our initial dataset into two parts: the global training/evaluation subset (80%) for model building and selection, and the held-out subset (20%) for predictions and final model performance evaluation. Shuffle before you split:If you download the original data from the Vanderbilt University website,you will notice that it is ordered by pclass, the class of the passenger and by alphabetical order of the name column. The first 323 rows correspond to the 1st class followed by 2nd (277) and 3rd (709) class passengers. It is important to shuffle the data before you split it so that all the different variables have have similar distributions in each training and held-out subsets. You can shuffle the data directly in the spreadsheet by creating a new column, generating a random number for each row and then ordering by that column. On GitHub: You will find an already shuffledtitanic.csv file at https://github.com/alexperrier/packt-aml/blob/master/ch4/titanic.csv. In addition to shuffling the data, we have removed punctuation in the name column: commas, quotes, and parenthesis, which can add confusion when parsing a csv file. We end up with two files:titanic_train.csv with 1047 rows and titanic_heldout.csv with 263rows. These files are also available in the GitHub repo (https://github.com/alexperrier/packt-aml/blob/master/ch4). The next step is to upload these files on S3 so that Amazon ML can access them. Loading data on S3 AWS S3 is one of the main AWS services dedicated to hosting files and managing their access. Files in S3 can be public and open to the internet or have access restricted to specific users, roles, or services.S3 is also used extensively by AWS for operations such as storing log files or results (predictions, scripts, queries, and so on). Files in S3 are organized around the notion of buckets. Buckets are placeholders with unique names similar to domain names for websites. A file in S3 will have a unique locator URI: s3://bucket_name/{path_of_folders}/filename. The bucket name is unique across S3. In this section, we will create a bucket for our data, upload the titanic training file, and open its access to Amazon ML. Go to https://console.aws.amazon.com/s3/home, and open an S3 account if you don’t have one yet. S3 pricing:S3 charges for the total volume of files you host and the volume of file transfers depends on the region where the files are hosted. At time of writing, for less than 1TB, AWS S3 charges $0.03/GB per month in the US east region. All S3 prices are available at https://aws.amazon.com/s3/pricing/. See also http://calculator.s3.amazonaws.com/index.htmlfor the AWS cost calculator. Creating a bucket Once you have created your S3 account, the next step is to create a bucket for your files.Click on the Create bucket button: Choose a name and a region, since bucket names are unique across S3, you must choose a name for your bucket that has not been already taken. We chose the name aml.packt for our bucket, and we will use this bucket throughout. Regarding the region, you should always select a region that is the closest to the person or application accessing the files in order to reduce latency and prices. Set Versioning, Logging, and Tags, versioning will keep a copy of every version of your files, which prevents from accidental deletions. Since versioning and logging induce extra costs, we chose to disable them. Set permissions. Review and save. Loading the data To upload the data, simply click on the upload button and select the titanic_train.csv file we created earlier on. You should, at this point, have the training dataset uploaded to your AWS S3 bucket. We added a/data folder in our aml.packt bucket to compartmentalize our objects. It will be useful later on when the bucket will also contain folders created by S3. At this point, only the owner of the bucket (you) is able to access and modify its contents. We need to grant the Amazon ML service permissions to read the data and add other files to the bucket. When creating the Amazon ML datasource, we will be prompted to grant these permissions inthe Amazon ML console. We can also modify the bucket’s policy upfront. Granting permissions We need to edit the policy of the aml.packt bucket. To do so, we have to perform the following steps: Click into your bucket. Select the Permissions tab. In the drop down, select Bucket Policy as shown in the following screenshot. This will open an editor: Paste in the following JSON. Make sure to replace {YOUR_BUCKET_NAME} with the name of your bucket and save: { “Version”: “2012-10-17”, “Statement”: [ { “Sid”: “AmazonML_s3:ListBucket”, “Effect”: “Allow”, “Principal”: { “Service”: “machinelearning.amazonaws.com” }, “Action”: “s3:ListBucket”, “Resource”: “arn:aws:s3:::{YOUR_BUCKET_NAME}”, “Condition”: { “StringLike”: { “s3:prefix”: “*” } } }, { “Sid”: “AmazonML_s3:GetObject”, “Effect”: “Allow”, “Principal”: { “Service”: “machinelearning.amazonaws.com” }, “Action”: “s3:GetObject”, “Resource”: “arn:aws:s3:::{YOUR_BUCKET_NAME}/*” }, { “Sid”: “AmazonML_s3:PutObject”, “Effect”: “Allow”, “Principal”: { “Service”: “machinelearning.amazonaws.com” }, “Action”: “s3:PutObject”, “Resource”: “arn:aws:s3:::{YOUR_BUCKET_NAME}/*” } ] } Further details on this policy are available at http://docs.aws.amazon.com/machine-learning/latest/dg/granting-amazon-ml-permissions-to-read-your-data-from-amazon-s3.html. Once again, this step is optional since Amazon ML will prompt you for access to the bucket when you create the datasource. Formatting the data Amazon ML works on comma separated values files (.csv)--a very simple format where each rowis an observation and each column is a variable or attribute. There are, however, a few conditionsthat shouldbe met: The data must be encoded in plain text using a character set, such asASCII, Unicode, or EBCDIC All values must be separated by commas; if a value contains a comma, it should be enclosed by double quotes Each observation (row) must be smaller than 100k There are also conditions regarding end of line characters that separate rows. Special care must be taken when using Excel on OS X (Mac) as explained on this page: http://docs.aws.amazon.com/machine-learning/latest/dg/understanding-the-data-format-for-amazon-ml.html What about other data file formats? Unfortunately, Amazon ML datasource are only compatible with csv files and Redshift databases and does not accept formats such as JSON, TSV, or XML. However, other services such as Athena, a serverless database service, do accept a wider range of formats. Summary In this article we learnt about how to use and work around with datasets using Amazon web services and Titanic datasets. We also learnt how prepare data and Amazon S3 services.  Resources for Article: Further resources on this subject: Processing Massive Datasets with Parallel Streams – the MapReduce Model [article] Processing Next-generation Sequencing Datasets Using Python [article] Combining Vector and Raster Datasets [article]
Read more
  • 0
  • 1
  • 17111

article-image-active-directory-domain-services-2016
Packt
09 May 2017
23 min read
Save for later

Active Directory Domain Services 2016

Packt
09 May 2017
23 min read
In this article, by Dishan Francis, the author of the book Mastering Active Directory, we will see AD DS features, privileged access management, time based group memberships. Microsoft, released Active Directory domain services 2016 at a very interesting time in technology. Today identity infrastructure requirements for enterprise are challenging, most of the companies uses cloud services for their operations (Software as a Service—SaaS) and lots moved infrastructure workloads to public clouds. (For more resources related to this topic, see here.) AD DS 2016 features Active Directory domain service (AD DS) improvements are bind with its forest and domain functional levels. Upgrading operating system or adding domain controllers which runs Windows Server 2016 to existing AD infrastructure not going to upgrade forest and domain functional levels. In order to use or test these new AD DS 2016 features you need to have forest and domain function levels set to Windows Server 2016. The minimum forest and domain functional levels you can run on your identity infrastructure depend on the lowest domain controller version running. For example, if you have Windows Server 2008 domain controller in your infrastructure, even though you add Windows Server 2016 domain controller, the domain and forest functional level need to maintain as Windows Server 2008 until last Windows Server 2008 demote from the infrastructure. Privileged access management Privileged access management (PAM) is one of the best topics which is discussed on presentations, tech shows, IT forums, IT groups, blogs and meetings for last few years (after 2014) around identity management. It has become a trending topic especially after the Windows Server 2016 previews released. For last year, I was travelling to countries, cities and had involved with many presentations, discussions about PAM.  First of all, this is not a feature that you can enable with few clicks. It is a combination of many technologies and methodologies which came together and make a workflow or in other words way of living for administrators. AD DS 2016 includes features and capabilities that support PAM in infrastructure but it is not the only thing. This is one of the greatest challenge I see about this new way of thinking and new way of working. Replacing a product is easy but changing a process is more complicated and challenging.   I started my career with one of the largest north American hosting company around 2003. I was a system administrator that time and one of my tasks was to identify hacking attempts and prevent workloads getting compromised. In order to do that I had to review lot of logs on different systems. But around that time most of the attacks from individual or groups were to put names on websites and prove that they can hack websites. Average hacking attempts per server was around 20 to 50 per day. Some collocation customers were even running their websites, workloads without any protection (even though not recommended). But as the time goes year by year number of attempts were dramatically increased and we start to talk about hundreds of thousands attempts per day. The following graph is taken from latest Symantec Internet Security Threat Report (2016) and it confirms number of web-based attacks increased by more than 117% from year 2014.  Web attacks blocked per month (Source - Symantec Internet Security Threat Report (2016)) It has not only changed the numbers, it also changed the purpose of attacks. As I said in earlier days it was script kiddies who were after fame. Then later as users started to use more and more online services, purpose of attacks changed to financial values. Attackers started to focus on websites which stores credit card information. For last 10 years, I had to change my credit card 4 times as my credit card information were exposed along with the websites I had used it with. These type of attacks are still happening in the industry.  When considering the types of threats after the year 2012, most of the things changed. Instead of fame or financial, attackers started to target identities. In earlier days, the data about a person were in different formats. For example, when I used to walk into my medical center 15 years ago, before seeing the doctor, administration staff had to go and find the file containing my name. They had number of racks filled with files and papers which included patient records, treatment history, test reports, and so on. But now things have changed, when I walk in, no one in administration need to worry about the file. Doctor can see all my records from his computer screen with few clicks. So, the data is being transformed into the digital format. More and more data about people is transforming into digital formats. In that health system, I become an identity and my identity is attached to the data and also to a certain privileges. Think about your bank, online banking system. You got your own username and password to type in, when you log in to the portal. So, you have your own identity in the bank system. Once you log in, you can access all your accounts, transfer money, make payments. Bank has granted some privileges to your identity. With your privileges, you cannot look into your neighbor’s bank account. But your bank manager can view your account and your neighbor’s account too. That means the privileges attached to the bank manager’s identity is different. Amount of data which can be retrieved from systems are dependent on the identity privileges. Not only that, some of these identities are integrated with different systems. Industries use different systems related to their operations. It can be email system, CMS or billing system. Each of these systems hold data. To make operations smooth these systems are integrated with one identity infrastructure and provides single sign-on experience instead of using different identities for each and every application. It is making identities more and more powerful within any system. For an attacker, what is more worth? To focus on one system or target on identity which is attached to data and privileges to many different systems? Which one can make more damage? If the identity which is the target, has more privileged access to the systems, its a total disaster. Is it all about usernames, passwords or admin accounts? No it's not, identities can make more damage than that. Usernames and passwords are just making it easy. Just think about the recent world famous cyber-attacks. Back in July 2015, a group called The Impact Team threatened to expose user account information of Ashley Madison dating site, if its parent company Avid Life Media didn't shut down the Ashley Madison and Established Men websites completely. For example, Ashley Madison website hack, is it that the financial value made it more dangerous? It was the identities which made damages to people’s lives. It was just enough to expose the names and make someones life to be humiliated. It ruined families and children lost their parents love and care. It proves it’s not only about permissions attached to an identity, individual identities itself are more important in modern big data phenomenon. It’s only been few months from the USA presidential election and by now we can see how much news it can make with a single tweet. It wasn’t needed to have special privileges to do a tweet, it was the identity which made that tweet important. In other hand if that twitter account got hacked and someone tweeted something fake on behalf of the actual person who owns it, what kind of damage it can make to whole world? In order to do that, does it need to hack the Jack Dorsey’s account? Value  of individual identity is more powerful than twitter CEO. According to following latest reports, it shows that majority of information exposed by identity attacks, are people names, addresses, medical reports, and government identity numbers. Source - Symantec Internet Security Threat Report (2016) The attacks targeted on identities are rising day by day. The following graph shows the number of identities been exposed, compared to the number of incidents. Source - Symantec Internet Security Threat Report (2016) In December 2015, there were only 11 incidents and 195 million identities were exposed. It shows how much damage these types of attacks can make.  Each and every time this kind of attack happens, most common answers from engineers are “Those attacks were so sophisticated”, “It was too complex to identify”, “They were so clever”, “It was zero-day attack”. Is that really true?  Zero-days attacks are based on unknown system bugs, errors to vendors. Latest reports show the average time of explores are less than 7 days and 1 day to release to patch. Source - Symantec Internet Security Threat Report (2016) Microsoft Security Intelligence Report Volume 21 | January through June, 2016 report contains the following figure which explains the complexity of the vulnerabilities. It clearly shows the majority of the vulnerabilities are less complex to exploit. High complexity vulnerabilities are still less than 5% from total vulnerability disclosures. It proves the attackers are still after low hanging fruits. Source: Microsoft Security Intelligence Report Volume 21 | January through June, 2016 Microsoft Active Directory is the leader in identity infrastructure solution provider. With all this constant news about identity breaches, Microsoft Active Directory name also appears. Then people start to question why Microsoft can’t fix it? But if you analyse these problems, it’s obvious that just providing technology rich product is not enough to solve these issues. With each and every new server operating system version, Microsoft releases new Active Directory version. Every time it contains new features to improve the identity infrastructure security. But when I go for the Active Directory released project, I see a majority of engineers not even following the security best practices defined by 10 years’ older Active Directory version. Think about a car race, its categories are usually based on the engine power. It can be 1800cc, 2000cc or more. In the race, most of the time it's the same models and same manufactured cars. If it's same manufacture, and if it's same engine capacity how one can win and the other lose? It’s the car tuning and the driving skills which decide a winner and loser. If Active Directory domain service 2016 can fix all the identity threats that’s really good but giving a product or technology doesn’t seem to be work so far. That’s why we need to change the way we think towards identity infrastructure security. We should not forget we are fighting against human adversaries. The tactics, methods, approaches they use, are changing every day. The products we use, do not have such frequent updates but we can change their ability to execute an attack on infrastructure by understanding fundamentals and use the products, technologies, workflows to prevent it. Before we move into identity theft prevention mechanism let’s look into typical identity infrastructure attack. Microsoft Tiered administration model is based on three tiers. All these identity attacks are starting with gaining some kind of access to the identity infrastructure and then move laterally until they have keys to the kingdom which is domain admin or enterprise administrator credentials. Then they have full ownership of entire identity infrastructure. As the preceding diagram shows that the first step on identity attack, is to get some kind of access to the system. They do not target domain admin or enterprise admin account first. Getting access to a typical user account is much easier than domain admin account. All they need is some kind of beach head. For this, still the most common attack technique is to send out phishing email. It’s typical that someone will still fall for that and click on it. Now they have some sort of access to your identity infrastructure and next step is to start moving laterally to gain more privileges. How many of you completely eliminated local administrator accounts in your infrastructure? I’m sure the answer will be almost none. Sometimes, users are asked for software installations, system level modifications frequently in their systems and most of the time engineers are ending up assigning local administrator privileges. If the compromised account used to be local administrator its becomes extremely easy to move to the next level. If not, they will make systems to misbehave. Then who will come to the rescue? It's the super powered IT help-desk peoples. In lots of organizations, IT help-desk engineers are domain administrators. If not at least local administrators to the systems. So, once they receive the call about a misbehaving computer, they RDP or login locally using the privileged account. If you are using RDP, it always sends your credentials via clear text. If the attacker is running any password harvesting tool it's extremely easy to capture the credentials. You may think if account (which is compromised) is a typical user account how it can execute such programs. But Windows operating systems are not preventing users from running any application on its user context. It will not allow to change any system level settings but it will still allow to run scripts or user level executable. Once they gain access to some identity in organization, the next level of privileges to own will be Tier 1. This is where the application administrators, data administrators, SaaS application administrators accounts live. In today's infrastructures, we have too many administrators. Primarily we have domain admins, enterprise administrators, then we have local administrators. Different applications running on the infrastructure have its own administrators such as exchange administrators, SQL administrators, and SharePoint administrators. The other third-party applications such as CMS, billing portal may have its own administrators. If you are using cloud services, SaaS applications, it has another set of administrators. Are we really aware of activities happening on these accounts? Mostly engineers are only worrying about protecting domain admin accounts, but at the same time forgetting about the other kinds of administrators in the infrastructure. Some of these administrator roles can make more damage than domain admin to a business. These application and services are decentralizing the management in the organization. In order to move latterly with privileges, these attackers only need to log into a machine or server where these administrators used to log in.  Local Security Authority Subsystem Service(LSASS) stores credentials in its memory for active Windows sessions. This prevents users from entering credentials for each and every service they access. This also stores Kerberos tickets. This allows attackers to perform a pass of the hash attack and retrieve locally stored credentials. Decentralized management of admin accounts make this process easier. There are features, security best practices which can be used to prevent the pass of the hash attacks in identity infrastructure.  Another problem with these types of accounts is once it becomes service admin accounts, eventually its becomes domain admin or enterprise administrator accounts. I have seen engineers created service accounts and when they can’t figure out the exact permission required for the program, as an easy fix it will add to the domain admin group. It’s not only the infrastructure attack that can expose such credentials. Service admins are attached to the application too, compromise on application can also expose the identities. In such scenario, it will be easier for attackers to gain keys to the kingdom.  Tier 0 is where the domain admin, enterprise admins operates. This is what the ultimate goal for identity infrastructure attack, once they obtain access to Tier 0, it means they own your entire identity infrastructure. Latest reports show once there is initial breach, it only takes less than 48 hours to gain Tier 0 privileges. According to the reports, once they gain access it will take up to 7-8 months minimum to identify the breach. Because once they have highest privileges they can make backdoors, clean up logs and hide forever if needed. Systems we use, always treat administrators as trustworthy people. It’s no longer valid statement for modern world. How many times you check systems logs to see what your domain admins are doing? Even though engineers look for the logs for other users, majority rarely check about domain admin accounts. The same thing applies for internal security breach too, as I said most people are good but you never know. Most of world famous identity attacks have proved that already. When I have discussion with engineers and customers about identity infrastructure security, following are the common comments I hear, "We have too many administrator accounts" "We do not know how many administrator account we got" "We got fast changing IT teams, so it’s hard to manage permissions" "We do not have visibility over administrator accounts activities" "If there is identity infrastructure breach or attempt, how do we identify?" Answer for all of these is PAM. As I said in the beginning, this is not one product. It’s a workflow and a new way of working. Main components for this process is listed as follows: Apply pass-the-hash prevention features to existing identity infrastructure. Install Microsoft Advanced Threat Analytics to monitor the domain controller traffic to identify potential real-time identity infrastructure threats. Install and configure Microsoft Identity Manager 2016—this product is allowing to manage privilege access of existing Active Directory forest by providing task-based time limited privilege access.  What is it to do with AD DS 2016? AD DS 2016 is now allowing time based group membership which makes this whole process possible. Users will add to the groups with TTL value and once its expires, the user will be removed from the group automatically. For example, let’s assume your CRM application has administrator rights assign to CRM Admin security group. The users in this group only log into the system once a month to do some maintenance. But the admin rights for the members in that group remain untouched for 29 days—24x7. So, it gives enough opportunity for attackers to try and gain access to the privileged accounts during that time. But if it’s admin rights can be limited at least for the day it needed isn’t it more useful? Then we know majority of days in month, CRM application do not have risk of been compromised by an account in CRM Admin group. What is the logic behind PAM? PAM product is built, based on Just-In-Time (JIT) administration concept. Back in 2014, Microsoft release PowerShell tool kit which allows Just-Enough-Administration. Let’s assume you are running a web server in your infrastructure. As part of the operation, every month you need to collect some logs to make a report. You already setup a PowerShell script for it. Someone in your team need to log into the system and need to run it. In order to do that, it requires administration privileges. Using JEA, it is possible to assign required permissions for the user to run only that particular program. In that way, user doesn't need to be added to the domain admin group. User will not be allowed to run any other program with assigned permission and it will not apply for another computer either. JIT administration is bound with time. Users will have required privileges only when they need it. Users will not hold privileged access rights all the time. PAM operations can be divide in to 4 major steps: Source - https://docs.microsoft.com/en-gb/microsoft-identity-manager/pam/privileged-identity-management-for-active-directory-domain-services Prepare: First step is to identify the privileged access groups in your exciting Active Directory forest and start to remove users from those. You may also need to do certain changes in your application infrastructure to support this setup. For example, if you assign privileged access to user accounts instead of security groups (in applications or services) it will need to change. Then next step is to setup equivalent groups in bastion forest without any members. When setup MIM, it will use a bastion forest to manage privileged access in existing Active Directory forest. This is a special forest and it cannot use for other infrastructure operations. This forest running with minimum of Windows Server 2012 R2 Active Directory forest functional level. When identity infrastructure compromised and attackers gain access to Tier 0, they can hide their activities for months or years. How we can be sure our existing identity infrastructure is not compromised already? if we implement this to same forest it will not achieve its core targets. Also, domain upgrades are painful it need time and budget. But because of the bastion forest, this solution can be applied to your existing identity infrastructure with minimum changes.  Protect: Next step is in the list to setup a workflow for authentications and authorization. Define how user can request privileges access when they are required. It can be via MIM portal or existing support portal (with integrated MIM REST API). It is possible to setup system to use Multi-Factor authentications (MFA) during this request process to prevent any unauthorized activity. Also, its important to define how the requests will be handled. It can be automatic approval or manual approval process. Operate: Once privilege access request approved, the user account will be added to the security group in bastion forest. The group itself have a SID value. In both forests, the group will have exact same SID value. Therefore the application or service will not see a difference between two groups in two different forest. Once the permission is granted it is only valid for the time defined by the authorization policy. Once it reaches the time limit, the user account will be removed from the security group automatically. Monitor: PAM provides visibility over the privilege access requests. Each and every request, events will be recorded and it is possible to review and also generate reports for audit purposes. It helps to fine tune the process and also to identify potential threats.  Let’s see how it’s really works: REBELADMIN CORP. uses a CRM system for its operations. The application got administrator role and REBELADMIN/CRMAdmins security group assigned to it. Any member of that group will have administrator privileges to the application. Recently PAM been introduced to the REBELADMIN CORP. As an engineer, I have identified REBELADMIN/CRMAdmins as privileged group and going to protect it using PAM. The first step is to remove the members of the REBELADMIN/CRMAdmins group. After that I have setup same group in the bastion forest. Not only the name is same, but also both the groups got the same SID value 1984.  User Dennis used to be a member of the REBELADMIN/CRMAdmins group and was running monthly report. At the end of the month, he tried to run it and now figured he do not have the required permissions. Next step for him is to request the required permission via MIM Portal. According to the policies, as part of the request, system wants Dennis to use MFA. Once Dennis verifies the PIN number the request logs in the portal. As administrator, I received the alert about the request and I log into system to review the request. It's legitimate request and I approve his access to the system for 8 hours. Then the system automatically added the user account for Dennis into BASTION/CRMAdmins group. This group have the same SID value as the production group. Therefore, the member of BASTION/CRMAdmins group will be treated as administrator by CRM application. This group membership contains TTL value too. After it passes 8 hours from approval, Dennis’s account will be automatically removed from BASTION/CRMAdmins group. In this process, we didn’t add any member to the production security group which is REBELADMIN/CRMAdmins. So, production forest stay untouched and protected. In here the most important thing we need to understand is the legacy approach for identity protection is no longer valid. We are against human adversaries. Identity is our new perimeter in infrastructure and to protect it we need to understand how adversaries doing it and stay step ahead. The new PAM with AD DS 2016 is new approach to the right direction.  Time based group memberships Time based group membership is part of that boarder topic. This allows administrators to assign temporarily group membership which is expressed by Time-To-Live (TTL) value. This value will add to the Kerberos ticket. This is also called as Expiring-Link feature. When a user is assigned to a temporarily group membership, his login Kerberos ticket granting ticket (TGT) life time will be equal to lowest TTL value he has. For example, let’s assume you granted temporarily group membership to user A to be a member of domain admin group. It is only valid for 60 minutes. But user logged in only after 50 minutes from original assign and only have 10 minutes left to be a member of domain admin group. Based on that domain controller will issue TGT only valid for 10 minutes for user A.  This feature is not enabled by default. The reason for that is, to use this feature the forest function level must be Windows Server 2016. Also, once this feature is enabled, it cannot be disabled.  Let’s see how it works in real world: I have Windows domain controller installed and it is running with Windows Server 2016 forest functional level. It can be verified using the following PowerShell command: Get-ADForest | fl Name,ForestMode Then we need to enable the Expiring Link feature. It can be enabled using the following command: Enable-ADOptionalFeature ‘Privileged Access Management Feature’ -Scope ForestOrConfigurationSet -Target rebeladmin.com The rebeladmin.com link can be replaced with your FQDN: I have a user called Adam Curtiss to whom I need to assign Domain Admins group membership for 60 minutes: Get-ADGroupMember “Domain Admins” The preceding command will list the current member of domain admin group:  Next step is to add the user Adam Curtiss to the Domain Admins group for 60 minutes: Add-ADGroupMember -Identity ‘Domain Admins’ -Members ‘acurtiss’ -MemberTimeToLive (New-TimeSpan -Minutes 60)  Once its run, we can verify the TTL value remaining for the group membership using the following command:  Get-ADGroup ‘Domain Admins’ -Property member -ShowMemberTimeToLive Once I log in as the user and list the Kerberos ticket it shows the renew time with less than 60 minutes as I log in as user after few minutes of granting. Once the TGT renewal comes, the user will no longer be a member of Domain Admins group. Summary In this article we looked at the new features and enhancements that come with AD DS 2016. One of the biggest improvement was Microsoft's new approach towards the PAM. This is not just a feature that can be enabled via AD DS, it's just a part of the border solution. It helps to protect identity infrastructures from adversaries as traditional techniques and technologies no longer valid with rising threats. Resources for Article: Further resources on this subject: Deploying and Synchronizing Azure Active Directory [article] How to Recover from an Active Directory Failure [article] Active Directory migration [article]
Read more
  • 0
  • 0
  • 4810

article-image-building-strong-foundation
Packt
04 May 2017
28 min read
Save for later

Building a Strong Foundation

Packt
04 May 2017
28 min read
In this article, by Mickey Macdonald, author of the book Mastering C++ Game Development, we will cover how these libraries can work together and build some of the libraries needed to round out the structure. (For more resources related to this topic, see here.) To get started, we will focus on, arguably one of the most important aspects of any game project, the rendering system. Proper, performant implementations not only takes a significant amount of time, but it also takes specialized knowledge of video driver implementations and mathematics for computer graphics. Having said that, it is not, in fact, impossible to create a custom low-level graphics library yourself, it's just not overly recommended if your end goal is just to make video games. So instead of creating a low-level implementation themselves, most developers turn to a few different libraries to provide them abstracted access to the bare metal of the graphics device. We will be using a few different graphic APIs to help speed up the process and help provide coherence across platforms. These APIs include the following: OpenGL (https://www.opengl.org/): The Open Graphics Library (OpenGL) is an open cross-language, cross-platform application programming interface, or API, used for rendering 2D and 3D graphics. The API provides low-level access to the graphics processing unit (GPU). SDL (https://www.libsdl.org/): Simple DirectMedia Layer (SDL) is a cross-platform software development library designed to deliver a low-level hardware abstraction layer to multimedia hardware components. While it does provide its own mechanism for rendering, SDL can use OpenGL to provide full 3D rendering support. While these APIs save us time and effort by providing us some abstraction when working with the graphics hardware, it will quickly become apparent that the level of abstraction will not be high enough. You will need another layer of abstraction to create an efficient way of reusing these APIs in multiple projects. This is where the helper and manager classes come in. These classes will provide the needed structure and abstraction for us and other coders. They will wrap all the common code needed to set up and initialize the libraries and hardware. The code that is required by any project regardless of gameplay or genre can be encapsulated in these classes and will become part of the "engine." In this article, we will cover the following topics: Building helper classes Encapsulation with managers Creating interfaces Building helper classes In object-oriented programming, a helper class is used to assist in providing some functionality, which is not, directly the main goal of the application in which it is used. Helper classes come in many forms and are often a catch-all term for classes that provide functionality outside of the current scope of a method or class. Many different programming patterns make use of helper classes. In our examples, we too will make heavy use of helper classes. Here is just one example. Let's take a look at the very common set of steps used to create a Window. It's safe to say that most of the games you will create will have some sort of display and will generally be typical across different targets, in our case Windows and the macOS. Having to retype the same instructions constantly over and over for each new project seems like kind of a waste. That sort of situation is perfect for abstracting away in a helper class that will eventually become part of the engine itself. The code below is the header for the Window class included in the demo code examples. To start, we have a few necessary includes, SDL, glew which is a Window creation helper library, and lastly, the standard string class is included: #pragma once #include <SDL/SDL.h> #include <GL/glew.h> #include <string> Next, we have an enum WindowFlags. We use this for setting some bitwise operations to change the way the window will be displayed; invisible, full screen, or borderless. You will notice that I have wrapped the code in the namespace BookEngine, this is essential for keeping naming conflicts from happening and will be very helpful once we start importing our engine into projects: namespace BookEngine { enum WindowFlags //Used for bitwise passing { INVISIBLE = 0x1, FULLSCREEN = 0x2, BORDERLESS = 0x4 }; Now we have the Window class itself. We have a few public methods in this class. First the default constructor and destructor. It is a good idea to include a default constructor and destructor even if they are empty, as shown here, despite the compiler, including its own, these specified ones are needed if you plan on creating intelligent or managed pointers, such as unique_ptr, of the class: class Window { public: Window(); ~Window(); Next we have the Create function, this function will be the one that builds or creates the window. It takes a few arguments for the creation of the window such as the name of the window, screen width and height, and any flags we want to set, see the previously mentioned enum. void Create(std::string windowName, int screenWidth, int screenHeight, unsigned int currentFlags); Then we have two getter functions. These functions will just return the width and height respectively: int GetScreenWidth() { return m_screenWidth; } int GetScreenHeight() { return m_screenHeight; } The last public function is the SwapBuffer function; this is an important function that we will take a look at in more depth shortly. void SwapBuffer(); To close out the class definition, we have a few private variables. The first is a pointer to a SDL_Window* type, named appropriate enough m_SDL_Window. Then we have two holder variables to store the width and height of our screen. This takes care of the definition of the new Window class, and as you can see it is pretty simple on face value. It provides easy access to the creation of the Window without the developer calling it having to know the exact details of the implementation, which is one aspect that makes Object Orientated Programming and this method is so powerful: private: SDL_Window* m_SDL_Window; int m_screenWidth; int m_screenHeight; }; } To get a real sense of the abstraction, let's walk through the implementation of the Window class and really see all the pieces it takes to create the window itself. #include "Window.h" #include "Exception.h" #include "Logger.h" namespace BookEngine { Window::Window() { } Window::~Window() { } The Window.cpp files starts out with the need includes, of course, we need to include Window.h, but you will also note we need to include the Exception.h and Logger.h header files also. These are two other helper files created to abstract their own processes. The Exception.h file is a helper class that provides an easy-to-use exception handling system. The Logger.h file is a helper class that as its name says, provides an easy-to-use logging system. After the includes, we again wrap the code in the BookEngine namespace and provide the empty constructor and destructor for the class. The Create function is the first to be implemented. In this function are the steps needed to create the actual window. It starts out setting the window display flags using a series of if statements to create a bitwise representation of the options for the window. We use the enum we created before to make this easier to read for us humans. void Window::Create(std::string windowName, int screenWidth, int screenHeight, unsigned int currentFlags) { Uint32 flags = SDL_WINDOW_OPENGL; if (currentFlags & INVISIBLE) { flags |= SDL_WINDOW_HIDDEN; } if (currentFlags & FULLSCREEN) { flags |= SDL_WINDOW_FULLSCREEN_DESKTOP; } if (currentFlags & BORDERLESS) { flags |= SDL_WINDOW_BORDERLESS; } After we set the window's display options, we move on to using the SDL library to create the window. As I mentioned before, we use libraries such as SDL to help us ease the creation of such structures. We start out wrapping these function calls in a Try statement; this will allow us to catch any issues and pass it along to our Exception class as we will see soon: try { //Open an SDL window m_SDL_Window = SDL_CreateWindow(windowName.c_str(), SDL_WINDOWPOS_CENTERED, SDL_WINDOWPOS_CENTERED, screenWidth, screenHeight, flags); The first line sets the private member variable m_SDL_Window to a newly created window using the passed in variables, for the name, width, height, and any flags. We also set the default window's spawn point to the screen center by passing the SDL_WINDOWPOS_CENTERED define to the function. if (m_SDL_Window == nullptr) throw Exception("SDL Window could not be created!"); After we have attempted to create the window, it is a good idea to check and see if the process did succeed. We do this with a simple if statement and check to see if the variable m_SDL_Window is set to a nullptr; if it is, we throw an Exception. We pass the Exception the string "SDL Window could not be created!". This is the error message that we can then print out in a catch statement. Later on, we will see an example of this. Using this method, we provide ourselves some simple error checking. Once we have created our window and have done some error checking, we can move on to setting up a few other components. One of these components is the OpenGL library which requires what is referred to as a context to be set. An OpenGL context can be thought of as a set of states that describes all the details related to the rendering of the application. The OpenGLcontext must be set before any drawing can be done. One problem is that creating a window and an OpenGL context is not part of the OpenGL specification itself. What this means is that every platform can handle this differently. Luckily for us, the SDL API again abstracts the heavy lifting for us and allows us to do this all in one line of code. We create a SDL_GLContext variable named glContext. We then assign glContext to the return value of the SDL_GL_CreateContext function that takes one argument, the SDL_Window we created earlier. After this we, of course, do a simple check to make sure everything worked as intended, just like we did earlier with the window creation: //Set up our OpenGL context SDL_GLContext glContext = SDL_GL_CreateContext(m_SDL_Window); if (glContext == nullptr) throw Exception("SDL_GL context could not be created!"); The next component we need to initialize is GLEW. Again this is abstracted for us to one simple command, glewInit(). This function takes no arguments but does return an error status code. We can use this status code to perform a similar error check like we did with the window and OpenGL. This time instead checking it against the defined GLEW_OK. If it evaluates to anything other than GLEW_OK, we throw an Exception to be caught later on. //Set up GLEW (optional) GLenum error = glewInit(); if (error != GLEW_OK) throw Exception("Could not initialize glew!"); Now that the needed components are initialized, now is a good time to log some information about the device running the application. You can log all kinds of data about the device which can provide valuable insights when trying to track down obscure issues. In this case, I am polling the system for the version of OpenGL that is running the application and then using the Logger helper class printing this out to a "Runtime" text file: //print some log info std::string versionNumber = (const char*)glGetString(GL_VERSION); WriteLog(LogType::RUN, "*** OpenGL Version: " + versionNumber + "***"); Now we set the clear color or the color that will be used to refresh the graphics card. In this case, it will be the background color of our application. The glClearColor function takes four float values that represent the red, green, blue, and alpha values in a range of 0.0 to 1.0. Alpha is the transparency value where 1.0f is opaque, and 0.0f is completely transparent. //Set the background color to blue glClearColor(0.0f, 0.0f, 1.0f, 1.0f); The next line sets the VSYNC value, which is a mechanism that will attempt to match the application's framerate to that of the physical display. The SDL_GL_SetSwapInterval function takes one argument, an integer that can be 1 for on or 0 for off. //Enable VSYNC SDL_GL_SetSwapInterval(1); The last two lines that make up the try statement block, enable blending and set the method used when performing alpha blending. For more information on these specific functions, check out the OpenGL development documents: //Enable alpha blend glEnable(GL_BLEND); glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA); } After our try block, we now have to include the catch block or blocks. This is where we will capture any of the thrown errors that have occurred. In our case, we are just going to grab all the Exceptions. We use the WriteLog function from the Logger helper class to add the exception message, e.reason to the error log text file. This is a very basic case, but of course, we could do more here, possibly even recover from an error if possible. catch (Exception e) { //Write Log WriteLog(LogType::ERROR, e.reason); } } Finally, the last function in the Window.cpp file is the SwapBuffer function. Without going too deep on the implementation, what swapping buffers does is exchange the front and back buffers of the GPU. This in a nutshell allows smoother drawing to the screen. It is a complicated process that again has been abstracted by the SDL library. Our SwapBuffer function, abstracts this process again so that when we want to swap the buffers we simply call SwapBuffer instead of having to call the SDL function and specify the window, which is what is exactly done in the function. void Window::SwapBuffer() { SDL_GL_SwapWindow(m_SDL_Window); } } So as you can see, building up these helper functions can go a long way in making the process of development and iteration much quicker and simpler. Next, we will look at another programming method that again abstracts the heavy lifting from the developer's hands and provides a form of control over the process, a management system. Encapsulation with managers When working with complex systems such as input and audio systems, it can easily become tedious and unwieldy to control and check each state and other internals of the system directly. This is where the idea of the "Manager" programming pattern comes in. Using abstraction and polymorphism we can create classes that allow us to modularize and simplify the interaction with these systems. Manager classes can be found in many different use cases. Essentially if you see a need to have structured control over a certain system, this could be a candidate for a Manger class. Stepping away from the rendering system for a second, let's take a look at a very common task that any game will need to perform, handling input. Since every game needs some form of input, it only makes sense to move the code that handles this to a class that we can use over and over again. Let's take a look at the InputManager class, starting with the header file: #pragma once #include <unordered_map> #include <glm/glm.hpp> namespace BookEngine { class InputManager { public: InputManager(); ~InputManager(); The InputManager class starts just like the others, we have the includes needed and again we wrap the class in the BookEngine namespace for convince and safety. The standard constructor and destructor are also defined. Next, we have a few more public functions. First the Update function, which will not surprisingly update the input system. Then we have the KeyPressed and KeyReleased functions, these functions both take an integer value corresponding to a keyboard key. The following functions fire off when the key is pressed or released respectively. void Update(); void KeyPress(unsigned int keyID); void KeyRelease(unsigned int keyID); After the KeyPress and KeyRelease functions, we have two more key related functions the isKeyDown and isKeyPressed . Like the KeyPress and KeyRelease functions the isKeyDown and isKeyPressed functions take integer values that correspond to keyboard keys. The noticeable difference is that these functions return a Boolean value based on the status of the key. We will see more about this in the implementation file coming up. bool isKeyDown(unsigned int keyID); //Returns true if key is held bool isKeyPressed(unsigned int keyID); //Returns true if key was pressed this update The last two public functions in the InputManager class are SetMouseCoords and GetMouseCoords which do exactly as the names suggest and set or get the mouse coordinates respectively. void SetMouseCoords(float x, float y); glm::vec2 GetMouseCoords() const { return m_mouseCoords; }; Moving on to the private members and functions, we have a few variables declared to store some information about the keys and mouse. First, we have a Boolean value that stores the state of the key being pressed down or not. Next, we have two unordered maps that will store the current keymap and previous keymaps. The last value we store is the mouse coordinates. We us a vec2 construct from another helper library the Graphic Library Math library or GLM. We use this vec2, which is just a two-dimensional vector, to store the x and y coordinate values of the mouse cursor since it is on a 2D plane, the screen. If you are looking for a refresher on vectors and the Cartesian coordinate system, I highly recommend the Beginning Math Concepts for Game Developers book by Dr. John P Flynt: private: bool WasKeyDown(unsigned int keyID); std::unordered_map<unsigned int, bool> m_keyMap; std::unordered_map<unsigned int, bool> m_previousKeyMap; glm::vec2 m_mouseCoords; }; } Now let's look at the implementation, the InputManager.cpp file. Again we start out with the includes and the namespace wrapper. Then we have the constructor and destructor. The highlight to note here is the setting of the m_mouseCoords to 0.0f in the constructor: namespace BookEngine { InputManager::InputManager() : m_mouseCoords(0.0f) { } InputManager::~InputManager() { } Next is the Update function. This is a simple update where we are stepping through each key in the key map and copying it over to the previous key map holder: m_previousKeyMap. void InputManager::Update() { for (auto& iter : m_keyMap) { m_previousKeyMap[iter.first] = iter.second; } } The next function is the KeyPress function. In this function, we use the trick of an associative array to test and insert the key pressed whichmatches the ID passed in. The trick is that if the item located at the index of the keyID index does not exist, it will automatically be created: void InputManager::KeyPress(unsigned int keyID) { m_keyMap[keyID] = true; } . We do the same for the KeyRelease function below. void InputManager::KeyRelease(unsigned int keyID) { m_keyMap[keyID] = false; } The KeyRelease function is the same setup as the KeyPressed function, except that we are setting the keyMap item at the keyID index to false. bool InputManager::isKeyDown(unsigned int keyID) { auto key = m_keyMap.find(keyID); if (key != m_keyMap.end()) return key->second; // Found the key return false; } After the KeyPress and KeyRelease functions, we implement the isKeyDown and isKeyPressed functions. First the isKeydown function; here we want to test if a key is already pressed down. In this case, we take a different approach to testing the key than in the KeyPress and KeyRelease functions and avoid the associative array trick. This is because we don't want to create a key if it does not already exist, so instead, we do it manually: bool InputManager::isKeyPressed(unsigned int keyID) { if(isKeyDown(keyID) && !m_wasKeyDown(keyID)) { return true; } return false; } The isKeyPressed function is quite simple. Here we test to see if the key that matches the passed in ID is pressed down, by using the isKeyDown function, and that it was not already pressed down by also passing the ID to m_wasKeyDown. If both of these conditions are met, we return true, or else we return false. Next, we have the WasKeyDown function, much like the isKeyDown function, we do a manual lookup to avoid accidentally creating the object using the associative array trick: bool InputManager::WasKeyDown(unsigned int keyID) { auto key = m_previousKeyMap.find(keyID); if (key != m_previousKeyMap.end()) return key->second; // Found the key return false; } The final function in the InputManger is SetMouseCoords. This is a very simple "setter" function that takes the passed in floats and assigns them to the x and y members of the two-dimensional vector, m_mouseCoords. void InputManager::SetMouseCoords(float x, float y) { m_mouseCoords.x = x; m_mouseCoords.y = y; } } Creating interfaces Sometimes you are faced with a situation where you need to describe capabilities and provide access to general behaviors of a class without committing to a particular implementation. This is where the idea of interfaces or abstract classes comes into play. Using interfaces provides a simple base class that other classes can then inherit from without having to worry about the intrinsic details. Building strong interfaces can enable rapid development by providing a standard class to interact with. While interfaces could, in theory, be created of any class, it is more common to see them used in situations where the code is commonly being reused. Let's take a look at an interface from the example code in the repository. This interface will provide access to the core components of the Game. I have named this class IGame, using the prefix I to identify this class as an interface. The following is the implementation beginning with the definition file IGame.h. To begin with, we have the needed includes and the namespace wrapper. You will notice that the files we are including are some of the ones we just created. This is a prime example of the continuation of the abstraction. We use these building blocks to continue to build the structure that will allow this seamless abstraction: #pragma once #include <memory> #include "BookEngine.h" #include "Window.h" #include "InputManager.h" #include "ScreenList.h" namespace BookEngine { Next, we have a forward declaration. This declaration is for another interface that has been created for screens. The full source code to this interface and its supporting helper classes are available in the code repository. class IScreen;using forward declarations like this is a common practice in C++. If the definition file only requires the simple definition of a class, not adding the header for that class will speed up compile times. Moving onto the public members and functions, we start off the constructor and destructor. You will notice that this destructor in this case is virtual. We are setting the destructor as virtual to allow us to call delete on the instance of the derived class through a pointer. This is handy when we want our interface to handle some of the cleanup directly as well. class IGame { public: IGame(); virtual ~IGame(); Next we have declarations for the Run function and the ExitGame function. void Run(); void ExitGame(); We then have some pure virtual functions, OnInit, OnExit, and AddScreens. Pure virtual functions are functions that must be overridden by the inheriting class. By adding the =0; to the end of the definition, we are telling the compiler that these functions are purely virtual. When designing your interfaces, it is important to be cautious when defining what functions must be overridden. It's also very important to note that having pure virtual function implicitly makes the class it is defined for abstract. Abstract classes cannot be instantiated directly because of this and any derived classes need to implement all inherited pure virtual functions. If they do not, they too will become abstract. virtual void OnInit() = 0; virtual void OnExit() = 0; virtual void AddScreens() = 0; After our pure virtual function declarations, we have a function OnSDLEvent which we use to hook into the SDL event system. This provides us support for our input and other event-driven systems: void OnSDLEvent(SDL_Event& event); The public function in the IGame interface class is a simple helper function GetFPS that returns the current FPS. Notice the const modifiers, they identify quickly that this function will not modify the variable's value in any way: declarations const float GetFPS() const { return m_fps; } In our protected space, we start with a few function declarations. First is the Init or initialization function. This will be the function that handles a good portion of the setup. Then we have two virtual functions Update and Draw. Like pure virtual functions, a virtual function is a function that can be overridden by a derived class's implementation. Unlike a pure virtual function, the virtual function does not make the class abstract by default and does not have to be overridden. Virtual and pure virtual functions are keystones of polymorphic design. You will quickly see their benefits as you continue your development journey: protected: bool Init(); virtual void Update(); virtual void Draw(); To close out the IGame definition file, we have a few members to house different objects and values. I am not going to go through these line by line since I feel they are pretty self-explanatory: declarations std::unique_ptr<ScreenList> m_screenList = nullptr; IGameScreen* m_currentScreen = nullptr; Window m_window; InputManager m_inputManager; bool m_isRunning = false; float m_fps = 0.0f; }; } Now that we have taken a look at the definition of our interface class, let's quickly walk through the implementation. The following is the IGame.cpp file. To save time and space, I am going to highlight the key points. For the most part, the code is self-explanatory, and the source located in the repository is well commented for more clarity: #include "IGame.h" #include "IScreen.h" #include "ScreenList.h" #include "Timing.h" namespace BookEngine { IGame::IGame() { m_screenList = std::make_unique<ScreenList>(this); } IGame::~IGame() { } Our implementation starts out with the constructor and destructor. The constructor is simple, its only job is to add a unique pointer of a new screen using this IGame object as the argument to pass in. See the IScreen class for more information on screen creation. Next, we have the implementation of the Run function. This function, when called will set the engine in motion. Inside the function, we do a quick check to make sure we have already initialized our object. We then use yet another helper class, FPSlimiter, to set the max fps that our game can run. After that, we set the isRunning boolean value to true, which we then use to control the game loop: void IGame::Run() { if (!Init()) return; FPSLimiter fpsLimiter; fpsLimiter.SetMaxFPS(60.0f); m_isRunning = true; Next is the game loop. In the game loop, we do a few simple calls. First, we start the fps limiter. We then call the update function on our input manager. It is a good idea always to check input before doing other updates or drawing since their calculations are sure to use the new input values. After we update the input manager, we recursively call our Update and Draw class, which we will see shortly. We close out the loop by ending the fpsLimiter function and calling SwapBuffer on the Window object. ///Game Loop while (m_isRunning) { fpsLimiter.Begin(); m_inputManager.Update(); Update(); Draw(); m_fps = fpsLimiter.End(); m_window.SwapBuffer(); } } The next function we implement is the ExitGame function. Ultimately, this will be the function that will be called on the final exit of the game. We close out, destroy, and free up any memory that the screen list has created and set the isRunning Boolean to false, which will put an end to the loop: void IGame::ExitGame() { m_currentScreen->OnExit(); if (m_screenList) { m_screenList->Destroy(); m_screenList.reset(); //Free memory } m_isRunning = false; } Next up is the Init function. This function will initialize all the internal object settings and call the initialization on the connected systems. Again, this is an excellent example of OOP or object orientated programming and polymorphism. Handling initialization in this manner allows the cascading effect, keeping the code modular and easier to modify: bool IMainGame::Init() { BookEngine::Init(); SDL_GL_SetAttribute(SDL_GL_ACCELERATED_VISUAL, 1); m_window.Create("BookEngine", 1024, 780, 0); OnInit(); AddScreens(); m_currentScreen = m_screenList->GetCurrentScreen(); m_currentScreen->OnEntry(); m_currentScreen->Run(); return true; } Next, we have the update function. In this Update function, we create a structure to allow us to execute certain code based on a state that the current screen is in. We accomplish this using a simple Switch case method with the enumerated elements of the ScreenState type as the cases. This setup is considered a simple finite state machine and is a very powerful design method used throughout game development: void IMainGame::Update() { if (m_currentScreen) { switch (m_currentScreen->GetScreenState()) { case ScreenState::RUNNING: m_currentScreen->Update(); break; case ScreenState::CHANGE_NEXT: m_currentScreen->OnExit(); m_currentScreen = m_screenList->MoveToNextScreen(); if (m_currentScreen) { m_currentScreen->Run(); m_currentScreen->OnEntry(); } break; case ScreenState::CHANGE_PREVIOUS: m_currentScreen->OnExit(); m_currentScreen = m_screenList->MoveToPreviousScreen(); if (m_currentScreen) { m_currentScreen->Run(); m_currentScreen->OnEntry(); } break; case ScreenState::EXIT_APP: ExitGame(); break; default: break; } } else { //we have no screen so exit ExitGame(); } } After our Update, we implement the Draw function. In our function, we only do a couple of things. First, we reset the Viewport as a simple safety check, then if the current screen's state matches the enumerated value RUNNING, we again use polymorphism to pass the Draw call down the object line: void IGame::Draw() { //For safety glViewport(0, 0, m_window.GetScreenWidth(), m_window.GetScreenHeight()); //Check if we have a screen and that the screen is running if (m_currentScreen && m_currentScreen->GetScreenState() == ScreenState::RUNNING) { m_currentScreen->Draw(); } } The last function we need to implement is the OnSDLEvent function. Like I mention in the definition section of this class, we will use this function to connect our input manger system to the SDL built in event system. Every key press or mouse movement is handled as an event. Based on the type of event that has occurred, we again use a Switch case statement to create a simple finite state machine. void IGame::OnSDLEvent(SDL_Event & event) { switch (event.type) { case SDL_QUIT: m_isRunning = false; break; case SDL_MOUSEMOTION: m_inputManager.SetMouseCoords((float)event.motion.x, (float)event.motion.y); break; case SDL_KEYDOWN: m_inputManager.KeyPress(event.key.keysym.sym); break; case SDL_KEYUP: m_inputManager.KeyRelease(event.key.keysym.sym); break; case SDL_MOUSEBUTTONDOWN: m_inputManager.KeyPress(event.button.button); break; case SDL_MOUSEBUTTONUP: m_inputManager.KeyRelease(event.button.button); break; } } } Well, that takes care of the IGame interface. With this created, we can now create a new project that can utilize this and other interfaces in the example engine to create a game and initialize it all with just a few lines of code: #pragma once #include <BookEngine/IMainGame.h> #include "GamePlayScreen.h" class App : public BookEngine::IGame { public: App(); ~App(); virtual void OnInit() override; virtual void OnExit() override; virtual void AddScreens() override; private: std::unique_ptr<GameplayScreen> m_gameplayScreen = nullptr; }; The highlights to note here are that, one, the App class inherits from the BookEngine::IGame interface and two, we have all the necessary overrides that the inherited class requires. Next, if we take a look at the main.cpp file, the entry point for our application, you will see the simple commands to set up and kick off all the amazing things our interfaces, managers, and helpers abstract for us: #include <BookEngine/IMainGame.h> #include "App.h" int main(int argc, char** argv) { App app; app.Run(); return 0; } As you can see, this is far simpler to type out every time we want to create a new project than having to recreate the framework constantly from scratch. To see the output of the framework, build the BookEngine project, then build and run the example project. On Windows, the example project when run will look like the following: On macOS, the example project when run will look like the following: Summary In this article, we covered quite a bit. We took a look at the different methods of using object oriented programming and polymorphism to create a reusable structure for all your game projects. We walked through the differences in Helper, Managers, and Interfaces classes with examples from real code. Resources for Article: Further resources on this subject: Game Development Using C++ [article] C++, SFML, Visual Studio, and Starting the first game [article] Common Game Programming Patterns [article]
Read more
  • 0
  • 0
  • 2045

article-image-deploying-first-container
Packt
11 Apr 2017
11 min read
Save for later

Deploying First Container

Packt
11 Apr 2017
11 min read
In this article by Srikant Machiraju, author of the book Learning Windows Server Containers, we will get acquainted with containers and containerization. Containerization helps you build software in layers, containers inspire distributed development, packaging, and publishing in the form of containers. Developers or IT administrators just have to choose a BaseOS Image, create customized layers as per their requirements, and distribute using Public or Private Repositories.Microsoft and Docker together have provided an amazing toolset that helps you build and deploy containers within no time. It is very easy to setup a dev/test environment as well. Microsoft Windows Server Operating System or Windows 10 Desktop OS comes with plug and play features for running Windows Server containers or Hyper-V Containers.Docker Hub,a public repository for images,serves as a huge catalogue of customized images built by community or docker enthusiasts. The images on DockerHub are freely available for anyone to download, customize,and distribute images. In this article, we will learn how to create and configure container development environments. The following are a few more concepts that you will learn in this article: Preparing Windows Server Containers Environment Pulling images from Docker Hub Installing Base OS Images (For more resources related to this topic, see here.) Preparing Development Environment In order to start creating Windows Containers,you need an instance of Windows Server 2016 or Windows 10 Enterprise/Professional Edition (with Anniversary Update). Irrespective of the environment, the PowerShell/Docker commands described in thisarticlefor creating and packaging containers/imagesare the same.The following are the options we have to setup a windows server container development environment: Windows 10: Using Windows 10 Enterprise or Professional Edition with Anniversary update you can create Hyper-V Containers by enabling the containers role. Docker or PowerShell can be used to manage the containers. We will learn how to configure the Windows 10 environment in the following section. Important: Windows 10 only supports Hyper-V Containers created using NanoServer Base OS Image; it does not support Windows Server Containers. Windows Server 2016: There are two options for working with containers on Windows Server 2016: You can download the Windows Server 2016 ISO from here (https://www.microsoft.com/en-in/evalcenter/evaluate-windows-server-technical-preview) and install it on a virtual machine running on Hyper-V or Virtual Box.For running Windows Server 2016 the host machine should have Hyper-V virtualization enabled.Additionally, for the containers to access theInternet, ensure that thenetwork is sharable between the host and Hyper-VVMs. Windows Azure provides a readymade instance of Windows Server 2016 with Containers configured. This so far is the easiest option available. In this article,I will be using Windows Server 2016 with Containers enabled on Azure to create and manageWindow Server Containers. Windows Server 2016 is still in preview and the latest version available at the time of writing is Technical Preview 5. Containerson Windows 10 The following steps explain how to setup a dev/test environment on Windows 10 for learning container development using Hyper-V Containers. Before continuing further, ensure thatyou're running Windows 10 Professional/Enterprise version with anniversary update. For validating Windows Edition on Windows 10,click Start and type This PC, andthen right-click on This PC and click on Properties. Check the Windows Edition section for the Windows 10 edition. If your PC shows Windows 10 Enterprise or Professional, you can download and install the Anniversary update from here (https://support.microsoft.com/en-us/help/12387/windows-10-update-history). If you do not have any of the above, please proceed to the following section, which explains how to work with containers using Windows Server 2016 environment on Azure and on-premises. Follow these steps to configure Hyper-V containers on Windows 10: Click on Start Menu and type powershell. Right-click powershell CLI (Command Line Interface) and Run as administrator. Run the following command to install the containers feature on Windows 10: Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All Run the following command to install the Hyper-V containers feature on Windows 10. We will only be using Windows Server Containers in this article: Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All Restart the PC by running the following command: Restart-Computer –Force Run the following command to update the registry settings: Set-ItemProperty -Path 'HKLM:SOFTWAREMicrosoftWindows NTCurrentVersionVirtualizationContainers' -Name VSmbDisableOplocks -Type DWord -Value 1 -Force Although PowerShell can be used to manage and run containers, Docker commands give a full wealth of options for container management. Microsoft PowerShell support for Windows Container development is still a work in progress, so we will mix and match PowerShell and Docker as per the scenarios. Run the following set of steps one by one to install and configure Docker on Windows 10: Invoke-WebRequest"https://master.dockerproject.org/windows/amd64/docker-1.13.0-dev.zip" -OutFile"$env:TEMPdocker-1.13.0-dev.zip" –UseBasicParsing Expand-Archive -Path "$env:TEMPdocker-1.13.0-dev.zip" -DestinationPath $env:ProgramFiles $env:path += ";c:program filesdocker" [Environment]::SetEnvironmentVariable("Path", $env:Path + ";C:Program FilesDocker", [EnvironmentVariableTarget]::Machine) dockerd --register-service Start the Docker Service by running the following command: Start-Service docker In order to develop windows server containers, we need any Windows Base OS Image,such as windowsservercore or nanoserver. Since Windows 10 supports nanoservers only, run the following command to download thenanoservercorebase OS image. The following command might take some time depending on your bandwidth; it downloadsand extracts thenanoserver base OS image, which is 970 MB in size approximately: docker pull microsoft/nanoserver Important: At the time of writing, Windows 10 can only run Nano Server Images. Even though the windowsservercore image gets download successfully, running containers using this image will fail due to incompatibility with the OS. Ensure that the images are successfully downloaded by running the following command: docker images Windows Server Containers On-Premise This section will help you to download and install Windows Server 2016 on a Virtual Machine using a hosted virtualization software such as Hyper-V or Virtual Box. Windows Server 2016 comes with two installation options, Windows Server 2016 Core and Windows Server 2016 Full feature, as shown in the following screenshot: Windows Server 2016 Core is a No GUI version of Windows with minimalistic server features and bare minimum size, whereas thefull version is the traditional Server Operating System and it comes with all features installed. No matter which installation option you choose, you will be able to install and run Windows Server Containers. You cannot change the NoGUI version to full version post installation, so make sure you are choosing the right option during installation. You can download the Windows Server ISO from here (https://www.microsoft.com/en-in/evalcenter/evaluate-windows-server-technical-preview, make sure you copy the Activation Keyas well from here) and setup ahostedvirtualization software such as Virtual Box or Hyper-V. Once you start installing from the ISO you will be presented with the following screen, which lets you select full installation with Desktop experience or Just the core with Command Line Interface. Windows Server Containers on Azure On Azure we are going to use an image that comes preinstalled with the containers feature. You can also start with plain Windows Server 2016 on Azure and install the Windows Containers role from Add/Remove Windows Features and then install Docker Engine or use the steps mentioned in Containers on Windows 10. In this article, we are going to create Windows Server 2016 Virtual Machine on Microsoft Azure. The name of the image on Windows Azure is Windows Server 2016 with Containers Tech Preview 5. Microsoft Azure does not support Hyper-V containers on Azure. However, you can deploy these container types when running WS 2016 on premises or on Windows 10. Note:For creating a VMon Azure you would need an Azure account. Microsoft provides a free account for beginners to learn and practice Azure for 30 days/$200. More details regarding creating a free account can be found here (https://azure.microsoft.com/en-in/free/). Container options on WS 2016 TP5 Windows Server 2016 supports two types of containers: Windows Server Containers and Hyper-V Containers. You can run any of these Containers on Windows Server 2016. But for running Hyper-V Containers we would need a container hostthat supports Hyper-V Nested virtualization enabled,which is not mandatory for Windows Server containers. Create Windows Server 2016 TP5 on Azure Azure VMs can be created using a variety of options such as Management Portal (https://manage.windowsazure.com), PowerShell,or Azure CLI. We will be using the new Azure Management Portal (codename Ibiza) for creating an Azure VM. Please follow these steps to create a new Windows Server 2016 Containers Virtual Machine: Login to the Azure Management portal, https://portal.azure.com. Click on +New and select Virtual Machines. Click on See All to get a full list of Virtual Machines. Search using text Windows Server 2016 (with double quotes for full phrase search). You will find three main flavors of Windows Server 2016, as shown in the following screenshot: Click Windows Server 2016with Containers Technical Preview 5 and then click Create on the new blade. Select Resource Manager as thedeployment model. Fill in the parameters as follows: Basic settings:    Name: The name of the Virtual Machine or the host name.    VM Disk Type:SSD (Solid State Disk).    User name: Administrator account username of the machine. This will be used while remotely connecting to the machine using RDP.    Password: Administrator password.    Confirm Password: Repeat administrator password.    Subscription: Select the Azure Subscription to be used to create the machine.    Resource Group: Resource group is a group name for logically grouping resources of a single project. You can use an existing resource group or create a new one.    Location: The geographical location of the Azure Data Center. Choose the nearest available to you. Click Ok. Select Size and click on DS3_V2. For this exercise,we will be using DS3_V2 Standard, which comes with four cores and 14 GB memory. SelectDS3_V2 Standard and click Select: Click on Settings. The Settings section will have most of the parameters pre-filled with defaults. You can leave the defaults unless you want to change anything and then click OK. Check a quick summary of the selections made using the summary tab and click OK. Azure starts creating the VM. The progress will be shown on a new tile added to the dashboard screen,as shown in the following screenshot: Azure might take a couple of minutes or more to create the Virtual Machine and configure extensions. You can check the status on the newly created dashboard tile. Installing Base OS Images and Verifying Installation The following steps will explain how to connect to an Azure VM and verify if the machine is ready for windows server container development: Once the VM is created and running, the status of the VM on the tile will be shown as Running: We can connect to Azure VMs primarily in two ways using RDP and Remote PowerShell. In this sample, we will be using RDP to connect to the Azure VM. Select the tile and click on the Connect icon. This downloads a remote desktop client to your local machine, as shown in the following screenshot: Note: If you are using any browser other than IE 10 or Edge,please check for the .rdp file in the respective downloads folder. Double-click the.rdp file and click Connect to connect to the VM. Enter the Username and Password used while creating the VM to login to the VM. Ignore the Security Certificate Warning and click on Yes. Ensure that the containers feature is installed by running the following command. This should show the Installed State of containers,as shown in the following screenshot: Get-WindowsFeature -Name Containers Ensure that thedocker client and engine is installed using the following commands on PowerShell CLI and verify the version: Docker version Docker Info provides the current state of thedocker daemon,such as the Number of Containers running, paused or stopped, the number of images, and so on, as shown in the following screenshot: Summary In this article, we have learnt how to create and configure windows server container and Hyper-V environment on Windows 10 or Windows Server 2016. Windows 10 professional or Enterprise with Anniversary update can be used to run Hyper-V Containers only. Resources for Article:  Further resources on this subject: Understanding Container Scenarios and Overview of Docker [article] HTML5: Generic Containers [article] OpenVZ Container Administration [article]
Read more
  • 0
  • 0
  • 2239
article-image-integrating-messages-app
Packt
06 Apr 2017
17 min read
Save for later

Integrating with Messages App

Packt
06 Apr 2017
17 min read
In this article by Hossam Ghareeb, the author of the book, iOS Programming Cookbook, we will cover the recipe Integrating iMessage app with iMessage app. (For more resources related to this topic, see here.) Integrating iMessage app with iMessage app Using iMessage apps will let users use your apps seamlessly from iMessage without having to leave the iMessage. Your app can share content in the conversation, make payment, or do any specific job that seems important or is appropriate to do within a Messages app. Getting ready Similar to the Stickers app we created earlier, you need Xcode 8.0 or later version to create an iMessage app extension and you can test it easily in the iOS simulator. The app that we are going to build is a Google drive picker app. It will be used from an iMessage extension to send a file to your friends just from Google Drive. Before starting, ensure that you follow the instructions in Google Drive API for iOS from https://developers.google.com/drive/ios/quickstart to get a client key to be used in our app. Installing the SDK in Xcode will be done via CocoaPods. To get more information about CocoaPods and how to use it to manage dependencies, visit https://cocoapods.org/ . How to do it… We Open Xcode and create a new iMessage app, as shown, and name itFiles Picker:   Now, let's install Google Drive SDK in iOS using CocoaPods. Open terminal and navigate to the directory that contains your Xcode project by running this command: cd path_to_directory Run the following command to create a Pod file to write your dependencies: Pod init It will create a Pod file for you. Open it via TextEdit and edit it to be like this: use_frameworks! target 'PDFPicker' do end target 'MessagesExtension' do pod 'GoogleAPIClient/Drive', '~> 1.0.2' pod 'GTMOAuth2', '~> 1.1.0' end Then, close the Xcode app completely and run the pod install command to install the SDK for you. A new workspace will be created. Open it instead of the Xcode project itself. Prepare the client key from the Google drive app you created as we mentioned in the Getting ready section, because we are going to use it in the Xcode project. Open MessagesViewController.swift and add the following import statements: import GoogleAPIClient import GTMOAuth2 Add the following private variables just below the class declaration and embed your client key in the kClientID constant, as shown: private let kKeychainItemName = "Drive API" private let kClientID = "Client_Key_Goes_HERE" private let scopes = [kGTLAuthScopeDrive] private let service = GTLServiceDrive() Add the following code in your class to request authentication to Google drive if it's not authenticated and load file info: override func viewDidLoad() { super.viewDidLoad() // Do any additional setup after loading the view. if let auth = GTMOAuth2ViewControllerTouch.authForGoogleFromKeychain(forName: kKeychainItemName, clientID: kClientID, clientSecret: nil) { service.authorizer = auth } } // When the view appears, ensure that the Drive API service is authorized // and perform API calls override func viewDidAppear(_ animated: Bool) { if let authorizer = service.authorizer, canAuth = authorizer.canAuthorize where canAuth { fetchFiles() } else { present(createAuthController(), animated: true, completion: nil) } } // Construct a query to get names and IDs of 10 files using the Google Drive API func fetchFiles() { print("Getting files...") if let query = GTLQueryDrive.queryForFilesList(){ query.fields = "nextPageToken, files(id, name, webViewLink, webContentLink, fileExtension)" service.executeQuery(query, delegate: self, didFinish: #selector(MessagesViewController.displayResultWithTicket(ticket:finishedWit hObject:error:))) } } // Parse results and display func displayResultWithTicket(ticket : GTLServiceTicket, finishedWithObject response : GTLDriveFileList, if let error = error { showAlert(title: "Error", message: error.localizedDescription) return } var filesString = "" let files = response.files as! [GTLDriveFile] if !files.isEmpty{ filesString += "Files:n" for file in files{ filesString += "(file.name) ((file.identifier) ((file.webViewLink) ((file.webContentLink))n" } } else { filesString = "No files found." } print(filesString) } // Creates the auth controller for authorizing access to Drive API private func createAuthController() -> GTMOAuth2ViewControllerTouch { let scopeString = scopes.joined(separator: " ") return GTMOAuth2ViewControllerTouch( scope: scopeString, clientID: kClientID, clientSecret: nil, keychainItemName: kKeychainItemName, delegate: self, finishedSelector: #selector(MessagesViewController.viewController(vc:finishedWithAuth:error:) ) ) } // Handle completion of the authorization process, and update the Drive API // with the new credentials. func viewController(vc : UIViewController, finishedWithAuth authResult : GTMOAuth2Authentication, error : NSError?) { if let error = error { service.authorizer = nil showAlert(title: "Authentication Error", message: error.localizedDescription) return } service.authorizer = authResult dismiss(animated: true, completion: nil) fetchFiles() } // Helper for showing an alert func showAlert(title : String, message: String) { let alert = UIAlertController( title: title, message: message, preferredStyle: UIAlertControllerStyle.alert ) let ok = UIAlertAction( title: "OK", style: UIAlertActionStyle.default, handler: nil ) alert.addAction(ok) self.present(alert, animated: true, completion: nil) } The code now requests authentication, loads files, and then prints them in the debug area. Now, try to build and run, you will see the following: Click on the arrow button in the bottom right corner to maximize the screen and try to log in with any Google account you have. Once the authentication is done, you will see the files' information printed in the debug area. Now, let's add a table view that will display the files' information and once a user selects a file, we will download this file to send it as an attachment to the conversation. Now, open theMainInterface.storyboard, drag a table view from Object Library, and add the following constraints: Set the delegate and data source of the table view from interface builder by dragging while holding down the Ctrl key to theMessagesViewController. Then, add an outlet to the table view, as follows, to be used to refresh the table with the files:  Drag a UITabeView cell from Object Library and drop it in the table view. For Attribute Inspector, set the cell style to Basic and the identifier to cell. Now, return to MessagesViewController.swift. Add the following property to hold the current display files: private var currentFiles = [GTLDriveFile]() Edit the displayResultWithTicket function to be like this: // Parse results and display func displayResultWithTicket(ticket : GTLServiceTicket, finishedWithObject response : GTLDriveFileList, error : NSError?) { if let error = error { showAlert(title: "Error", message: error.localizedDescription) return } var filesString = "" let files = response.files as! [GTLDriveFile] self.currentFiles = files if !files.isEmpty{ filesString += "Files:n" for file in files{ filesString += "(file.name) ((file.identifier) ((file.webViewLink) ((file.webContentLink))n" } } else { filesString = "No files found." } print(filesString) self.filesTableView.reloadData() } Now, add the following method for the table view delegate and data source: // MARK: - Table View methods - func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int { return self.currentFiles.count } func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell { let cell = tableView.dequeueReusableCell(withIdentifier: "cell") let file = self.currentFiles[indexPath.row] cell?.textLabel?.text = file.name return cell! } func tableView(_ tableView: UITableView, didSelectRowAt indexPath: IndexPath) { let file = self.currentFiles[indexPath.row] // Download File here to send as attachment. if let downloadURLString = file.webContentLink{ let url = NSURL(string: downloadURLString) if let name = file.name{ let downloadedPath = (documentsPath() as NSString).appendingPathComponent("(name)") let fetcher = service.fetcherService.fetcher(with: url as! URL) let destinationURL = NSURL(fileURLWithPath: downloadedPath) as URL fetcher.destinationFileURL = destinationURL fetcher.beginFetch(completionHandler: { (data, error) in if error == nil{ self.activeConversation?.insertAttachment(destinationURL, withAlternateFilename: name, completionHandler: nil) } }) } } } private func documentsPath() -> String{ let paths = NSSearchPathForDirectoriesInDomains(.documentDirectory, .userDomainMask, true) return paths.first ?? "" } Now, build and run the app, and you will see the magic: select any file and the app will download and save it to the local disk and send it as an attachment to the conversation, as illustrated: How it works… We started by installing the Google Drive SDK to the Xcode project. This SDK has all the APIs that we need to manage drive files and user authentication. When you visit the Google developers' website, you will see two options to install the SDK: manually or using CocoaPods. I totally recommend using CocoaPods to manage your dependencies as it is simple and efficient. Once the SDK has been installed via CocoaPods, we added some variables to be used for the Google Drive API and the most important one is the client key. You can access this value from the project you have created in the Google Developers Console. In the viewDidLoad function, first, we check if we have an authentication saved in KeyChain, and then, we use it. We can do that by calling GTMOAuth2ViewControllerTouch.authForGoogleFromKeychain, which takes the Keychain name and client key as parameters to search for authentication. It's useful as it helps you remember the last authentication and there is no need to ask for user authentication again if a user has already been authenticated before. In viewDidAppear, we check if a user is already authenticated; so, in that case, we start fetching files from the drive and, if not, we display the authentication controller, which asks a user to enter his Google account credentials. To display the authentication controller, we present the authentication view controller created in the createAuthController() function. In this function, the Google Drive API provides us with the GTMOAuth2ViewControllerTouch class, which encapsulates all logic for Google account authentication for your app. You need to pass the client key for your project, keychain name to save the authentication details there, and the finished  viewController(vc : UIViewController, finishedWithAuth authResult : GTMOAuth2Authentication, error : NSError?) selector that will be called after the authentication is complete. In that function, we check for errors and if something wrong happens, we display an alert message to the user. If no error occurs, we start fetching files using the fetchFiles() function. In the fetchFiles() function, we first create a query by calling GTLQueryDrive.queryForFilesList(). The GTLQueryDrive class has all the information you need about your query, such as which fields to read, for example, name, fileExtension, and a lot of other fields that you can fetch from the Google drive. You can specify the page size if you are going to call with pagination, for example, 10 by 10 files. Once you are happy with your query, execute it by calling service.executeQuery, which takes the query and the finished selector to be called when finished. In our example, it will call the displayResultWithTicket function, which prepares the files to be displayed in the table view. Then, we call self.filesTableView.reloadData() to refresh the table view to display the list of files. In the delegate function of table view didSelectRowAt indexPath:, we first read the webContentLink property from the GTLDriveFile instance, which is a download link for the selected file. To fetch a file from the Google drive, the API provides us with GTMSessionFetcher that can fetch a file and write it directly to a device's disk locally when you pass a local path to it. To create GTMSessionFetcher, use the service.fetcherService factory class, which gives you instance to a fetcher via the file URL. Then, we create a local path to the downloaded file by appending the filename to the documents path of your app and then, pass it to fetcher via the following command: fetcher.destinationFileURL = destinationURL Once you set up everything, call fetcher.beginFetch and pass a completion handler to be executed after finishing the fetching. Once the fetching is completed successfully, you can get a reference to the current conversation so that you can insert the file to it as an attachment. To do this, just call the following function: self.activeConversation?.insertAttachment(destinationURL, withAlternateFilename: name, completionHandler: nil) There's more… Yes, there's more that you can do in the preceding example to make it fancier and more appealing to users. Check the following options to make it better: You can show a loading indicator or progress bar while a file is downloading. Checks if the file is already downloaded, and if so, there is no need to download it again. Adding pagination to request only 10 files at a time. Options to filter documents by type, such as PDF, images, or even by date. Search for a file in your drive. Showing Progress indicator As we said, one of the features that we can add in the preceding example is the ability to show a progress bar indicating the downloading progress of a file. Before starting how to show a progress bar, let's install a library that is very helpful in managing/showing HUD indicators, which is MBProgressHUD. This library is available in GitHub at https://github.com/jdg/MBProgressHUD. As we agreed before, all packages are managed via CocoaPods, so now, let's install the library via CocoaPods, as shown: Open the Podfile and update it to be as follows: use_frameworks! target 'PDFPicker' do end target 'MessagesExtension' do pod 'GoogleAPIClient/Drive', '~> 1.0.2' pod 'GTMOAuth2', '~> 1.1.0' pod 'MBProgressHUD', '~> 1.0.0' end Run the following command to install the dependencies: pod install Now, at the top of the MessagesViewController.swift file, add the following import statement to import the library: Now, let's edit the didSelectRowAtIndexPath function to be like this: func tableView(_ tableView: UITableView, didSelectRowAt indexPath: IndexPath) { let file = self.currentFiles[indexPath.row] // Download File here to send as attachment. if let downloadURLString = file.webContentLink{ let url = NSURL(string: downloadURLString) if let name = file.name{ let downloadedPath = (documentsPath() as NSString).appendingPathComponent("(name)") let fetcher = service.fetcherService.fetcher(with: url as! URL) let destinationURL = NSURL(fileURLWithPath: downloadedPath) as URL fetcher.destinationFileURL = destinationURL var progress = Progress() let hud = MBProgressHUD.showAdded(to: self.view, animated: true) hud.mode = .annularDeterminate; hud.progressObject = progress fetcher.beginFetch(completionHandler: { (data, error) in if error == nil{ hud.hide(animated: true) self.activeConversation?.insertAttachment(destinationURL, withAlternateFilename: name, completionHandler: nil) } }) fetcher.downloadProgressBlock = { (bytes, written, expected) in let p = Double(written) * 100.0 / Double(expected) print(p) progress.totalUnitCount = expected progress.completedUnitCount = written } } } } First, we create an instance of MBProgressHUD and set its type to annularDeterminate, which means to display a circular progress bar. HUD will update its progress by taking a reference to the NSProgress object. Progress has two important variables to determine the progress value, which are totalUnitCount and completedUnitCount. These two values will be set inside the progress completion block, downloadProgressBlock, in the fetcher instance. HUD will be hidden in the completion block that will be called once the download is complete. Now build and run; after authentication, when you click on a file, you will see something like this: As you can see, the progressive view is updated with the percentage of download to give the user an overview of what is going on. Request files with pagination Loading all files at once is easy from the development side, but it's incorrect from the user experience side. It will take too much time at the beginning when you get the list of all the files and it would be great if we could request only 10 files at a time with pagination. In this section, we will see how to add the pagination concept to our example and request only 10 files at a time. When a user scrolls to the end of the list, we will display a loading indicator, call the next page, and append the results to our current results. Implementation of pagination is pretty easy and requires only a few changes in our code. Let's see how to do it: We will start by adding the progress cell design in MainInterface.storyboard. Open the design of MessagesViewController and drag a new cell along with our default cell. Drag a UIActivityIndicatorView from ObjectLibrary and place it as a subview to the new cell. Add center constraints to center it horizontally and vertically as shown: Now, select the new cell and go to attribute inspector to add an identifier to the cell and disable the selection as illustrated: Now, from the design side, we are ready. Open MessagesViewController.swift to add some tweaks to it. Add the following two variables to the list of our current variables: private var doneFetchingFiles = false private var nextPageToken: String! The doneFetchingFiles flag will be used to hide the progress cell when we try to load the next page from Google Drive and returns an empty list. In that case, we know that we are done with the fetching files and there is no need to display the progress cell any more. The nextPageToken contains the token to be passed to the GTLQueryDrive query to ask it to load the next page. Now, go to the fetchFiles() function and update it to be as shown: func fetchFiles() { print("Getting files...") if let query = GTLQueryDrive.queryForFilesList(){ query.fields = "nextPageToken, files(id, name, webViewLink, webContentLink, fileExtension)" query.mimeType = "application/pdf" query.pageSize = 10 query.pageToken = nextPageToken service.executeQuery(query, delegate: self, didFinish: #selector(MessagesViewController.displayResultWithTicket(ticket:finishedWit hObject:error:))) } } The only difference you can note between the preceding code and the one before that is setting the pageSize and pageToken. For pageSize, we set how many files we require for each call and for pageToken, we pass the token to get the next page. We receive this token as a response from the previous page call. This means that, at the first call, we don't have a token and it will be passed as nil. Now, open the displayResultWithTicket function and update it like this: // Parse results and display func displayResultWithTicket(ticket : GTLServiceTicket, finishedWithObject response : GTLDriveFileList, error : NSError?) { if let error = error { showAlert(title: "Error", message: error.localizedDescription) return } var filesString = "" nextPageToken = response.nextPageToken let files = response.files as! [GTLDriveFile] doneFetchingFiles = files.isEmpty self.currentFiles += files if !files.isEmpty{ filesString += "Files:n" for file in files{ filesString += "(file.name) ((file.identifier) ((file.webViewLink) ((file.webContentLink))n" } } else { filesString = "No files found." } print(filesString) self.filesTableView.reloadData() } As you can see, we first get the token that is to be used to load the next page. We get it by calling response.nextPageToken and setting it to our new  nextPageToken property so that we can use it while loading the next page. The doneFetchingFiles will be true only if the current page we are loading has no files, which means that we are done. Then, we append the new files we get to the current files we have. We don't know when to fire the calling of the next page. We will do this once the user scrolls down to the refresh cell that we have. To do so, we will implement one of the UITableViewDelegate methods, which is willDisplayCell, as illustrated: func tableView(_ tableView: UITableView, willDisplay cell: UITableViewCell, forRowAt indexPath: IndexPath) { if !doneFetchingFiles && indexPath.row == self.currentFiles.count { // Refresh cell fetchFiles() return } } For any cell that is going to be displayed, this function will be triggered with indexPath of the cell. First, we check if we are not done with the fetching files and the row is equal to the last row, then, we fire fetchFiles() again to load the next page. As we added a new refresh cell at the bottom, we should update our UITableViewDataSource functions, such as numbersOfRowsInSection and cellForRow. Check our updated functions, shown as follows: func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int { return doneFetchingFiles ? self.currentFiles.count : self.currentFiles.count + 1 } func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell { if !doneFetchingFiles && indexPath.row == self.currentFiles.count{ return tableView.dequeueReusableCell(withIdentifier: "progressCell")! } let cell = tableView.dequeueReusableCell(withIdentifier: "cell") let file = self.currentFiles[indexPath.row] cell?.textLabel?.text = file.name return cell! } As you can see, the number of rows will be equal to the current files' count plus one for the refresh cell. If we are done with the fetching files, we will return only the number of files. Now, everything seems perfect. When you build and run, you will see only 10 files listed, as shown: And when you scroll down you would see the progress cell that and 10 more files will be called. Summary In this article, we learned how to integrate iMessage app with iMessage app. Resources for Article: Further resources on this subject: iOS Security Overview [article] Optimizing JavaScript for iOS Hybrid Apps [article] Testing our application on an iOS device [article]
Read more
  • 0
  • 0
  • 32818

article-image-pros-and-cons-serverless-model
Packt
06 Apr 2017
22 min read
Save for later

Pros and Cons of the Serverless Model

Packt
06 Apr 2017
22 min read
In this article by Diego Zanon, author of the book Building Serverless Web Applications, we give you an introduction to the Serverless model and the pros and cons that you should consider before building a serverless application. (For more resources related to this topic, see here.) Introduction to the Serverless model Serverless can be a model, a kind of architecture, a pattern or anything else you prefer to call it. For me, serverless is an adjective, a word that qualifies a way of thinking. It's a way to abstract how the code that you write will be executed. Thinking serverless is to not think in servers. You code, you test, you deploy and that's (almost) enough. Serverless is a buzzword. You still need servers to run your applications, but you should not worry about them that much. Maintaining a server is none of your business. The focus is on the development, writing code, and not in the operation. DevOps is still necessary, although with a smaller role. You need to automate deployment and have at least a minimal monitoring of how your application is operating and costing, but you won't start or stop machines to match the usage and you'll neither replace failed instances nor apply security patches to the operating system. Thinking serverless A serverless solution is entirely request-driven. Every time that a user requests some information, a trigger will notify your cloud vendor to pick your code and execute it to retrieve the answer. In contrast, a traditional solution also works to answer requests, but the code is always up and running, consuming machine resources that were reserved specifically for you, even when no one is using your website. In a serverless architecture, it's not necessary to load the entire codebase into a running machine to process a single request. For a faster loading step, only the code that is necessary to answer the request is selected to run. This small piece of the solution is referenced to as a function. So we only run functions on demand. Although we call it simply as a function, it's usually a zipped package that contains a piece of code that runs as an entry point and all the modules that this code depends to execute. What makes the Serverless model so interesting is that you are only billed for the time that was needed to execute your function, which is usually measured in fractions of seconds, not hours of use. If no one is using your service, you pay nothing. Also, if you have a sudden peak of users accessing your app, the cloud service will load different instances to handle all simultaneous requests. If one of those cloud machines fails, another one will be made available automatically, without your interference. Serverless and PaaS Serverless is often confused with Platform as a Service (PaaS). PaaS is a kind of cloud computing model that allows developers to launch applications without worrying about the infrastructure. According to this definition, they have the same objective, which is true. Serverless is like a rebranding of PaaS or you can call it as the next generation of PaaS. The main difference between PaaS and Serverless is that in PaaS you don't manage machines, but you know that they exist. You are billed by provisioning them, even if there is no user actively browsing your website. In PaaS, your code is always running and waiting for new requests. In Serverless, there is a service that is listening for requests and will trigger your code to run only when necessary. This is reflected in your bill. You will pay only for the fractions of seconds that your code was executed and the number of requests that were made to this listener. IaaS and on-premises Besides PaaS, Serverless is frequently compared to Infrastructure as a Service (IaaS) and the on-premises solutions to expose its advantages. IaaS is another strategy to deploy cloud solutions where you hire virtual machines and is allowed to connect to them to configure everything that you need in the guest operating system. It gives you greater flexibility, but comes along with more responsibilities. You need to apply security patches, handle occasional failures and set up new servers to handle usage peaks. Also, you pay the same per hour if you are using 5% or 100% of the machine's CPU. On-premises are the traditional kind of solution where you buy the physical computers and run them inside your company. You get total flexibility and control with this approach. Hosting your own solution can be cheaper, but it happens only when your traffic usage is extremely stable. Over or under provisioning computers is so frequent that it's hard to have real gains using this approach, also when you add the risks and costs to hire a team to manage those machines. Cloud providers may look expensive, but several detailed use cases prove that the return on investment (ROI) is larger running on the cloud than on-premises. When using the cloud, you benefit from the economy of scale of many gigantic data centers. Running by your own exposes your business to a wide range of risks and costs that you'll never be able to anticipate. The main goals of serverless To define a service as serverless, it must have at least the following features: Scale as you need: There is no under or over provisioning Highly available: Fault-tolerant and always online Cost-efficient: Never pay for idle servers Scalability Regarding IaaS, you can achieve infinite scalability with any cloud service. You just need to hire new machines as your usage grows. You can also automate the process of starting and stopping servers as your demand changes. But this is not a fast way to scale. When you start a new machine, you usually need something like 5 minutes before it can be usable to process new requests. Also, as starting and stopping machines is costly, you only do this after you are certain that you need. So, your automated process will wait for some minutes to confirm that your demand has changed before taking any action. IaaS is able to handle well-behaved usage changes, but can't handle unexpected high peaks that happen after announcements or marketing campaigns. With serverless, your scalability is measured in seconds and not minutes. Besides being scalable, it's very fast to scale. Also, it scales per request, without needing to provision capacity. When you consider a high usage frequency in a scale of minutes, IaaS suffers to satisfy the needed capacity while serverless meets even higher usages in less time. In the following figure, the left graph shows how scalability occurs with IaaS. The right graph shows how well the demand can be satisfied using a serverless solution: With an on-premises approach, this is an even bigger problem. As the usage grows, new machines must be bought and prepared. However, increasing the infrastructure requires purchase orders to be created and approved, delay waiting the new servers to arrive and time for the team to configure and test them. It can take weeks to grow, or even months if the company is very big and requests many steps and procedures to be filled in. Availability A highly available solution is one that is fault-tolerant to hardware failures. If one machine goes out, you can keep running with a satisfactory performance. However, if you lose an entire data center due to a power outage, you need machines in another data center to keep the service online. It generally means duplicating your entire infrastructure by placing each half in a different data center. Highly available solutions are usually very expensive in IaaS and on-premises. If you have multiple machines to handle your workload, placing them in different physical places and running a load balance service can be enough. If one data center goes out, you keep the traffic in the remaining machines and scale to compensate. However, there are cases where you pay extra without using those machines. For example, if you have a huge relational database that scaled vertically, you will end up paying for another expensive machine just as a slave to keep the availability. Even for NoSQL databases, if you set a MongoDB replica set in a consistent model, you pay for instances that will act only as secondaries, without serving to alleviate read requests. Instead of running idle machines, you can set them in a cold start state, meaning that the machine is prepared, but is off to reduce costs. However, if you run a website that sells products or services, you can lose customers even in small downtimes. Cold start for web servers can take a few minutes to recover, but needs several minutes for databases. Considering these scenarios, you get high availability for free in serverless. The cost is already considered in what you pay to use. Another aspect of availability is how to handle Distributed Denial of Service (DDoS) attacks. When you receive a huge load of requests in a very short time, how do you handle it? There are some tools and techniques that help mitigate the problem, for example, blacklisting IPs that go over a specific request rate, but before those tools start to work, you need to scale the solution, and it needs to scale really fast to prevent the availability to be compromised. In this, again, serverless has the best scaling speed. Cost efficiency It's impossible to match the traffic usage with what you have provisioned. Considering IaaS or on-premises, as a rule of thumb, CPU and RAM usage must always be lower than 90% to consider the machine healthy, and it is desirable to have a CPU using less than 20% of the capacity on normal traffic. In this case, you are paying for 80% of the waste, where the capacity is in an idle state. Paying for computer resources that you don't use is not efficient. Many cloud providers advertise that you just pay for what you use, but they usually offer significant discounts when you provision for 24 hours of usage in long term (one year or more). This means that you pay for machines that will keep running even in very low traffic hours. Also, even if you want to shut down machines in hours with very low traffic, you need to keep at least a minimum infrastructure 24/7 to keep you web server and databases always online. Regarding high availability, you need extra machines to add redundancy. Again, it's a waste of resources. Another efficiency problem is related with the databases, especially relational ones. Scaling vertically is a very troublesome task, so relational databases are always provisioned considering max peaks. It means that you pay for an expensive machine when most of the time you don't need one. In serverless, you shouldn't worry about provisioning or idle times. You should pay exactly for the CPU and RAM time that is used, measured in fractions of seconds and not hours. If it's a serverless database, you need to store data permanently, so this represents a cost even if no one is using your system. However, storage is usually very cheap. The higher cost, which is related with the database engine that runs queries and manipulates data, will only be billed by the amount of time used without considering idle times. Running a serverless system continuously for one hour has a much higher cost than one hour in a traditional system. However, the difference is that it will never keep one machine with 100% for one hour straight. The cost efficiency of serverless is perceived clearer in websites with varying traffic and not in the ones with flat traffic. Pros We can list the following strengths for the Serverless model: Fast scalability High availability Efficient usage of resources Reduced operational costs Focus on business, not on infrastructure System security is outsourced Continuous delivery Microservices friendly Cost model is startup friendly Let's skip the first three benefits, since they were already covered in the previous section and take a look into the others. Reduced operational costs As the infrastructure is fully managed by the cloud vendor, it reduces the operational costs since you don't need to worry about hardware failures, neither applying security patches to the operating system, nor fixing network issues. It effectively means that you need to spend less sysadmin hours to keep your application running. Also, it helps reducing risks. If you make an investment to deploy a new service and that ends up as a failure, you don't need to worry about selling machines or how to dispose of the data center that you have built. Focus on business Lean software development advocates that you must spend time in what aggregates value to the final product. In a serverless project, the focus is on business. Infrastructure is a second-class citizen. Configuring a large infrastructure is a costly and time consuming task. If you want to validate an idea through a Minimum Viable Product (MVP), without losing time to market, consider using serverless to save time. There are tools that automate the deployment, such as the Serverless Framework, which helps the developer to launch a prototype with minimum effort. If the idea fails, infrastructure costs are minimized since there are no payments in advance. System security The cloud vendor is responsible to manage the security of the operating system, runtime, physical access, networking, and all related technologies that enable the platform to operate. The developer still needs to handle authentication, authorization, and code vulnerabilities, but the rest is outsourced to the cloud provider. It's a positive feature if you consider that a large team of specialists is focused to implement the best security practices and patches new bug fixes as soon as possible to serve hundreds of their customers. That's economy of scale. Continuous delivery Serverless is based on breaking a big project into dozens of packages, each one represented by a top-level function that handles requests. Deploying a new version of a function means uploading a ZIP package to replacing the previous one and updating the endpoint configuration that specifies how this function can be triggered. Executing this task manually, for dozens of functions, is an exhaustive task. Automation is a must-have feature when working in a serverless project. For this task, we can use the Serverless Framework, which helps developers to manage and organize solutions, making a deployment task as simple as executing a one-line command. With automation, continuous delivery is a consequence that brings many benefits like the ability to deploy short development cycles and easier rollbacks anytime. Another related benefit when the deployment is automated is the creation of different environments. You can create a new test environment, which is an exact duplicate of the development environment, using simple commands. The ability to replicate the environment is very important to favor acceptance tests and deployment to production. Microservices friendly A microservices architecture is encouraged in a serverless project. As your functions are single units of deployment, you can have different teams working concurrently on different use cases. You can also use different programming languages in the same project and take advantage of emerging technologies or team skills. Cost model Suppose that you have built a serverless store. The average user will make some requests to see a few products and a few more requests to decide if they will buy something or not. In serverless, a single unit of code has a predictable time to execute for a given input. After collecting some data, you can predict how much a single user costs in average, and this unit cost will almost be constant as your application grows in usage. Knowing how much a single user cost and keeping this number fixed is very important for a startup. It helps to decide how much you need to charge for a service or earn through ads or sales to make a profit. In a traditional infrastructure, you need to make payments in advance, and scaling your application means increasing your capacity in steps. So, calculating the unit cost of a user is more difficult and it's a variable number. The following chart makes this difference easier to understand: Cons Serverless is great, but no technology is a silver bullet. You should be aware of the following issues: Higher latency Constraints Hidden inefficiencies Vendor dependency Debugging Atomic deploys Uncertainties We will discuss these drawbacks in detail now. Higher latency Serverless is request-driven and your code is not running all the time. When a request is made, it triggers a service that finds your function, unzips the package, loads into a container, and makes it available to be executed. The problem is that those steps takes time—up to a few hundreds of milliseconds. This issue is called cold start delay and is a trade-off that exists between serverless cost-effective model and a lower latency of traditional hosting. There are some solutions to fix this performance problem. For example, you can configure your function to reserve more RAM memory. It gives a faster start and overall performance. The programming language is also important. Java has a much higher cold start time then JavaScript (Node.js). Another solution is to benefit from the fact that the cloud provider may cache the loaded code, which means that the first execution will have a delay but further requests will benefit by a smaller latency. You can optimize a serverless function by aggregating a large number of functionalities into a single function. The consequence is that this package will be executed with a higher frequency and will frequently skip the cold start issue. The problem is that a big package will take more time to load and provoke a higher first start time. At a last resort, you could schedule another service to ping your functions periodically, like one time per couple of minutes, to prevent them to be put to sleep. It will add costs, but remove the cold start problem. There is also a concept of serverless databases that references services where the database is fully managed by the vendor and it charges only the storage and the time to execute the database engine. Those solutions are wonderful, but they add even more delay for your requests when compared with traditional databases. Proceed with caution when selecting those. Constraints If you go serverless, you need to know what the vendor constraints are. For example, on Amazon Web Services (AWS), you can't run a Lambda function for more than 5 minutes. It makes sense because if you're doing this, you are using it wrong. Serverless was designed to be cost efficient in short bursts. For constant and predictable processing, it will be expensive. Another constraint on AWS Lambda is the number of concurrent executions across all functions within a given region. Amazon limits this to 100. Suppose that your functions need 100 milliseconds in average to execute. In this scenario, you can handle up to 1000 users per second. The reasoning behind this restriction is to avoid excessive costs due to programming errors that may create potential runways or recursive iterations. AWS Lambda has a default limit of 100 concurrent executions. However, you can file a case into AWS Support Center to raise this limit. If you say that your application is ready for production and that you understand the risks, they will happily increase this value. When monitoring your Lambda functions using Amazon CloudWatch, there is an option called Throttles. Each invocation that exceeds the safety limit of concurrent calls is counted as one throttle. Hidden inefficiencies Some people are calling serverless as a NoOps solution. That's not true. DevOps is still necessary. You don't need to worry much about servers because they are second-class citizens and the focus is on your business. However, adding metrics and monitoring your applications will always be a good practice. It's so easy to scale that a specific function may be deployed with a poor performance that takes much more time than necessary and remains unnoticed forever because no one is monitoring the operation. Also, over or under provisioning is also possible (in a smaller sense) since you need to configure your function setting the amount of RAM memory that it will reserve and the threshold to timeout the execution. It's a very different scale of provisioning, but you need to keep it in mind to avoid mistakes. Vendor dependency When you build a serverless solution, you trust your business in a third-party vendor. You should be aware that companies fail and you can suffer downtimes, security breaches, and performance issues. Also, the vendor may change the billing model, increase costs, introduce bugs into their services, have poor documentations, modify an API forcing you to upgrade, and terminate services. A whole bunch of bad things may happen. What you need to weigh is whether it's worth trusting in another company or making a big investment to build all by yourself. You can mitigate these problems by doing a market search before selecting a vendor. However, you still need to count on luck. For example, Parse was a vendor that offered managed services with really nice features. It was bought by Facebook in 2013, which gave more reliability due to the fact that it was backed by a big company. Unfortunately, Facebook decided to shutdown all servers in 2016 giving one year of notice for customers to migrate to other vendors. Vendor lock-in is another big issue. When you use cloud services, it's very likely that one specific service has a completely different implementation than another vendor, making those two different APIs. You need to rewrite code in case you decide to migrate. It's already a common problem. If you use a managed service to send e-mails, you need to rewrite part of your code before migrating to another vendor. What raises a red flag here is that a serverless solution is entirely based into one vendor and migrating the entire codebase can be much more troublesome. To mitigate this problem, some tools like the Serverless Framework are moving to include multivendor support. Currently, only AWS is supported but they expect to include Microsoft, Google, and IBM clouds in the future, without requiring code rewrites to migrate. Multivendor support represents safety for your business and gives power to competitiveness. Debugging Unit testing a serverless solution is fairly simple because any code that your functions relies on can be separated into modules and unit tested. Integration tests are a little bit more complicated because you need to be online to test with external services. You can build stubs, but you lose some fidelity in your tests. When it comes to debugging to test a feature or fix an error, it's a whole different problem. You can't hook into an external service and make slow processing steps to see how your code behaves. Also, those serverless APIs are not open source, so you can't run them in-house for testing. All you have is the ability to log steps, which is a slow debugging approach, or you can extract the code and adapt it to host into your own servers and make local calls. Atomic deploys Deploying a new version of a serverless function is easy. You update the code and the next time that a trigger requests this function, your newly deployed code will be selected to run. This means that, for a brief moment, two instances of the same function can be executed concurrently with different implementations. Usually, that's not a problem, but when you deal with persistent storage and databases, you should be aware that a new piece of code can insert data into a format that an old version can't understand. Also, if you want to deploy a function that relies on a new implementation of another function, you need to be careful in the order that you deploy those functions. Ordering is often not secured by the tools that automate the deployment process. The problem here is that current serverless implementations consider that deployment is an atomic process for each function. You can't batch deploy a group of functions atomically. You can mitigate this issue by disabling the event source while you deploy a specific group, but that means introducing a downtime into the deployment process, or you can use a monolith approach instead of a microservices architecture for serverless applications. Uncertainties Serverless is still a pretty new concept. Early adopters are braving this field testing what works and which kind patterns and technologies can be used. Emerging tools are defining the development process. Vendors are releasing and improving new services. There are high expectations for the future, but the future hasn't come yet. Some uncertainties still worry developers when it comes to build large applications. Being a pioneer can be rewarding, but risky. Technical debt is a concept that compares software development with finances. The easiest solution in the short run is not always the best overall solution. When you take a bad decision in the beginning, you pay later with extra hours to fix it. Software is not perfect. Every single architecture has pros and cons that append technical debt in the long run. The question is: how much technical debt does serverless aggregate to the software development process? Is it more, less, or equivalent to the kind of architecture that you are using today? Summary In this article, you learned about the Serverless model and how it is different from other traditional approaches. You already know what the main benefits and advantages are that it may offer for your next application. Also, you are aware that no technology is a silver bullet. You know what kind of problems you may have with serverless and how mitigate some of them. Resources for article Refer to Building Serverless Architectures, available at https://www.packtpub.com/application-development/building-serverless-architectures for further resources on this subject. Resources for Article: Further resources on this subject: Deployment and DevOps [article] Customizing Xtext Components [article] Heads up to MvvmCross [article]
Read more
  • 0
  • 0
  • 2049

article-image-introduction-network-security
Packt
06 Apr 2017
18 min read
Save for later

Introduction to Network Security

Packt
06 Apr 2017
18 min read
In this article by Warun Levesque, Michael McLafferty, and Arthur Salmon, the authors of the book Applied Network Security, we will be covering the following topics, which will give an introduction to network security: Murphy's law The definition of a hacker and the types The hacking process Recent events and statistics of network attacks Security for individual versus company Mitigation against threats Building an assessment This world is changing rapidly with advancing network technologies. Unfortunately, sometimes the convenience of technology can outpace its security and safety. Technologies like the Internet of things is ushering in a new era of network communication. We also want to change the mindset of being in the field of network security. Most current cyber security professionals practice defensive and passive security. They mostly focus on mitigation and forensic tactics to analyze the aftermath of an attack. We want to change this mindset to one of Offensive security. This article will give insight on how a hacker thinks and what methods they use. Having knowledge of a hacker's tactics, will give the reader a great advantage in protecting any network from attack. (For more resources related to this topic, see here.) Murphy's law Network security is much like Murphy's law in the sense that, if something can go wrong it will go wrong. To be successful at understanding and applying network security, a person must master the three Ps. The three Ps are, persistence, patience, and passion. A cyber security professional must be persistent in their pursuit of a solution to a problem. Giving up is not an option. The answer will be there; it just may take more time than expected to find it. Having patience is also an important trait to master. When dealing with network anomalies, it is very easy to get frustrated. Taking a deep breath and keeping a cool head goes a long way in finding the correct solution to your network security problems. Finally, developing a passion for cyber security is critical to being a successful network security professional. Having that passion will drive you to learn more and evolve yourself on a daily basis to be better. Once you learn, then you will improve and perhaps go on to inspire others to reach similar aspirations in cyber security. The definition of a hacker and the types A hacker is a person who uses computers to gain unauthorized access to data. There are many different types of hackers. There are white hat, grey hat, and black hat hackers. Some hackers are defined for their intention. For example, a hacker that attacks for political reasons may be known as a hacktivist. A white hat hackers have no criminal intent, but instead focuses on finding and fixing network vulnerabilities. Often companies will hire a white hat hacker to test the security of their network for vulnerabilities. A grey hat hacker is someone who may have criminal intent but not often for personal gain. Often a grey hat will seek to expose a network vulnerability without the permission from the owner of the network. A black hat hacker is purely criminal. They sole objective is personal gain. Black hat hackers take advantage of network vulnerabilities anyway they can for maximum benefit. A cyber-criminal is another type of black hat hacker, who is motivated to attack for illegal financial gain. A more basic type of hacker is known as a script kiddie. A script kiddie is a person who knows how to use basic hacking tools but doesn't understand how they work. They often lack the knowledge to launch any kind of real attack, but can still cause problems on a poorly protected network. Hackers tools There are a range of many different hacking tools. A tool like Nmap for example, is a great tool for both reconnaissance and scanning for network vulnerabilities. Some tools are grouped together to make toolkits and frameworks, such as the Social Engineering Toolkit and Metasploit framework. The Metasploit framework is one of the most versatile and supported hacking tool frameworks available. Metasploit is built around a collection of highly effective modules, such as MSFvenom and provides access to an extensive database of exploits and vulnerabilities. There are also physical hacking tools. Devices like the Rubber Ducky and Wi-Fi Pineapple are good examples. The Rubber Ducky is a usb payload injector, that automatically injects a malicious virus into the device it's plugged into. The Wi-Fi Pineapple can act as a rogue router and be used to launch man in the middle attacks. The Wi-Fi Pineapple also has a range of modules that allow it to execute multiple attack vectors. These types of tools are known as penetration testing equipment. The hacking process There are five main phases to the hacking process: Reconnaissance: The reconnaissance phase is often the most time consuming. This phase can last days, weeks, or even months sometimes depending on the target. The objective during the reconnaissance phase is to learn as much as possible about the potential target. Scanning: In this phase the hacker will scan for vulnerabilities in the network to exploit. These scans will look for weaknesses such as, open ports, open services, outdated applications (including operating systems), and the type of equipment being used on the network. Access: In this phase the hacker will use the knowledge gained in the previous phases to gain access to sensitive data or use the network to attack other targets. The objective of this phase is to have the attacker gain some level of control over other devices on the network. Maintaining access: During this phase a hacker will look at various options, such as creating a backdoor to maintain access to devices they have compromised. By creating a backdoor, a hacker can maintain a persistent attack on a network, without fear of losing access to the devices they have gained control over. Although when a backdoor is created, it increases the chance of a hacker being discovered. Backdoors are noisy and often leave a large footprint for IDS to follow. Covering your tracks: This phase is about hiding the intrusion of the network by the hacker as to not alert any IDS that may be monitoring the network. The objective of this phase is to erase any trace that an attack occurred on the network. Recent events and statistics of network attacks The news has been full of cyber-attacks in recent years. The number and scale of attacks are increasing at an alarming rate. It is important for anyone in network security to study these attacks. Staying current with this kind of information will help in defending your network from similar attacks. Since 2015, the medical and insurance industry have been heavily targeted for cyber-attacks. On May 5th, 2015 Premera Blue Cross was attacked. This attack is said to have compromised at least 11 million customer accounts containing personal data. The attack exposed customer's names, birth dates, social security numbers, phone numbers, bank account information, mailing, and e-mail addresses. Another attack that was on a larger scale, was the attack on Anthem. It is estimated that 80 million personal data records were stolen from customers, employees, and even the Chief Executive Officer of Anthem. Another more infamous cyber-attack recently was the Sony hack. This hack was a little different than the Anthem and Blue Cross attacks, because it was carried out by hacktivist instead of cyber-criminals. Even though both types of hacking are criminal, the fundamental reasoning and objectives of the attacks are quite different. The objective in the Sony attack was to disrupt and embarrass the executives at Sony as well as prevent a film from being released. No financial data was targeted. Instead the hackers went after personal e-mails of top executives. The hackers then released the e-mails to the public, causing humiliation to Sony and its executives. Many apologies were issued by Sony in the following weeks of the attack. Large commercial retailers have also been a favorite target for hackers. An attack occurred against Home Depot in September 2014. That attack was on a large scale. It is estimated that over 56 million credit cards were compromised during the Home Depot attack. A similar attack but on a smaller scale was carried out against Staples in October 2014. During this attack, over 1.4 million credit card numbers were stolen. The statistics on cyber security attacks are eye opening. It is estimated by some experts that cybercrime has a worldwide cost of 110 billion dollars a year. In a given year, over 15 million Americans will have their identified stolen through cyber-attacks, it is also estimated that 1.5 million people fall victim to cybercrime every day. These statistics are rapidly increasing and will continue to do so until more people take an active interest in network security. Our defense The baseline for preventing a potential security issues typically begins with hardening the security infrastructure, including firewalls, DMZ, and physical security platform. Entrusting only valid sources or individuals with personal data and or access to that data. That also includes being compliant with all regulations that apply to a given situation or business. Being aware of the types of breaches as well as your potential vulnerabilities. Also understanding is an individual or an organization a higher risk target for attacks. The question has to be asked, does one's organization promote security? This is done both at the personal and the business level to deter cyber-attacks? After a decade of responding to incidents and helping customers recover from and increase their resilience against breaches. Organization may already have a security training and awareness (STA) program, or other training and program could have existed. As the security and threat landscape evolves organizations and individuals need to continually evaluate practices that are required and appropriate for the data they collect, transmit, retain, and destroy. Encryption of data at rest/in storage and in transit is a fundamental security requirement and the respective failure is frequently being cited as the cause for regulatory action and lawsuits. Enforce effective password management policies. Least privilege user access (LUA) is a core security strategy component, and all accounts should run with as few privileges and access levels as possible. Conduct regular security design and code reviews including penetration tests and vulnerability scans to identify and mitigate vulnerabilities. Require e-mail authentication on all inbound and outbound mail servers to help detect malicious e-mail including spear phishing and spoofed e-mail. Continuously monitor in real-time the security of your organization's infrastructure including collecting and analyzing all network traffic, and analyzing centralized logs (including firewall, IDS/IPS, VPN, and AV) using log management tools, as well as reviewing network statistics. Identify anomalous activity, investigate, and revise your view of anomalous activity accordingly. User training would be the biggest challenge, but is arguably the most important defense. Security for individual versus company One of the fundamental questions individuals need to ask themselves, is there a difference between them the individual and an organization? Individual security is less likely due to attack service area. However, there are tools and sites on the internet that can be utilized to detect and mitigate data breaches for both.https://haveibeenpwned.com/ or http://map.norsecorp.com/ are good sites to start with. The issue is that individuals believe they are not a target because there is little to gain from attacking individuals, but in truth everyone has the ability to become a target. Wi-Fi vulnerabilities Protecting wireless networks can be very challenging at times. There are many vulnerabilities that a hacker can exploit to compromise a wireless network. One of the basic Wi-Fi vulnerabilities is broadcasting the Service Set Identifier (SSID) of your wireless network. Broadcasting the SSID makes the wireless network easier to find and target. Another vulnerability in Wi-Fi networks is using Media Access Control (MAC) addressesfor network authentication. A hacker can easily spoof or mimic a trusted MAC address to gain access to the network. Using weak encryption such as Wired Equivalent Privacy (WEP) will make your network an easy target for attack. There are many hacking tools available to crack any WEP key in under five minutes. A major physical vulnerability in wireless networks are the Access Points (AP). Sometimes APs will be placed in poor locations that can be easily accessed by a hacker. A hacker may install what is called a rogue AP. This rogue AP will monitor the network for data that a hacker can use to escalate their attack. Often this tactic is used to harvest the credentials of high ranking management personnel, to gain access to encrypted databases that contain the personal/financial data of employees and customers or both. Peer-to-peer technology can also be a vulnerability for wireless networks. A hacker may gain access to a wireless network by using a legitimate user as an accepted entry point. Not using and enforcing security policies is also a major vulnerability found in wireless networks. Using security tools like Active Directory (deployed properly) will make it harder for a hacker to gain access to a network. Hackers will often go after low hanging fruit (easy targets), so having at least some deterrence will go a long way in protecting your wireless network. Using Intrusion Detection Systems (IDS) in combination with Active Directory will immensely increase the defense of any wireless network. Although the most effective factor is, having a well-trained and informed cyber security professional watching over the network. The more a cyber security professional (threat hunter) understands the tactics of a hacker, the more effective that Threat hunter will become in discovering and neutralizing a network attack. Although there are many challenges in protecting a wireless network, with the proper planning and deployment those challenges can be overcome. Knowns and unknowns The toughest thing about an unknown risk to security is that they are unknown. Unless they are found they can stay hidden. A common practice to determine an unknown risk would be to identify all the known risks and attempt to mitigate them as best as possible. There are many sites available that can assist in this venture. The most helpful would be reports from CVE sites that identify vulnerabilities. False positives   Positive Negative True TP: correctly identified TN: correctly rejected False FP: incorrectly identified FN: incorrectly rejected As it related to detection for an analyzed event there are four situations that exist in this context, corresponding to the relation between the result of the detection for an analyzed event. In this case, each of the corresponding situations mentioned in the preceding table are outlined as follows: True positive (TP): It is when the analyzed event is correctly classified as intrusion or as harmful/malicious. For example, a network security administrator enters their credentials into the Active Directory server and is granted administrator access. True negative (TN): It is when the analyzed event is correctly classified and correctly rejected. For example, an attacker uses a port like 4444 to communicate with a victim's device. An intrusion detection system detects network traffic on the authorized port and alerts the cyber security team to this potential malicious activity. The cyber security team quickly closes the port and isolates the infected device from the network. False positive (FP): It is when the analyzed event is innocuous or otherwise clean as it relates to perspective of security, however, the system classifies it as malicious or harmful. For example, a user types their password into a website's login text field. Instead of being granted access, the user is flagged for an SQL injection attempt by input sanitation. This is often caused when input sanitation is misconfigured. False negative (FN): It is when the analyzed event is malicious but it is classified as normal/innocuous. For example, an attacker inputs an SQL injection string into a text field found on a website to gain unauthorized access to database information. The website accepts the SQL injection as normal user behavior and grants access to the attacker. As it relates to detection, having systems correctly identify the given situations in paramount. Mitigation against threats There are many threats that a network faces. New network threats are emerging all the time. As a network security, professional, it would be wise to have a good understanding of effective mitigation techniques. For example, a hacker using a packet sniffer can be mitigated by only allowing the network admin to run a network analyzer (packet sniffer) on the network. A packet sniffer can usually detect another packet sniffer on the network right away. Although, there are ways a knowledgeable hacker can disguise the packet sniffer as another piece of software. A hacker will not usually go to such lengths unless it is a highly-secured target. It is alarming that; most businesses do not properly monitor their network or even at all. It is important for any business to have a business continuity/disaster recovery plan. This plan is intended to allow a business to continue to operate and recover from a serious network attack. The most common deployment of the continuity/disaster recovery plan is after a DDoS attack. A DDoS attack could potentially cost a business or organization millions of dollars is lost revenue and productivity. One of the most effective and hardest to mitigate attacks is social engineering. All the most devastating network attacks have begun with some type of social engineering attack. One good example is the hack against Snapchat on February 26th, 2016. "Last Friday, Snapchat's payroll department was targeted by an isolated e-mail phishing scam in which a scammer impersonated our Chief Executive Officer and asked for employee payroll information," Snapchat explained in a blog post. Unfortunately, the phishing e-mail wasn't recognized for what it was — a scam — and payroll information about some current and former employees was disclosed externally. Socially engineered phishing e-mails, same as the one that affected Snapchat are common attack vectors for hackers. The one difference between phishing e-mails from a few years ago, and the ones in 2016 is, the level of social engineering hackers are putting into the e-mails. The Snapchat HR phishing e-mail, indicated a high level of reconnaissance on the Chief Executive Officer of Snapchat. This reconnaissance most likely took months. This level of detail and targeting of an individual (Chief Executive Officer) is more accurately know as a spear-phishing e-mail. Spear-phishing campaigns go after one individual (fish) compared to phishing campaigns that are more general and may be sent to millions of users (fish). It is like casting a big open net into the water and seeing what comes back. The only real way to mitigate against social engineering attacks is training and building awareness among users. By properly training the users that access the network, it will create a higher level of awareness against socially engineered attacks. Building an assessment Creating a network assessment is an important aspect of network security. A network assessment will allow for a better understanding of where vulnerabilities may be found within the network. It is important to know precisely what you are doing during a network assessment. If the assessment is done wrong, you could cause great harm to the network you are trying to protect. Before you start the network assessment, you should determine the objectives of the assessment itself. Are you trying to identify if the network has any open ports that shouldn't be? Is your objective to quantify how much traffic flows through the network at any given time or a specific time? Once you decide on the objectives of the network assessment, you will then be able to choose the type of tools you will use. Network assessment tools are often known as penetration testing tools. A person who employs these tools is known as a penetration tester or pen tester. These tools are designed to find and exploit network vulnerabilities, so that they can be fixed before a real attack occurs. That is why it is important to know what you are doing when using penetration testing tools during an assessment. Sometimes network assessments require a team. It is important to have an accurate idea of the scale of the network before you pick your team. In a large enterprise network, it can be easy to become overwhelmed by tasks to complete without enough support. Once the scale of the network assessment is complete, the next step would be to ensure you have written permission and scope from management. All parties involved in the network assessment must be clear on what can and cannot be done to the network during the assessment. After the assessment is completed, the last step is creating a report to educate concerned parties of the findings. Providing detailed information and solutions to vulnerabilities will help keep the network up to date on defense. The report will also be able to determine if there are any viruses lying dormant, waiting for the opportune time to attack the network. Network assessments should be conducted routinely and frequently to help ensure strong networksecurity. Summary In this article we covered the fundamentals of network security. It began by explaining the importance of having network security and what should be done to secure the network. It also covered the different ways physical security can be applied. The importance of having security policies in place and wireless security was discussed. This article also spoke about wireless security policies and why they are important. Resources for Article: Further resources on this subject: API and Intent-Driven Networking [article] Deploying First Server [article] Point-to-Point Networks [article]
Read more
  • 0
  • 0
  • 23871
article-image-systems-and-logics
Packt
06 Apr 2017
19 min read
Save for later

Systems and Logics

Packt
06 Apr 2017
19 min read
In this article by Priya Kuber, Rishi Gaurav Bhatnagar, and Vijay Varada, authors of the book Arduino for Kids explains structure and various components of a code: How does a code work What is code What is a System How to download, save and access a file in the Arduino IDE (For more resources related to this topic, see here.) What is a System? Imagine system as a box which in which a process is completed. Every system is solving a larger problem, and can be broken down into smaller problems that can be solved and assembled. Sort of like a Lego set! Each small process has 'logic' as the backbone of the solution. Logic, can be expressed as an algorithm and implemented in code. You can design a system to arrive at solutions to a problem. Another advantage to breaking down a system into small processes is that in case your solution fails to work, you can easily spot the source of your problem, by checking if your individual processes work. What is Code? Code is a simple set of written instructions, given to a specific program in a computer, to perform a desired task. Code is written in a computer language. As we all know by now, a computer is an intelligent, electronic device capable of solving logical problems with a given set of instructions. Some examples of computer languages are Python, Ruby, C, C++ and so on. Find out some more examples of languages from the internet and write it down in your notebook. What is an Algorithm? A logical set by step process, guided by the boundaries (or constraints) defined by a problem, followed to find a solution is called an algorithm. In a better and more pictorial form, it can be represented as follows: (Logic + Control = Algorithm) (A picture depicting this equation) What does that even mean? Look at the following example to understand the process. Let's understand what an algorithm means with the help of an example. It's your friend's birthday and you have been invited for the party (Isn't this exciting already?). You decide to gift her something. Since it's a gift, let's wrap it. What would you do to wrap the gift? How would you do it? Look at the size of the gift Fetch the gift wrapping paper Fetch the scissors Fetch the tape Then you would proceed to place the gift inside the wrapping paper. You will start start folding the corners in a way that it efficiently covers the Gift. In the meanwhile, to make sure that your wrapping is tight, you would use a scotch tape. You keep working on the wrapper till the whole gift is covered (and mind you, neatly! you don't want mommy scolding you, right?). What did you just do? You used a logical step by step process to solve a simple task given to you. Again coming back to the sentence: (Logic + Control = Algorithm) 'Logic' here, is the set of instructions given to a computer to solve the problem. 'Control' are the words making sure that the computer understands all your boundaries. Logic Logic is the study of reasoning and when we add this to the control structures, they become algorithms. Have you ever watered the plants using a water pipe or washed a car with it? How do you think it works? The pipe guides the water from the water tap to the car. It makes sure optimum amount of water reaches the end of the pipe. A pipe is a control structure for water in this case. We will understand more about control structures in the next topic. How does a control structure work? A very good example to understand how a control structure works, is taken from wikiversity. (https://en.wikiversity.org/wiki/Control_structures) A precondition is the state of a variable before entering a control structure. In the gift wrapping example, the size of the gift determines the amount of gift wrapping paper you will use. Hence, it is a condition that you need to follow to successfully finish the task. In programming terms, such condition is called precondition. Similarly, a post condition is the state of the variable after exiting the control structure. And a variable, in code, is an alphabetic character, or a set of alphabetic characters, representing or storing a number, or a value. Some examples of variables are x, y, z, a, b, c, kitten, dog, robot Let us analyze flow control by using traffic flow as a model. A vehicle is arriving at an intersection. Thus, the precondition is the vehicle is in motion. Suppose the traffic light at the intersection is red. The control structure must determine the proper course of action to assign to the vehicle. Precondition: The vehicle is in motion. Control Structure: Is the traffic light green? If so, then the vehicle may stay in motion. Is the traffic light red? If so, then the vehicle must stop. End of Control Structure: Post condition: The vehicle comes to a stop. Thus, upon exiting the control structure, the vehicle is stopped. If you wonder where you learnt to wrap the gift, you would know that you learnt it by observing other people doing a similar task through your eyes. Since our microcontroller does not have eyes, we need to teach it to have a logical thinking using Code. The series of logical steps that lead to a solution is called algorithm as we saw in the previous task. Hence, all the instructions we give to a micro controller are in the form of an algorithm. A good algorithm solves the problem in a fast and efficient way. Blocks of small algorithms form larger algorithms. But algorithm is just code! What will happen when you try to add sensors to your code? A combination of electronics and code can be called a system. Picture: (block diagram of sensors + Arduino + code written in a dialogue box ) Logic is universal. Just like there can be multiple ways to fold the wrapping paper, there can be multiple ways to solve a problem too! A micro controller takes the instructions only in certain languages. The instructions then go to a compiler that translates the code that we have written to the machine. What language does your Arduino Understand? For Arduino, we will use the language 'processing'. Quoting from processing.org, Processing is a flexible software sketchbook and a language for learning how to code within the context of the visual arts. Processing is an open source programming language and integrated development environment (IDE). Processing was originally built for designers and it was extensively used in electronics arts and visual design communities with the sole purpose of teaching the fundamentals of computer sciences in a visual context. This also served as the foundations of electronic sketchbooks. From the previous example of gift wrapping, you noticed that before you need to bring in the paper and other stationery needed, you had to see the size of the problem at hand (the gift). What is a Library? In computer language, the stationery needed to complete your task, is called "Library". A library is a collection of reusable code that a programmer can 'call' instead of writing everything again. Now imagine if you had to cut a tree, make paper, then color the paper into the beautiful wrapping paper that you used, when I asked you to wrap the gift. How tiresome would it be? (If you are inventing a new type of paper, sure, go ahead chop some wood!) So before writing a program, you make sure that you have 'called' all the right libraries. Can you search the internet and make a note of a few arduino libraries in your inventor's diary? Please remember, that libraries are also made up of code! As your next activity, we will together learn more about how a library is created. Activity: Understanding the Morse Code During the times before the two-way mobile communication, people used a one-way communication called the Morse code. The following image is the experimental setup of a Morse code. Do not worry; we will not get into how you will perform it physically, but by this example, you will understand how your Arduino will work. We will show you the bigger picture first and then dissect it systematically so that you understand what a code contains. The Morse code is made up of two components "Short" and "Long" signals. The signals could be in the form of a light pulse or sound. The following image shows how the Morse code looks like. A dot is a short signal and a dash is a long signal. Interesting, right? Try encrypting your message for your friend with this dots and dashes. For example, "Hello" would be: The image below shows how the Arduino code for Morse code will looks like. The piece of code in dots and dashes is the message SOS that I am sure you all know, is an urgent appeal for help. SOS in Morse goes: dot dot dot; dash dash dash; dot dot dot. Since this is a library, which is being created using dots and dashes, it is important that we define how the dot becomes dot, and dash becomes dash first. The following sections will take smaller sections or pieces of main code and explain you how they work. We will also introduce some interesting concepts using the same. What is a function? Functions have instructions in a single line of code, telling the values in the bracket how to act. Let us see which one is the function in our code. Can you try to guess from the following screenshot? No? Let me help you! digitalWrite() in the above code is a Function, that as you understand, 'writes' on the correct pin of the Arduino. delay is a Function that tells the controller how frequently it should send the message. The higher the delay number, the slower will be the message (Imagine it as a way to slow down your friend who speaks too fast, helping you to understand him better!) Look up the internet to find out what is the maximum number that you stuff into delay. What is a constant? A constant is an identifier with pre-defined, non-changeable values. What is an identifier you ask? An identifier is a name that labels the identity of a unique object or value. As you can see from the above piece of code, HIGH and LOW are Constants. Q: What is the opposite of Constant? Ans: Variable The above food for thought brings us to the next section. What is a variable? A variable is a symbolic name for information. In plain English, a 'teacher' can have any name; hence, the 'teacher' could be a variable. A variable is used to store a piece of information temporarily. The value of a variable changes, if any action is taken on it for example; Add, subtract, multiply etc. (Imagine how your teacher praises you when you complete your assignment on time and scolds you when you do not!) What is a Datatype? Datatypes are sets of data that have a pre-defined value. Now look at the first block of the example program in the following image: int as shown in the above screenshot, is a Datatype The following table shows some of the examples of a Datatype. Datatype Use Example int describes an integer number is used to represent whole numbers 1, 2, 13, 99 etc float used to represent that the numbers are decimal 0.66, 1.73 etc char represents any character. Strings are written in single quotes 'A', 65 etc str represent string "This is a good day!" With the above definition, can we recognize what pinMode is? Every time you have a doubt in a command or you want to learn more about it, you can always look it up at Arduino.cc website. You could do the same for digitalWrite() as well! From the pinMode page of Arduino.cc we can define it as a command that configures the specified pin to behave either as an input or an output. Let us now see something more interesting. What is a control structure? We have already seen the working of a control structure. In this section, we will be more specific to our code. Now I draw your attention towards this specific block from the main example above: Do you see void setup() followed by a code in the brackets? Similarly void loop() ? These make the basics of the structure of an Arduino program sketch. A structure, holds the program together, and helps the compiler to make sense of the commands entered. A compiler is a program that turns code understood by humans into the code that is understood by machines. There are other loop and control structures as you can see in the following screenshot: These control structures are explained next. How do you use Control Structures? Imagine you are teaching your friend to build 6 cm high lego wall. You ask her to place one layer of lego bricks, and then you further ask her to place another layer of lego bricks on top of the bottom layer. You ask her to repeat the process until the wall is 6 cm high. This process of repeating instructions until a desired result is achieved is called Loop. A micro-controller is only as smart as you program it to be. Hence, we will move on to the different types of loops. While loop: Like the name suggests, it repeats a statement (or group of statements) while the given condition is true. The condition is tested before executing the loop body. For loop: Execute a sequence of statements multiple times and abbreviates the code that manages the loop variable. Do while loop: Like a while statement, except that it tests the condition at the end of the loop body Nested loop: You can use one or more loop inside any another while, for or do..while loop. Now you were able to successfully tell your friend when to stop, but how to control the micro controller? Do not worry, the magic is on its way! You introduce Control statements. Break statements: Breaks the flow of the loop or switch statement and transfers execution to the statement that is immediately following the loop or switch. Continue statements: This statement causes the loop to skip the remainder of its body and immediately retest its condition before reiterating. Goto statements: This transfers control to a statement which is labeled . It is no advised to use goto statement in your programs. Quiz time: What is an infinite loop? Look up the internet and note it in your inventor-notebook. The Arduino IDE The full form of IDE is Integrated Development Environment. IDE uses a Compiler to translate code in a simple language that the computer understands. Compiler is the program that reads all your code and translates your instructions to your microcontroller. In case of the Arduino IDE, it also verifies if your code is making sense to it or not. Arduino IDE is like your friend who helps you finish your homework, reviews it before you give it for submission, if there are any errors; it helps you identify them and resolve them. Why should you love the Arduino IDE? I am sure by now things look too technical. You have been introduced to SO many new terms to learn and understand. The important thing here is not to forget to have fun while learning. Understanding how the IDE works is very useful when you are trying to modify or write your own code. If you make a mistake, it would tell you which line is giving you trouble. Isn't it cool? The Arduino IDE also comes with loads of cool examples that you can plug-and-play. It also has a long list of libraries for you to access. Now let us learn how to get the library on to your computer. Ask an adult to help you with this section if you are unable to succeed. Make a note of the following answers in your inventor's notebook before downloading the IDE. Get your answers from google or ask an adult. What is an operating system?What is the name of the operating system running on your computer?What is the version of your current operating system?Is your operating system 32 bit or 64 bit?What is the name of the Arduino board that you have? Now that we did our homework, let us start playing! How to download the IDE? Let us now, go further and understand how to download something that's going to be our playground. I am sure you'd be eager to see the place you'll be working in for building new and interesting stuff! For those of you wanting to learn and do everything my themselves, open any browser and search for "Arduino IDE" followed by the name of your operating system with "32 bits" or "64 bits" as learnt in the previous section. Click to download the latest version and install! Else, the step-by-step instructions are here: Open your browser (Firefox, Chrome, Safari)(Insert image of the logos of firefox, chrome and safari) Go to www.arduino.ccas shown in the following screenshot Click on the 'Download' section of the homepage, which is the third option from your left as shown in the following screenshot. From the options, locate the name of your operating system, click on the right version (32 bits or 64 bits) Then click on 'Just Download' after the new page appears. After clicking on the desired link and saving the files, you should be able to 'double click' on the Arduino icon and install the software. If you have managed to install successfully, you should see the following screens. If not, go back to step 1 and follow the procedure again. The next screenshot shows you how the program will look like when it is loading. This is how the IDE looks when no code is written into it. Your first program Now that you have your IDE ready and open, it is time to start exploring. As promised before, the Arduino IDE comes with many examples, libraries, and helping tools to get curious minds such as you to get started soon. Let us now look at how you can access your first program via the Arduino IDE. A large number of examples can be accessed in the File > Examples option as shown in the following screenshot. Just like we all have nicknames in school, a program, written in in processing is called a 'sketch'. Whenever you write any program for Arduino, it is important that you save your work. Programs written in processing are saved with the extension .ino The name .ino is derived from the last 3 letters of the word ArduINO. What are the other extensions are you aware of? (Hint: .doc, .ppt etc) Make a note in your inventor's notebook. Now ask yourself why do so many extensions exist. An extension gives the computer, the address of the software which will open the file, so that when the contents are displayed, it makes sense. As we learnt above, that the program written in the Arduino IDE is called a 'Sketch'. Your first sketch is named 'blink'. What does it do? Well, it makes your Arduino blink! For now, we can concentrate on the code. Click on File | Examples | Basics | Blink. Refer to next image for this. When you load an example sketch, this is how it would look like. In the image below you will be able to identify the structure of code, recall the meaning of functions and integers from the previous section. We learnt that the Arduino IDE is a compiler too! After opening your first example, we can now learn how to hide information from the compiler. If you want to insert any extra information in plain English, you can do so by using symbols as following. /* your text here*/ OR // your text here Comments can also be individually inserted above lines of code, explaining the functions. It is good practice to write comments, as it would be useful when you are visiting back your old code to modify at a later date. Try editing the contents of the comment section by spelling out your name. The following screenshot will show you how your edited code will look like. Verifying your first sketch Now that you have your first complete sketch in the IDE, how do you confirm that your micro-controller with understand? You do this, by clicking on the easy to locate Verify button with a small . You will now see that the IDE informs you "Done compiling" as shown in the screenshot below. It means that you have successfully written and verified your first code. Congratulations, you did it! Saving your first sketch As we learnt above, it is very important to save your work. We now learn the steps to make sure that your work does not get lost. Now that you have your first code inside your IDE, click on File > SaveAs. The following screenshot will show you how to save the sketch. Give an appropriate name to your project file, and save it just like you would save your files from Paintbrush, or any other software that you use. The file will be saved in a .ino format. Accessing your first sketch Open the folder where you saved the sketch. Double click on the .ino file. The program will open in a new window of IDE. The following screen shot has been taken from a Mac Os, the file will look different in a Linux or a Windows system. Summary Now we know about systems and how logic is used to solve problems. We can write and modify simple code. We also know the basics of Arduino IDE and studied how to verify, save and access your program. Resources for Article:  Further resources on this subject: Getting Started with Arduino [article] Connecting Arduino to the Web [article] Functions with Arduino [article]
Read more
  • 0
  • 0
  • 24358

article-image-learning-cassandra
Packt
06 Apr 2017
17 min read
Save for later

Learning Cassandra

Packt
06 Apr 2017
17 min read
In this article by Sandeep Yarabarla, the author of the book Learning Apache Cassandra - Second Edition, we will built products using relational databases like MySQL and PostgreSQL, and perhaps experimented with NoSQL databases including a document store like MongoDB or a key-value store like Redis. While each of these tools has its strengths, you will now consider whether a distributed database like Cassandra might be the best choice for the task at hand. In this article, we'll begin with the need for NoSQL databases to satisfy the conundrum of ever growing data. We will see why NoSQL databases are becoming the de facto choice for big data and real-time web applications. We will also talk about the major reasons to choose Cassandra from among the many database options available to you. Having established that Cassandra is a great choice, we'll go through the nuts and bolts of getting a local Cassandra installation up and running (For more resources related to this topic, see here.) What is Big Data Big Data is a relatively new term which has been gathering steam over the past few years. Big Data is a term used for data sets that are relatively large to be stored in a traditional database system or processed by traditional data processing pipelines. This data could be structured, semi-structured or unstructured data. The data sets that belong to this category usually scale to terabytes or petabytes of data. Big Data usually involves one or more of the following: Velocity: Data moves at an unprecedented speed and must be dealt with it in a timely manner. Volume: Organizations collect data from a variety of sources, including business transactions, social media and information from sensor or machine-to-machine data. This could involve terabytes to petabytes of data. In the past, storing it would've been a problem – but new technologies have eased the burden. Variety: Data comes in all sorts of formats ranging from structured data to be stored in traditional databases to unstructured data (blobs) such as images, audio files and text files. These are known as the 3 Vs of Big Data. In addition to these, we tend to associate another term with Big Data. Complexity: Today's data comes from multiple sources, which makes it difficult to link, match, cleanse and transform data across systems. However, it's necessary to connect and correlate relationships, hierarchies and multiple data linkages or your data can quickly spiral out of control. It must be able to traverse multiple data centers, cloud and geographical zones. Challenges of modern applications Before we delve into the shortcomings of relational systems to handle Big Data, let's take a look at some of the challenges faced by modern web facing and Big Data applications. Later, this will give an insight into how NoSQL data stores or Cassandra in particular helps solve these issues. One of the most important challenges faced by a web facing application is it should be able to handle a large number of concurrent users. Think of a search engine like google which handles millions of concurrent users at any given point of time or a large online retailer. The response from these applications should be swift even as the number of users keeps on growing. Modern applications need to be able to handle large amounts of data which can scale to several petabytes of data and beyond. Consider a large social network with a few hundred million users: Think of the amount of data generated in tracking and managing those users Think of how this data can be used for analytics Business critical applications should continue running without much impact even when there is a system failure or multiple system failures (server failure, network failure, and so on.). The applications should be able to handle failures gracefully without any data loss or interruptions. These applications should be able to scale across multiple data centers and geographical regions to support customers from different regions around the world with minimum delay. Modern applications should be implementing fully distributed architectures and should be capable of scaling out horizontally to support any data size or any number of concurrent users. Why not relational Relational database systems (RDBMS) have been the primary data store for enterprise applications for 20 years. Lately, NoSQL databases have been picking a lot of steam and businesses are slowly seeing a shift towards non-relational databases. There are a few reasons why relational databases don't seem like a good fit for modern web applications. Relational databases are not designed for clustered solutions. There are some solutions that shard data across servers but these are fragile, complex and generally don't work well. Sharding solutions implemented by RDBMS MySQL's product MySQL cluster provides clustering support which adds many capabilities of non-relational systems. It is actually a NoSQL solution that integrates with MySQL relational database. It partitions the data onto multiple nodes and the data can be accessed via different APIs. Oracle provides a clustering solution, Oracle RAC which involves multiple nodes running an oracle process accessing the same database files. This creates a single point of failure as well as resource limitations in accessing the database itself. They are not a good fit current hardware and architectures. Relational databases are usually scaled up by using larger machines with more powerful hardware and maybe clustering and replication among a small number of nodes. Their core architecture is not a good fit for commodity hardware and thus doesn't work with scale out architectures. Scale out vs Scale up architecture Scaling out means adding more nodes to a system such as adding more servers to a distributed database or filesystem. This is also known as horizontal scaling. Scaling up means adding more resources to a single node within the system such as adding more CPU, memory or disks to a server. This is also known as vertical scaling. How to handle Big Data Now that we are convinced the relational model is not a good fit for Big Data, let's try to figure out ways to handle Big Data. These are the solutions that paved the way for various NoSQL databases. Clustering: The data should be spread across different nodes in a cluster. The data should be replicated across multiple nodes in order to sustain node failures. This helps spread the data across the cluster and different nodes contain different subsets of data. This improves performance and provides fault tolerance. A node is an instance of database software running on a server. Multiple instances of the same database could be running on the same server. Flexible schema: Schemas should be flexible unlike the relational model and should evolve with data. Relax consistency: We should embrace the concept of eventual consistency which means data will eventually be propagated to all the nodes in the cluster (in case of replication). Eventual consistency allows data replication across nodes with minimum overhead. This allows for fast writes with the need for distributed locking. De-normalization of data: De-normalize data to optimize queries. This has to be done at the cost of writing and maintaining multiple copies of the same data. What is Cassandra and why Cassandra Cassandra is a fully distributed, masterless database, offering superior scalability and fault tolerance to traditional single master databases. Compared with other popular distributed databases like Riak, HBase, and Voldemort, Cassandra offers a uniquely robust and expressive interface for modeling and querying data. What follows is an overview of several desirable database capabilities, with accompanying discussions of what Cassandra has to offer in each category. Horizontal scalability Horizontal scalability refers to the ability to expand the storage and processing capacity of a database by adding more servers to a database cluster. A traditional single-master database's storage capacity is limited by the capacity of the server that hosts the master instance. If the data set outgrows this capacity, and a more powerful server isn't available, the data set must be sharded among multiple independent database instances that know nothing of each other. Your application bears responsibility for knowing to which instance a given piece of data belongs. Cassandra, on the other hand, is deployed as a cluster of instances that are all aware of each other. From the client application's standpoint, the cluster is a single entity; the application need not know, nor care, which machine a piece of data belongs to. Instead, data can be read or written to any instance in the cluster, referred to as a node; this node will forward the request to the instance where the data actually belongs. The result is that Cassandra deployments have an almost limitless capacity to store and process data. When additional capacity is required, more machines can simply be added to the cluster. When new machines join the cluster, Cassandra takes careof rebalancing the existing data so that each node in the expanded cluster has a roughly equal share. Also, the performance of a Cassandra cluster is directly proportional to the number of nodes within the cluster. As you keep on adding instances, the read and write throughput will keep increasing proportionally. Cassandra is one of the several popular distributed databases inspired by the Dynamo architecture, originally published in a paper by Amazon. Other widely used implementations of Dynamo include Riak and Voldemort. You can read the original paper at http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf High availability The simplest database deployments are run as a single instance on a single server. This sort of configuration is highly vulnerable to interruption: if the server is affected by a hardware failure or network connection outage, the application's ability to read and write data is completely lost until the server is restored. If the failure is catastrophic, the data on that server might be lost completely. Master-slave architectures improve this picture a bit. The master instance receives all write operations, and then these operations are replicated to follower instances. The application can read data from the master or any of the follower instances, so a single host becoming unavailable will not prevent the application from continuing to read data. A failure of the master, however, will still prevent the application from performing any write operations, so while this configuration provides high read availability, it doesn't completely provide high availability. Cassandra, on the other hand, has no single point of failure for reading or writing data. Each piece of data is replicated to multiple nodes, but none of these nodes holds the authoritative master copy. All the nodes in a Cassandra cluster are peers without a master node. If a machine becomes unavailable, Cassandra will continue writing data to the other nodes that share data with that machine, and will queue the operations and update the failed node when it rejoins the cluster. This means in a typical configuration, multiple nodes must fail simultaneously for there to be any application-visible interruption in Cassandra's availability. How many copies? When you create a keyspace - Cassandra's version of a database, you specify how many copies of each piece of data should be stored; this is called the replication factor. A replication factor of 3 is a common choice for most use cases. Write optimization Traditional relational and document databases are optimized for read performance. Writing data to a relational database will typically involve making in-place updates to complicated data structures on disk, in order to maintain a data structure that can be read efficiently and flexibly. Updating these data structures is a very expensive operation from a standpoint of disk I/O, which is often the limiting factor for database performance. Since writes are more expensive than reads, you'll typically avoid any unnecessary updates to a relational database, even at the expense of extra read operations. Cassandra, on the other hand, is highly optimized for write throughput, and in fact never modifies data on disk; it only appends to existing files or creates new ones. This is much easier on disk I/O and means that Cassandra can provide astonishingly high write throughput. Since both writing data to Cassandra, and storing data in Cassandra, are inexpensive, denormalization carries little cost and is a good way to ensure that data can be efficiently read in various access scenarios. Because Cassandra is optimized for write volume; you shouldn't shy away from writing data to the database. In fact, it's most efficient to write without reading whenever possible, even if doing so might result in redundant updates. Just because Cassandra is optimized for writes doesn't make it bad at reads; in fact, a well-designed Cassandra database can handle very heavy read loads with no problem. Structured records The first three database features we looked at are commonly found in distributed data stores. However, databases like Riak and Voldemort are purely key-value stores; these databases have no knowledge of the internal structure of a record that's stored at a particular key. This means useful functions like updating only part of a record, reading only certain fields from a record, or retrieving records that contain a particular value in a given field are not possible. Relational databases like PostgreSQL, document stores like MongoDB, and, to a limited extent, newer key-value stores like Redis do have a concept of the internal structure of their records, and most application developers are accustomed to taking advantage of the possibilities this allows. None of these databases, however, offer the advantages of a masterless distributed architecture. In Cassandra, records are structured much in the same way as they are in a relational database—using tables, rows, and columns. Thus, applications using Cassandra can enjoy all the benefits of masterless distributed storage while also getting all the advanced data modeling and access features associated with structured records. Secondary indexes A secondary index, commonly referred to as an index in the context of a relational database, is a structure allowing efficient lookup of records by some attribute other than their primary key. This is a widely useful capability: for instance, when developing a blog application, you would want to be able to easily retrieve all of the posts written by a particular author. Cassandra supports secondary indexes; while Cassandra's version is not as versatile as indexes in a typical relational database, it's a powerful feature in the right circumstances. Materialized views Data modelling principles in Cassandra compel us to denormalize data as much as possible. Prior to Cassandra 3.0, the only way to query on a non-primary key column was to create a secondary index and query on it. However, secondary indexes have a performance tradeoff if it contains high cardinality data. Often, high cardinality secondary indexes have to scan data on all the nodes and aggregate them to return the query results. This defeats the purpose of having a distributed system. To avoid secondary indexes and client-side denormalization, Cassandra introduced the feature of Materialized views which does server-side denormalization. You can create views for a base table and Cassandra ensures eventual consistency between the base and view. This lets us do very fast lookups on each view following the normal Cassandra read path. Materialized views maintain a correspondence of one CQL row each in the base and the view, so we need to ensure that each CQL row which is required for the views will be reflected in the base table's primary keys. Although, materialized view allows for fast lookups on non-primary key indexes; this comes at a performance hit to writes. Efficient result ordering It's quite common to want to retrieve a record set ordered by a particular field; for instance, a photo sharing service will want to retrieve the most recent photographs in descending order of creation. Since sorting data on the fly is a fundamentally expensive operation, databases must keep information about record ordering persisted on disk in order to efficiently return results in order. In a relational database, this is one of the jobs of a secondary index. In Cassandra, secondary indexes can't be used for result ordering, but tables can be structured such that rows are always kept sorted by a given column or columns, called clustering columns. Sorting by arbitrary columns at read time is not possible, but the capacity to efficiently order records in any way, and to retrieve ranges of records based on this ordering, is an unusually powerful capability for a distributed database. Immediate consistency When we write a piece of data to a database, it is our hope that that data is immediately available to any other process that may wish to read it. From another point of view, when we read some data from a database, we would like to be guaranteed that the data we retrieve is the most recently updated version. This guarantee is calledimmediate consistency, and it's a property of most common single-master databases like MySQL and PostgreSQL. Distributed systems like Cassandra typically do not provide an immediate consistency guarantee. Instead, developers must be willing to accept eventual consistency, which means when data is updated, the system will reflect that update at some point in the future. Developers are willing to give up immediate consistency precisely because it is a direct tradeoff with high availability. In the case of Cassandra, that tradeoff is made explicit through tunable consistency. Each time you design a write or read path for data, you have the option of immediate consistency with less resilient availability, or eventual consistency with extremely resilient availability. Discretely writable collections While it's useful for records to be internally structured into discrete fields, a given property of a record isn't always a single value like a string or an integer. One simple way to handle fields that contain collections of values is to serialize them using a format like JSON, and then save the serialized collection into a text field. However, in order to update collections stored in this way, the serialized data must be read from the database, decoded, modified, and then written back to the database in its entirety. If two clients try to perform this kind of modification to the same record concurrently, one of the updates will be overwritten by the other. For this reason, many databases offer built-in collection structures that can be discretely updated: values can be added to, and removed from collections, without reading and rewriting the entire collection. Neither the client nor Cassandra itself needs to read the current state of the collection in order to update it, meaning collection updates are also blazingly efficient. Relational joins In real-world applications, different pieces of data relate to each other in a variety of ways. Relational databases allow us to perform queries that make these relationships explicit, for instance, to retrieve a set of events whose location is in the state of New York (this is assuming events and locations are different record types). Cassandra, however, is not a relational database, and does not support anything like joins. Instead, applications using Cassandra typically denormalize data and make clever use of clustering in order to perform the sorts of data access that would use a join in a relational database. For datasets that aren't already denormalized, applications can also perform client-side joins, which mimic the behavior of a relational database by performing multiple queries and joining the results at the application level. Client-side joins are less efficient than reading data that has been denormalized in advance, but offer more flexibility. Summary In this article, you explored the reasons to choose Cassandra from among the many databases available, and having determined that Cassandra is a great choice, you installed it on your development machine. You had your first taste of the Cassandra Query Language when you issued your first few commands through the CQL shell in order to create a keyspace, table, and insert and read data. You're now poised to begin working with Cassandra in earnest. Resources for Article: Further resources on this subject: Setting up a Kubernetes Cluster [article] Dealing with Legacy Code [article] Evolving the data model [article]
Read more
  • 0
  • 0
  • 2233
Modal Close icon
Modal Close icon