How-To Tutorials

In this article by Gigi Sayfan, the author of the book Mastering Kubernetes, we will learn how to do the monitoring Kubernetes with Heapster. (For more resources related to this topic, see here.) Monitoring Kubernetes with Heapster Heapster is a Kubernetes project that provides a robust monitoring solution for Kubernetes clusters. It runs as a pod (of course), so it can be managed by Kubernetes itself. Heapster supports Kubernetes and CoreOS clusters. It has a very modular and flexible design. Heapster collects both operational metrics and events from every node in the cluster, stores them in a persistent backend (with a well-defined schema) and allows visualization and programmatic access. Heapster can be configured to use different backends (or sinks, in Heapster’s parlance) and their corresponding visualization frontends. The most common combination is InfluxDB as backend and Grafana as frontend. The Google Cloud platform integrates Heapster with the Google monitoring service. There are many other less common backends, such as the following: Log InfluxDB Google Cloud monitoring Google Cloud logging Hawkular-Metrics(metrics only) OpenTSDB Monasca (metrics only) Kafka (metrics only) Riemann (metrics only) Elasticsearch You can use multiple backends by specifying sinks on the command-line: --sink=log --sink=influxdb:http://monitoring-influxdb:80/ cAdvisor cAdvisor is part of the kubelet, which runs on every node. It collects information about the CPU/cores usage, memory, network,and file systems of each container. It provides a basic UI on port 4194, but, most importantly for Heapster, it provides all this information through the kubelet. Heapster records the information collected by cAdvisor on each node and stores it in its backend for analysis and visualization. The cAdvisor UI is useful if you want to quickly verify that a particular node is setup correctly, for example, while creating a new cluster when Heapster is not hooked up yet. Here is what it looks same as shown following: InfluxDB backend InfluxDB is a modern and robust distributed time-series database. It is very well-suited and used broadly for centralized metrics and logging. It is also the preferred Heapster backend (outside the Google Cloud platform). The only thing is InfluxDB clustering, high availability is part of enterprise offering. The storageschema The InfluxDB storage schema defines the information that Heapster stores in InfluxDB and is available for querying and graphing later. The metrics are divided into multiple categories, called measurements. You can treat and query each metric separately, or you can query a whole category as one measurement and receive the individual metrics as fields. The naming convention is <category>/<metrics name> (except for uptime, which has a single metric). If you have a SQL background you can think of measurements as tables. Each metrics are stored per container. Each metric is labeled with the following information: pod_id – Unique ID of a pod pod_name – User-provided name of a pod pod_namespace – The namespace of a pod container_base_image – Base image for the container container_name – User-provided name of the container or full cgroup name for system containers host_id – Cloud-provider-specified or user-specified Identifier of a node hostname – Hostname where the container ran labels – Comma-separated list of user-provided labels; format is key:value’ namespace_id – UID of the namespace of a pod resource_id – A unique identifier used to differentiate multiple metrics of the same type, for example, FS partitions under filesystem/usage Here are all the metrics grouped by category. As you can see, it is quite extensive. CPU cpu/limit – CPU hard limit in millicores cpu/node_capacity – CPU capacity of a node cpu/node_allocatable – CPU allocatable of a node cpu/node_reservation – Share of CPU that is reserved on the node allocatable cpu/node_utilization – CPU utilization as a share of node allocatable cpu/request – CPU request (the guaranteed amount of resources) in millicores cpu/usage – Cumulative CPU usage on all cores cpu/usage_rate – CPU usage on all cores in millicores File system filesystem/usage – Total number of bytes consumed on a filesystem filesystem/limit – The total size of the filesystem in bytes filesystem/available – The number of available bytes remaining in the filesystem Memory memory/limit – Memory hard limit in bytes memory/major_page_faults – Number of major page faults memory/major_page_faults_rate – Number of major page faults per second memory/node_capacity – Memory capacity of a node memory/node_allocatable – Memory allocatable of a node memory/node_reservation – Share of memory that is reserved on the node allocatable memory/node_utilization – Memory utilization as a share of memory allocatable memory/page_faults – Number of page faults memory/page_faults_rate – Number of page faults per second memory/request – Memory request (the guaranteed amount of resources) in bytes memory/usage – Total memory usage memory/working_set – Total working set usage; working set is the memory being used and not easily dropped by the kernel Network network/rx – Cumulative number of bytes received over the network network/rx_errors – Cumulative number of errors while receiving over the network network/rx_errors_rate – Number of errors per second while receiving over the network network/rx_rate – Number of bytes received over the network per second network/tx – Cumulative number of bytes sent over the network network/tx_errors – Cumulative number of errors while sending over the network network/tx_errors_rate – Number of errors while sending over the network network/tx_rate – Number of bytes sent over the network per second Uptime uptime – Number of milliseconds since the container was started You can work with InfluxDB directly if you’re familiar with it. You can either connect to it using its own API or use its web interface. Type the following command to find its port: k describe service monitoring-influxdb --namespace=kube-system | grep NodePort Type: NodePort NodePort: http 32699/TCP NodePort: api 30020/TCP Now you can browse to the InfluxDB web interface using the HTTP port. You’ll need to configure it to point to the API port. The username and password are root and root by default: Once you’re setup you can select what database to use (see top-right corner). The Kubernetes database is called k8s. You can now query the metrics using the InfluxDB query language. Grafana visualization Grafana runs in its own container and serves a sophisticated dashboard that works well with InfluxDB as a data source. To locate the port, type the following command: k describe service monitoring-influxdb --namespace=kube-system | grep NodePort Type: NodePort NodePort: <unset> 30763/TCP Now you can access the Grafana web interface on that port. The first thing you need to do is setup the data source to point to the InfluxDB backend: Make sure to test the connection and then go explore the various options in the dashboards. There are several default dashboards, but you should be able to customize it to your preferences. Grafana is designed to let adapt it to your needs. Summary In this article we have learned how to do monitoring Kubernetes with Heapster.  Resources for Article: Further resources on this subject: The Microsoft Azure Stack Architecture [article] Building A Recommendation System with Azure [article] Setting up a Kubernetes Cluster [article]

The ability of computers to perform tasks such as identifying objects has always been a humongous task for both the software and the required architecture. This isn't the case anymore since the likes of Google, Amazon, and a few other companies have done all the hard work, providing the infrastructure and making it available as a cloud service. It should be noted that they are as easy to access as making REST API calls. Read Also: Machine Learning as a Service (MLaaS): How Google Cloud Platform, Microsoft Azure, and AWS are democratizing Artificial Intelligence This article is taken from the book Learning Kotlin by building Android Applications by Eunice Adutwumwaa Obugyei and Natarajan Raman. This book will teach you programming in Kotlin including data types, flow control, lambdas, object-oriented, and functional programming while building  Android Apps. In this tutorial, you will learn how to use the face detection API from Google's Mobile Vision API to detect faces and add fun functionalities such as adding rabbit ears to a user's picture. We will cover the following topics: Identifying human faces in an image Tracking human faces from a camera feed Identifying specific parts of the face (for example, eyes, ears, nose, and mouth) Drawing graphics on specific parts of a detected face in an image (for example, rabbit ears over the user's ears) Introduction to Mobile Vision The Mobile Vision API provides a framework for finding objects in photos and videos. The framework can locate and describe visual objects in images or video frames, and it has an event-driven API that tracks the position of those objects. Currently, the Mobile Vision API includes face, barcode, and text detectors. Faces API concepts Before diving into coding the features, it is necessary that you understand the underlying concepts of the face detection capabilities of the face detection API. From the official documentation: Face detection is the process of automatically locating human faces in visual media (digital images or video). A face that is detected is reported at a position with an associated size and orientation. Once a face is detected, it can be searched for landmarks such as the eyes and nose. A key point to note is that only after a face is detected, will landmark such as eyes and a nose be searched for. As part of the API, you could opt out of detecting these landmarks. Note the difference between face detection and face recognition. While the former is able to recognize a face from an image or video, the latter does the same and is also able to tell that a face has been seen before. The former has no memory of a face it has detected before. We will be using a couple of terms in this section, so let me give you an overview of each of these before we go any further: Face tracking extends face detection to video sequences. When a face appears in a video for any length of time, it can be identified as the same person and can be tracked. It is important to note that the face that you are tracking must appear in the same video. Also, this is not a form of face recognition; this mechanism just makes inferences based on the position and motion of the face(s) in a video sequence. A landmark is a point of interest within a face. The left eye, right eye, and nose base are all examples of landmarks. The Face API provides the ability to find landmarks on a detected face. Classification is determining whether a certain facial characteristic is present. For example, a face can be classified with regards to whether its eyes are open or closed or smiling or not. Getting started – detecting faces You will first learn how to detect a face in a photo and its associated landmarks. We will need some requirements in order to pursue this. With a minimum of Google Play Services 7.8, you can use the Mobile Vision APIs, which provide the face detection APIs. Make sure you update your Google Play Services from the SDK manager so that you meet this requirement. Get an Android device that runs Android 4.2.2 or later or a configured Android Emulator. The latest version of the Android SDK includes the SDK tools component. Creating the FunyFace project Create a new project called FunyFace. Open up the app module's build.gradle file and update the dependencies to include the Mobile Vision APIs: dependencies { implementation fileTree(dir: 'libs', include: ['*.jar']) implementation"org.jetbrains.kotlin:kotlin-stdlib-jre7:$kotlin_version" implementation 'com.google.android.gms:play-services-vision:11.0.4' ... } Now, update your AndroidManifest.xml to include meta data for the faces API: <meta-data android:name="com.google.android.gms.vision.DEPENDENCIES" android:value="face" /> Now, your app is ready to use the face detection APIs. To keep things simple, for this lab, you're just going to process an image that is already present in your app. Add the following image to your res/drawable folder: Now, this is how you will go about performing face detection. You will first load the image into memory, get a Paint instance, and create a temporary bitmap based on the original, from which you will create a canvas. Create a frame using the bitmap and then call the detect method on FaceDetector, using this frame to get back SparseArray of face objects. Well, let's get down to business—this is where you will see how all of these play out. First, open up your activity_main.xml file and update the layout so that it has an image view and a button. See the following code: <?xml version="1.0" encoding="utf-8"?> <FrameLayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" android:layout_width="match_parent" android:layout_height="match_parent" xmlns:app="http://schemas.android.com/apk/res-auto" tools:context="com.packtpub.eunice.funyface.MainActivity"> <ImageView android:id="@+id/imageView" android:layout_width="match_parent" android:layout_height="match_parent" android:src="@mipmap/ic_launcher_round" app:layout_constraintBottom_toTopOf="parent" android:scaleType="fitCenter"/> <Button android:id="@+id/button" android:layout_width="wrap_content" android:layout_height="wrap_content" android:layout_gravity="bottom|center" android:text="Detect Face"/> </FrameLayout> That is all you need to do here so that you have FrameLayout with ImageView and a button. Now, open up MainActivity.kt and add the following import statements. This is just to make sure that you import from the right packages as you move along. In your onCreate() method, attach a click listener to the button in your MainActivity layout file: package com.packtpub.eunice.funface import android.graphics.* import android.graphics.drawable.BitmapDrawable import android.os.Bundle import android.support.v7.app.AlertDialog import android.support.v7.app.AppCompatActivity import com.google.android.gms.vision.Frame import com.google.android.gms.vision.face.FaceDetector import kotlinx.android.synthetic.main.activity_main.* class MainActivity : AppCompatActivity() { override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setContentView(R.layout.activity_main) button.setOnClickListener { detectFace() } } } Loading the image In your detectFace() method, you will first load your image from the drawable folder into memory and create a bitmap image from it. Since you will be updating this bitmap to paint over it when the face is detected, you need to make it mutable. This is what makes your bitmap mutable: options.inMutable=true See the following implementation: private fun detectFace() { // Load the image val bitmapOptions = BitmapFactory.Options() bitmapOptions.inMutable = true val myBitmap = BitmapFactory.decodeResource( applicationContext.resources, R.drawable.children_group_picture, bitmapOptions) } Creating a Paint instance Use the Paint API to get an instance of the Paint class. You will only draw around the face, and not paint the whole face. To do this, set a thin stroke, give it a color, which in our case is red, and set the style of paint to STROKE using Paint.Style.STROKE: // Get a Paint instance val myRectPaint = Paint() myRectPaint.strokeWidth = 5F myRectPaint.color = Color.RED myRectPaint.style = Paint.Style.STROKE The Paint class holds the information related to the style and color related to the text, bitmap, and various shapes. Creating a canvas To get the canvas, first create a bitmap using the dimensions from the bitmap you created earlier. With this canvas, you will paint over the bitmap to draw the outline of the face after it has been detected: // Create a canvas using the dimensions from the image's bitmap val tempBitmap = Bitmap.createBitmap(myBitmap.width, myBitmap.height, Bitmap.Config.RGB_565) val tempCanvas = Canvas(tempBitmap) tempCanvas.drawBitmap(myBitmap, 0F, 0F, null) The Canvas class is used to hold the call made to draw. A canvas is a drawing surface and it provides various methods for drawing onto a bitmap. Creating the face detector All you have done thus far is basically housekeeping. You will now access the FaceDetector API by which you will, well, detect the face in the image. You will disable tracking for now, as you only want to detect the face at this stage. Note that on its first run, the Play Services SDK will take some time to initialize the Faces API. It may or may not have completed this process at the time you intend to use it. Therefore, as a safety check, you need to ensure its availability before using it. In this case, you will show a simple dialog to the user if the FaceDetector is not ready at the time the app is run. Also, note that you may need an internet connection as the SDK initializes. You also need to ensure you have enough space, as the initialization may download some native library onto the device: // Create a FaceDetector val faceDetector = FaceDetector.Builder(applicationContext).setTrackingEnabled(false) .build() if (!faceDetector.isOperational) { AlertDialog.Builder(this) .setMessage("Could not set up the face detector!") .show() return } Detecting the faces Now, you will use the detect() method from the faceDetector instance to get the faces and their metadata. The result will be SparseArray of Face objects: // Detect the faces val frame = Frame.Builder().setBitmap(myBitmap).build() val faces = faceDetector.detect(frame) Drawing rectangles on the faces Now that you have the faces, you will iterate through this array to get the coordinates of the bounding rectangle for the face. Rectangles require x, y of the top left and bottom right corners, but the information available only gives the left and top positions, so you have to calculate the bottom right using the top left, width, and height. Then, you need to release the faceDetector to free up resources. Here's the code: // Mark out the identified face for (i in 0 until faces.size()) { val thisFace = faces.valueAt(i) val left = thisFace.position.x val top = thisFace.position.y val right = left + thisFace.width val bottom = top + thisFace.height tempCanvas.drawRoundRect(RectF(left, top, right, bottom), 2F, 2F, myRectPaint) } imageView.setImageDrawable(BitmapDrawable(resources, tempBitmap)) // Release the FaceDetector faceDetector.release() Results All set. Run the app, press the DETECT FACE button, and wait a while...:   The app should detect the face and a square box should appear around the face, voila: Okay, let's move on and add some fun to their faces. To do this, you need to identify the position of the specific landmark you want, then draw over it. To find out the landmark's representation, you label them this time around, then later draw your filter to the desired position. To label, update the for loop which drew the rectangle around the face: // Mark out the identified face for (i in 0 until faces.size()) { ... for (landmark in thisFace.landmarks) { val x = landmark.position.x val y = landmark.position.y when (landmark.type) { NOSE_BASE -> { val scaledWidth = eyePatchBitmap.getScaledWidth(tempCanvas) val scaledHeight = eyePatchBitmap.getScaledHeight(tempCanvas) tempCanvas.drawBitmap(eyePatchBitmap, x - scaledWidth / 2, y - scaledHeight / 2, null) } } } } Run the app and take note of the labels of the various landmarks: There you have it! That's funny, right? Summary In this tutorial, we learned how to use the Mobile Vision APIs, in this case, the Faces API. There are a few things to note here. This program is not optimized for production. Some things you can do on your own are, load the image and do the processing in a background thread. You can also provide a functionality to allow the user to pick and choose images from different sources other than the static one used. You can get more creative with the filters and how they are applied too. Also, you can enable the tracking feature on the FaceDetector instance, and feed in a video to try out face tracking. To know more about Kotlin APIs as a preliminary for building stunning applications for Android, read our book Learning Kotlin by building Android Applications. 6 common challenges faced by Android App developers Google plans to let the AMP Project have an open governance model, soon! Entry level phones to taste the Go edition of the Android 9.0 Pie version

In this article by Steven Noble, the author of the book Building Modern Networks, we will discuss networking concepts such as hyper-scale networking, software-defined networking, network hardware and software design along with a litany of network design ideas utilized in NGN. (For more resources related to this topic, see here.) The term Next Generation Network (NGN) has been around for over 20 years and refers to the current state of the art network equipment, protocols and features. A big driver in NGN is the constant newer, better, faster forwarding ASICs coming out of companies like Barefoot, Broadcom, Cavium, Nephos (MediaTek) and others. The advent of commodity networking chips has shortened the development time for generic switches, allowing hyper scale networking end users to build equipment upgrades into their network designs. At the time of writing, multiple companies have announced 6.4 Tbps switching chips. In layman terms, a 6.4 Tbps switching chip can handle 64x100GbE of evenly distributed network traffic without losing any packets. To put the number in perspective, the entire internet in 2004 was about 4 Tbps, so all of the internet traffic in 2004 could have crossed this one switching chip without issue. (Internet Traffic 1.3 EB/month http://blogs.cisco.com/sp/the-history-and-future-of-internet-traffic) A hyper-scale network is one that is operated by companies such as Facebook, Google, Twitter and other companies that add hundreds if not thousands of new systems a month to keep up with demand. Examples of next generation networking At the start of the commercial internet age (1994), software routers running on minicomputers such as BBNs PDP-11 based IP routers designed in the 1970's were still in use and hubs were simply dumb hardware devices that broadcast traffic everywhere. At that time, the state of the art in networking was the Cisco 7000 series router, introduced in 1993. The next generation router was the Cisco 7500 (1995), while the Cisco 12000 series (gigabit) routers and the Juniper M40 were only concepts. When we say next generation, we are speaking of the current state of the art and the near future of networking equipment and software. For example, 100 GB Ethernet is the current state of the art, while 400 GB Ethernet is in the pipeline. The definition of a modern network is a network that contains one or more of the following concepts: Software-defined Networking (SDN) Network design concepts Next generation hardware Hyper scale networking Open networking hardware and software Network Function Virtualization (NFV) Highly configurable traffic management Both Open and Closed network hardware vendors have been innovating at a high rate of speed with the help of and due to hyper-scale companies like Google, Facebook and others who have the need for next generation high speed network devices. This provides the network architect with a reasonable pipeline of equipment to be used in designs. Google and Facebook are both companies with hyper scale networks. A hyper scale network is one where the data stored, transferred, and updated on the network grows exponentially. Hyper scale companies deploy new equipment, software, and configurations weekly or even daily to support the needs of their customers. These companies have needs that are outside of the normal networking equipment available, so they must innovate by building their own next generation network devices, designing multi-tiered networks (like a three stage Clos network) and automating the installation and configuration of the next generation networking devices. The need of hyper scalers is well summed up by Google's Amin Vahdat in a 2014 Wired article "We couldn't buy the hardware we needed to build a network of the size and speed we needed to build". Terms and concepts in networking Here you will find the definition of some terms that are important in networking. They have been broken into groups of similar concepts. Routing and switching concepts In network devices and network designs there are many important concepts to understand. Here we begin with the way data is handled. The easiest way to discuss networking is to look at the OSI layer and point out where each device sits. OSI Layer with respect to routers and switches: Layer 1 (Physical): Layer 1 includes cables, hub, and switch ports. This is how all of the devices connect to each other including copper cables (CatX), fiber optics and Direct Attach Cables (DAC) which connect SFP ports without fiber. Layer 2 (Data link Layer): Layer 2 includes the raw data sent over the links and manages the Media Access Control (MAC) addresses for Ethernet Layer 3 (Network layer): Layer 3 includes packets that have more than just layer 2 data, such as IP, IPX (Novell Networks protocol), AFP (Apple's protocol) Routers and switches In a network you will have equipment that switches and/or routes traffic. A switch is a networking device that connects multiple devices such as servers, provides local connectivity and provides an uplink to the core network. A router is a network device that computes paths to remote and local devices, providing connectivity to devices across a network. Both switches and routers can use copper and fiber connections to interconnect. There are a few parts to a networking device, the forwarding chip, the TCAM, and the network processor. Some newer switches have Baseboard Management Controllers (BMCs) which manage the power, fans and other hardware, lessening the burden on the NOS to manage these devices. Currently routers and switches are very similar as there are many Layer 3 forwarding capable switches and some Layer 2 forwarding capable routers. Making a switch Layer 3 capable is less of an issue than making a router Layer 2 forwarding as the switch already is doing Layer 2 and adding Layer 3 is not an issue. A router does not do Layer 2 forwarding in general, so it has to be modified to allow for ports to switch rather than route. Control plane The control plane is where all of the information about how packets should be handled is kept. Routing protocols live in the control plane and are constantly scanning information received to determine the best path for traffic to flow. This data is then packed into a simple table and pushed down to the data plane. Data plane The data plane is where forwarding happens. In a software router, this would be done in the devices CPU, in a hardware router, this would be done using the forwarding chip and associated memories. VLAN/VXLAN A Virtual Local Area Network (VLAN) is a way of creating separate logical networks within a physical network. VLANs are generally used to separate/combine different users, or network elements such as phones, servers, workstations, and so on. You can have up to 4,096 VLANs on a network segment. A Virtual Extensible LAN (VXLAN) was created to all for large, dynamic isolated logical networks for virtualized and multiple tenant networks. You can have up to 16 million VXLANs on a network segment. A VXLAN Tunnel Endpoint (VTEP) is a set of two logical interfaces inbound which encapsulates incoming traffic into VXLANs and outbound which removes the encapsulation of outgoing traffic from VXLAN back to its original state.  Network design concepts Network design requires the knowledge of the physical structure of the network so that the proper design choices are made. For example, in data center you would have a local area network, if you have multiple data centers near each other, they would be considered a metro area network. LAN A Local Area Network (LAN), generally considered to be within the same building. These networks can be bridged (switched) or routed. In general LANs are segmented into areas to avoid large broadcast domains. MAN A Metro Area Network (MAN), generally defined as multiple sites in the same geographic area or city, that is, metropolitan area. A MAN generally runs at the same speed as a LAN but is able to cover larger distances. WAN A Wide Area Network (WAN), essentially everything that is not a LAN or MAN is a WAN. WANs generally use fiber optic cables to transmit data from one location to another. WAN circuits can be provided via multiple connections and data encapsulations including MPLS, ATM, and Ethernet. Most large network providers utilize Dense Wavelength Division Multiplexing (DWDM) to put more bits on their fiber networks. DWDM puts multiple colors of light onto the fiber, allowing up to 128 different wavelengths to be sent down a single fiber. DWDM has just entered open networking with the introduction of Facebook's Voyager system. Leaf-Spine design In a Leaf-Spine network design, there are Leaf switches (the switches that connect to the servers) sometimes called Top of Rack (ToR) switches connected to a set of Spine (switches that connect leafs together) sometimes called End of Rack (EoR) switches. Clos network A Clos network is one of the ways to design a multi-stage network. Based on the switching network design by Charles Clos in 1952, a three stage Clos is the smallest version of a Clos network. It has an ingress, a middle, and an egress stage. Some hyper scale networks are using five stage Clos where the middle is replaced with another three stage Clos. In a five stage Clos there is an ingress, a middle ingress, a middle, a middle egress and an egress stage. All stages are connected to their neighbor, so in the example shown, Ingress 1 is connected to all four of the middle stages just as Egress 1 is connected to all four of the middle stages. A Clos network can be built in odd numbers starting with 3, so a 5, 7, and so on stage Clos is possible. For even numbered designs, Benes designs are usable. Benes network A Benes design is a non-blocking Clos design where the middle stage is 2x2 instead of NxN. A Benes network can have even numbers of stages. Here is a four stage Benes network. Network controller concepts Here we will discuss the concepts of network controllers. Every networking device has a controller, whether built in or external to manage the forwarding of the system. Controller A controller is a computer that sits on the network and manages one or more network devices. A controller can be built into a device, like the Cisco Supervisor module or standalone like an OpenFlow controller. The controller is responsible for managing all of the control plane data and deciding what should be sent down to the data plane. Generally, a controller will have a Command-line Interface (CLI) and more recently a web configuration interface. Some controllers will even have an Application Programming Interface (API). OpenFlow controller An OpenFlow controller, as it sounds is a controller that uses the OpenFlow protocol to communicate with network devices. The most common OpenFlow controllers that people hear about are OpenDaylight and ONOS. People who are working with OpenFlow would also know of Floodlight and RYU. Supervisor module A route processor is a computer that sits inside of the chassis of the network device you are managing. Sometimes the route processor is built in to the system, while other times it is a module that can be replaced/upgraded. Many vendor multi-slot systems have multiple route processors for redundancy. An example of a removable route processor is the Cisco 9500 series Supervisor module. There are multiple versions available including revision A, with a 4 core processor and 16 GB of RAM and revision B with a 6 core processor and 24 GB of RAM. Previous systems such as the Cisco Catalyst 7600 had options such as the SUP720 (Supervisor Module 720) of which they offered multiple versions. The standard SUP720 had a limited number of routes that it could support (256k) versus the SUP720 XL which could support up to 1M routes. Juniper Route Engine In Juniper terminology, the controller is called a Route Engine. They are similar to the Cisco Route Processor/Supervisor modules. Unlike Cisco Supervisor modules which utilize special CPUS, Juniper's REs generally use common x86 CPUs. Like Cisco, Juniper multi-slot systems can have redundant processors. Juniper has recently released the information about the NG-REs or Next Generation Route Engines. One example is the new RE-S-X6-64G, a 6-core x86 CPU based routing engine with 64 GB DRAM and 2x 64 GB SSD storage available for the MX240/MX480/MX960. These NG-REs allow for containers and other virtual machines to be run directly. Built in processor When looking at single rack unit (1 RU) or pizza box design switches there are some important design considerations. Most 1 RU switches do not have redundant processors, or field replaceable route processors. In general the field replaceable units (FRUs) that the customer can replace are power supplies and fans. If the failure is outside of the available FRUs, the entire switch must be replaced in the event of a failure. With white box switches this can be a simple process as white box switches can be used in multiple locations of your network including the customer edge, provider edge and core. Sparing (keeping a spare switch) is easy when you have the same hardware in multiple parts of the network. Recently commodity switch fabric chips have come with built-in low power ARM CPUs that can be used to manage the entire system, leading to cheaper and less power hungry designs. Facebook Wedge microserver The Facebook Wedge is different from most white box switches as it has its controller as an add in module, the same board that is used in some of the OCP servers. By separating the controller board from the switch, different boards can be put in place, such as higher memory, faster CPUs, different CPU types, and so on. Routing protocols A routing protocol is a daemon that runs on a controller and communicates with other network devices to exchange route information. For this section we will use common words to demonstrate the way the routing protocol is working, these should not be construed as the actual way that the protocols talk. BGP Border Gateway Protocol (BGP) is a path vector based External Gateway Protocol (EGP) protocol that makes routing decisions based on paths, network policies, or rules (route-maps on Cisco). Though designed as a EGP, BGP can be used as both an interior (iboga) and exterior (eBGP) routing protocol. BGP uses keep alive packets (are you there?) to confirm that neighbors are still accessible. BGP is the protocol that is utilized to route traffic across the internet, exchanging routing information between different Autonomous Systems (AS). An AS is all of the connected networks under the control of a single entity such as Level 3 (AS1) or Sprint (AS1239). When two different ASes interconnect, BGP peering sessions are setup between two or more network devices that have direct connections to each other. In an eBGP scenario, AS1 and AS1239 would setup BGP peering sessions that would allow traffic to route between their AS. In an iBGP scenario, the same AS would peer with other routers with the same AS and transfer the routes that are defined on the system. While iBGP is used internally in most networks, iBGP is used in large corporate networks because other Interior Gateway Protocols (IGPs) may not scale. Examples: iBGP next hop self In this scenario AS1 and AS2 are peered with each other and exchanging one prefix each. AS1 advertises 192.168.1.0/24 and AS2 advertises 192.168.2.0/24. Each network has two routers, one border router, which connects to other ASes and one internal router which gets its routes from the border router. The routes are advertised internally with the next-hop set as the border router. This is a standard scenario when you are not running an IGP inside to distribute the routes for the border router external interfaces. The conversation goes like this: AS1 -> AS2: Hi AS2, I am AS1 AS2 -> AS1: Hi AS1, I am AS2 AS1 -> AS2: I have the following route, 192.168.1.0/24 AS2 - AS1: I have received the route, I have 192.168.2.0/24 AS1 - AS2: I have received the route AS1 -> Internal Router AS1: I have this route, 192.168.2.0/24, you can reach it through me at 10.1.1.1 AS2 -> Internal Router AS2: I have this route, 192.168.1.0/24, you can reach it through me at 10.1.1.1 iBGP next-hop unmodified In the next scenario the border routers are the same, but the internal routers are given a next-hop of the external (Other AS) border router. The last scenario is where you peer with a router server, a system that handles peering, filtering the routes based on what you have specified you send. The routes are then forwarded onto your peers with your IP as the next hop. OSPF Open Shortest Path First (OSPF) is a relatively simple protocol. Different links on the same router are put into the same or different areas. For example, you would use area 1 for the interconnects between campuses but you would use another area, such as area 10 for the campus itself. By separating areas, you can reduce the amount of cross talk that happens between devices. There are two versions of OSPF, v2 and v3. The main difference between v2 and v3 is that v2 is for IPv4 networks and v3 is for IPv6 networks. When there are multiple paths that can be taken, the cost of the links must be taken into account. Below you can see where there are two paths, one has a total cost of 20 (5+5+10), the other 16 (8+8) so the traffic will take the lowest cost link. IS-IS IS-IS is a link-state routing protocol, operating by flooding link state information throughout a network of routers using NETs (Network Entity Title). Each IS-IS router has its own database of the network topology, built by aggregating the flooded network information. IS-IS is used by companies who are looking for Fast convergence, scalability and Rapid flooding of new information. IS-IS uses the concept of levels instead of areas as in OSPF. There are two levels in IS-IS, Level 1 - area and Level2 - backbone. A Level 1 Intermediate System (IS), keeps track of the destinations within its area, while a Level 2 IS keep track of paths to the Level 1 areas. EIGRP Enhanced Interior Gateway Routing Protocol (EIGRP) is Cisco's proprietary routing protocol. It is hardly ever seen in current networks but if you see it in yours, then you need to plan accordingly. Replacing EIGRP with OSPF is suggested so that you can interoperate with non-cisco devices. RIP If Routing Information Protocol (RIP) is being used in your network, it must be replaced during the design. Most newer routing stacks do not support RIP. RIP is one of the original routing protocols, using the number of hops (routed ports) between the device and remote location to determine the optimal path. RIP sends its entire routing database out every 30 seconds. When routing tables were small, many years ago, RIP worked fine. With larger tables, the traffic bursts and resulting re-computing by other routers in the network causes routers to run at almost 100 percent CPU all the time. Cables Here we will review the major types of cables. Copper Copper cables have been around for a very long time, originally network devices were connected together using coax cable (the same cable used for television antennas and cable).  These days there are a few standard cables that are used. RJ45 Cables Cat5 - A 100Mb capable cable, used for both 10Mb and 100Mb connections  Cat5E - 1GbE capable cable but not suggested for 1GbE networks (Cat6 is better and the price difference is nominal). Cat6 - A 1GbE capable cable, can be used for any speed at or below 1GbE including 100Mb and 10Mb. SFPs SFP - Small Form-factor Pluggable port. Capable of up to 1GbE connections SFP+ - Same size as the SFP, capable of up to 10Gb connections SFP28 - Same size as the SFP, capable of up to 25Gb connections QSFP - Quad Small Form-factor Pluggable - A bit wider than the SFP but capable of multiple GbE connections QSFP+ - Same size as the QSFP - capable of 40GbE as 4x10GbE on the same cable QSFP28 - Same size as the QSFP - capable of 100GbE DAC - A direct attach cable that fits into a SFP or QSFP port Fiber/Hot pluggable Breakout Cables As routers and switches continue to become more dense, where the number of ports on the front of the device can no longer fit in the space, manufacturers have moved to what we call breakout cables. For example, if you have a switch that can handle 3.2Tb/s of traffic, you need to provide 3200Gbp/s of port capacity. The easiest way to do that is to use 32 100Gb ports which will fit on the front of a 1U device.  You cannot fit 128 10Gb ports without using either a breakout patch panel (which will then use another few rack units (RUs), or a breakout cable. For a period of time in the 1990's, Cisco used RJ21 connectors to provide up to 96 ethernet ports per slot Network engineers would then create breakout cables to go from the RJ21 to RJ45. These days, we have both DAC (Direct Attach Cable) and Fiber breakout cables. For example, here you can see a 1x4 breakout cable, providing 4 10g or 25G ports from a single 40G or 100G port. If you build a LAN network that only includes switches that provide layer 2 connectivity, any devices you want to connect together need to be in the same IP block. If you have a router in your network, it can route traffic between IP blocks. Part 1: What defines a modern network There is a litany of concepts that define a modern network, from simple principles to full feature sets. In general, a next-generation data center design enables you to move to a widely distributed non-blocking fabric with uniform chipset, bandwidth, and buffering characteristics in a simple architecture. In one example, to support these requirements, you would begin with a true three-tier Clos switching architecture with Top of Rack (ToR), spine, and fabric layers to build a data center network. Each ToR would have access to multiple fabrics and have the ability to select a desired path based on application requirement or network availability. Following the definition of a modern network from the introduction, here we layout the general definition of the parts. Modern network pieces Here we will discuss the concepts that build a Next Generation Network (NGN). Software Defined Networks Software defined networks can be defined in multiple ways. The general definition of a Software defined network is one which can be controlled as a singular unit instead of at a system by system basis. The control-plane which would normally be in the device and using routing protocols is replaced with a controller. Software defined networks can be built using many different technologies including OpenFlow, overlay networks and automation tools. Next generation networking and hyper scale networks As we mention in the introduction, twenty years ago NGN hardware would have been the Cisco GSR (officially introduced in 1997) or the Juniper M40 (officially released in 1998). Large Cisco and Juniper customers would have been working with the companies to help come up with the specifications and determining how to deploy the devices (possibly Alpha or Beta versions) in their networks. Today we can look at the hyper scale networking companies to see what a modern network looks like. A hyper scale network is one where the data stored, transferred and updated on the network grows exponentially. Technology such as 100Gb Ethernet, software defined networking, Open networking equipment and software are being deployed by hyper scale companies. Open networking hardware overview Open Hardware has been around for about 10 years, first in the consumer space and more recently in the enterprise space. Enterprise open networking hardware companies such as Quanta and Accton provide a significant amount of the hardware currently utilized in networks today. Companies such as Google and Facebook have been building their own hardware for many years. Facebook's routers such as the Wedge 100 and Backpack are available publicly for end users to utilize. Some examples of Open Networking hardware are: The Dell S6000-ON - a 32x40G switch with 32 QSFP ports on the front. The Quanta LY8 - a 48x10G + 6x40G switch with 48 SFP+ ports and 6 QSFP ports. The Facebook Wedge 100 - a 32x100G switch with 32 QSFP28 ports on the front. Open networking software overview To use open networking hardware, you need an operating system. The operating system manages the system devices such as fans, power, LEDs and temperature. On top of the operating system you will run a forwarding agent, examples of forwarding agents are Indigo, the open source OpenFlow daemon and Quagga, an open source routing agent. Closed networking hardware overview Cisco and Juniper are the leaders in the Closed Hardware and Software space. Cisco produces switches like the Nexus series (3000, 7000, 9000) with the 9000 programmable by ACI. Juniper provides the MX series (480, 960, 2020) with the 2020 being the highest end forwarding system they sell. Closed networking software overview Cisco has multiple network operating systems including IOS, NX-OS, IOS-XR. All Cisco NOSs are closed source and proprietary to the system that they run on. Cisco has what the industry calls a "industry standard CLI" which is emulated by many other companies. Juniper ships a single NOS, JunOS which can install on multiple different systems. JunOS is a closed source BSD based NOS. The JunOS CLI is significantly different from IOS and is more focused on engineers who program. Network Virtualization Not to be confused with Network Function Virtualization (NFV), Network virtualization is the concept of re-creating the hardware interfaces that exist in a traditional network in software. By creating a software counterpart to the hardware interfaces, you decouple the network forwarding from the hardware. There are a few companies and software projects that allow the end user to enable network virtualization. The first one is NSX which comes from the same team that developed OvS (Open Virtual Switch) Nicira, which was acquired by VMWare in 2012. Another project is Big Switch Networks Big Cloud Fabric, which utilizes a heavily modified version of Indigo, an OpenFlow controller. Network Function Virtualization Network Function Virtualization can be summed up by the statement that: "Due to recent network focused advancements in PC hardware, any service able to be delivered on proprietary, application specific hardware should be able to be done on a virtual machine". Essentially: routers, firewalls, load balancers and other network devices all running virtualized on commodity hardware. Traffic Engineering Traffic engineering is a method of optimizing the performance of a telecommunications network by dynamically analyzing, predicting and regulating the behavior of data transmitted over that network. Part 2: Next generation networking examples In my 25 or so years of networking, I have dealt with a lot of different networking technologies, each iteration (supposedly) better than the last. Starting with Thin Net (10BASE2), moving through ArcNet, 10BASE-T, Token Ring, ATM to the Desktop, FDDI and onwards. Generally, the technology improved for each system until it was swapped out. A good example is the change from a literal ring for token ring to a switching design where devices hung off of a hub (as in 10BASE-T). ATM to the desktop was a novel idea, providing up to 25Mbps to connected devices, but the complexity of configuring and managing it was not worth the gain. Today almost everything is Ethernet as shown by the Facebook Voyager DWDM system, which uses Ethernet over both traditional SFP ports and the DWDM interfaces.  Ethernet is simple, well supported and easy to manage. Example 1 - Migration from FDDI to 100Base-T In late 1996, early 1997, the Exodus network used FDDI rings (Fiber Distributed Data Interface) to connect the main routers together at 100Mbps. As the network grew we had to decide between two competing technologies, FDDI switches and Fast Ethernet (100Base-T) both providing 100Mbp/s. FDDI switches from companies like DEC (FDDI Gigaswitch) were used in most of the Internet Exchange Points (IXPs) and worked reasonably well with one minor issue, head of line blocking (HOLB), which also impacted other technologies. Head of line blocking occurs when a packet is destined for an interface that is already full, so a queue is built, if the interface continues to be full, eventually the queue will be dropped. While we were testing the DEC FDDI Gigaswitches, we were also in deep discussions with Cisco about the availability of Fast Ethernet (FE) and working on designs. Because FE was new, there were concerns about how it would perform and how we would be able to build a redundant network design. In the end, we decided to use FE, connect the main routers in a full mesh and use routing protocols to manage fail-over. Example 2 - NGN Failure - LANE (LAN Emulation) During the high growth period at Exodus communications, there was a request to connect a new data center to the original one and allow customers to put servers in both locations using the same address space. To do this, we chose LAN Emulation or LANE which allows a ATM network to be used like a LAN. On paper, LANE looked like a great idea, the ability to extend the LAN so that customers could use the same IP space in two different locations. In reality, it was very different. For hardware, we were using Cisco 5513 switches which provided a combination of Ethernet and ATM ports. There were multiple issues with this design: First, the customer is provided with an ethernet interface, which runs over an ATM optical interface.  Any error on the physical connection between switches or the ATM layer would cause errors on the Ethernet layer. Second, monitoring was very hard, when there were network issues, you had to look in multiple locations to determine where the errors were happening. After a few weeks, we did a midnight swap putting Cisco 7500 routers in to replace the 5500 switches and moving customers onto new blocks for the new data center. Part 3: Designing a modern network When designing a new network, some of the following might be important to you: Simple, focused yet non-blocking IP fabric Multistage parallel fabrics based on Clos network concept Simple merchant silicon Distributed control plane with some centralized controls Wide multi-path (ECMP) Uniform chipset, bandwidth, and buffering 1:1 oversubscribed (non-blocking fabric) Minimize the hardware necessary to carry east–west traffic Ability to support a large number of bare metal servers without adding an additional layer Limit fabric to a 5 stage Clos within the data center to minimize lookups and switching latency. Support host attachment at 10G, 25G, 50G and 100G Ethernet Traffic management In a modern network one of the first decisions is whether you will use a centralized controller or not. If you use a centralized controller, you will be able to see and control the entire network from one location. If you do not use a centralized controller, you will need to either manage each system directly or via automation. There is a middle space where you can use some software defined network pieces to manage parts of the network, such as an OpenFlow controller for the WAN or VMware NSX for your virtualized workloads. Once you know what the general management goal is, the next decision is whether to use open, proprietary, or a combination of both open and proprietary networking equipment. Open networking equipment is a concept that has been around less than a decade and started when very large network operators decided that they wanted a better control of the cost and features of the equipment in their networks. Google is a good example. In the following figure, you can see how Facebook used both their own hardware, 6-Pack/Backpack and legacy vendor hardware for their interoperability and performance testing. Google wanted to build a high-speed backbone, but was not looking to pay the prices that the incumbent proprietary vendors such as Cisco and Juniper wanted. Google set a price per port (1G/10G/40G) that they wanted to hit and designed equipment around that. Later companies like Facebook decided to go the same direction and contracted with commodity manufacturers to build network switches that met their needs. Proprietary vendors can offer the same level of performance or better using their massive teams of engineers to design and optimize hardware. This distinction even applies on the software side where companies like VMware and Cisco have created software defined networking tools such as NSX and ACI. With the large amount of networking gear available, designing and building a modern network can appear to be a complex concept. Designing a modern network requires research and a good understanding of networking equipment. While complex, the task is not hard if you follow the guidelines. These are a few of the stages of planning that need to be followed before the modern network design is started: The first step is to understand the scope of the project (single site, multi-site, multi-continent, multi-planet). The second step is to determine if the project is a green field (new) or brown field deployment (how many of the sites already exist and will/will not be upgraded). The third step is to determine if there will be any software defined networking (SDN), next generation networking (NGN) or Open Networking pieces. Finally, it is key that the equipment to be used is assembled and tested to determine if the equipment meets the needs of the network. Summary In this article, we have discussed many different concepts that tie NGN together. The term NGN refers to the latest and near-term networking equipment and designs. We looked at networking concepts such as local, metro and wide area networks, network controllers, routers and switches. Routing protocols such as BGP, IS-IS, OSPF and RIP. Then we discussed many pieces that are used either singularly or together that create a modern network. In the end, we also learned some guidelines that should be followed while designing a network. Resources for Article:   Further resources on this subject: Analyzing Social Networks with Facebook [article] Social Networks [article] Point-to-Point Networks [article]

In this article by Lentin Joseph, the author of the book ROS Robotic Projects, we learn how one of the capability in most of the service and social robots is face detection and tracking. The robot can identify faces and it can move its head according to the human face move around it. There are numerous implementation of face detection and tracking system in web. Most of the trackers are having a pan and tilt mechanism and a camera is mounted on the top of the servos. In this article, we are going to see a simple tracker which is having only pan mechanism. We are going to use a USB webcam which is mounted on AX-12 Dynamixel servo. (For more resources related to this topic, see here.) You can see following topics on this article: Overview of the project Hardware and software prerequisites Overview of the project The aim of the project is to build a simple face tracker which can track face only in the horizontal axis of camera. The tracker is having a webcam, Dynamixel servo called AX-12 and a supporting bracket to mount camera on the servo. The servo tracker will follow the face until it align to the center of the image which is getting from webcam. Once it reaches the center, it will stop and wait for the face movement. The face detection is done using OpenCV and ROS interface, and controlling the servo is done using Dynamixel motor driver in ROS. We are going to create two ROS packages for this complete tracking system, one is for face detection and finding centroid of face and next is for sending commands to servo to track the face using the centroid values. Ok!! Let's start discussing the hardware and software prerequisites of this project. Hardware and software prerequisites Following table of hardware components which can be used for building this project. You can also see a rough price of each component and purchase link of the same. List of hardware components: No Component name Estimated price (USD) Purchase link 1 Webcam 32 https://amzn.com/B003LVZO8S 2 Dynamixel AX -12 A servo with mounting bracket 76 https://amzn.com/B0051OXJXU 3 USB To Dynamixel Adapter 50 http://www.robotshop.com/en/robotis-usb-to-dynamixel-adapter.html 4 Extra 3 pin cables for AX-12 servos 12 http://www.trossenrobotics.com/p/100mm-3-Pin-DYNAMIXEL-Compatible-Cable-10-Pack 5 Power adapter 5 https://amzn.com/B005JRGOCM 6 6 Port AX/MX Power Hub 5 http://www.trossenrobotics.com/6-port-ax-mx-power-hub 7 USB extension cable 1 https://amzn.com/B00YBKA5Z0   Total Cost + Shipping + Tax ~ 190 - 200   The URLs and price can vary. If the links are not available, you can do a google search may do the job. The shipping charges and tax are excluded from the price. If you are thinking that, the total cost is not affordable for you, then there are cheap alternatives to do this project too. The main heart of this project is Dynamixel servo. We may can replace this servo with RC servos which only cost around $10 and using an Arduino board cost around $20 can be used to control the servo too, so you may can think about porting the face tracker project work using Arduino and RC servo Ok, let's look on to the software prerequisites of the project. The prerequisites include ROS framework, OS version and ROS packages: No Name of software Estimated price (USD) Download link 1 Ubuntu 16.04 L.T.S Free http://releases.ubuntu.com/16.04/ 2 ROS Kinetic L.T.S Free http://wiki.ros.org/kinetic/Installation/Ubuntu 3 ROS usb_cam package Free http://wiki.ros.org/usb_cam 3 ROS cv_bridge package Free http://wiki.ros.org/cv_bridge 4 ROS Dynamixel controller Free https://github.com/arebgun/dynamixel_motor 5 Windows 7 or higher ~ $120 https://www.microsoft.com/en-in/software-download/windows7 7 RoboPlus (Windows application) Free http://www.robotis.com/download/software/RoboPlusWeb%28v1.1.3.0%29.exe The above table will gives you an idea about which all are the software we are going to be used for this project. We may need both Windows and Ubuntu for doing this project. It will be great if you have dual operating system on your computer Let's see how to install these software first Installing dependent ROS packages We have already installed and configured Ubuntu 16.04 and ROS Kinetic on it. Here are the dependent packages we need to install for this project. Installing usb_cam ROS package Let's see the use of usb_cam package in ROS first. The usb_cam package is ROS driver for Video4Linux (V4L) USB camera. The V4L is a collection of devices drivers in Linux for real time video capture from webcams. The usb_cam ROS package work using the V4L devices and publish the video stream from devices as ROS image messages. We can subscribe it and do our own processing using it. The official ROS page of this package is given in the above table. You may can check this page for different settings and configuration this package offers. Creating ROS workspace for dependencies Before starting installing usb_cam package, let's create a ROS workspace for keeping the dependencies of the entire projects mentioned in the book. We may can create another workspace for keeping the project code. Create a ROS workspace called ros_project_dependencies_ws in home folder. Clone the usb_cam package into the src folder: $ git clone https://github.com/bosch-ros-pkg/usb_cam.git Build the workspace using catkin_make After building the package, install v4l-util Ubuntu package. It is a collection of command line V4L utilities which is using by usb_cam package: $ sudo apt-get install v4l-utils Configuring webcam on Ubuntu 16.04 After installing these two, we can connect the webcam to PC to check it properly detected in our PC. Take a terminal and execute dmesg command to check the kernel logs. If your camera is detected in Linux, it may give logs like this{ $ dmesg Kernels logs of webcam device You can use any webcam which has driver support in Linux. In this project, iBall Face2Face (http://www.iball.co.in/Product/Face2Face-C8-0--Rev-3-0-/90) webcam is used for tracking. You can also go for a popular webcam which is mentioned as a hardware prerequisite. You can opt that for better performance and tracking. If our webcam has support in Ubuntu, we may can open the video device using a tool called cheese. Cheese is simply a webcam viewer. Enter the command cheese in the terminal, if it is not available you can install it using following command: $ sudo apt-get install cheese If the driver and device are proper, you may get the video stream from webcam like this: Webcam video streaming using cheese Congratulation!!, your webcam is working well in Ubuntu, but are we done with everything? No. The next thing is to test the ROS usb_cam package. We have to make sure that is working well in ROS!! Interfacing Webcam to ROS Let's test the webcam using usb_cam package. The following command is used to launch the usb_cam nodes to display images from webcam and publishing ROS image topics at the same time: $ roslaunch usb_cam usb_cam-test.launch If everything works fine, you will get the image stream and logs in the terminal as shown below: Working of usb_cam package in ROS The image is displayed using image_view package in ROS, which is subscribing the topic called /usb_cam/image_raw Here are the topics, that usb_cam node is publishing: Figure 4: The topics publishing by usb_cam node We have just done with interfacing a webcam in ROS. So what's next? We have to interface AX-12 Dynamixel servo to ROS. Before proceeding to interfacing, we have to do something to configure this servo. Next we are going to see how to configure a Dynamixel servo AX-12A. Configuring a Dynamixel servo using RoboPlus The configuring of Dynamixel servo can be done using a software called RoboPlus providing by ROBOTIS INC (http://en.robotis.com/index/), the manufacturer of Dynamixel servos. For configuring Dynamixel, you have to switch your operating system to Windows. The tool RoboPlus will work on Windows. In this project, we are going to configure the servo in Windows 7. Here is the link to download RoboPlus: http://www.robotis.com/download/software/RoboPlusWeb%28v1.1.3.0%29.exe. If the link is not working, you can just search in google to get the RoboPlus 1.1.3 version. After installing the software, you will get the following window, navigate to Expert tab in the software for getting the application for configuring Dynamixel: Dynamixel Manager in RoboPlus Before taking the Dynamixel Wizard and do configuring, we have to connect the Dynamixel and properly powered. Following image of AX-12A servo that we are using for this project and its pin connection. AX-12A Dynamixel and its connection diagram Unlike other RC servos, AX-12 is an intelligent actuator which is having a microcontroller which can monitoring every parameters of servo and customize all the servo parameters. It is having a geared drive and the output of the servo is connected to servo horn. We may can connect any links on this servo horn. There are two connection ports behind each servo. Each port is having pins such as VCC, GND and Data. The ports of Dynamixel are daisy chained so that we can connect another servo from one servo. Here is the connection diagram of Dynamixel with PC. AX-12A Dynamixel and its connection diagram The main hardware component which interfacing Dynamixel to PC is called USB to Dynamixel. This is a USB to serial adapter which can convert USB to RS232, RS 484 and TTL. In AX-12 motors, the data communication is using TTL. From the Figure AX 12A Dynamixel and its connection diagram, we can seen that there are three pins in each port. The data pin is used to send and receive from AX-12 and power pins are used to power the servo. The input voltage range of AX-12A Dynamixel is from 9V to 12V. The second port in each Dynamixel can be used for daisy chaining. We can connect up to 254 servos using this chaining Official links of AX-12A servo and USB to Dynamixel AX-12A: http://www.trossenrobotics.com/dynamixel-ax-12-robot-actuator.aspx USB to Dynamixel: http://www.trossenrobotics.com/robotis-bioloid-usb2dynamixel.aspx For working with Dynamixel, we should know some more things. Let's have a look on some of the important specification of AX-12A servo. The specifications are taken from the servo manual. Figure 8: AX-12A Specification The Dynamixel servos can communicate to PC to a maximum speed of 1 Mbps. It can also give feedback of various parameters such as its position, temperature and current load. Unlike RC servos, this can rotate up to 300 degrees and communication is mainly using digital packets. Powering and connecting Dynamixel to PC Now we are going to connect Dynamixel to PC. Given below a standard way of connecting Dynamixel to PC: Connecting Dynamixel to PC The three pin cable can be first connected to any of the port of AX-12 and other side have to connect to the way to connect 6 port power hub. From the 6-port power hub, connect another cable to the USB to Dynamixel. We have to select the switch of USB to Dynamixel to TTL mode. The power can be either be connected through a 12V adapter or through battery. The 12V adapter is having 2.1X5.5 female barrel jack, so you should check the specification of male adapter plug while purchasing. Setting USB to Dynamixel driver on PC As we have already discussed the USB to Dynamixel adapter is a USB to serial convertor, which is having an FTDI chip (http://www.ftdichip.com/) on it. We have to install a proper FTDI driver on the PC for detecting the device. The driver may need for Windows but not for Linux, because FTDI drivers are built in the Linux kernel. If you install the RoboPlus software, the driver may be already installed along with it. If it is not, you can manually install from the RoboPlus installation folder. Plug the USB to Dynamixel to the Windows PC, and check the device manager. (Right click on My Computer | Properties | Device Manager). If the device is properly detected, you can see like following figure: Figure 10: COM Port of USB to Dynamixel If you are getting a COM port for USB to Dynamixel, then you can start the Dynamixel Manager from RoboPlus. You can connect to the serial port number from the list and click the Search button to scan for Dynamixel as shown in following figure. Select the COM port from the list and connecting to the port is marked as 1. After connecting to the COM port, select the default baud rate as 1 Mbps and click the Start searching button: COM Port of USB to Dynamixel If you are getting a list of servo in the left side panel, it means that your PC have detected a Dynamixel servo. If the servo is not detecting, you can do following steps to debug: Make sure that supply is proper and connections are proper using a multi meter. Make sure that servo LED on the back is blinking when power on. If it is not coming, it can be a problem with servo or power supply. Upgrade the firmware of servo using Dynamixel Manager from the option marked as 6. The wizard is shown in the following figure. During wizard, you may need power off the supply and ON it again for detecting the servo. After detecting the servo, you have to select the servo model and install the new firmware. This may help to detect the servo in the Dynamixel manager if the existing servo firmware is out dated. Dynamixel recovery wizard If the servos are listing on the Dynamixel manager, click on a servo and you can see its complete configuration. We have to modify some values inside the configurations for our current face tracker project. Here are the parameters: ID : Set the ID as 1 Baud rate: 1 Moving Speed: 100 Goal Position: 512 The modified servo settings are shown in the following figure: Modified Dynamixel firmware settings After doing these settings, you can check the servo is working good or not by changing its Goal position. Yes!! Now you are done with Dynamixel configuration, Congratulation!! What's next? We want to interface Dynamixel to ROS. Summary This article was about building a face tracker using webcam and Dynamixel motor. The software we have used was ROS and OpenCV. Initially you can see how to configure the webcam and Dynamixel motor and after configuring, we were trying to build two package for tracking. One package does the face detection and second package is a controller which can send position command to Dynamixel to track the face. We have discussed the use of all files inside the packages and did a final run to show the complete working of the system. Resources for Article: Further resources on this subject: Using ROS with UAVs [article] Hardware Overview [article] Arduino Development [article]

Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!This article is an excerpt from the book, Data Cleaning with Power BI, by Gus Frazer. Unlock the full potential of your data by mastering the art of cleaning, preparing, and transforming data with Power BI for smarter insights and data visualizationsIntroductionFor those who have never heard of Microsoft  Copilot, it is a new technology that Microsoft has released across a number of its platforms that combines generative AI with your data to enhance productivity. Copilot for Power BI harnesses cutting-edge generative AI alongside your dataset, revolutionizing the process of uncovering and disseminating insights with unprecedented speed. Seamlessly integrated into your workflow, Copilot offers an array of functionalities aimed at streamlining your reporting experience.When it comes to report creation, Copilot streamlines the process by allowing users to effortlessly generate reports by articulating the insights they seek or posing questions regarding their dataset using NLP. Copilot then analyzes the data, pulling together relevant information to craft visually striking reports, thereby transforming raw data into actionable insights instantaneously. Moreover, Copilot has the ability to read your data and suggest the best position to begin your analysis, which can then be tailored to suit the direction you want to take the analysis in.This is great, but how can it help you clean and prepare data for analysis? Well, Copilot can be leveraged on multiple data tools from within the Microsoft  Fabric platform. For those who are not aware, Power BI has now become part of the Fabric platform. Depending on what type of license you have for Power BI, you might already have access to this. Any customers with Premium capacity licensing for the Power BI service would have automatically been given access to Microsoft  Fabric, and more importantly, Copilot.That being said, currently, Copilot has only been made available to customers with a P1 (or above) Premium capacity or a Fabric license of F64 (or above), which is the equivalent licensing available directly from the Azure portal.If you would like to follow along with the next example, you will need to set up a Fabric capacity within your Azure portal. Don’t worry, you can pause this service when it’s not being used to ensure you are only charged for the time you’re using it. Alternatively, follow the steps to see the outcome:1. Log in to the Azure portal that you set up in the previous section of this chapter.2. Select the search bar at the top of the page and type in Microsoft Fabric. Select the service in the menu that appears below the search bar, which should take you to the page where you can manage your capacities.3. Select Create a Fabric capacity. Note that you will need to use an organizational account in order to create a Fabric capacity as opposed to a personal account. You can sign up for a Microsoft  Fabric trial for your organization within the window. Further details on how to do this are provided here: https://learn.microsoft.com/en-us/power-bi/ enterprise/service-admin-signing-up-for-power-bi-with-a-newoffice-365-trial.4. Select the subscription and resource group you would like to use for this Fabric capacity.5. Then, under capacity details, you can enter your capacity name. In this example, you can call it cleaningdata.6. The Region field should populate with the region of your tenant, but you can change this if you like. However, this may have implications on performance, which it should warn you about with a message.7. Set the capacity to F64.8. Then, click on select Review + create.9. Review the terms and then click on Create, which will begin the deployment of your capacity.10. Once deployed, select Go to resource to view your Fabric capacity. Take note that this will be active once deployed. Make sure to return here aft er testing to pause or delete your Fabric capacity to prevent yourself from getting charged for this service.Now you will need to ensure you have activated the Copilot settings from within your Fabric capacity. To do this, go to https://app.powerbi.com/admin-portal/ to log in and access the admin portal.Important tipIf you can’t see the Tenant settings tab, then you will need to ensure you have been set up as an admin within your Microsoft  365 admin center. If you have just created a new account, then you will need to set this up. Follow the next links to assign roles:• https://learn.microsoft.com/en-us/microsoft-365/admin/addusers/assign-admin-roles• https://learn.microsoft.com/en-us/fabric/admin/microsoftfabric-admin11. Scroll to the  bottom of Tenant settings until you see the Copilot and Azure OpenAI service (preview) section as shown:Figure  – The tenant settings from within Power BI12. Ensure both settings are set to Enabled and then click on Apply.Now that you have created your Fabric capacity, let’s jump into an example of how we can use Copilot to help with the cleaning of data. As we have created a new capacity, you will have to create a new workspace that uses this new capacity:1. Navigate back to Workspaces using the left navigation bar. Then, select New Workspace.2. Name your workspace CleaningData(Copilot), then select the dropdown for advanced configuration settings.3. Ensure you have selected Fabric capacity in the license mode, which in turn will have selected your capacity below, and then select Apply. You have now created your capacity!4. Now let’s use Fabric to create a new dataflow using the latest update of Datafl ow Gen2. Select New from within the workspace and then select More options.5. This will navigate you to a page with all the possible actions to create items within your Fabric workspace. Under Data Factory, select Datafl ow Gen2.6. This will load a Datafl ow Gen2 instance called Datafl ow 1. On the top row, you should now see the Copilot logo within the Home ribbon as highlighted:Figure – The ribbon within a Dataflow Gen2 instance7. Select Copilot to open the Copilot window on the right-hand side of the page. As you have not connected to any data, it will prompt you to select get data.8. Select Text/CSV and then enter the following into the File path or URL box:https://raw.githubusercontent.com/PacktPublishing/Data-Cleaningwith-Power-BI/main/Retail%20Store%20Sales%20Data.csv9. Leave the rest of the settings as their defaults and click on Next.10. This will then open a preview of the file data. Click on Create to load this data into your Datafl ow Gen2 instance. You will see that the Copilot window will have now changed to prompt you as to what you would like to do (if it hasn’t, then simply close the Copilot window and reopen):Figure – Data loaded into Dataflow Gen211. In this example, we can see that the data includes a column called Order Date but we don’t have a fi eld for the fi scal year. Enter the following prompt to ask Copilot to help with the transformation:There's a column in the data named Order Date, which shows when an order was placed. However, I need to create a new column from this that shows the Fiscal Year. Can you extract the year from the date and call this Fiscal Year? Set this new column to type number also.12. Proceed using the arrow key or press Enter. Copilot will then begin working on your request. As you will see in the resulting output, the model has added a function (or step) called Custom to the query that we had selected.13. Scroll to the far side and you will see that this has added a new column called Fiscal Year.14. Now add the following prompt to narrow down our data and press Enter:Can you now remove all columns leaving me with just Order ID, Order Date, Fiscal year, category, and Sales?15. This will then add another function or step called Choose columns. Finally, add the following prompt to aggregate this data and press Enter:Can you now group this data by Category, Fiscal year, and aggregated by Sum of Sales?As you can see, Copilot has now added another function called Custom 1 to the applied steps in this query, resulting in this table:Figure – The results from asking Copilot to transform the dataTo view the M query that  Copilot has added, select Advanced editor, which will show the functions that Copilot has added for you:Figure – The resulting M query created by Copilot to carry out the request transformations to clean the dataIn this example, you explored the new technologies available with Copilot and how they help to transform the data using tools such as Datafl ow Gen2.While it’s great to understand the amazing possibilities AI brings to data, it’s also crucially important that you understand the challenges it presents.ConclusionIn conclusion, Microsoft Copilot offers a groundbreaking approach to enhancing productivity and efficiency in data analysis and report generation within Power BI. By seamlessly integrating generative AI technology, Copilot revolutionizes the way insights are discovered and data is prepared, providing users with unprecedented speed and accuracy. Whether streamlining report creation or optimizing data management tasks, Copilot empowers users to unlock the full potential of their data, paving the way for more informed decision-making and actionable insights.Author BioGus Frazer is a seasoned Analytics Consultant focused on Business Intelligence solutions. With over 7 years of experience working for the two market-leading platforms, Power BI & Tableau, has amassed a wealth of knowledge and expertise. Gus has helped hundreds of customers to drive their digital and data transformations, scope data requirements, drive actionable insights, and most important of all, cleanse data ready for analysis. Most recently helping to set up, organize and run the Power BI UK community at Microsoft. He holds 6 Azure and Power BI certifications, including the PL-300 and DP-500 certifications. In this book, Gus offers readers invaluable guidance on ingesting, preparing, and cleansing data for analysis in Power BI. --This text refers to an out of print or unavailable edition of this title.

In the last few years, a major shift has occurred within the software industry in terms of how companies organize themselves and manage product responsibilities. The result is a merging of development and operations roles under a single umbrella known as DevOps. However, that’s not where the story ends. More and more businesses are beginning to realize that cybersecurity strategy should not be treated as an independent activity. Software reliability is completely dependent upon security and risk management; otherwise, you leave your organization vulnerable to external attacks. The result of this thinking has been an increase in the scope of the DevOps role, adding a security aspect as well also known as DevSecOps. But not every company can easily shift to the DevSecOps model overnight, especially when communication issues get in the way. In this article, we'll cover five of the most common roadblocks and ways to avoid them. 1. Unclear responsibilities Now that the DevSecOps trend has gone mainstream in the software industry, you'll see many new job listings popping up in this area. Unfortunately, some companies are using the term as a catch-all to throw various disconnected duties at a single person. The result can be quite frustrating. Leadership and management need to set a clear scope for the responsibilities of DevSecOps engineers and integrate them directly with other parts of the organization, including development, quality assurance, and cloud administrators. DevSecOps can only work as a strategy if it's fully adopted and understood by people at every level. It's also important to be careful not to characterize the DevSecOps concept as a group of tools. The focus should always remain on the individuals performing their duties and not the applications that they use. Clarifying this line of distinction can make it easier to communicate within your organization. 2. Missing connection to end-users Engineers in the DevSecOps role should be involved at every phase of the software development lifecycle. Otherwise, they will not have a holistic view of the platform's security status and how it is functioning. In a worst-case scenario, the DevSecOps team is disconnected from end-users entirely. If an engineer does not understand how real people are using their application, then the product as a whole is likely doomed. User requirements should form the basis of every coding project, and supporting the development lifecycle is only possible if that link exists and is maintained. 3. Too many (and unsecured) communication tools Engineers in a DevSecOps role often spend the majority of their days coordinating between other groups within the organization. This activity can't succeed unless there is a strong communication tool set at their disposal. One mistake many companies make is deciding to invest in dozens of different chat, messaging, and conferencing apps in hopes that it will make things easier. The problem is that easy online communication comes at the price of privacy. Some platforms retain your data for their own internal use or even to sell - in the form of uniquely identifiable IP addresses - to advertisers. Though it’s relatively easy to hide your IP address, do you want to trust an app that plays fast and loose with your information in the first place? The problem is that all this ease of communication comes at a price which often includes data retention or sharing with third parties of uniquely identifiable information like IP addresses, which can be hidden a few different ways - but why trust an app that reveals it in the first place? One way a DevSecOps team can address this issue is to emphasize the security risk to decision-makers in regards to many popular tools. Slack, WhatsApp, Snapchat and others are recent examples of a popular messaging apps that are now taking flak because of different security risks they pose. When it comes to email, even Gmail, the most popular email service, has been caught allowing unfettered access to user email addresses. Our advice is to use an encrypted email tool such as ProtonMail or Mailfence rather than rely on the usual suspects with with better name recognition. The more communication tools you use, the larger the threat surface vulnerable to hackers. 4. Alert fatigue One key part of the DevSecOps suite of responsibilities is to streamline all monitoring and alerting functions across the organization. The goal is to make it easy for both managers and engineers to find out about live issues and quickly navigate to the source of the problem. In some cases, DevSecOps engineers will be asked to set very sensitive alerting protocols so that no potential problem is missed. There is a downside, though, because having too many notifications can lead to alert fatigue. This is especially true if your monitoring tools are throwing false positives all day long. A string of unnecessary alerts will cause people to stop paying attention to them. An approach to alerting should be well thought out and clearly documented in runbooks by the DevSecOps team. A runbook should explain exactly what an alert means and the steps required to address it. This level of documentation allows DevSecOps engineers to outsource incident response to a larger group. 5. Hidden dependencies Because of the wide scope of the DevSecOps role, sometimes organizations expect engineers to be fortune-tellers and be able to predict how changes will impact code, tests, security. This level of confidence cannot be reached unless there is clear and consistent communication across the company. Take for example a decision to add a firewall protection around a database server to block outside threats. This will probably seem like a simple change for the engineers working on the system, but they may not realize that a new firewall could cut off connections to other services within the same infrastructure. If DevSecOps was involved in the meetings and decision making, then this type of hidden dependency could have been uncovered earlier. The DevSecOps model can only succeed if the organization has a strong policy of change management. Any modification to a live system should be thoroughly vetted by representatives of all teams. At that time, risks can be weighed and approvals can be made. Changes should be scheduled at times when the impact will be minimal. Final thoughts When browsing job listings, you'll surely see an influx of roles mentioning both DevOps and DevSecOps. These roles can have an incredibly wide scope and often play a critical role in the success of a software company. But breakdowns in communication have the potential to derail the goals of DevSecOps putting the entire organization at risk. Modern software development is all about being agile, meaning that requirements gathering and coding all happen with great flexibility and fluidity. The same should be true for DevSecOps. The duties of this role should be evaluated and tweaked on a regular basis and clear communication is the best way to go about it. Author Bio Gary Stevens is a front-end developer. He’s a full-time blockchain geek and a volunteer working for the Ethereum foundation as well as an active Github contributor. Why do IT teams need to transition from DevOps to DevSecOps? The seven deadly sins of web design Dark Web Phishing Kits: Cheap, plentiful and ready to trick you

Can it be real? The complete DevOps toolchain integrated into one tool, one UI and one process? GitLab seems to think so. GitLab has already made huge strides in terms of centralizing the DevOps process into a single tool. Up until now, most of the focus has been on creating a seamless development system and operations have not been as important. What’s new is the extension of the tool to include the operating side of DevOps as well as the development side. Let's talk a little bit about what DevOps is in order to fully appreciate the advances offered by GitLab. DevOps is basically a holistic approach to software development, quality assurance, and operations. While each of these elements of software creation is distinct, they are all heavily reliant on the other elements to be effective. The DevOps approach is to acknowledge this interdependence and then try to leverage the interdepence to increase productivity and to enhance the final user experience. Two of the most talked about elements of DevOps are continous integration and continuous deployment. Continuous integration and deployment Continuous integration and deployment are aimed at continuously integrating changes to a codebase, potentially from multiple sources, and then continuously deploying these changes into production. These tools require a pretty sophisticated automation and testing framework in order to be really effective. There are plenty of tools for one or the other, but the notion behind GitLab is essentially that if you can affect both of these processes from the same UI, these processes would be that much more efficient. GitLab has shown this to be true.  There is also the human side to consider, that is, coming up with what tasks need to be performed, assigning these tasks to developers and monitoring their progress. GitLab offers tools that help streamline this process as well. You can track issues, create issue boards to organize workflow and these issue boards can be sliced a number of different ways so that most imaginable human organizational needs can be met. Monitoring and delivery So far, we’ve seen that DevOps is about bringing everything together into a smooth process, and GitLab wants that process to occur in one place. GitLab can help you from planning to deployment and everywhere in between. But, GitLab isn’t satisfied with stopping at deployment, and they shouldn’t be. When we think about the three legs of DevOps, development, operations, and quality assurance and testing, what I’ve said about GitLab really only applies to the development leg. This is an unfortunately common problem with DevOps tools and organizational strategies. They seem to cater to developers and basically no one else. Maybe devs complain the most, I don’t know. GitLab has basically solved the DevOps problems between planning and deployment and, naturally, wants to move on to the monitoring and delivery of applications. This is a really exciting direction. After all, software is ultimately about making things happen. Sometimes it's easy to lose sight of this and only focus on the tools that make the software. It is sometimes tempting to view software development as being inherently important, but it's really not; it's a process of making stuff for people to use. If you get too far away from that truth, things can get sticky. I think this is part of the reason the Ops side of DevOps is often overlooked. Operations is concerned with managing the software out there in the wild. This includes dealing with network and hardware considerations and end users. GitLab wants operations to take place using the same UI as development. And why not? It’s the same application isn’t it? And in addition to technical performance, what about how the users are interacting with the application? If the application is somehow monetized, why shouldn’t that information also be available in the same UI as everything else having to do with this application? Again, it's still the same application. One tool to rule them all If you take a minute to step back and appreciate the vision of GitLab’s current direction, I think you can see why this is so exciting. If GitLab is successful in the long-term of extending out their reach into every element of an application's lifecycle including user interactions, productivity would skyrocket.  This idea isn’t really new. The ‘one tool to rule them all’ isn’t even that imaginative of a concept. It's just that no one has ever really created this ‘one tool.’ I believe we are about to enter, or have already entered, a DevOps space race. I believe GitLab is comfortably leading the pack, but they will need to keep working hard if they want it to stay that way. I believe we will be getting the one tool to rule them all, and I believe it is going to be soon. The way things are looking, GitLab is going to be the one to bring it to us, but only time will tell. Erik Kappelman wears many hats including blogger, developer, data consultant, economist, and transportation planner. He lives in Helena, Montana and works for the Department of Transportation as a transportation demand modeler.

The Common UNIX Printing System (CUPS) is actually a printer management tool, and thus monitoring CUPS always remains a very essential activity to to make the best use of the resources available. Monitoring CUPS will allow us to take action quickly should something go wrong. Using the lpstat Command The lpstat command displays the status of the CUPS service, printers, classes, and jobs. It supports a number of options. If the command is used without any options, it displays the job queues for the current user: $lpstatcupstest-3 kajol 8192 Tue Aug 05 13:24:43 2008cupstest-4 kajol 8192 Tue Aug 05 13:25:34 2008 To check whether the CUPS server is running, use the -r option. $lpstat -rscheduler is running$lpstat -dsystem default destination: cupsclass The above command gives information about the default destination printer or class. The output following the command shows that the default destination of the system is cupsclass. $lpstat -c cupsclass This shows the printer class and the member printers belonging to that class. If a particular class is not specified, then the output shows all classes along with their member printers. members of class cupsclass:cupsprinter1cupsprinter2$lpstat -v cupsprinter2 The command above will show the device to which cupsprinter1 is attached. If no printers are specified, then the output will list all printers along with device-uri information. device for cupsprinter2: ipp://cupsserver.cupsgroup.org/printers/cupsprinter2$lpstat -s This shows a status summary for all printers and classes on the network. The summary includes the default destination, a list of classes and their member printers, and a list of printers and their associated devices. The output is equivalent to using the -d, -c, and -v options simultaneously. system default destination: cupsclassmembers of class cupsclass:cupsprinter1cupsprinter2device for cupsprinter1: lpd://192.168.0.11/printers/cupsprinter1device for cupsprinter2: ipp://cupsserver.cupsgroup.org/printers/cupsprinter2$lpstat -a This command shows if printers are currently accepting jobs. If no printers are specified, then it will list all printers. cupsprinter1 accepting requests since Mon 16 Jun 2008 02:28:14 PM ISTcupsprinter2 accepting requests since Wed 18 Jun 2008 11:07:23 AM IST$lpstat -p cupsprinter1 This shows whether the printer cupsprinter1 is enabled and if it is currently printing a job. If no printers are specified then all the printers are listed. printer cupsprinter1 is idle. enabled since Mon 16 Jun 2008 02:28:14 PM IST$lpstat -o This shows the job queues on the specified destinations. If no destinations are specified then all jobs are shown. $lpstat -t This displays status information for all printers, which is equivalent to using the -r, -d, -c, -v, -a, -p, and -o options. scheduler is runningsystem default destination: cupsclassmembers of class cupsclass:cupsprinter1cupsprinter2device for cupsprinter1: lpd://192.168.0.11/printers/cupsprinter1device for cupsprinter2: ipp://cupsserver.cupsgroup.org/printers/cupsprinter2cupsprinter1 accepting requests since Mon 16 Jun 2008 02:28:14 PM ISTcupsprinter2 accepting requests since Wed 18 Jun 2008 11:07:23 AM ISTprinter cupsprinter1 is idle. enabled since Mon 16 Jun 2008 02:28:14 PM ISTprinter cupsprinter2 now printing cupsprinter2-5711. enabled since Wed 18 Jun 2008 03:12:55 PM ISTPrinter is now on-line.cupsprinter2-5711 kajol 1449984 Wed 18 Jun 2008 03:12:55 PM IST$lpstat -l This command displays printers, classes, or jobs in a long list. $lpstat -u This shows a list of print jobs queued by the specified users. If no users are specified, it lists the jobs queued by the current user. cupsprinter2-5711 kajol 1449984 Wed 18 Jun 2008 03:12:55 PM IST$lpstat -h 192.168.0.11:631 The above command specifies an alternative server for CUPS. It uses the port number that is specified along with the server. If no port is specified, then it will connect to the default port 631. $lpstat -U username You can specify an alternative username with the -U option $lpstat -R : $lpstat -W all This shows the ranking of print jobs. This command specifies which jobs to show, complete, incomplete (the default), or all. This option must appear before the -o option and any printer names: $lpstat -W completed -o cupsprinter2 The output will be as follows cupsprinter2-5709 kajol 483328 Wed 18 Jun 2008 03:10:45 PM ISTcupsprinter2-5710 kajol 97280 Wed 18 Jun 2008 03:12:18 PM ISTcupsprinter2-5711 kajol 1449984 Wed 18 Jun 2008 03:12:55 PM ISTcupsprinter2-5712 kajol 8192 Wed 18 Jun 2008 03:22:31 PM ISTcupsprinter2-5713 kajol 9216 Wed 18 Jun 2008 03:23:38 PM ISTcupsprinter2-5714 kajol 9216 Wed 18 Jun 2008 03:24:23 PM IST$lpstat -E This command forces encryption when connecting to a print server.

Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!IntroductionHave you ever tried asking a Generative AI-powered chatbot the question: “What is the meaning of life?”. In case you have not tried that yet, here’s the response I got when I tried that myself using a custom chatbot app I built with a managed machine learning (ML) service called Amazon SageMaker.                                              Image 01 — Asking a chatbot the meaning of lifeYou would be surprised that I built this quick demo application myself in just a few hours! In this post, I will teach you how to deploy your own Large Language Models (LLMs) in a SageMaker Inference Endpoint (that is, a machine learning-powered server that responds to inputs) with just a few lines of code.                                                   Image 02 — Deploying an LLM to a SageMaker Inference EndpointWhile most tutorials available teach us how to utilize existing Application Programming Interfaces (APIs) to prepare chatbot applications, it’s best that we also know how to deploy LLMs in our own servers in order to guarantee data privacy and compliance. In addition to this, we’ll be able to manage the long-term costs of our AI-powered systems as well. One of the most powerful solutions available for these types of requirements is Amazon SageMaker which helps us focus on the work we need to do instead of worrying about cloud infrastructure management.We’ll divide the hands-on portion into the following sections:●  Section I: Preparing the SageMaker Notebook Instance●  Section II: Deploying an LLM using the SageMaker Python SDK to a SageMaker Inference Endpoint●  Section III: Enabling Data Capture with SageMaker Model Monitor (discussed in Part 2)●  Section IV: Invoking the SageMaker inference endpoint using the boto3 client (discussed in Part 2)●  Section V: Preparing a Demo UI for our chatbot application (discussed in Part 2)●  Section VI: Cleaning Up (discussed in Part 2) Without further ado, let’s begin!Section I: Preparing the SageMaker Notebook InstanceLet’s start by creating a SageMaker Notebook instance. Note that while we can also do this in SageMaker Studio, running the example in a Sagemaker Notebook Instance should do the trick. If this is your first time launching a SageMaker Notebook instance, you can think of it as your local machine with several tools pre-installed already where we can run our scripts.STEP # 01: Sign in to your AWS account and navigate to the SageMaker console by typing sagemaker in the search box similar to what we have in the following image:                                                           Image 03 — Navigating to the SageMaker consoleChoose Amazon SageMaker from the list of options available as highlighted in Image 03.STEP # 02: In the sidebar, locate and click Notebook instances under Notebook:                                 Image 04 — Locating Notebook instances in the sidebar STEP # 03: Next, locate and click the Create notebook instance button.STEP # 04: In the Create notebook instance page, you’ll be asked to input a few configuration parameters before we’re able to launch the notebook instance where we’ll be running our code:                                                          Image 05 — Creating a new SageMaker Notebook instanceSpecify a Notebook instance name (for example, llm-demo) and select a Notebook instance type. For best results, you may select a relatively powerful instance type (ml.m4.xlarge) where we will run the scripts. However, you may decide to choose a smaller instance type such as ml.t3.medium (slower but less expensive). Note that we will not be deploying our LLM inside this notebook instance as the model will be deployed in a separate inference endpoint (which will require a more powerful instance type such as an ml.g5.2xlarge).STEP # 05:Create an IAM role by choosing Create a new role from the list of options available in the IAM role dropdown (under Permissions and encryption).                                                                             Image 06 — Opening the Jupyter appThis will open the following popup window. Given that we’re just working on a demo application, the default security configuration should do the trick. Click the Create role button.Important Note: Make sure to have a more secure configuration when dealing with production (or staging) work environments.Won’t dive deep into how cloud security works in this post so feel free to look for other resources and references to further improve the current security setup. In case you are interested to learn more about cloud security, feel free to check my 3rd book “Building and Automating Penetration Testing Labs in the Cloud”. In the 7th Chapter of the book (Setting Up an IAM Privilege Escalation Lab), you’ll learn how misconfigured machine learning environments on AWS can easily be exploited with the right sequence of steps.STEP #06: Click the Create notebook instance button. Wait for about 5-10 minutes for the SageMaker Notebook instance to be ready.Important Note: Given that this will launch a resource that will run until you turn it off (or delete it), make sure to complete all the steps in the 2nd part of this post and clean up the created resources accordingly.STEP # 07:Once the instance is ready, click Open Jupyter similar to what we have in Image 07:                                                                            Image 07 — Opening the Jupyter appThis will open the Jupyter application in a browser tab. If this is your first time using this application, do not worry as detailed instructions will be provided in the succeeding steps to help you get familiar with this tool.STEP # 08:Create a new notebook by clicking New and selecting conda_python3 from the list of options available: Image 08 — Creating a new notebook using the conda_python3 kernelIn case you are wondering about what a kernel is, it is simply an “engine” or “environment” with pre-installed libraries and prerequisites that executes the code specified in the notebook cells. You’ll see this in action in a bit.STEP # 09:At this point, we should see the following interface where we can run various types of scripts and blocks of code:                                                                              Image 09 — New Jupyter notebookFeel free to rename the Jupyter Notebook before proceeding to the next step. If you have not used a Jupyter Notebook before, you may run the following line of code by typing the following in the text field and pressing SHIFT + ENTER. print('hello')This should print the output hello right below the text field where we placed our code.Section II: Deploying an LLM using the SageMaker Python SDK to a SageMaker Inference EndpointSTEP # 01: With everything ready, let’s start by installing a specific version of the SageMaker Python SDK: !pip install sagemaker==2.192.1Here, we’ll be using v2.192.1. This will help us ensure that you won’t encounter breaking changes even if you work on the hands-on solutions in this post at a later date.In case you are wondering what the SageMaker Python SDK is, it is simply a software development kit (SDK) with the set of tools and APIs to help developers interact with and utilize the different features and capabilities of Amazon SageMaker.STEP # 02: Next, let’s import and prepare a few prerequisites by running the following block of code: import sagemaker import time sagemaker_session = sagemaker.Session() region = sagemaker_session.boto_region_name role = sagemaker.get_execution_role()STEP # 03: Let’s import HuggingFaceModel and get_huggingface_llm_image_uri as well:from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uriSTEP # 04: Next, let’s define the generate_random_label() function which we’ll use later when naming our resources:from string import ascii_uppercase from random import choice def generate_random_label():    letters = ascii_uppercase      return ''.join(choice(letters) for i in range(10))This will help us avoid naming conflicts when creating and configuring our resources.STEP # 05: Use the get_huggingface_llm_image_uri function we imported in an earlier step to retrieve the container image URI for our LLM. In addition to this, let’s define the model_name we’ll use later when deploying our LLM to a SageMaker endpoint:image_uri = get_huggingface_llm_image_uri( backend="huggingface", region=region, version="1.1.0" ) model_name = "MistralLite-" + generate_random_label()STEP # 06: Before, we proceed with the actual deployment, let’s quickly inspect what we have in the image_uri variable:image_uriThis will output the following variable value:'763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-pytorch-tgi-inference:2.0.1-tgi1.1.0-gpu-py39-cu118-ubuntu20.04'STEP # 07: Similarly, let’s check the variable value of model_name model_nameThis will give us the following:'MistralLite-HKGKFRXURT'Note that you’ll get a different model_name value since we’re randomly generating a portion of the model nameSTEP # 08: Let’s prepare the hub model configuration as well:hub_env = { 'HF_MODEL_ID': 'amazon/MistralLite', 'HF_TASK': 'text-generation', 'SM_NUM_GPUS': '1', "MAX_INPUT_LENGTH": '16000', "MAX_TOTAL_TOKENS": '16384', "MAX_BATCH_PREFILL_TOKENS": '16384', "MAX_BATCH_TOTAL_TOKENS":  '16384', }Here, we specify that we’ll be using the MistralLite model. If this is your first time hearing out MistralLite, it is a fine-tuned Mistral-7B-v0.1 language model. It can perform significantly better on several long context retrieve and answering tasks. For more information, feel free to check: https://huggingface.co/amazon/MistralLite.STEP # 09: Let’s initialize the HuggingFaceModel object using some of the prerequisites and variables we’ve prepared in the earlier steps:model = HuggingFaceModel(    name=model_name,    env=hub_env,    role=role,    image_uri=image_uri )STEP # 10: Now, let’s proceed with the deployment of the model using the deploy() method:predictor = model.deploy( initial_instance_count=1, instance_type="ml.g5.2xlarge", endpoint_name=model_name, )Here, we’re using an ml.g5.2xlarge for our inference endpoint.Given that this step may take about 10-15 minutes to complete, feel free to grab a cup of coffee or tea while waiting!Important Note: Given that this will launch a resource that will run until you turn it off (or delete it), make sure to complete all the steps in the 2nd part of this post and clean up the created resources accordingly.STEP # 11: Now, let’s prepare our first input data:question = "What is the meaning of life?" input_data = { "inputs": f"<|prompter|>{question}</s><|assistant|>", "parameters": {    "do_sample": False,    "max_new_tokens": 2000,    "return_full_text": False, } }STEP # 12: With the prerequisites ready, let’s have our deployed LLM process the input data we prepared in the previous step:result = predictor.predict(input_data)[0]["generated_text"] print(result)This should yield the following output:The meaning of life is a philosophical question that has been debated by thinkers and philosophers for centuries. There is no single answer that can be definitively proven, as the meaning of life is subjective and can vary greatly from person to person. ...Looks like our SageMaker Inference endpoint (where the LLM is deployed) is working just fine!ConclusionThat wraps up the first part of this post. At this point, you should have a good idea of how to deploy LLMs using Amazon SageMaker. However, there’s more in store for us in the second part as we’ll build on top of what we have already and enable data capture to help us collect and analyze the data (that is, the input requests and output responses) that pass through the inference endpoint. In addition to this, we’ll prepare a demo user interface utilizing the ML model we deployed in this post.If you’re looking for the link to the second part, here it is: Deploying LLMs with Amazon SageMaker - Part 2We are just scratching the surface as there is a long list of capabilities and features available in SageMaker. If you want to take things to the next level, feel free to read 2 of my books focusing heavily on SageMaker: “Machine Learning with Amazon SageMaker Cookbook” and “Machine Learning Engineering on AWS”.Author BioJoshua Arvin Lat is the Chief Technology Officer (CTO) of NuWorks Interactive Labs, Inc. He previously served as the CTO of 3 Australian-owned companies and also served as the Director for Software Development and Engineering for multiple e-commerce startups in the past. Years ago, he and his team won 1st place in a global cybersecurity competition with their published research paper. He is also an AWS Machine Learning Hero and he has been sharing his knowledge in several international conferences to discuss practical strategies on machine learning, engineering, security, and management. He is also the author of the books "Machine Learning with Amazon SageMaker Cookbook", "Machine Learning Engineering on AWS", and "Building and Automating Penetration Testing Labs in the Cloud". Due to his proven track record in leading digital transformation within organizations, he has been recognized as one of the prestigious Orange Boomerang: Digital Leader of the Year 2023 award winners.

In this article we'll understand how functional programming can be applied to web services in Python. This article is an extract from the 2nd edition of the bestseller, Functional Python Programming, written by Steven Lott. We'll look at a RESTful web service, which can slice and dice a source of data and provide downloads as JSON, XML, or CSV files. We'll provide an overall WSGI-compatible wrapper. The functions that do the real work of the application won't be narrowly constrained to fit the WSGI standard. We'll use a simple dataset with four subcollections: the Anscombe Quartet. It's a small set of data but it can be used to show the principles of a RESTful web service. We'll split our application into two tiers: a web tier, which will be a simple WSGI application, and data service tier, which will be more typical functional programming. We'll look at the web tier first so that we can focus on a functional approach to provide meaningful results. We need to provide two pieces of information to the web service: The quartet that we want: this is a slice and dice operation. The idea is to slice up the information by filtering and extracting meaningful subsets. The output format we want. The data selection is commonly done through the request path. We can request /anscombe/I/ or /anscombe/II/ to pick specific datasets from the quartet. The idea is that a URL defines a resource, and there's no good reason for the URL to ever change. In this case, the dataset selectors aren't dependent on dates or some organizational approval status, or other external factors. The URL is timeless and absolute. The output format is not a first-class part of the URL. It's just a serialization format, not the data itself. In some cases, the format is requested through the HTTP Accept header. This is hard to use from a browser, but easy to use from an application using a RESTful API. When extracting data from the browser, a query string is commonly used to specify the output format. We'll use the ?form=json method at the end of the path to specify the JSON output format. A URL we can use will look like this: http://localhost:8080/anscombe/III/?form=csv This would request a CSV download of the third dataset. Creating the Web Server Gateway Interface First, we'll use a simple URL pattern-matching expression to define the one and only routing in our application. In a larger or more complex application, we might have more than one such pattern: import re path_pat= re.compile(r"^/anscombe/(?P<dataset>.*?)/?$") This pattern allows us to define an overall script in the WSGI sense at the top level of the path. In this case, the script is anscombe. We'll take the next level of the path as a dataset to select from the Anscombe Quartet. The dataset value should be one of I, II, III, or IV. We used a named parameter for the selection criteria. In many cases, RESTful APIs are described using a syntax, as follows: /anscombe/{dataset}/ We translated this idealized pattern into a proper, regular expression, and preserved the name of the dataset selector in the path. Here are some example URL paths that demonstrate how this pattern works: >>> m1 = path_pat.match( "/anscombe/I" ) >>> m1.groupdict() {'dataset': 'I'} >>> m2 = path_pat.match( "/anscombe/II/" ) >>> m2.groupdict() {'dataset': 'II'} >>> m3 = path_pat.match( "/anscombe/" ) >>> m3.groupdict() {'dataset': ''} Each of these examples shows the details parsed from the URL path. When a specific series is named, this is located in the path. When no series is named, then an empty string is found by the pattern. Here's the overall WSGI application: import traceback import urllib.parse def anscombe_app( environ: Dict, start_response: SR_Func ) -> Iterable[bytes]: log = environ['wsgi.errors'] try: match = path_pat.match(environ['PATH_INFO']) set_id = match.group('dataset').upper() query = urllib.parse.parse_qs(environ['QUERY_STRING']) print(environ['PATH_INFO'], environ['QUERY_STRING'], match.groupdict(), file=log) dataset = anscombe_filter(set_id, raw_data()) content_bytes, mime = serialize( query['form'][0], set_id, dataset) headers = [ ('Content-Type', mime), ('Content-Length', str(len(content_bytes))), ] start_response("200 OK", headers) return [content_bytes] except Exception as e: # pylint: disable=broad-except traceback.print_exc(file=log) tb = traceback.format_exc() content = error_page.substitute( title="Error", message=repr(e), traceback=tb) content_bytes = content.encode("utf-8") headers = [ ('Content-Type', "text/html"), ('Content-Length', str(len(content_bytes))), ] start_response("404 NOT FOUND", headers) return [content_bytes] This application will extract two pieces of information from the request: the PATH_INFO and the QUERY_STRING keys in the environment dictionary. The PATH_INFO request will define which set to extract. The QUERY_STRING request will specify an output format. It's important to note that query strings can be quite complex. Rather than assume it is simply a string like ?form=json, we've used the urllib.parse module to properly locate all of the name-value pairs in the query string. The value with the 'form' key in the dictionary extracted from the query string can be found in query['form'][0]. This should be one of the defined formats. If it isn't, an exception will be raised, and an error page displayed. After locating the path and query string, the application processing is highlighted in bold. These two statements rely on three functions to gather, filter, and serialize the results: The raw_data() function reads the raw data from a file. The result is a dictionary with lists of Pair objects. The anscombe_filter() function accepts a selection string and the dictionary of raw data and returns a single list of Pair objects. The list of pairs is then serialized into bytes by the serialize() function. The serializer is expected to produce byte's, which can then be packaged with an appropriate header, and returned. We elected to produce an HTTP Content-Length header as part of the result. This header isn't required, but it's polite for large downloads. Because we decided to emit this header, we are forced to create a bytes object with the serialization of the data so we can count the bytes. If we elected to omit the Content-Length header, we could change the structure of this application dramatically. Each serializer could be changed to a generator function, which would yield bytes as they are produced. For large datasets, this can be a helpful optimization. For the user watching a download, however, it might not be so pleasant because the browser can't display how much of the download is complete. A common optimization is to break the transaction into two parts. The first part computes the result and places a file into a Downloads directory. The response is a 302 FOUND with a Location header that identifies the file to download. Generally, most clients will then request the file based on this initial response. The file can be downloaded by Apache httpd or Nginx without involving the Python application. For this example, all errors are treated as a 404 NOT FOUND error. This could be misleading, since a number of individual things might go wrong. More sophisticated error handling could give more try:/except: blocks to provide more informative feedback. For debugging purposes, we've provided a Python stack trace in the resulting web page. Outside the context of debugging, this is a very bad idea. Feedback from an API should be just enough to fix the request, and nothing more. A stack trace provides too much information to potentially malicious users. Getting raw data Here's what we're using for this application: from Chapter_3.ch03_ex5 import ( series, head_map_filter, row_iter) from typing import ( NamedTuple, Callable, List, Tuple, Iterable, Dict, Any) RawPairIter = Iterable[Tuple[float, float]] class Pair(NamedTuple): x: float y: float pairs: Callable[[RawPairIter], List[Pair]] \ = lambda source: list(Pair(*row) for row in source) def raw_data() -> Dict[str, List[Pair]]: with open("Anscombe.txt") as source: data = tuple(head_map_filter(row_iter(source))) mapping = { id_str: pairs(series(id_num, data)) for id_num, id_str in enumerate( ['I', 'II', 'III', 'IV']) } return mapping The raw_data() function opens the local data file, and applies the row_iter() function to return each line of the file parsed into a row of separate items. We applied the head_map_filter() function to remove the heading from the file. The result created a tuple-of-list structure, which is assigned the variable data. This handles parsing the input into a structure that's useful. The resulting structure is an instance of the Pair subclass of the NamedTuple class, with two fields that have float as their type hints. We used a dictionary comprehension to build the mapping from id_str to pairs assembled from the results of the series() function. The series() function extracts (x, y) pairs from the input document. In the document, each series is in two adjacent columns. The series named I is in columns zero and one; the series() function extracts the relevant column pairs. The pairs() function is created as a lambda object because it's a small generator function with a single parameter. This function builds the desired NamedTuple objects from the sequence of anonymous tuples created by the series() function. Since the output from the raw_data() function is a mapping, we can do something like the following example to pick a specific series by name: >>> raw_data()['I'] [Pair(x=10.0, y=8.04), Pair(x=8.0, y=6.95), ... Given a key, for example, 'I', the series is a list of Pair objects that have the x, y values for each item in the series. Applying a filter In this application, we're using a simple filter. The entire filter process is embodied in the following function: def anscombe_filter( set_id: str, raw_data_map: Dict[str, List[Pair]] ) -> List[Pair]: return raw_data_map[set_id] We made this trivial expression into a function for three reasons: The functional notation is slightly more consistent and a bit more flexible than the subscript expression We can easily expand the filtering to do more We can include separate unit tests in the docstring for this function While a simple lambda would work, it wouldn't be quite as convenient to test. For error handling, we've done exactly nothing. We've focused on what's sometimes called the happy path: an ideal sequence of events. Any problems that arise in this function will raise an exception. The WSGI wrapper function should catch all exceptions and return an appropriate status message and error response content. For example, it's possible that the set_id method will be wrong in some way. Rather than obsess over all the ways it could be wrong, we'll simply allow Python to throw an exception. Indeed, this function follows the Python advice that, it's better to seek forgiveness than to ask permission. This advice is materialized in code by avoiding permission-seeking: there are no preparatory if statements that seek to qualify the arguments as valid. There is only forgiveness handling: an exception will be raised and handled in the WSGI wrapper. This essential advice applies to the preceding raw data and the serialization that we will see now. Serializing the results Serialization is the conversion of Python data into a stream of bytes, suitable for transmission. Each format is best described by a simple function that serializes just that one format. A top-level generic serializer can then pick from a list of specific serializers. The picking of serializers leads to the following collection of functions: Serializer = Callable[[str, List[Pair]], bytes] SERIALIZERS: Dict[str, Tuple[str, Serializer]]= { 'xml': ('application/xml', serialize_xml), 'html': ('text/html', serialize_html), 'json': ('application/json', serialize_json), 'csv': ('text/csv', serialize_csv), } def serialize( format: str, title: str, data: List[Pair] ) -> Tuple[bytes, str]: mime, function = SERIALIZERS.get( format.lower(), ('text/html', serialize_html)) return function(title, data), mime The overall serialize() function locates a specific serializer in the SERIALIZERS dictionary, which maps a format name to a two-tuple. The tuple has a MIME type that must be used in the response to characterize the results. The tuple also has a function based on the Serializer type hint. This function will transform a name and a list of Pair objects into bytes that will be downloaded. The serialize() function doesn't do any data transformation. It merely maps a name to a function that does the hard work of transformation. Returning a function permits the overall application to manage the details of memory or file-system serialization. Serializing to the file system, while slow, permits larger files to be handled. We'll look at the individual serializers below. The serializers fall into two groups: those that produce strings and those that produce bytes. A serializer that produces a string will need to have the string encoded as bytes for download. A serializer that produces bytes doesn't need any further work. For the serializers, which produce strings, we can use function composition with a standardized convert-to-bytes function. Here's a decorator that can standardize the conversion to bytes: from typing import Callable, TypeVar, Any, cast from functools import wraps def to_bytes( function: Callable[..., str] ) -> Callable[..., bytes]: @wraps(function) def decorated(*args, **kw): text = function(*args, **kw) return text.encode("utf-8") return cast(Callable[..., bytes], decorated) We've created a small decorator named @to_bytes. This will evaluate the given function and then encode the results using UTF-8 to get bytes. Note that the decorator changes the decorated function from having a return type of str to a return type of bytes. We haven't formally declared parameters for the decorated function, and used ... instead of the details. We'll show how this is used with JSON, CSV, and HTML serializers. The XML serializer produces bytes directly and doesn't need to be composed with this additional function. We could also do the functional composition in the initialization of the serializers mapping. Instead of decorating the function definition, we could decorate the reference to the function object. Here's an alternative definition for the serializer mapping: SERIALIZERS = { 'xml': ('application/xml', serialize_xml), 'html': ('text/html', to_bytes(serialize_html)), 'json': ('application/json', to_bytes(serialize_json)), 'csv': ('text/csv', to_bytes(serialize_csv)), } This replaces decoration at the site of the function definition with decoration when building this mapping data structure. It seems potentially confusing to defer the decoration. Serializing data into JSON or CSV formats The JSON and CSV serializers are similar because both rely on Python's libraries to serialize. The libraries are inherently imperative, so the function bodies are strict sequences of statements. Here's the JSON serializer: import json @to_bytes def serialize_json(series: str, data: List[Pair]) -> str: """ >>> data = [Pair(2,3), Pair(5,7)] >>> serialize_json( "test", data ) b'[{"x": 2, "y": 3}, {"x": 5, "y": 7}]' """ obj = [dict(x=r.x, y=r.y) for r in data] text = json.dumps(obj, sort_keys=True) return text We created a list-of-dict structure and used the json.dumps() function to create a string representation. The JSON module requires a materialized list object; we can't provide a lazy generator function. The sort_keys=True argument value is helpful for unit testing. However, it's not required for the application and represents a bit of overhead. Here's the CSV serializer: import csv import io @to_bytes def serialize_csv(series: str, data: List[Pair]) -> str: """ >>> data = [Pair(2,3), Pair(5,7)] >>> serialize_csv("test", data) b'x,y\\r\\n2,3\\r\\n5,7\\r\\n' """ buffer = io.StringIO() wtr = csv.DictWriter(buffer, Pair._fields) wtr.writeheader() wtr.writerows(r._asdict() for r in data) return buffer.getvalue() The CSV module's readers and writers are a mixture of imperative and functional elements. We must create the writer, and properly create headings in a strict sequence. We've used the _fields attribute of the Pair namedtuple to determine the column headings for the writer. The writerows() method of the writer will accept a lazy generator function. In this case, we used the _asdict() method of each Pair object to return a dictionary suitable for use with the CSV writer. Serializing data into XML We'll look at one approach to XML serialization using the built-in libraries. This will build a document from individual tags. A common alternative approach is to use Python introspection to examine and map Python objects and class names to XML tags and attributes. Here's our XML serialization: import xml.etree.ElementTree as XML def serialize_xml(series: str, data: List[Pair]) -> bytes: """ >>> data = [Pair(2,3), Pair(5,7)] >>> serialize_xml( "test", data ) b'<series name="test"><row><x>2</x><y>3</y></row><row><x>5</x><y>7</y></row></series>' """ doc = XML.Element("series", name=series) for row in data: row_xml = XML.SubElement(doc, "row") x = XML.SubElement(row_xml, "x") x.text = str(row.x) y = XML.SubElement(row_xml, "y") y.text = str(row.y) return cast(bytes, XML.tostring(doc, encoding='utf-8')) We created a top-level element, <series>, and placed <row> sub-elements underneath that top element. Within each <row> sub-element, we've created <x> and <y> tags, and assigned text content to each tag. The interface for building an XML document using the ElementTree library tends to be heavily imperative. This makes it a poor fit for an otherwise functional design. In addition to the imperative style, note that we haven't created a DTD or XSD. We have not properly assigned a namespace to our tags. We also omitted the <?xml version="1.0"?> processing instruction that is generally the first item in an XML document. The XML.tostring() function has a type hint that states it returns str. This is generally true, but when we provide the encoding parameter, the result type changes to bytes. There's no easy way to formalize the idea of variant return types based on parameter values, so we use an explicit cast() to inform mypy of the actual type. A more sophisticated serialization library could be helpful here. There are many to choose from. Visit https://wiki.python.org/moin/PythonXml for a list of alternatives. Serializing data into HTML In our final example of serialization, we'll look at the complexity of creating an HTML document. The complexity arises because in HTML, we're expected to provide an entire web page with a great deal of context information. Here's one way to tackle this HTML problem: import string data_page = string.Template("""\ <html> <head><title>Series ${title}</title></head> <body> <h1>Series ${title}</h1> <table> <thead><tr><td>x</td><td>y</td></tr></thead> <tbody> ${rows} </tbody> </table> </body> </html> """) @to_bytes def serialize_html(series: str, data: List[Pair]) -> str: """ >>> data = [Pair(2,3), Pair(5,7)] >>> serialize_html("test", data) #doctest: +ELLIPSIS b'<html>...<tr><td>2</td><td>3</td></tr>\\n<tr><td>5</td><td>7</td></tr>... """ text = data_page.substitute( title=series, rows="\n".join( "<tr><td>{0.x}</td><td>{0.y}</td></tr>".format(row) for row in data) ) return text Our serialization function has two parts. The first part is a string.Template() function that contains the essential HTML page. It has two placeholders where data can be inserted into the template. The ${title} method shows where title information can be inserted, and the ${rows} method shows where the data rows can be inserted. The function creates individual data rows using a simple format string. These are joined into a longer string, which is then substituted into the template. While workable for simple cases like the preceding example, this isn't ideal for more complex result sets. There are a number of more sophisticated template tools to create HTML pages. A number of these include the ability to embed the looping in the template, separate from the function that initializes serialization. If you found this tutorial useful and would like to learn more such techniques, head over to get Steven Lott's bestseller, Functional Python Programming. What is the difference between functional and object-oriented programming? Should you move to Python 3? 7 Python experts’ opinions Is Python edging R out in the data science wars?

Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights and books. Don't miss out – sign up today!IntroductionWe are living in the era of Generative AI and Chatbots have evolved from simple rule-based systems to sophisticated AI-driven entities capable of holding intricate conversations. With tools like OpenAI's ChatGPT, creating a chatbot has never been more accessible. This article dives deep into how one can specialize these chatbots by fine-tuning them for specific industries and applications, using the Science QA dataset as an example.PrerequisitesBefore diving in, please ensure you have the necessary tools installed:OpenAI - Create an account on OpenAI and Generate API keysInstall Streamlit.pip install streamlitA dataset in CSV format for trainingFor this article, we have used the publicly available Science QA dataset for training and validation.Train dataset - ScienceQA-trainValidation dataset- ScienceQA-valWhile OpenAI powers the chatbot's brain, Streamlit will serve as the platform to interact with it.Understanding Fine-tuningFine-tuning is the process of training a pre-trained model on a new dataset to adapt to specific tasks. It's like teaching a general doctor to specialize in a field. The nuances and specific knowledge in the specialized field would aid and enhance to create your specialized dataset.Fine-tuning with OpenAIOpenAI's API simplifies the fine-tuning process. By providing your dataset, OpenAI trains your model to specialize in the subject of your data. First, we download the two CSV files from Science QA and save them in your project folder.Next, we process these CSV files and convert them into JSONL format which is required by OpenAI.import openai import pandas as pd # Use your Open AI access key openai.api_key = "sk-**************************************" # Load the training and validation datasets train_df = pd.read_csv("science-train.csv") val_df = pd.read_csv("science-val.csv") def convert_dataset_to_jsonl(df, file_name): df["conversation"] = df.apply(format_chat, axis=1)   with open(file_name, 'w') as jsonl_file:      for example in df["conversation"]:         jsonl_file.write(example + '\n') # Convert the training and validation datasets convert_dataset_to_jsonl(train_df, "fine_tune_train_data.jsonl") convert_dataset_to_jsonl(val_df, "fine_tune_val_data.jsonl")After converting the datasets to JSONL format, we will upload these datasets to OpenAI for further consumption for model fine-tuning.train = openai.File.create( file=open("fine_tune_train_data.jsonl", "rb"),purpose='fine-tune',) val = openai.File.create( file=open("fine_tune_val_data.jsonl", "rb"),purpose='fine-tune',) print(train) print(val) After printing, save these files names for train and validation sets. Next, we will create the fine-tuned model with your train and test data files. Pass the train and test data file names that we printed in step 3. model = openai.FineTuningJob.create( model = "gpt-3.5-turbo", training_file = train_data, validation_file = val_data, suffix = "scienceqa") print(model)Now, you have successfully created your fine-tuned model. You can also check the status in your OpenAI account.Building the Streamlit AppStreamlit is a game-changer for Python enthusiasts looking to deploy applications without the intricacies of web development. Integrating the fine-tuned model involves invoking OpenAI's API within the Streamlit interface. With customization features, tailor your app to resonate with the theme of your chatbot.You will also add some styling to your web app to create a visually appealing platform for your fine-tuned model.import streamlit as st # Set OpenAI API key openai.api_key = "sk-**********************" # Use your fine-tuned model ID here: FINE_TUNED_MODEL = "ft:gpt-3.5-turbo-****************" # Message system setup messages = [{ "role": "system", "content": "You are an AI specialized in Science and Tech.. "},] def get_chat(): return messages def chatbot(input): if input: chat = get_chat() chat.append({"role": "user", "content": input}) response = openai.ChatCompletion.create( model="ft:gpt-3.5-turbo-*****************", messages=chat, max_tokens=150, ) message = response['choices'][0]['message']['content'] chat.append({"role": "assistant", "content": message}) return message # Custom vibrant styling st.title('AI Chatbot Specialized in Science') st.write("Ask me about Science and Technology) # Sidebar for additional information or actions with st.sidebar: st.write("Instructions:") st.write("1. Ask anything related to Science.") st.write("2. Wait for the AI to provide an insightful answer!") # Main Chat Interface user_input = st.text_area("You: ", "") if st.button("Ask"): user_output = chatbot(user_input) st.text_area("AI's Response:", user_output, height=200)After saving the code, run the code using the following command-strealmit run 'yourpyfilename.py'Voila!! The app is ready!!The app opens on a new web page in your browser. Here is how it looks- Deployment ConsiderationsEnsure your API keys remain confidential. As for scalability, remember each query costs money; caching frequent queries can be a cost-effective strategy.Happy Fine Tuning!!ConclusionFine-tuning a model like GPT-3.5 Turbo for specific industries can immensely boost its effectiveness in those domains. Using tools like Streamlit, the deployment becomes hassle-free. Experiment, iterate, and watch your specialized chatbot come to life!Author BioSwagata Ashwani serves as a Principal Data Scientist at Boomi, where she leads the charge in deploying cutting-edge AI solutions, with a particular emphasis on Natural Language Processing (NLP). With a stellar track record in AI research, she is always on the lookout for the next state-of-the-art tool or technique to revolutionize the industry. Beyond her technical expertise, Swagata is a fervent advocate for women in tech. She believes in giving back to the community, regularly contributing to open-source initiatives that drive the democratization of technology.Swagata's passion isn't limited to the world of AI; she is a nature enthusiast, often wandering beaches and indulging in the serenity they offer. With a cup of coffee in hand, she finds joy in the rhythm of dance and the tranquility of the great outdoors.

In May 2019, DataStax hosted the Accelerate conference for Apache Cassandra™ inviting community members, DataStax customers, and other users to come together, discuss the latest developments around Cassandra, and find out more about the development of Cassandra. Nate McCall, Apache Cassandra Project Chair, presented the road to version 4.0 and what the community is focusing on for the future. So, what does the future really hold for Cassandra? The project has been going for ten years already, so what has to be added?  First off, listening to Nate’s keynote, the approach to development has evolved. As part of the development approach around Cassandra, it’s important to understand who is committing updates to Cassandra. The number of organisations contributing to Cassandra has increased, while the companies involved in the Project Management Committee includes some of the biggest companies in the world.  The likes of Instagram, Facebook and Netflix have team members contributing and leading the development of Cassandra because it is essential to their businesses. For DataStax, we continue to support the growth and development of Cassandra as an open source project through our own code contributions, our development and training, and our drivers that are available for the community and for our customers alike.  Having said all this, there are still areas where Cassandra can improve as we get ready for 4.0. From a development standpoint, the big things to look forward to as mentioned in Nate’s keynote are:  An improved Repair model For a distributed database, being able to carry on through any failure event is critical. After a failure, those nodes will have to be brought back online, and then catch up with the transactions that they missed. Making nodes consistent is a big task, covered by the Repair function. In Cassandra 4.0, the aim is to make Repair smarter. For example, Cassandra can preview the impact of a repair on a host to check that the operation will go through successfully, and specific pull requests for data can also be supported. Alongside this, a new transient replication feature should reduce the cost and bandwidth overhead associated with repair. By replicating temporary copies of data to supplement full copies, the overall cluster should be able to achieve higher levels of availability but at the same time reduce the overall volume of storage required significantly. For companies running very large clusters, the cost savings achievable here could be massive. A Messaging rewrite Efficient messaging between nodes is essential when your database is distributed. Cassandra 4.0 will have a new messaging system in place based on Netty, an asynchronous event-driven network application framework. In practice, using Netty will improve performance of messaging between nodes within clusters and between clusters. On top of this change, zero copy support will provide the ability to improve how quickly data can be streamed between nodes. Zero copy support achieves this by modifying the streaming path to add additional information into the streaming header, and then using ZeroCopy APIs to transfer bytes to and from the network and disk. This allows nodes to transfer large files faster. Cassandra and Kubernetes support Adding new messaging support and being able to transfer SSTables means that Cassandra can add more support for Kubernetes, and for Kubernetes to do interesting things around Cassandra too. One area that has been discussed is around dynamic cluster management, where the number of nodes and the volume of storage can be increased or decreased on demand. Sidecars Sidecars are additional functional tools designed to work alongside a main process. These sidecars fill a gap that is not part of the main application or service, and that should remain separate but linked. For Cassandra, running sidecars allows developers to add more functionality to their operations, such as creating events on an application. Java 11 support Java 11 support has been added to the Cassandra trunk version and will be present in 4.0. This will allow Cassandra users to use Java 11, rather than version 8 which is no longer supported.  Diagnostic events and logging This will make it easier for teams to use events for a range of things, from security requirements through to logging activities and triggering tools.  As part of the conference, there were two big trends that I took from the event. The first is – as Nate commented in his keynote – that there was a definite need for more community events that can bring together people who care about Cassandra and get them working together.   The second is that Apache Cassandra is essential to many companies today. Some of the world’s largest internet companies and most valuable brands out there rely on Cassandra in order to achieve what they do. They are contributors and committers to Cassandra, and they have to be sure that Cassandra is ready to meet their requirements. For everyone using Cassandra, this means that versions have to be ready for use in production rather than having issues to be fixed. Things will get released when they are ready, rather than to meet a particular deadline. And the community will take the lead in ensuring that they are happy with any release.  Cassandra 4.0 is nearing release. It’ll be out when it is ready. Whether you are looking at getting involved with the project through contributions, developing drivers or through writing documentation, there is a warm welcome for everyone in the run up to what should be a great release.  I’m already looking forward to ApacheCon later this year! Author Bio Patrick McFadin is the vice president of developer relations at DataStax, where he leads a team devoted to making users of DataStax products successful. Previously, he was chief evangelist for Apache Cassandra and a consultant for DataStax, where he helped build some of the largest and most exciting deployments in production; a chief architect at Hobsons; and an Oracle DBA and developer for over 15 years.

In this article, we'll cover how to use corpus readers and create custom corpora. At the same time, you'll learn how to use the existing corpus data that comes with NLTK. We'll also cover creating custom corpus readers, which can be used when your corpus is not in a file format that NLTK already recognizes, or if your corpus is not in files at all, but instead is located in a database such as MongoDB. Setting up a custom corpus A corpus is a collection of text documents, and corpora is the plural of corpus. So a custom corpus is really just a bunch of text files in a directory, often alongside many other directories of text files. Getting ready You should already have the NLTK data package installed, following the instructions at http://www.nltk.org/data. We'll assume that the data is installed to C:nltk_data on Windows, and /usr/share/nltk_data on Linux, Unix, or Mac OS X. How to do it... NLTK defines a list of data directories, or paths, in nltk.data.path. Our custom corpora must be within one of these paths so it can be found by NLTK. So as not to conflict with the official data package, we'll create a custom nltk_data directory in our home directory. Here's some Python code to create this directory and verify that it is in the list of known paths specified by nltk.data.path: >>> import os, os.path >>> path = os.path.expanduser('~/nltk_data') >>> if not os.path.exists(path): ... os.mkdir(path) >>> os.path.exists(path) True >>> import nltk.data >>> path in nltk.data.path True If the last line, path in nltk.data.path, is True, then you should now have a nltk_ data directory in your home directory. The path should be %UserProfile%nltk_data on Windows, or ~/nltk_data on Unix, Linux, or Mac OS X. For simplicity, I'll refer to the directory as ~/nltk_data. If the last line does not return True, try creating the nltk_data directory manually in your home directory, then verify that the absolute path is in nltk.data.path. It's essential to ensure that this directory exists and is in nltk.data.path before continuing. Once you have your nltk_data directory, the convention is that corpora reside in a corpora subdirectory. Create this corpora directory within the nltk_data directory, so that the path is ~/nltk_data/corpora. Finally, we'll create a subdirectory in corpora to hold our custom corpus. Let's call it cookbook, giving us the full path of ~/nltk_data/corpora/cookbook. Now we can create a simple word list file and make sure it loads. The source code for this article can be downloaded here. Consider a word list file called mywords.txt. Put this file into ~/nltk_data/corpora/cookbook/. Now we can use nltk.data.load() to load the file. >>> import nltk.data >>> nltk.data.load('corpora/cookbook/mywords.txt', format='raw') 'nltkn' We need to specify format='raw' since nltk.data.load() doesn't know how to interpret .txt files. As we'll see, it does know how to interpret a number of other file formats. How it works... The nltk.data.load() function recognizes a number of formats, such as 'raw', 'pickle', and 'yaml'. If no format is specified, then it tries to guess the format based on the file's extension. In the previous case, we have a .txt file, which is not a recognized extension, so we have to specify the 'raw' format. But if we used a file that ended in .yaml, then we would not need to specify the format. Filenames passed in to nltk.data.load() can be absolute or relative paths. Relative paths must be relative to one of the paths specified in nltk.data.path. The file is found using nltk.data.find(path), which searches all known paths combined with the relative path. Absolute paths do not require a search, and are used as is. There's more... For most corpora access, you won't actually need to use nltk.data.load, as that will be handled by the CorpusReader classes covered in the following recipes. But it's a good function to be familiar with for loading .pickle files and .yaml files, plus it introduces the idea of putting all of your data files into a path known by NLTK. Loading a YAML file If you put the synonyms.yaml file into ~/nltk_data/corpora/cookbook (next to mywords.txt), you can use nltk.data. load() to load it without specifying a format. >>> import nltk.data >>> nltk.data.load('corpora/cookbook/synonyms.yaml') {'bday': 'birthday'} This assumes that PyYAML is installed. If not, you can find download and installation instructions at http://pyyaml.org/wiki/PyYAML. See also In the next recipes, we'll cover various corpus readers, and then in the Lazy corpus loading recipe, we'll use the LazyCorpusLoader, which expects corpus data to be in a corpora subdirectory of one of the paths specified by nltk.data.path. Creating a word list corpus The WordListCorpusReader is one of the simplest CorpusReader classes. It provides access to a file containing a list of words, one word per line. Getting ready We need to start by creating a word list file. This could be a single column CSV file, or just a normal text file with one word per line. Let's create a file named wordlist that looks like this: nltk corpus corpora wordnet How to do it... Now we can instantiate a WordListCorpusReader that will produce a list of words from our file. It takes two arguments: the directory path containing the files, and a list of filenames. If you open the Python console in the same directory as the files, then '.' can be used as the directory path. Otherwise, you must use a directory path such as: 'nltk_data/corpora/ cookbook'. >>> from nltk.corpus.reader import WordListCorpusReader >>> reader = WordListCorpusReader('.', ['wordlist']) >>> reader.words() ['nltk', 'corpus', 'corpora', 'wordnet'] >>> reader.fileids() ['wordlist'] How it works... WordListCorpusReader inherits from CorpusReader, which is a common base class for all corpus readers. CorpusReader does all the work of identifying which files to read, while WordListCorpus reads the files and tokenizes each line to produce a list of words. Here's an inheritance diagram: When you call the words() function, it calls nltk.tokenize.line_tokenize() on the raw file data, which you can access using the raw() function. >>> reader.raw() 'nltkncorpusncorporanwordnetn' >>> from nltk.tokenize import line_tokenize >>> line_tokenize(reader.raw()) ['nltk', 'corpus', 'corpora', 'wordnet'] There's more... The stopwords corpus is a good example of a multi-file WordListCorpusReader. Names corpus Another word list corpus that comes with NLTK is the names corpus. It contains two files: female.txt and male.txt, each containing a list of a few thousand common first names organized by gender. >>> from nltk.corpus import names >>> names.fileids() ['female.txt', 'male.txt'] >>> len(names.words('female.txt')) 5001 >>> len(names.words('male.txt')) 2943 English words NLTK also comes with a large list of English words. There's one file with 850 basic words, and another list with over 200,000 known English words. >>> from nltk.corpus import words >>> words.fileids() ['en', 'en-basic'] >>> len(words.words('en-basic')) 850 >>> len(words.words('en')) 234936 Creating a part-of-speech tagged word corpus Part-of-speech tagging is the process of identifying the part-of-speech tag for a word. Most of the time, a tagger must first be trained on a training corpus. Let us take a look at how to create and use a training corpus of part-of-speech tagged words. Getting ready The simplest format for a tagged corpus is of the form "word/tag". Following is an excerpt from the brown corpus: The/at-tl expense/nn and/cc time/nn involved/vbn are/ber astronomical/ jj ./. Each word has a tag denoting its part-of-speech. For example, nn refers to a noun, while a tag that starts with vb is a verb. How to do it... If you were to put the previous excerpt into a file called brown.pos, you could then create a TaggedCorpusReader and do the following: >>> from nltk.corpus.reader import TaggedCorpusReader >>> reader = TaggedCorpusReader('.', r'.*.pos') >>> reader.words() ['The', 'expense', 'and', 'time', 'involved', 'are', ...] >>> reader.tagged_words() [('The', 'AT-TL'), ('expense', 'NN'), ('and', 'CC'), …] >>> reader.sents() [['The', 'expense', 'and', 'time', 'involved', 'are', 'astronomical', '.']] >>> reader.tagged_sents() [[('The', 'AT-TL'), ('expense', 'NN'), ('and', 'CC'), ('time', 'NN'), ('involved', 'VBN'), ('are', 'BER'), ('astronomical', 'JJ'), ('.', '.')]] >>> reader.paras() [[['The', 'expense', 'and', 'time', 'involved', 'are', 'astronomical', '.']]] >>> reader.tagged_paras() [[[('The', 'AT-TL'), ('expense', 'NN'), ('and', 'CC'), ('time', 'NN'), ('involved', 'VBN'), ('are', 'BER'), ('astronomical', 'JJ'), ('.', '.')]]] How it works... This time, instead of naming the file explicitly, we use a regular expression, r'.*.pos', to match all files whose name ends with .pos. We could have done the same thing as we did with the WordListCorpusReader, and pass ['brown.pos'] as the second argument, but this way you can see how to include multiple files in a corpus without naming each one explicitly. TaggedCorpusReader provides a number of methods for extracting text from a corpus. First, you can get a list of all words, or a list of tagged tokens. A tagged token is simply a tuple of (word, tag). Next, you can get a list of every sentence, and also every tagged sentence, where the sentence is itself a list of words or tagged tokens. Finally, you can get a list of paragraphs, where each paragraph is a list of sentences, and each sentence is a list of words or tagged tokens. Here's an inheritance diagram listing all the major methods: There's more... The functions demonstrated in the previous diagram all depend on tokenizers for splitting the text. TaggedCorpusReader tries to have good defaults, but you can customize them by passing in your own tokenizers at initialization time. Customizing the word tokenizer The default word tokenizer is an instance of nltk.tokenize.WhitespaceTokenizer. If you want to use a different tokenizer, you can pass that in as word_tokenizer. >>> from nltk.tokenize import SpaceTokenizer >>> reader = TaggedCorpusReader('.', r'.*.pos', word_ tokenizer=SpaceTokenizer()) >>> reader.words() ['The', 'expense', 'and', 'time', 'involved', 'are', ...] Customizing the sentence tokenizer The default sentence tokenizer is an instance of nltk.tokenize.RegexpTokenize with 'n' to identify the gaps. It assumes that each sentence is on a line all by itself, and individual sentences do not have line breaks. To customize this, you can pass in your own tokenizer as sent_tokenizer. >>> from nltk.tokenize import LineTokenizer >>> reader = TaggedCorpusReader('.', r'.*.pos', sent_ tokenizer=LineTokenizer()) >>> reader.sents() [['The', 'expense', 'and', 'time', 'involved', 'are', 'astronomical', '.']] Customizing the paragraph block reader Paragraphs are assumed to be split by blank lines. This is done with the default para_ block_reader, which is nltk.corpus.reader.util.read_blankline_block. There are a number of other block reader functions in nltk.corpus.reader.util, whose purpose is to read blocks of text from a stream. Their usage will be covered in more detail in the later recipe, Creating a custom corpus view, where we'll create a custom corpus reader. Customizing the tag separator If you don't want to use '/' as the word/tag separator, you can pass an alternative string to TaggedCorpusReader for sep. The default is sep='/', but if you want to split words and tags with '|', such as 'word|tag', then you should pass in sep='|'. Simplifying tags with a tag mapping function If you'd like to somehow transform the part-of-speech tags, you can pass in a tag_mapping_ function at initialization, then call one of the tagged_* functions with simplify_ tags=True. Here's an example where we lowercase each tag: >>> reader = TaggedCorpusReader('.', r'.*.pos', tag_mapping_ function=lambda t: t.lower()) >>> reader.tagged_words(simplify_tags=True) [('The', 'at-tl'), ('expense', 'nn'), ('and', 'cc'), …] Calling tagged_words() without simplify_tags=True would produce the same result as if you did not pass in a tag_mapping_function. There are also a number of tag simplification functions defined in nltk.tag.simplify. These can be useful for reducing the number of different part-of-speech tags. >>> from nltk.tag import simplify >>> reader = TaggedCorpusReader('.', r'.*.pos', tag_mapping_ function=simplify.simplify_brown_tag) >>> reader.tagged_words(simplify_tags=True) [('The', 'DET'), ('expense', 'N'), ('and', 'CNJ'), ...] >>> reader = TaggedCorpusReader('.', r'.*.pos', tag_mapping_ function=simplify.simplify_tag) >>> reader.tagged_words(simplify_tags=True) [('The', 'A'), ('expense', 'N'), ('and', 'C'), ...]

In this article, Indermohan Singh, the author of Ionic 2 Blueprints we will create a conference app. We will create an app which will provide list of speakers, schedule, directions to the venue, ticket booking, and lots of other features. We will learn the following things: Using the device's native features Leveraging localStorage Ionic menu and tabs Using RxJS to build a perfect search filter (For more resources related to this topic, see here.) Conference app is a companion application for conference attendees. In this application, we are using Lanyrd JSON Exportand hardcoded JSON file as our backend. We will have a tabs and side menu interface, just like our e-commerce application. When a user opens our app, the app will show a tab interface with SpeakersPageopen. It will have SchedulePage for conference schedule and AboutPage for information about conference. We will also make this app work offline, without any Internet connection. So, your user will still be able to view speakers, see the schedule, and do other stuff without using the Internet at all. JSON data In the application, we have used a hardcoded JSON file as our Database. But in the truest sense, we are actually using a JSON export of a Lanyrd event. I was trying to make this article using Lanyrd as the backend, but unfortunately, Lanyrd is mostly in maintenance mode. So I was not able to use it. In this article, I am still using a JSON export from Lanyrd, from a previous event. So, if you are able to get a JSON export for your event, you can just swap the URL and you are good to go. Those who don't want to use Lanyrd and instead want to use their own backend, must have a look at the next section. I have described the structure of JSON, which I have used to make this app. You can create your REST API accordingly. Understanding JSON Let's understand the structure of the JSON export. The whole JSON database is an object with two keys, timezone and sessions, like the following: { timezone: "Australia/Brisbane", sessions: [..] } Timezone is just a string, but sessions key is an array of lists of all the sessions of our conference. Items in the sessions array are divided according to days of the conference. Each item represents a day of the conference and has the following structure: { day: "Saturday 21st November", sessions: [..] } So, the sessions array of each day has actual sessions as items. Each item has the following structure: { start_time: "2015-11-21 09:30:00", topics: [], web_url: "url of event times: "9:30am - 10:00am", id: "sdtpgq", types: [ ], end_time_epoch: 1448064000, speakers: [], title: "Talk Title", event_id: "event_id", space: "Space", day: "Saturday 21st November", end_time: "2015-11-21 10:00:00", other_url: null, start_time_epoch: 1448062200, abstract: "<p>Abstract of Talk</p>" }, Here, the speakers array has a list of all speakers. We will use that speakers array to create a list of all speakers in an array. You can see the whole structure here: That's all we need to understand for JSON. Defining the app In this section, we will define various functionalities of our application. We will also show the architecture of our app using an app flow diagram. Functionalities We will be including the following functionalities in our application: List of speakers Schedule detail Search functionality using session title, abstract, and speaker's names Hide/Show any day of the schedule Favorite list for sessions Adding favorite sessions to the device calendar Ability to share sessions to other applications Directions to venue Offline working App flow This is how the control will flow inside our application: Let's understand the flow: RootComponent: RootComponent is the root Ionic component. It is defined inside the /app/app.ts file. TabsPage: TabsPage acts as a container for our SpeakersPage, SchedulePage, and AboutPage. SpeakersPage: SpeakersPage shows a list of all the speakers of our conference. SchedulePage: SchedulePage shows us the schedule of our conference and allows us various filter features. AboutPage: AboutPage provides us information about the conference. SpeakersDetail: SpeakerDetail page shows the details of the speaker and a list of his/her presentations in this conference. SessionDetail: SessionDetail page shows the details of a session with the title and abstract of the session. FavoritePage: FavoritePage shows a list of the user's favorite sessions. Summary In this article, we discussed about the JSON files that will used as database in our app. We also defined the the functionalities of our app and understood the flow of our app. Resources for Article:  Further resources on this subject: First Look at Ionic [article] Ionic JS Components [article] Creating Our First App with Ionic [article]

[box type="note" align="" class="" width=""]The following is an excerpt from the book Scala for Machine Learning, Second Edition, written by Patrick R. Nicolas. The book will help you leverage Scala and Machine Learning to study and construct systems that can learn from data. [/box] This article aims to teach how a mathematical formula can be converted into a machine learning model in three easy steps: Stackable traits enable developers to follow a strict mathematical formalism while implementing a model in Scala. Scientists use a universally accepted template to solve mathematical problems: Declare the variables relevant to the problem. Define a model (equation, algorithm, formulas…) as the solution to the problem. Instantiate the variables and execute the model to solve the problem. Let's consider the example of the concept of kernel functions, a model that consists of the composition of two mathematical functions, and its potential implementation in Scala. Step 1 – variable declaration The implementation consists of wrapping (scope) the two functions into traits and defining these functions as abstract values. The mathematical formalism is as follows: The Scala implementation is represented here: type V = Vector[Double] trait F{ val f: V => V} trait G{ val g: V => Double } Step 2 – model definition The model is defined as the composition of the two functions. The stack of traits G, F describes the type of compatible functions that can be composed using the self-referenced constraint self: G with F: Formalism h = f o g The implementation is as follows: class H {self: G with F =>def apply(v:V): Double =g(f(v))} Step 3 – instantiation The model is executed once the variable f and g are instantiated. The formalism is as follows: The implementation is as follows: val h = new H with G with F { val f: V=>V = (v: V) =>v.map(exp(_)) val g: V => Double = (v: V) =>v.sum Note: Lazy value trigger In the preceding example, the value of h(v) = g(f(v)) can be automatically computed as soon as g and f are initialized, by declaring h as a lazy value. Clearly, Scala preserves the formalism of mathematical models, making it easier for scientists and developers to migrate their existing projects written in scientific oriented languages such as R. Note: Emulation of R Most data scientists use the language R to create models and apply learning strategies. They may consider Scala as an alternative to R in some cases, as Scala preserves the mathematical formalism used in models implemented in R. In this article we explained the concept preservation of mathematical formalism. This needs to be further extended to dynamic creation of workflows using traits. Design patterns, such as the cake pattern, can be used to do this. If you enjoyed this excerpt, be sure to check out the book Scala Machine Learning, Second Edition to gain solid foundational knowledge in machine learning with Scala.