Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-evade-intrusion-detection-systems-using-proxy-cannon-tutorial
Packt Editorial Staff
26 Mar 2019
10 min read
Save for later

Adrian Pruteanu shows how to evade Intrusion Detection Systems using Proxy Cannon [Tutorial]

Packt Editorial Staff
26 Mar 2019
10 min read
These days, it is fairly common for mature companies to implement Intrusion detection system (IDS), intrusion prevention systems (IPS), and security information and event management (SIEM) when they detect abuse against a particular application. When an unknown IP is performing too many operations in a short time on a protected application, IDS or IPS may take action against the source. If we are conducting a password spraying attack, we may avoid lockouts but we're still hammering the server from one source: our machine. A good way to evade these types of detection systems is to distribute the connection requests from the attacker machine over many IPs, which is commonly done by malicious actors through networks of compromised hosts. With the advent of cloud computing and computing time becoming increasingly cheap, even free in some cases, we don't have to stray outside of the law and build a botnet. In this article we'll see how to use Proxy cannon to evade intrusion detection systems (IDS). This article is taken from the book Becoming the Hacker written by Adrian Pruteanu. This book will teach you how to approach web penetration testing with an attacker's mindset. While testing web applications for performance is common, the ever-changing threat landscape makes security testing much more difficult for the defender. The Tor Project was started to provide a way for users to browse the internet anonymously. It is by far the best way to anonymize traffic and best of all, it's free. It is an effective way to change the public IP during an attack. The Tor network Tor is a network of independently operated nodes interconnected to form a network through which packets can be routed. The following graphic shows how a user, Alice, can connect to Bob through a randomly generated path or circuit, through the Tor network: Figure 1: The Tor network traffic flow (source: https://www.torproject.org/) Instead of connecting directly to the destination, the client connection from Alice to Bob will be routed through a randomly chosen set of nodes in the Tor network. Each packet is encrypted and every node can only decrypt enough information to route it to the next hop along the path. The exit node is the final node in the chain, which will make the connection to the intended destination on behalf of the client. When the packet arrives at Bob's machine, the request will look like it's coming from the exit node and not Alice's public IP. Note: More information on Tor can be found on the official site: https://www.torproject.org. While Tor is important for anonymity, we're not really concerned with staying completely anonymous. We can, however, leverage the randomly chosen exit nodes to mask the public IP when attacking an application. There are a couple of issues with conducting attacks through the Tor network. The routing protocol is inherently slower than a more direct connection. This is because Tor adds several layers of encryption to each transmission, and each transmission is forwarded through three Tor nodes on top of the normal routing that internet communication requires. This process improves anonymity but also increases communication delay significantly. The lag is noticeable for normal web browsing, but this is a tolerable trade-off. For large volume scans, it may not be the ideal transport. Warning: It should also be noted that Tor is used heavily in regions of the world where privacy is of utmost importance. Conducting large volume attacks through Tor is discouraged, as it can lead to unnecessary network slowdowns and can impact legitimate users. Low and slow attacks shouldn't cause any problems. Some red-team engagements may even require testing from the Tor network to verify related IDS/IPS rules are working as intended, but caution should be taken when launching attacks through a limited-resource public medium. Proxy cannon An alternative to using Tor for diversifying our attack IPs is to simply use the cloud. There are countless Infrastructure as a Service (IaaS) providers, each with a large IP space available for free to VM instances. VMs are cheap and sometimes free as well, so routing our traffic through them should be fairly cost effective. Amazon, Microsoft, and Google all have an easy-to-use API for automating the management of VM instances. If we can spawn a new VM with a new external IP periodically, we can route our traffic to the target application through it and mask our true origin. This should make it much more difficult for automated systems to detect and alert on our activities. Cue ProxyCannon, a great tool that does all the heavy lifting of talking to Amazon's AWS API, creating and destroying VM instances, rotating external IPs, and routing our traffic through them. Note: ProxyCannon was developed by Shellntel and is available on GitHub: https://github.com/Shellntel/scripts/blob/master/proxyCannon.py. ProxyCannon requires boto, a Python library that provides API access to Amazon's AWS. We can use Python's pip command to install the required dependency: root@kali:~/tools# pip install -U boto Collecting boto  Downloading boto-2.48.0-py2.py3-none-any.whl (1.4MB) [...] Installing collected packages: boto Successfully installed boto-2.48.0 The ProxyCannon tool should now be ready to use with the -h option showing all of the available options: root@kali:~/tools# python proxyCannon.py -h usage: proxyCannon.py [-h] [-id [IMAGE_ID]] [-t [IMAGE_TYPE]]             [--region [REGION]] [-r] [-v] [--name [NAME]]             [-i [INTERFACE]] [-l]             num_of_instances positional arguments:  num_of_instances   The number of amazon instances you'd like to launch. optional arguments:  -h, --help         show this help message and exit  -id [IMAGE_ID], --image-id [IMAGE_ID]             Amazon ami image ID. Example: ami-d05e75b8. If not             set, ami-d05e75b8.  -t [IMAGE_TYPE], --image-type [IMAGE_TYPE]             Amazon ami image type Example: t2.nano. If not             set, defaults to t2.nano.  --region [REGION] Select the region: Example: us-east-1. If             not set, defaults to us-east-1. positional arguments:  num_of_instances   The number of amazon instances you'd like to launch. optional arguments:  -h, --help         show this help message and exit  -id [IMAGE_ID], --image-id [IMAGE_ID]             Amazon ami image ID. Example: ami-d05e75b8. If not             set, ami-d05e75b8.  -t [IMAGE_TYPE], --image-type [IMAGE_TYPE]             Amazon ami image type Example: t2.nano. If not             set, defaults to t2.nano.  --region [REGION] Select the region: Example: us-east-1. If             not set, defaults to us-east-1. Output is to /tmp/ By default, ProxyCannon creates t2.nano virtual instances in AWS, which should be free for a limited time with new accounts. They have very little resources but are typically enough for most attacks. To change the type of instance, we can supply the -t switch. The default region is us-east-1 and can be adjusted using the --region switch. ProxyCannon will create as many instances as specified in the num_of_instances and using the -r switch, it will rotate them regularly. The -l switch is also useful to keep track of what public IPs ProxyCannon is using over the course of the execution. This is useful for reporting purposes: the blue team may need a list of all the IPs used in the attack. In order for the tool to be able to communicate with our AWS account and to manage instances automatically, we have to create API access keys in the AWS console. The interface is fairly straightforward and can be accessed in the account Security Credentials page. The access key ID and the secret keys are randomly generated and should be stored securely. Once the engagement is over, you should delete the keys in the AWS console. Figure 2: Generating a new AWS API access key We can start ProxyCannon using the -r and -l switches, and specify that we want 3 instances running at the same time. root@kali:~/tools# python proxyCannon.py -r -l 3 What is the AWS Access Key Id: d2hhdCBhcmUgeW91IGRvaW5n What is the AWS Secret Access Key: dW5mb3J0dW5hdGVseSB0aGlzIGlzIG5vdCB0aGUgcmVhbCBrZXku [...] Upon first run, ProxyCannon will ask you for these values and store them in the ~/.boto file. root@kali:~/tools# cat ~/.boto [default] aws_access_key_id = d2hhdCBhcmUgeW91IGRvaW5n aws_secret_access_key = dW5mb3J0dW5hdGVseSB0aGlzIGlzIG5vdCB0aGUgcmVhbCBrZXku As you can see, these are stored in plaintext, so make sure this file is properly protected. Amazon recommends that these keys are rotated frequently. It's probably a good idea to create new ones for each engagement and delete them from AWS as soon as they're not required anymore. ProxyCannon will connect to Amazon EC2, setup the SSH keys, adjust the security groups, and start the VM instances. This process may take a couple of minutes to complete. [*] Connecting to Amazon's EC2... [*] Generating ssh keypairs... [*] Generating Amazon Security Group... [~] Starting 3 instances, please give about 4 minutes for them to fully boot [====================] 100% ProxyCannon will overwrite the current system iptables configuration to properly route all traffic through whatever instance is chosen: [*] Provisioning Hosts..... [*] Saving existing iptables state [*] Building new iptables... [*] Done! +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + Leave this terminal open and start another to run your commands.+ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ [~] Press ctrl + c to terminate the script gracefully. [...] As promised, ProxyCannon will periodically rotate our effective external IP using SSH tunnels and by modifying the routing table. All of this is done automatically, in the background, while Burp Suite or ZAP runs the password spraying attack. The following is the periodic output from ProxyCannon showing the IPs being rotated: [*] Rotating IPs. [*] Replaced 107.21.177.36 with 34.207.187.254 on tun0 [*] Replaced 34.234.91.233 with 52.91.91.157 on tun1 [*] Replaced 34.202.237.230 with 34.228.167.195 on tun2 [*] Replaced 34.207.187.254 with 34.228.158.208 on tun0 [*] Replaced 52.91.91.157 with 54.198.223.114 on tun1 On the AWS console, we can see the started t2.nano instances and their public IPs: Figure 3: AWS instances created to route our traffic through We can test ProxyCannon by repeating a curl request to our target application using the watch command. We don't need to drop in a shell similar to torsocks because ProxyCannon modifies the local system routing to help us change our external IP. root@kali:~# watch -n30 curl http://c2.spider.ml On the target application side, c2.spider.ml, the server log, shows connection attempts from various IPs belonging to the Amazon address space: 52.91.91.157 - - [13:01:16] "GET / HTTP/1.1" 200 - 52.91.91.157 - - [13:01:22] "GET / HTTP/1.1" 200 - 34.228.158.208 - - [13:01:43] "GET / HTTP/1.1" 200 - 34.228.158.208 - - [13:01:48] "GET / HTTP/1.1" 200 - 54.198.223.114 - - [13:06:34] "GET / HTTP/1.1" 200 - 54.198.223.114 - - [13:06:39] "GET / HTTP/1.1" 200 - It should be noted that there is a lower limit to how often we can rotate the IPs on Amazon or any cloud provider for that matter. It takes a while for instances to boot and IP addresses to be reserved, associated, and become active. ProxyCannon has a hardcoded value of about 90 seconds to ensure the effective IP actually changes. In this article, we looked at Proxy cannon for staying under the radar while conducting brute-force attacks during an engagement. Becoming the Hacker is a playbook to help you become an ethical hacker and protect the web. Learn about the tricks of a web attacker. 6 common use cases of Reverse Proxy scenarios MarioNet: A browser-based attack that allows hackers to run malicious code even if users’ exit a web page Black Hat hackers used IPMI cards to launch JungleSec Ransomware, affects most of the Linux servers
Read more
  • 0
  • 0
  • 25152

article-image-plotting-data-sage
Packt
05 May 2011
14 min read
Save for later

Plotting Data with Sage

Packt
05 May 2011
14 min read
Sage Beginner's Guide Unlock the full potential of Sage for simplifying and automating mathematical computing Confusion alert: Sage plots and matplotlib The 2D plotting capabilities of Sage are built upon a Python plotting package called matplotlib. The most widely used features of matplotlib are accessible through Sage functions. You can also import the matplotlib package into Sage, and use all of its features directly. This is very powerful, but it's also confusing, because there's more than one way to do the same thing. To further add to the confusion, matplotlib has two interfaces: the command-oriented Pyplot interface and an object-oriented interface. The examples in this chapter will attempt to clarify which interface is being used. Plotting in two dimensions Two-dimensional plots are probably the most important tool for visually presenting information in math, science, and engineering. Sage has a wide variety of tools for making many types of 2D plots. Plotting symbolic expressions with Sage We will start by exploring the plotting functions that are built in to Sage. They are generally less flexible than using matplotlib directly, but also tend to be easier to use. Time for action – plotting symbolic expressions Let's plot some simple functions. Enter the following code: p1 = plot(sin, (-2*pi, 2*pi), thickness=2.0, rgbcolor=(0.5, 1, 0), legend_label='sin(x)') p2 = plot(cos, (-2*pi, 2*pi), thickness=3.0, color='purple', alpha=0.5, legend_label='cos(x)') plt = p1 + p2 plt.axes_labels(['x', 'f(x)']) show(plt) If you run the code from the interactive shell, the plot will open in a separate window. If you run it from the notebook interface, the plot will appear below the input cell. In either case, the result should look like this: What just happened? This example demonstrated the most basic type of plotting in Sage. The plot function requires the following arguments: graphics_object = plot(callable symbolic expression, (independent_var, ind_var_min, ind_var_max)) The first argument is a callable symbolic expression, and the second argument is a tuple consisting of the independent variable, the lower limit of the domain, and the upper limit. If there is no ambiguity, you do not need to specify the independent variable. Sage automatically selects the right number of points to make a nice curve in the specified domain. The plot function returns a graphics object. To combine two graphics objects in the same image, use the + operator: plt = p1 + p2. Graphics objects have additional methods for modifying the final image. In this case, we used the axes_labels method to label the x and y axes. Finally, the show function was used to finish the calculation and display the image. The plot function accepts optional arguments that can be used to customize the appearance and format of the plot. To see a list of all the options and their default values, type: sage: plot.options {'fillalpha': 0.5, 'detect_poles': False, 'plot_points': 200, 'thickness': 1, 'alpha': 1, 'adaptive_tolerance': 0.01, 'fillcolor': 'automatic', 'adaptive_recursion': 5, 'exclude': None, 'legend_label': None, 'rgbcolor': (0, 0, 1), 'fill': False} Here is a summary of the options for customizing the appearance of a plot: KeywordDescriptionalphaTransparency of the line (0=opaque, 1=transparent)fillTrue to fill area below the linefillalphaTransparency of the filled-in area (0=opaque, 1=transparent)fillcolorColor of the filled-in areargbcolorColor of the line Sage uses an algorithm to determine the best number of points to use for the plot, and how to distribute them on the x axis. The algorithm uses recursion to add more points to resolve regions where the function changes rapidly. Here are the options that control how the plot is generated: KeywordDescriptionadaptive_recursionMax depth of recursion when resolving areas of the plot where the function changes rapidlyadaptive_toleranceTolerance for stopping recursiondetect_polesDetect points where function value approaches infinity (see next example)excludeA list or tuple of points to exclude from the plotplot_pointsNumber of points to use in the plot Specifying colors in Sage There are several ways to specify a color in Sage. For basic colors, you can use a string containing the name of the color, such as red or blue. You can also use a tuple of three floating-point values between 0 and 1.0. The first value is the amount of red, the second is the amount of green, and the third is the amount of blue. For example, the tuple (0.5, 0.0, 0.5) represents a medium purple color. Some functions "blow up" to plus or minus infinity at a certain point. A simplistic plotting algorithm will have trouble plotting these points, but Sage adapts. Time for action – plotting a function with a pole Let's try to plot a simple function that takes on infinite values within the domain of the plot: pole_plot = plot(1 / (x - 1), (0.8, 1.2), detect_poles='show', marker='.') print("min y = {0} max y = {1}".format(pole_plot.ymax(), pole_plot.ymin())) pole_plot.ymax(100.0) pole_plot.ymin(-100.0) # Use TeX to make nicer labels pole_plot.axes_labels([r'$x$', r'$1/(x-1)$']) pole_plot.show() The output from this code is as follows: What just happened? We did a few things differently compared to the previous example. We defined a callable symbolic expression right in the plot function. We also used the option detect_poles='show' to plot a dashed vertical line at the x value where the function returns infinite values. The option marker='.' tells Sage to use a small dot to mark the individual (x,y) values on the graph. In this case, the dots are so close together that they look like a fat line. We also used the methods ymin and ymax to get and set the minimum and maximum values of the vertical axis. When called without arguments, these methods return the current values. When given an argument, they set the minimum and maximum values of the vertical axis. Finally, we labeled the axes with nicely typeset mathematical expressions. As in the previous example, we used the method axes_labels to set the labels on the x and y axes. However, we did two special things with the label strings: r'$frac{1}{(x-1)}$' The letter r is placed in front of the string, which tells Python that this is a raw string. When processing a raw string, Python does not interpret backslash characters as commands (such as interpreting n as a newline). Note that the first and last characters of the string are dollar signs, which tells Sage that the strings contain mark-up that needs to be processed before being displayed. The mark-up language is a subset of TeX, which is widely used for typesetting complicated mathematical expressions. Sage performs this processing with a built-in interpreter, so you don't need to have TeX installed to take advantage of typeset labels. It's a good idea to use raw strings to hold TeX markup because TeX uses a lot of backslashes. To learn about the typesetting language, see the matplotlib documentation at: http://matplotlib.sourceforge.net/users/mathtext.html Time for action – plotting a parametric function Some functions are defined in terms of a parameter. Sage can easily plot parametric functions: var('t') pp = parametric_plot((cos(t), sin(t)), (t, 0, 2*pi), fill=True, fillcolor='blue') pp.show(aspect_ratio=1, figsize=(3, 3), frame=True) The output from this code is as follows: What just happened? We used two parametric functions to plot a circle. This is a convenient place to demonstrate the fill option, which fills in the space between the function and the horizontal axis. The fillcolor option tells Sage which color to use for the fill, and the color can be specified in the usual ways. We also demonstrated some useful options for the show method (these options also work with the show function). The option aspect_ratio=1 forces the x and y axes to use the same scale. In other words, one unit on the x axis takes up the same number of pixels on the screen as one unit on the y axis. Try changing the aspect ratio to 0.5 and 2.0, and see how the circle looks. The option figsize=(x_size,y_size) specifies the aspect ratio and relative size of the figure. The units for the figure size are relative, and don't correspond to an absolute unit like inches or centimetres. The option frame=True places a frame with tick marks around the outside of the plot. Time for action – making a polar plot Some functions are more easily described in terms of angle and radius. The angle is the independent variable, and the radius at that angle is the dependent variable. Polar plots are widely used in electrical engineering to describe the radiation pattern of an antenna. Some antennas are designed to transmit (or receive) electromagnetic radiation in a very narrow beam. The beam shape is known as the radiation pattern. One way to achieve a narrow beam is to use an array of simple dipole antennas, and carefully control the phase of the signal fed to each antenna. In the following example, we will consider seven short dipole antennas set in a straight line: # A linear broadside array of short vertical dipoles # located along the z axis with 1/2 wavelength spacing var('r, theta') N = 7 normalized_element_pattern = sin(theta) array_factor = 1 / N * sin(N * pi / 2 * cos(theta)) / sin(pi / 2 * cos(theta)) array_plot = polar_plot(abs(array_factor), (theta, 0, pi), color='red', legend_label='Array') radiation_plot = polar_plot(abs(normalized_element_pattern * array_factor), (theta, 0, pi), color='blue', legend_label='Radiation') combined_plot = array_plot + radiation_plot combined_plot.xmin(-0.25) combined_plot.xmax(0.25) combined_plot.set_legend_options(loc=(0.5, 0.3)) show(combined_plot, figsize=(2, 5), aspect_ratio=1) Execute the code. You should get a plot like this: What just happened? We plotted a polar function, and used several of the plotting features that we've already discussed. There are two subtle points worth mentioning. The function array_factor is a function of two variables, N and theta. In this example, N is more like a parameter, while theta is the independent variable we want to use for plotting. We use the syntax (theta, 0, pi) in the plot function to indicate that theta is the independent variable. The second new aspect of this example is that we used the methods xmin and xmax to set the limits of the x axis for the graphics object called combined_plot. We also used the set_legend_options of the graphics object to adjust the position of the legend to avoid covering up important details of the plot. Time for action – plotting a vector field Vector fields are used to represent force fields such as electromagnetic fields, and are used to visualize the solutions of differential equations. Sage has a special plotting function to visualize vector fields. var('x, y') a = plot_vector_field((x, y), (x, -3, 3), (y, -3, 3), color='blue') b = plot_vector_field((y, -x), (x, -3, 3), (y, -3, 3), color='red') show(a + b, aspect_ratio=1, figsize=(4, 4)) You should get the following image: What just happened? The plot_vector_field function uses the following syntax: plot_vector_field((x_function,y_function), (x,x_min,x_max), (y,y_ min,y_max)) The keyword argument color specifies the color of the vectors. Plotting data in Sage So far, we've been making graphs of functions. We specify the function and the domain, and Sage automatically chooses the points to make a nice-looking curve. Sometimes, we need to plot discrete data points that represent experimental measurements or simulation results. The following functions are used for plotting defined sets of points. Time for action – making a scatter plot Scatter plots are used in science and engineering to look for correlation between two variables. A cloud of points that is roughly circular indicates that the two variables are independent, while a more elliptical arrangement indicates that there may be a relationship between them. In the following example, the x and y coordinates are contrived to make a nice plot. In real life, the x and y coordinates would typically be read in from data files. Enter the following code: def noisy_line(m, b, x): return m * x + b + 0.5 * (random() - 0.5) slope = 1.0 intercept = -0.5 x_coords = [random() for t in range(50)] y_coords = [noisy_line(slope, intercept, x) for x in x_coords] sp = scatter_plot(zip(x_coords, y_coords)) sp += line([(0.0, intercept), (1.0, slope+intercept)], color='red') sp.show() The result should look similar to this plot. Note that your results won't match exactly, since the point positions are determined randomly. What just happened? We created a list of randomized x coordinates using the built-in random function. This function returns a random number in the range 0 <= x < 1. We defined a function called noisy_line that we then used to create a list of randomized y coordinates with a linear relationship to the x coordinates. We now have a list of x coordinates and a list of y coordinates, but the scatter_plot function needs a list of (x,y) tuples. The zip function takes the two lists and combines them into a single list of tuples. The scatter_plot function returns a graphics object called sp. To add a line object to the plot, we use the following syntax: sp += line([(x1, y1), (x2,y2)], color='red') The += operator is a way to increment a variable; x+=1 is a shortcut for x = x + 1. Because the + operator also combines graphics objects, this syntax can be used to add a graphics object to an existing graphics object. Time for action – plotting a list Sometimes, you need to plot a list of discrete data points. The following example might be found in an introductory digital signal processing (DSP) course. We will use lists to represent digital signals. We sample the analogue function cosine(t) at two different sampling rates, and plot the resulting digital signals. # Use list_plot to visualize digital signals # Undersampling and oversampling a cosine signal sample_times_1 = srange(0, 6*pi, 4*pi/5) sample_times_2 = srange(0, 6*pi, pi/3) data1 = [cos(t) for t in sample_times_1] data2 = [cos(t) for t in sample_times_2] plot1 = list_plot(zip(sample_times_1, data1), color='blue') plot1.axes_range(0, 18, -1, 1) plot1 += text("Undersampled", (9, 1.1), color='blue', fontsize=12) plot2 = list_plot(zip(sample_times_2, data2), color='red') plot2.axes_range(0, 18, -1, 1) plot2 += text("Oversampled", (9, 1.1), color='red', fontsize=12) g = graphics_array([plot1, plot2], 2, 1) # 2 rows, 1 column g.show(gridlines=["minor", False]) The result is as follows: What just happened? The function list_plot works a lot like scatter_plot from the previous example, so I won't explain it again. We used the method axes_range(x_min, x_max, y_min, y_max) to set the limits of the x and y axes all at once. Once again, we used the += operator to add a graphics object to an existing object. This time, we added a text annotation instead of a line. The basic syntax for adding text at a given (x,y) position is text('a string', (x,y)). To see the options that text accepts, type the following: sage: text.options {'vertical_alignment': 'center', 'fontsize': 10, 'rgbcolor': (0, 0, 1), 'horizontal_alignment': 'center', 'axis_coords': False} To display the two plots, we introduced a new function called graphics_array, which uses the basic syntax: graphics_array([plot_1, plot_2, ..., plot_n], num_rows, num_columns) This function returns another graphics object, and we used the show method to display the plots. We used the keyword argument gridlines=["minor", False] to tell Sage to display vertical lines at each of the minor ticks on the x axis. The first item in the list specifies vertical grid lines, and the second specifies horizontal grid lines. The following options can be used for either element: "major"Grid lines at major ticks"minor"Grid lines at major and minor ticksFalseNo grid lines Try playing with these options in the previous example.  
Read more
  • 0
  • 0
  • 25149

article-image-uber-ai-labs-senior-research-scientist-ankit-jain-tensorflow-updates-learning-machine-learning
Sugandha Lahoti
19 Dec 2019
10 min read
Save for later

Uber AI Labs senior research scientist, Ankit Jain on TensorFlow updates and learning machine learning by doing [Interview]

Sugandha Lahoti
19 Dec 2019
10 min read
No doubt, TensorFlow is one of the most popular machine learning libraries right now. However, newbie developers who want to experiment with TensorFlow often face difficulties in learning TensorFlow, relying just on tutorials.  Recently, we sat down with Ankit Jain, senior research scientist at Uber AI Labs and one of the authors of the book, TensorFlow Machine Learning Projects. Ankit talked about how real-world implementations can be a good way to learn for those developing TF models, specifically the ‘learn by doing’ approach. Talking about TensorFlow 2.0, he considers ‘eager execution by default’ a major paradigm shift and is all game for interoperability between TF 2.0 and other machine learning frameworks. He also gave us an insight into the limitations of AI algorithms (generalization, AI ethics, labeled data to name a few). Continue reading the full interview for a detailed perspective. On why TensorFlow 2 upgrade is paradigm-shifting in more ways than one TensorFlow 2 was released last month. What are some of your top features in TensorFlow 2.0? How do you think it has upgraded the machine learning ecosystem? TF 2.0 is a major upgrade from its predecessor in many ways. It addressed many of the shortcomings of TF 1.x and with this release, the difference between Pytorch and TF has narrowed. One of the biggest paradigm shifts in TF 2.0 is eager execution by default. This means you don’t have to pre-define a static computation graph, create sessions, deal with the unintuitive interface or have painful experience in debugging your deep learning model code. However, you lose on some performance in run time when you switch to complete eager mode. For that purpose, they have introduced tf.function decorator which can help you translate your Python functions to Tensorflow graphs. This way you can retain both code readability and ease of debugging while getting the performance of TensorFlow graphs.  Another major update is that many confusing redundancies have been consolidated and many functions are now integrated with Keras API. This will help to standardize the communication of data/models among various components of TensorFlow ecosystem. TF 2.0 also comes with backward compatibility to TF 1.X with an easy optional way to convert your TF 1.X code into TF 2.0. TF 1.X suffered from a lack of standardization in how we load/save trained machine learning models. TF 2.0 fixed this by defining a single API SavedModels. As SavedModels is integrated with the Tensorflow ecosystem, it becomes much easier to deploy models using Tensorflow Lite, Tensorflow.js to other devices/applications.   With the onset of TensorFlow 2, Tensorflow and Keras are integrated into one module (tf.keras). TF 2.0 now delivers Keras as the central high-level API used to build and train models. What is the future/benefits of TensorFlow + Keras?  Keras has been a very popular high-level API for faster prototyping and production and even for research. As the field of AI/ML is in nascent stages, ease of development can have a huge impact for people getting started in machine learning.  Previously, a developer new to machine learning started from Keras while an experienced researcher used only Tensorflow 1.x due to its flexibility to build custom models. With Keras integrated as a high level API for TF 2.0, we can expect both beginners and experts working on the same framework which can lead to better collaboration and better exchange of ideas in the community.  Additionally, a single high level easy to use API reduces confusion and streamlines consistency across use cases of production and research.  Overall, I think it’s a great step in the right direction by Google which will enable more developers to hop on the Tensorflow ecosystem.  On TensorFlow, NLP and structured learning Recently, Transformers 2.0, a popular OS NLP library, was released that provides TF 2.0 and PyTorch deep interoperability. What are your views on this development? One of the areas where deep learning has made an immense impact is Natural Language Processing (NLP). Research in NLP is moving very fast and it is hard to keep up with all the papers and code releases by various research groups around the world.  Hugging Face, the company behind the library “Transformers” has really eased the usage of state of the art (SOTA) models and process of building new models by simplifying the preprocessing and model building pipeline through an easy to use Keras like interface. “Transformers 2.0” is the recent release from the company and the most important feature is the interoperability between Pytorch and TF 2.0. TF 2.0 is more production-ready while Pytorch is more oriented towards research. With this upgrade, you can pretty much move from one framework to another for training, validation, and deployment of the model.  Interoperability between frameworks is very important for the AI community as it enables development velocity. Moreover, as none of the frameworks can be perfect at everything, it makes the framework developers focus more on their strengths and make those features seamless. This will create greater efficiency going forward. Overall, I think this is a great development and I expect other libraries in domains like Computer Vision, Graph Learning etc. to follow suit. This will enable a lot more application of state of the art models to production.  Google recently launched Neural Structured Learning (NSL), an open-source Tensorflow based framework for training neural networks with graphs and structured data. What are some of the potential applications of NSL? What do you think can be some Machine Learning Projects based around NSL? Neural structured learning is a concept of learning neural network parameters with structured signals other than features. Many real-world datasets contain some structured information like Knowledge graphs or molecular graphs in biology. Incorporating these signals can lead to a more accurate and robust model. From an implementation perspective, it boils down to adding a regularizer to the loss function such that the representation of neighboring nodes in the graph is similar.  Any application where the amount of labeled data is limited but has structural information like Knowledge Graph that can be exploited is a good candidate for these types of models. A possible example could be fraud detection in online systems. Fraud data generally has sparse labels and fraudsters create multiple accounts that are connected to each other through some information like devices etc. This structured information can be utilized to learn a better representation of fraud accounts.  There can be other applications is molecular data and other problems involving the knowledge graph. On Ankit’s experience working on his book, TensorFlow Machine Learning Project Tell us the motivation behind writing your book TensorFlow Machine Learning Projects. Why is TensorFlow ideal for building ML projects? What are some of your favorite machine learning projects from this book? When I started learning Tensorflow, I stumbled upon many tutorials (including the official ones) which explained various concepts on how Tensorflow works. While that was helpful in understanding the basics, most of my learning came from building projects with Tensorflow. That is when I realized the need for a resource that teaches using a ‘learn by doing’ approach. This book is unique in the way that it teaches machine learning theory, Tensorflow utilities and programming concepts all while developing a project in which you can have fun building and is also of practical use.  My favorite chapter from the book is “Generating Uncertainty in Traffic Signs Classifier using Bayesian Neural Networks”. With the development of self-driving cars, traffic signs detection is a major problem that needs to be solved. This chapter explains an advanced AI concept of Bayesian Neural Networks and shows step by step how to use those to detect traffic signs using Tensorflow. Some of the readers of the book have started to use this concept in their practical applications already. Machine Learning challenges and advice to those developing TensorFlow models What are the biggest challenges today in the field of Machine Learning and AI? What do you see as the greatest technology disruptors in the next 5 years? While AI and machine learning has seen huge success in recent years, there are few limitations of AI algorithms as we see today. Some of the major ones are: Labeled Data: Most of the success of AI has come from supervised learning. Many of the recent supervised deep learning algorithms require huge quantities of labeled data which is expensive to obtain. For example, obtaining huge amounts of clinical trial data for healthcare prediction is very challenging. The good news is that there is some research around building good ML models using sparse data labels. Explainability: Deep learning models are essentially a “black box” where you don’t know what factor(s) led to the prediction. For some applications like money lending, disease diagnosis, fraud detection etc. the explanations of predictions become very important. Currently, we see some nascent work in this direction with LIME and SHAP libraries. Generalization: In the current state of AI, we build one model for each application. We still don’t have good generality of models from one task to another. Generalization, if solved, can lead us to truly Artificial General Intelligence (AGI). Thankfully approaches like transfer learning and meta-learning are trying to solve this challenge. Bias, Fairness, and Ethics: An output of the machine learning model is heavily based on the input training data. Many a time, training data can have biases towards particular ethnicities, classes, religions, etc. We need more solutions in this direction to build trust in AI algorithms. Overall, I feel, AI is becoming mainstream and in the next 5 years we will see many traditional industries adopt AI to solve critical business problems and achieve more automation. At the same time, tooling for AI will keep on improving which will also help in its adoption. What is your advice for those developing machine learning projects on TensorFlow? Building projects with new techniques and technologies is a hard process. It requires patience, dealing with failures and hard work. For that reason, it is very important to pick up a project that you are passionate about. This way, you will continue building even if you are stuck somewhere. The selection of the right projects is by far the most important criterion in the project-based learning method.  About the Author Ankit currently works as a Senior Research Scientist at Uber AI Labs, the machine learning research arm of Uber. His work primarily involves the application of Deep Learning methods to a variety of Uber’s problems ranging from food recommendation system, forecasting to self-driving cars.  Previously, he has worked in a variety of data science roles at Bank of America, Facebook and other startups. Additionally, he has been a featured speaker in many of the top AI conferences and universities across the US, including UC Berkeley, OReilly AI conference etc. He completed his MS from UC Berkeley and a BS from IIT Bombay (India). You can find him on Linkedin, Twitter, and GitHub. About the Book With the help of this book, TensorFlow Machine Learning Projects you’ll not only learn how to build advanced projects using different datasets but also be able to tackle common challenges using a range of libraries from the TensorFlow ecosystem. To start with, you’ll get to grips with using TensorFlow for machine learning projects; you’ll explore a wide range of projects using TensorForest and TensorBoard for detecting exoplanets, TensorFlow.js for sentiment analysis, and TensorFlow Lite for digit classification. As you make your way through the book, you’ll build projects in various real-world domains. By the end of this book, you’ll have gained the required expertise to build full-fledged machine learning projects at work.  
Read more
  • 0
  • 0
  • 25113

article-image-scribus-importing-images
Packt
06 Apr 2011
13 min read
Save for later

Scribus: Importing Images

Packt
06 Apr 2011
13 min read
  Scribus 1.3.5: Beginner's Guide Create optimum page layouts for your documents using productive tools of Scribus.  Importing and exporting: The concepts To begin with, remember that there are two kinds of graphics you can add to your layout. You can have photos, generally taken from a digital camera, downloaded, or bought on some website. Photos will generally be stored in JPEG files, but you can also find PNG, TIFF, or many other file formats. The second kind of graphics is vector drawings such as logos and maps. They are computer-made drawings and are stored as EPS or SVG files. You will certainly need to work with both in most of your documents. The previous image shows the comparison between a photo on the left-hand side, and the same photo traced in vectors on the right-hand side. In the middle, see how the details of the photo are made up of square pixels that are sometimes difficult to handle to get good printing results. The first difference is that with photos, you will need to place them in a frame. There are some tips to automatically add the frame, but anyway, a frame will be needed. Vector drawings are imported as shapes and can be manipulated in Scribus like any other Scribus object. The second difference is that when working with photos, you will have to import them within Scribus. The term "import" is precise here. Most text processors don't import but insert images. In the case of "insert", the images are definitely stored in your document. So you can send this document to anyone via e-mail or can store it on any external storage device without caring about whether these images will still be present later. Scribus does it differently: it adds to the frame a reference to the imported file's own storage position. Scribus will look for the file each time it needs it, and if the file is not where it should be, problems may arise. Basically, the steps you go through while performing a DTP import and while performing a text processor insert are the same, but the global workflows are different because the professional needs to which they refer to are different. All the communication software, layout programs, website building software, as well as video editing software perform the same steps. Once you're used to it, it's really handy: it results in lighter and more adaptable files. However, while teaching Scribus, I have many times received mails from students who have trouble with the images in their documents, just because they didn't take enough care of this little difference. Remembering it will really help you to work peacefully. DTP versus text processors again Here we are talking about the default behavior of the software. As text processors can now work with frames, they often import the image as a link. Scribus itself should be able to embed pictures in its next release. Will the difference between these two pieces software one day disappear? The next difference is about the color space of the pictures. Cameras use RAW or JPEG (which are RGB documents). Offset printers, which everyone usually refers to, are based on the CMYK (Cyan, Magenta, Yellow, and Black) model. Traditionally, most proprietary packages ask the users to convert RGB files into CMYK. With Scribus there is no need to do so. You can import your graphics as RGB and Scribus will export them all as desired at the end of the process (which most of the time is exporting to PDF). So, if you use GIMP, you shouldn't be embarrassed by its default lack of CMYK support. CMYK in GIMP If you really want to import CMYK photos into Scribus, you can use the GIMP Separate plugin to convert an RGB document to a CMYK. Batch converting can be easily done using the convert utility of ImageMagick. Consider using the new CMYKTool software too, which gives some nice color test tools and options, including ink coverage. That said, we now have to see how it really works in Scribus. Importing photos To import photos, you simply have to: Create an Image Frame with the Insert Image Frame tool (I) or with a shape or polygon converted to an Image Frame. Go to File | Import | Get Image or use Ctrl + D. This is the most common way to do it. On some systems, you can even drag the picture from your folder into the frame. The thing that you should never do is copy the photo (wherever it is) and paste it into your frame. When doing this, Scribus has no knowledge of the initial picture's storage location (because the copied image is placed in a special buffer shared among the applications), and that's what it needs. There are other ways to import an image. You can use: The DirectImageImport script The Picture Browser on some very new and experimental Scribus versions The DirectImageImport script is placed in the Script | Scribus Scripts menu. It will display a window waiting for you to specify which image to import. Once you validate, Scribus automatically creates the frame and places the image within. Some people find it useful because it seems that you can save one step this way. But, in fact, the frame size is a default based on the page width, so you will have to customize it. You might have to draw it directly with a good size and it should be as easy. But anyway, you'll choose the one you prefer. Always remember that each image needs a frame. The image might be smaller or bigger than the frame. In the first case it won't fill it, and in the second case it will appear cropped. You can adapt the frame size as you want; the photo won't change at all. However, one thing you will certainly want to do is move the picture within the frame to select a better area. To do this, double-click on the frame to go into the Content Editing mode; you can now drag the picture inside to place it more precisely. The contextual menu of the Image Frame will give you two menus that will help to make the size of the image the same as that of the frame: Adjust Image to Frame will extend or reduce the photo so that it fills the frame but keeps the photo's height to width ratio Adjust Frame to Image will modify the frame size so that it fits the image In both cases, once applied, you can change the size of the frame as you need to and the photo will automatically be scaled to fit the frame size. Using such options is really interesting because it is fast and easy, but it is not without some risk. When a frame shape is changed after the image is imported, the image itself doesn't change. For example, if you use any of the skew buttons in the Node window (Edit button of the Shape tab of PP) you'll see that the frame will be skewed but not the picture itself. Image changes have to be made in an image manipulation program. Relinking photos Once your photos have been imported and they fit perfectly in your page content, you may wish to stop for a day or show this to some other people. It's always a good idea to make your decisions later, as you'll see more weaknesses a few days later. It is generally at this point that problems appear. If you send your Scribus .sla document for proofreading, the photos won't be displayed on the proofreader's screen as they are not embedded into the file. The same issue arises if you decide to move or rename some folders or photos on your own computer. Since Scribus just saves the location of your files and loads them when necessary, it won't be able to find them anymore. In this case, don't worry at all. Nothing is lost; it is just that Scribus has no idea of what you have done to your file—you will just need to tell it. At this point, you have two possibilities: Import the photos that have disappeared, again, so that Scribus can save their new location. Go to Extra | Manage Images. Show where the photos have gone by choosing the missing photo (shown with red diagonals crossing the image display area) and clicking on the Search button just below it. When relinking, you can link to another image if you wish; these images just need to have the same name to make Scribus guess it's the good one. So it will be very important that you give unique names to your photos. You can define some naming rules, or keep the camera automatic numbering. If you need to send your document to someone, you can either send a PDF that can't be modified, just annotated, or the whole Scribus document along with the photos. The best thing for this is not to compress the photo folder. It won't make the link into Scribus for they are absolute and the folder path name will differ on your reader's computer. They would need to relink every picture and that might take long for some documents. The best technique is to use the built-in File } Collect for Output window. It will ask you to create a new directory in which all the files needed for that document will be copied with relative paths, so that it will work everywhere. Compress this folder and send it. Time for action – creating a postcard As an example, let's create a postcard. It's an easy document that won't need many features and, therefore, we can concentrate on our image issues. We'll go through the dos and don'ts to let you experiment with the problems that you might have with a bad workflow. Let's create a two page, A6 landscape document with a small margin guide of 6mm. After creating the page, use Page | Manage Guides and add one vertical guide with a gap of 5mm and one horizontal guide with a gap of 20mm. In the first frame, import (File | Import | Get Image) the pict8-1.psd file. Notice that it is in a folder called Photos, with some other files that we will use in this document. When you're in the file selector, Scribus shows some file information: you can see the size in pixels (1552x2592), the resolution (72x72), and the colorspace (RGB). This image will look very big in the frame. Right-click and choose Adjust Image to Frame. It fits the height but not the width. Also we'd want the bottom of the flower image to be at the bottom of the frame. Open the Properties Palette and in the Image tab, select the Free Scaling option and change the X-scale and Y-scale up to 12 or a value close to it. Now it might be better. In the top right-hand side frame, import pict8-2.jpg and set it in the best way you can using the same procedure. Double-click on the frame and drag the picture to find the best placement. Then in the last frame of the first page, import the pict8-3.png file. You can add frame text and type something inside it, such as "Utopia through centuries", and set it nicely. On the second page we'll use one horizontal column without gap and one vertical column with a 5mm gap. On the right-hand part of the horizontal guide (use Page | Snap to Guides if you want to be sure) draw an horizontal line from one vertical to another bordering that one. Keep this line selected and go to Item | Multiple Duplicate. In the window, choose 4 copies, leave the next option selected, and define a 0.4 inch vertical gap. Here it is necessary to write the address. At the bottom left-hand corner of the same page, draw a new Image Frame in which you can import the pict8-4.tif file. In the Image properties, scale the image to the frame size and deselect Proportional so that the image fills the frame perfectly, even if it is distorted. Then in the XYZ tab of the PP, click on the Flip Horizontally button (the blue double arrow icon). We have now set our pages and the photos are placed correctly. Let's create some errors voluntarily. First, let's say we rename the Photos folder to PostcardPhotos. This must be done in your file browser; you cannot do it from Scribus. Go back to Scribus. Things might not have changed for now, but if you right-click on an image and choose Update Image you will see that it won't be displayed anymore. If you want everything to go well again, you can rename your folder again or go to Extras | Manage Images. You see there that each Image Frame of the document is listed and that they contain nothing because the photos are not found anymore. For each image selected in this window, you'll have to click on the Search button and specify which directory it has been moved into in the next window. Scribus will display the image found (image with a similar name only). Select one of the listed images and click on Select. Everything should go well now. (Move the mouse over the image to enlarge.) What just happened? After having created our page, we placed our photos inside the frames. By renaming the folder we broke the link between Scribus and the image files. The Manage Images window lets us see what happened. The full paths to the pictures are displayed here. In our example, all the pictures are in the same folder, but you could have imported pictures from several directories. In this case, only those being included in the renamed folder would have disappeared from the Scribus frames. By clicking on Search, we told Scribus what happened to those pictures and that those pictures still exist but somewhere else. Notice that if you deleted the photos they will be lost forever. The best advice would be to keep a copy of everything: the original as well as the modified photos. Notice that if your images are stored on an external device, it should be plugged and mounted. In fact, renaming the folder is not the only reason why an image might disappear. It happens when: We rename the image folder We rename any parent folder of the image folder We rename the picture We delete the picture We delete the whole folder containing the picture Giving the documents and the pictures to someone else by an e-mail or an USB-key will certainly be similar to the second case. In the first three cases, using the Manage Images window will help you find the images (it's better to know where they are). In the last two cases, you should be ready to look for new pictures and import them into the frames.  
Read more
  • 0
  • 1
  • 25097

article-image-aws-machine-learning-learning-aws-cli-to-execute-a-simple-amazon-ml-workflow-tutorial
Melisha Dsouza
13 Sep 2018
15 min read
Save for later

AWS machine learning: Learning AWS CLI to execute a simple Amazon ML workflow [Tutorial]

Melisha Dsouza
13 Sep 2018
15 min read
Using the AWS web interface to manage and run your projects is time-consuming. We will, therefore, start running our projects via the command line with the AWS Command Line Interface (AWS CLI). With just one tool to download and configure, multiple  AWS services can be controlled from the command line and they can be automated through scripts. The code files for this article are available on Github. This article is an excerpt from a book written by Alexis Perrier titled Effective Amazon Machine Learning. Getting started and setting up Creating a performing predictive model from raw data requires many trials and errors, much back and forth. Creating new features, cleaning up data, and trying out new parameters for the model are needed to ensure the robustness of the model. There is a constant back and forth between the data, the models, and the evaluations. Scripting this workflow either via the AWS CLI will give us the ability to speed up the create, test, select loop. Installing AWS CLI In order to set up your CLI credentials, you need your access key ID and your secret access key.  You can simply create them from the IAM console (https://console.aws.amazon.com/iam). Navigate to Users, select your IAM user name and click on the Security credentials tab. Choose Create Access Key and download the CSV file. Store the keys in a secure location. We will need the key in a few minutes to set up AWS CLI. But first, we need to install AWS CLI. Docker environment – This tutorial will help you use the AWS CLI within a docker container: https://blog.flowlog-stats.com/2016/05/03/aws-cli-in-a-docker-container/. A docker image for running the AWS CLI is available at https://hub.docker.com/r/fstab/aws-cli/. There is no need to rewrite the AWS documentation on how to install the AWS CLI. It is complete and up to date, and available at http://docs.aws.amazon.com/cli/latest/userguide/installing.html. In a nutshell, installing the CLI requires you to have Python and pip already installed. Then, run the following: $ pip install --upgrade --user awscli Add AWS to your $PATH: $ export PATH=~/.local/bin:$PATH Reload the bash configuration file (this is for OSX): $ source ~/.bash_profile Check that everything works with the following command: $ aws --version You should see something similar to the following output: $ aws-cli/1.11.47 Python/3.5.2 Darwin/15.6.0 botocore/1.5.10 Once installed, we need to configure the AWS CLI type: $ aws configure Now input the access keys you just created: $ aws configure AWS Access Key ID [None]: ABCDEF_THISISANEXAMPLE AWS Secret Access Key [None]: abcdefghijk_THISISANEXAMPLE Default region name [None]: us-west-2 Default output format [None]: json Choose the region that is closest to you and the format you prefer (JSON, text, or table). JSON is the default format. The AWS configure command creates two files: a config file and a credential file. On OSX, the files are ~/.aws/config and ~/.aws/credentials. You can directly edit these files to change your access or configuration. You will need to create different profiles if you need to access multiple AWS accounts. You can do so via the AWS configure command: $ aws configure --profile user2 You can also do so directly in the config and credential files: ~/.aws/config [default] output = json region = us-east-1 [profile user2] output = text region = us-west-2 You can edit Credential file as follows: ~/.aws/credentials [default] aws_secret_access_key = ABCDEF_THISISANEXAMPLE aws_access_key_id = abcdefghijk_THISISANEXAMPLE [user2] aws_access_key_id = ABCDEF_ANOTHERKEY aws_secret_access_key = abcdefghijk_ANOTHERKEY Refer to the AWS CLI setup page for more in-depth information: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html Picking up CLI syntax The overall format of any AWS CLI command is as follows: $ aws <service> [options] <command> <subcommand> [parameters] Here the terms are stated as: <service>: Is the name of the service you are managing: S3, machine learning, and EC2 [options] : Allows you to set the region, the profile, and the output of the command <command> <subcommand>: Is the actual command you want to execute  [parameters] : Are the parameters for these commands A simple example will help you understand the syntax better. To list the content of an S3 bucket named aml.packt, the command is as follows: $ aws s3 ls aml.packt Here, s3 is the service, ls is the command, and aml.packt is the parameter. The aws help command will output a list of all available services. There are many more examples and explanations on the AWS documentation available at http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-using.html. Passing parameters using JSON files For some services and commands, the list of parameters can become long and difficult to check and maintain. For instance, in order to create an Amazon ML model via the CLI, you need to specify at least seven different elements: the Model ID, name, type, the model's parameters, the ID of the training data source, and the recipe name and URI (aws machinelearning create-ml-model help ). When possible, we will use the CLI ability to read parameters from a JSON file instead of specifying them in the command line. AWS CLI also offers a way to generate a JSON template, which you can then use with the right parameters. To generate that JSON parameter file model (the JSON skeleton), simply add --generate-cli-skeleton after the command name. For instance, to generate the JSON skeleton for the create model command of the machine learning service, write the following: $ aws machinelearning create-ml-model --generate-cli-skeleton This will give the following output: { "MLModelId": "", "MLModelName": "", "MLModelType": "", "Parameters": { "KeyName": "" }, "TrainingDataSourceId": "", "Recipe": "", "RecipeUri": "" } You can then configure this to your liking. To have the skeleton command generate a JSON file and not simply output the skeleton in the terminal, add > filename.json: $ aws machinelearning create-ml-model --generate-cli-skeleton > filename.json This will create a filename.json file with the JSON template. Once all the required parameters are specified, you create the model with the command (assuming the filename.json is in the current folder): $ aws machinelearning create-ml-model file://filename.json Before we dive further into the machine learning workflow via the CLI, we need to introduce the dataset we will be using in this chapter. Introducing the Ames Housing dataset We will use the Ames Housing dataset that was compiled by Dean De Cock for use in data science education. It is a great alternative to the popular but older Boston Housing dataset. The Ames Housing dataset is used in the Advanced Regression Techniques challenge on the Kaggle website: https://www.kaggle.com/c/house-prices-advanced-regression-techniques/. The original version of the dataset is available: http://www.amstat.org/publications/jse/v19n3/decock/AmesHousing.xls and in the GitHub repository for this chapter. For more information on the genesis of this dataset and an in-depth explanation of the different variables, read the paper by Dean De Cock available in PDF at https://ww2.amstat.org/publications/jse/v19n3/decock.pdf. We will start by splitting the dataset into a train and a validate set and build a model on the train set. Both train and validate sets are available in the GitHub repository as ames_housing_training.csv and ames_housing_validate.csv. The entire dataset is in the ames_housing.csv file. Splitting the dataset with shell commands We will use shell commands to shuffle, split, and create training and validation subsets of the Ames Housing dataset: First, extract the first line into a separate file, ames_housing_header.csv and remove it from the original file: $ head -n 1 ames_housing.csv > ames_housing_header.csv We just tail all the lines after the first one into the same file: $ tail -n +2 ames_housing.csv > ames_housing_nohead.csv Then randomly sort the rows into a temporary file. (gshuf is the OSX equivalent of the Linux shuf shell command. It can be installed via brew install coreutils): $ gshuf ames_housing_nohead.csv -o ames_housing_nohead.csv Extract the first 2,050 rows as the training file and the last 880 rows as the validation file: $ head -n 2050 ames_housing_nohead.csv > ames_housing_training.csv $ tail -n 880 ames_housing_nohead.csv > ames_housing_validate.csv Finally, add back the header into both training and validation files: $ cat ames_housing_header.csv ames_housing_training.csv > tmp.csv $ mv tmp.csv ames_housing_training.csv $ cat ames_housing_header.csv ames_housing_validate.csv > tmp.csv $ mv tmp.csv ames_housing_validate.csv A simple project using AWS CLI We are now ready to execute a simple Amazon ML workflow using the CLI. This includes the following: Uploading files on S3 Creating a datasource and the recipe Creating a model Creating an evaluation Prediction batch and real time Let's start by uploading the training and validation files to S3. In the following lines, replace the bucket name aml.packt with your own bucket name. To upload the files to the S3 location s3://aml.packt/data/ch8/, run the following command lines: $ aws s3 cp ./ames_housing_training.csv s3://aml.packt/data/ch8/ upload: ./ames_housing_training.csv to s3://aml.packt/data/ch8/ames_housing_training.csv $ aws s3 cp ./ames_housing_validate.csv s3://aml.packt/data/ch8/ upload: ./ames_housing_validate.csv to s3://aml.packt/data/ch8/ames_housing_validate.csv An overview of Amazon ML CLI commands That's it for the S3 part. Now let's explore the CLI for Amazon's machine learning service. All Amazon ML CLI commands are available at http://docs.aws.amazon.com/cli/latest/reference/machinelearning/. There are 30 commands, which can be grouped by object and action. You can perform the following: create : creates the object describe: searches objects given some parameters (location, dates, names, and so on) get: given an object ID, returns information update: given an object ID, updates the object delete: deletes an object These can be performed on the following elements: datasource create-data-source-from-rds create-data-source-from-redshift create-data-source-from-s3 describe-data-sources delete-data-source get-data-source update-data-source ml-model create-ml-model describe-ml-models get-ml-model delete-ml-model update-ml-model evaluation create-evaluation describe-evaluations get-evaluation delete-evaluation update-evaluation batch prediction create-batch-prediction describe-batch-predictions get-batch-prediction delete-batch-prediction update-batch-prediction real-time end point create-realtime-endpoint delete-realtime-endpoint predict You can also handle tags and set waiting times. Note that the AWS CLI gives you the ability to create datasources from S3, Redshift, and RDS, while the web interface only allowed datasources from S3 and Redshift. Creating the datasource We will start by creating the datasource. Let's first see what parameters are needed by generating the following skeleton: $ aws machinelearning create-data-source-from-s3 --generate-cli-skeleton This generates the following JSON object: { "DataSourceId": "", "DataSourceName": "", "DataSpec": { "DataLocationS3": "", "DataRearrangement": "", "DataSchema": "", "DataSchemaLocationS3": "" }, "ComputeStatistics": true } The different parameters are mostly self-explanatory and further information can be found on the AWS documentation at http://docs.aws.amazon.com/cli/latest/reference/machinelearning/create-data-source-from-s3.html. A word on the schema: when creating a datasource from the web interface, you have the possibility to use a wizard, to be guided through the creation of the schema. The wizard facilitates the process by guessing the type of the variables, thus making available a default schema that you can modify. There is no default schema available via the AWS CLI. You have to define the entire schema yourself, either in a JSON format in the DataSchema field or by uploading a schema file to S3 and specifying its location, in the DataSchemaLocationS3 field. Since our dataset has many variables (79), we cheated and used the wizard to create a default schema that we uploaded to S3. Throughout the rest of the chapter, we will specify the schema location not its JSON definition. In this example, we will create the following datasource parameter file, dsrc_ames_housing_001.json: { "DataSourceId": "ch8_ames_housing_001", "DataSourceName": "[DS] Ames Housing 001", "DataSpec": { "DataLocationS3": "s3://aml.packt/data/ch8/ames_housing_training.csv", "DataSchemaLocationS3": "s3://aml.packt/data/ch8/ames_housing.csv.schema" }, "ComputeStatistics": true } For the validation subset (save to dsrc_ames_housing_002.json): { "DataSourceId": "ch8_ames_housing_002", "DataSourceName": "[DS] Ames Housing 002", "DataSpec": { "DataLocationS3": "s3://aml.packt/data/ch8/ames_housing_validate.csv", "DataSchemaLocationS3": "s3://aml.packt/data/ch8/ames_housing.csv.schema" }, "ComputeStatistics": true } Since we have already split our data into a training and a validation set, there's no need to specify the data DataRearrangement field. Alternatively, we could also have avoided splitting our dataset and specified the following DataRearrangement on the original dataset, assuming it had been already shuffled: (save to dsrc_ames_housing_003.json): { "DataSourceId": "ch8_ames_housing_003", "DataSourceName": "[DS] Ames Housing training 003", "DataSpec": { "DataLocationS3": "s3://aml.packt/data/ch8/ames_housing_shuffled.csv", "DataRearrangement": "{"splitting":{"percentBegin":0,"percentEnd":70}}", "DataSchemaLocationS3": "s3://aml.packt/data/ch8/ames_housing.csv.schema" }, "ComputeStatistics": true } For the validation set (save to dsrc_ames_housing_004.json): { "DataSourceId": "ch8_ames_housing_004", "DataSourceName": "[DS] Ames Housing validation 004", "DataSpec": { "DataLocationS3": "s3://aml.packt/data/ch8/ames_housing_shuffled.csv", "DataRearrangement": "{"splitting":{"percentBegin":70,"percentEnd":100}}", }, "ComputeStatistics": true } Here, the ames_housing.csv file has previously been shuffled using the gshuf command line and uploaded to S3: $ gshuf ames_housing_nohead.csv -o ames_housing_nohead.csv $ cat ames_housing_header.csv ames_housing_nohead.csv > tmp.csv $ mv tmp.csv ames_housing_shuffled.csv $ aws s3 cp ./ames_housing_shuffled.csv s3://aml.packt/data/ch8/ Note that we don't need to create these four datasources; these are just examples of alternative ways to create datasources. We then create these datasources by running the following: $ aws machinelearning create-data-source-from-s3 --cli-input-json file://dsrc_ames_housing_001.json We can check whether the datasource creation is pending: In return, we get the datasoure ID we had specified: { "DataSourceId": "ch8_ames_housing_001" } We can then obtain information on that datasource with the following: $ aws machinelearning get-data-source --data-source-id ch8_ames_housing_001 This returns the following: { "Status": "COMPLETED", "NumberOfFiles": 1, "CreatedByIamUser": "arn:aws:iam::178277xxxxxxx:user/alexperrier", "LastUpdatedAt": 1486834110.483, "DataLocationS3": "s3://aml.packt/data/ch8/ames_housing_training.csv", "ComputeStatistics": true, "StartedAt": 1486833867.707, "LogUri": "https://eml-prod-emr.s3.amazonaws.com/178277513911-ds-ch8_ames_housing_001/.....", "DataSourceId": "ch8_ames_housing_001", "CreatedAt": 1486030865.965, "ComputeTime": 880000, "DataSizeInBytes": 648150, "FinishedAt": 1486834110.483, "Name": "[DS] Ames Housing 001" } Note that we have access to the operation log URI, which could be useful to analyze the model training later on. Creating the model Creating the model with the create-ml-model command follows the same steps: Generate the skeleton with the following: $ aws machinelearning create-ml-model --generate-cli-skeleton > mdl_ames_housing_001.json Write the configuration file: { "MLModelId": "ch8_ames_housing_001", "MLModelName": "[MDL] Ames Housing 001", "MLModelType": "REGRESSION", "Parameters": { "sgd.shuffleType": "auto", "sgd.l2RegularizationAmount": "1.0E-06", "sgd.maxPasses": "100" }, "TrainingDataSourceId": "ch8_ames_housing_001", "RecipeUri": "s3://aml.packt/data/ch8 /recipe_ames_housing_001.json" } Note the parameters of the algorithm. Here, we used mild L2 regularization and 100 passes. Launch the model creation with the following: $ aws machinelearning create-ml-model --cli-input-json file://mdl_ames_housing_001.json The model ID is returned: { "MLModelId": "ch8_ames_housing_001" } This get-ml-model command gives you a status update on the operation as well as the URL to the log. $ aws machinelearning get-ml-model --ml-model-id ch8_ames_housing_001 The watch command allows you to repeat a shell command every n seconds. To get the status of the model creation every 10s, just write the following: $ watch -n 10 aws machinelearning get-ml-model --ml-model-id ch8_ames_housing_001 The output of the get-ml-model will be refreshed every 10s until you kill it. It is not possible to create the default recipe via the AWS CLI commands. You can always define a blank recipe that would not carry out any transformation on the data. However, the default recipe has been shown to be positively impacting the model performance. To obtain this default recipe, we created it via the web interface, copied it into a file that we uploaded to S3. The resulting file recipe_ames_housing_001.json is available in our GitHub repository. Its content is quite long as the dataset has 79 variables and is not reproduced here for brevity purposes. Evaluating our model with create-evaluation Our model is now trained and we would like to evaluate it on the evaluation subset. For that, we will use the create-evaluation CLI command: Generate the skeleton: $ aws machinelearning create-evaluation --generate-cli-skeleton > eval_ames_housing_001.json Configure the parameter file: { "EvaluationId": "ch8_ames_housing_001", "EvaluationName": "[EVL] Ames Housing 001", "MLModelId": "ch8_ames_housing_001", "EvaluationDataSourceId": "ch8_ames_housing_002" } Launch the evaluation creation: $ aws machinelearning create-evaluation --cli-input-json file://eval_ames_housing_001.json Get the evaluation information: $ aws machinelearning get-evaluation --evaluation-id ch8_ames_housing_001 From that output, we get the performance of the model in the form of the RMSE: "PerformanceMetrics": { "Properties": { "RegressionRMSE": "29853.250469108018" } } The value may seem big, but it is relative to the range of the salePrice variable for the houses, which has a mean of 181300.0 and std of 79886.7. So an RMSE of 29853.2 is a decent score. You don't have to wait for the datasource creation to be completed in order to launch the model training. Amazon ML will simply wait for the parent operation to conclude before launching the dependent one. This makes chaining operations possible. At this point, we have a trained and evaluated model. In this tutorial, we have successfully seen the detailed steps on how to get started with CLI and we have also implemented a  simple project to get comfortable with the same. To understand how to leverage Amazon's powerful platform for your predictive analytics needs,  check out this book Effective Amazon Machine Learning Part1. Learning AWS CLI Part2. ChatOps with Slack and AWS CLI Automate tasks using Azure PowerShell and Azure CLI [Tutorial]
Read more
  • 0
  • 0
  • 25076

article-image-use-sqlite-ionic-store-data
Oli Huggins
13 Jun 2016
10 min read
Save for later

How to use SQLite with Ionic to store data?

Oli Huggins
13 Jun 2016
10 min read
Hybrid Mobile apps have a challenging task of being as performant as native apps, but I always tell other developers that it depends on not the technology but how we code. The Ionic Framework is a popular hybrid app development library, which uses optimal design patterns to create awe-inspiring mobile experiences. We cannot exactly use web design patterns to create hybrid mobile apps. The task of storing data locally on a device is one such capability, which can make or break the performance of your app. In a web app, we may use localStorage to store data but mobile apps require much more data to be stored and swift access. Localstorage is synchronous, so it acts slow in accessing the data. Also, web developers who have experience of coding in a backend language such as C#, PHP, or Java would find it more convenient to access data using SQL queries than using object-based DB. SQLite is a lightweight embedded relational DBMS used in web browsers and web views for hybrid mobile apps. It is similar to the HTML5 WebSQL API and is asynchronous in nature, so it does not block the DOM or any other JS code. Ionic apps can leverage this tool using an open source Cordova Plugin by Chris Brody (@brodybits). We can use this plugin directly or use it with the ngCordova library by the Ionic team, which abstracts Cordova plugin calls into AngularJS-based services. We will create an Ionic app in this blog post to create Trackers to track any information by storing it at any point in time. We can use this data to analyze the information and draw it on charts. We will be using the ‘cordova-sqlite-ext’ plugin and the ngCordova library. We will start by creating a new Ionic app with a blank starter template using the Ionic CLI command, ‘$ ionic start sqlite-sample blank’. We should also add appropriate platforms for which we want to build our app. The command to add a specific platform is ‘$ ionic platform add <platform_name>’. Since we will be using ngCordova to manage SQLite plugin from the Ionic app, we have to now install ngCordova to our app. Run the following bower command to download ngCordova dependencies to the local bower ‘lib’ folder: bower install ngCordova We need to inject the JS file using a script tag in our index.html: <script src=“lib/ngCordova/dist/ng-cordova.js"></script> Also, we need to include the ngCordova module as a dependency in our app.js main module declaration: angular.module('starter', [‘ionic’,’ngCordova']) Now, we need to add the Cordova plugin for SQLite using the CLI command: cordova plugin add https://github.com/litehelpers/Cordova-sqlite-storage.git Since we will be using the $cordovaSQLite service of ngCordova only to access this plugin from our Ionic app, we need not inject any other plugin. We will have the following two views in our Ionic app: Trackers list: This list shows all the trackers we add to DB Tracker details: This is a view to show list of data entries we make for a specific tracker We would need to create the routes by registering the states for the two views we want to create. We need to add the following config block code for our ‘starter’ module in the app.js file only: .config(function($stateProvider,$urlRouterProvider){ $urlRouterProvider.otherwise('/') $stateProvider.state('home', { url: '/', controller:'TrackersListCtrl', templateUrl: 'js/trackers-list/template.html' }); $stateProvider.state('tracker', { url: '/tracker/:id', controller:'TrackerDetailsCtrl', templateUrl: 'js/tracker-details/template.html' }) }); Both views would have similar functionality, but will display different entities. Our view will display a list of trackers from the SQLite DB table and also provide a feature to add a new tracker or delete an existing one. Create a new folder named trackers-list where we can store our controller and template for the view. We will also abstract our code to access the SQLite DB into an Ionic factory. We will implement the following methods: initDB: This will initialize or create a table for this entity if it does not exist getAllTrackers: This will get all trackers list rows from the created table addNewTracker - This is a method to insert a new row for a new tracker into the table deleteTracker - This is a method to delete a specific tracker using its ID getTracker - This will get a specific Tracker from the cached list using an ID to display anywhere We will be injecting the $cordovaSQLite service into our factory to interact with our SQLite DB. We can open an existing DB or create a new DB using the command $cordovaSQLite.openDB(“myApp.db”). We have to store the object reference returned from this method call, so we will store it in a module-level variable called db. We have to pass this object reference to our future $cordovaSQLite service calls. $cordovaSQLite has a handful of methods to provide varying features: openDB: This is a method to establish a connection to the existing DB or create a new DB execute: This is a method to execute a single SQL command query insertCollection: This is a method to insert bulk values nestedExecute: This is a method to run nested queries deleted: This is a method to delete a particular DB We see the usage of openDB and execute the command in this post. In our factory, we will create a standard method runQuery to adhere to DRY(Don’t Repeat Yourself) principles. The code for the runQuery function is as follows: function runQuery(query,dataParams,successCb,errorCb) { $ionicPlatform.ready(function() { $cordovaSQLite.execute(db, query,dataParams).then(function(res) { successCb(res); }, function (err) { errorCb(err); }); }.bind(this)); } In the preceding code, we pass the query as a string, dataParams (dynamic query parameters) as an array, and successCB/errorCB as callback functions. We should always ensure that any Cordova plugin code should be called when the Cordova ready event is already fired, which is ensured by the $ionicPlatform.ready() method. We will then call the execute method of the $cordovaSQLite service passing the ‘db’ object reference, query, and dataParams as arguments. The method returns a promise to which we register callbacks using the ‘.then’ method. We pass the results or error using the success callback or error callback. Now, we will write code for each of the methods to initialize DB, insert a new row, fetch all rows, and then delete a row. initDB Method: function initDB() { db = $cordovaSQLite.openDB("myapp.db"); var query = "CREATE TABLE IF NOT EXISTS trackers_list (id in-teger autoincrement primary key, name string)"; runQuery(query,[],function(res) { console.log("table created "); }, function (err) { console.log(err); }); } In the preceding code, the openDB method is used to establish a connection with an existing DB or create a new DB. Then, we run the query to create a new table called ‘trackers_list’ if it does not exist. We define the columns ID with integer autoincrement primary key properties with the name string. addNewTracker Method: function addNewTracker(name) { var deferred = $q.defer(); var query = "INSERT INTO trackers_list (name) VALUES (?)"; runQuery(query,[name],function(response){ //Success Callback console.log(response); deferred.resolve(response); },function(error){ //Error Callback console.log(error); deferred.reject(error); }); return deferred.promise; } In the preceding code, we take ‘name’ as an argument, which will be passed into the insert query. We write the insert query and add a new row to the trackers_list table where ID will be auto-generated. We pass dynamic query parameters using the ‘?’ character in our query string, which will be replaced by elements in the dataParams array passed as the second argument to the runQuery method. We also use a $q library to return a promise to our factory methods so that controllers can manage asynchronous calls. getAllTrackers Method: This method is the same as the addNewTracker method, only without the name parameter, and it has the following query: var query = "SELECT * from trackers_list”; This method will return a promise, which when resolved will give the response from the $cordovaSQLite method. The response object will have the following structure: { insertId: <specific_id>, rows: {item: function, length: <total_no_of_rows>} rowsAffected: 0 } The response object has properties insertId representing the new ID generated for the row, rowsAffected giving the number of rows affected by the query and rows object with item method property, to which we can pass the index of the row to retrieve it. We will write the following code in the controller to convert the response.rows object into an utterable array of rows to be displayed using the ng-repeat directive: for(var i=0;i<response.rows.length;i++) { $scope.trackersList.push({ id:response.rows.item(i).id, name:response.rows.item(i).name }); } The code in the template to display the list of Trackers would be as follows: <ion-item ui-sref="tracker({id:tracker.id})" class="item-icon-right" ng-repeat="tracker in trackersList track by $index"> {{tracker.name}} <ion-delete-button class="ion-minus-circled" ng-click=“deleteTracker($index,tracker.id)"> </ion-delete-button> <i class="icon ion-chevron-right”></i> </ion-item> deleteTracker Method: function deleteTracker(id) { var deferred = $q.defer(); var query = "DELETE FROM trackers_list WHERE id = ?"; runQuery(query,[id],function(response){ … [Same Code as addNewTrackerMethod] } The delete tracker method has the same code as the addNewTracker method, where the only change is in the query and the argument passed. We pass ‘id’ as the argument to be used in the WHERE clause of delete query to delete the row with that specific ID. Rest of the Ionic App Code: The rest of the app code has not been discussed in this post because we have already discussed the code that is intended for integration with SQLite. You can implement your own version of this app or even use this sample code for any other use case. The trackers details view will be implemented in the same way to store data into the tracker_entries table with a foreign key, tracker_id, used for this table. It will also use this ID in the SELECT query to fetch entries for a specific tracker on its detail view. The GitHub link for the exact functioning code for complete app developed during this tutorial. About the author Rahat Khanna is a techno nerd experienced in developing web and mobile apps for many international MNCs and start-ups. He has completed his bachelors in technology with computer science and engineering as specialization. During the last 7 years, he has worked for a multinational IT service company and ran his own entrepreneurial venture in his early twenties. He has worked on projects ranging from static HTML websites to scalable web applications and engaging mobile apps. Along with his current job as a senior UI developer at Flipkart, a billion dollar e-commerce firm, he now blogs on the latest technology frameworks on sites such as www.airpair.com, appsonmob.com, and so on, and delivers talks at community events. He has been helping individual developers and start-ups in their Ionic projects to deliver amazing mobile apps.
Read more
  • 0
  • 0
  • 25068
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-whats-new-in-usb4-transfer-speeds-of-upto-40gb-second-with-thunderbolt-3-and-more
Sugandha Lahoti
04 Sep 2019
3 min read
Save for later

What’s new in USB4? Transfer speeds of upto 40GB/second with Thunderbolt 3 and more

Sugandha Lahoti
04 Sep 2019
3 min read
USB4 technical specifications were published yesterday. Along with removing space in stylization (USB4 instead of USB 4), the new version offers double the speed of it’s previous versions. USB4 architecture is based on Intel’s Thunderbolt; Intel had provided Thunderbolt 3 to the USB Promoter Group royalty-free earlier this year. Features of USB4 USB4 runs blazingly fast by incorporating Intel's Thunderbolt technology. It allows transfers at the rate of 40 gigabits per second, twice the speed of the latest version of USB 3 and 8 times the speed of the original USB 3 standard. 40Gbps speeds, can example, allow users to do things like connect two 4K monitors at once, or run high-end external GPUs with ease. Key characteristics as specified in the USB4 specification include: Two-lane operation using existing USB Type-C cables and up to 40 Gbps operation over 40 Gbps certified cables Multiple data and display protocols to efficiently share the total available bandwidth over the bus Backward compatibility with USB 3.2, USB 2.0 and Thunderbolt 3 Another good news is that USB4 will use the same USB-C connector design as USB 3, which means manufacturers will not need to introduce new USB4 ports into their devices. Why USB4 omits a space The change in stylization was to simplify things. In an interview with Tom’s Hardware, USB Promoter Group CEO Brad Saunders said this is to prevent the profusion of products sporting version number badges that could confuse consumers. “We don’t plan to get into a 4.0, 4.1, 4.2 kind of iterative path,” he explained. “We want to keep it as simple as possible. When and if it goes faster, we’ll simply have the faster version of the certification and the brand.” Is Thunderbolt 3 compatibility optional? The specification mentioned that using Thunderbolt 3 support is optional. The published spec states: It’s up to USB4 device makers to support Thunderbolt.  This was a major topic of discussion among people on Social media. https://twitter.com/KevinLozandier/status/1169106844289077248 The USB Implementers Forum released a detailed statement clarifying the issue. "Regarding USB4 specification’s optional support for Thunderbolt 3, USB-IF anticipates PC vendors to broadly support Thunderbolt 3 compatibility in their USB4 solutions given Thunderbolt 3 compatibility is now included in the USB4 specification and therefore royalty free for formal adopters," the USB-IF said in a statement. "That said, Intel still maintains the Thunderbolt 3 branding/certification so consumers can look for the appropriate Thunderbolt 3 logo and brand name to ensure the USB4 product in question has the expected Thunderbolt 3 compatibility. Furthermore, the decision was made not to make Thunderbolt 3 compatibility a USB4 specification requirement as certain manufacturers (e.g. smartphone makers) likely won’t need to add the extra capabilities that come with Thunderbolt 3 compatibility when designing their USB4 products," the statement added. Though the specification is released, it will be some time before USB4 compatible devices hit the market. We can expect to see devices that take advantage of the new version late 2020 or beyond. Read more in Hardware USB-IF launches ‘Type-C Authentication Program’ for better security Apple USB Restricted Mode: Here’s Everything You Need to Know USB 4 will integrate Thunderbolt 3 to increase the speed to 40Gbps
Read more
  • 0
  • 0
  • 25029

article-image-ragdoll-physics
Packt
19 Feb 2016
5 min read
Save for later

Ragdoll Physics

Packt
19 Feb 2016
5 min read
In this article we will learn how to apply Ragdoll physics to a character. (For more resources related to this topic, see here.) Applying Ragdoll physics to a character Action games often make use of Ragdoll physics to simulate the character's body reaction to being unconsciously under the effect of a hit or explosion. In this recipe, we will learn how to set up and activate Ragdoll physics to our character whenever she steps in a landmine object. We will also use the opportunity to reset the character's position and animations a number of seconds after that event has occurred. Getting ready For this recipe, we have prepared a Unity Package named Ragdoll, containing a basic scene that features an animated character and two prefabs, already placed into the scene, named Landmine and Spawnpoint. The package can be found inside the 1362_07_08 folder. How to do it... To apply ragdoll physics to your character, follow these steps: Create a new project and import the Ragdoll Unity Package. Then, from the Project view, open the mecanimPlayground level. You will see the animated book character and two discs: Landmine and Spawnpoint. First, let's set up our Ragdoll. Access the GameObject | 3D Object | Ragdoll... menu and the Ragdoll wizard will pop-up. Assign the transforms as follows:     Root: mixamorig:Hips     Left Hips: mixamorig:LeftUpLeg     Left Knee: mixamorig:LeftLeg     Left Foot: mixamorig:LeftFoot     Right Hips: mixamorig:RightUpLeg     Right Knee: mixamorig:RightLeg     Right Foot: mixamorig:RightFoot     Left Arm: mixamorig:LeftArm     Left Elbow: mixamorig:LeftForeArm     Right Arm: mixamorig:RightArm     Right Elbow: mixamorig:RightForeArm     Middle Spine: mixamorig:Spine1     Head: mixamorig:Head     Total Mass: 20     Strength: 50 Insert image 1362OT_07_45.png From the Project view, create a new C# Script named RagdollCharacter.cs. Open the script and add the following code: using UnityEngine; using System.Collections; public class RagdollCharacter : MonoBehaviour { void Start () { DeactivateRagdoll(); } public void ActivateRagdoll(){ gameObject.GetComponent<CharacterController> ().enabled = false; gameObject.GetComponent<BasicController> ().enabled = false; gameObject.GetComponent<Animator> ().enabled = false; foreach (Rigidbody bone in GetComponentsInChildren<Rigidbody>()) { bone.isKinematic = false; bone.detectCollisions = true; } foreach (Collider col in GetComponentsInChildren<Collider>()) { col.enabled = true; } StartCoroutine (Restore ()); } public void DeactivateRagdoll(){ gameObject.GetComponent<BasicController>().enabled = true; gameObject.GetComponent<Animator>().enabled = true; transform.position = GameObject.Find("Spawnpoint").transform.position; transform.rotation = GameObject.Find("Spawnpoint").transform.rotation; foreach(Rigidbody bone in GetComponentsInChildren<Rigidbody>()){ bone.isKinematic = true; bone.detectCollisions = false; } foreach (CharacterJoint joint in GetComponentsInChildren<CharacterJoint>()) { joint.enableProjection = true; } foreach(Collider col in GetComponentsInChildren<Collider>()){ col.enabled = false; } gameObject.GetComponent<CharacterController>().enabled = true; } IEnumerator Restore(){ yield return new WaitForSeconds(5); DeactivateRagdoll(); } } Save and close the script. Attach the RagdollCharacter.cs script to the book Game Object. Then, select the book character and, from the top of the Inspector view, change its tag to Player. From the Project view, create a new C# Script named Landmine.cs. Open the script and add the following code: using UnityEngine; using System.Collections; public class Landmine : MonoBehaviour { public float range = 2f; public float force = 2f; public float up = 4f; private bool active = true; void OnTriggerEnter ( Collider collision ){ if(collision.gameObject.tag == "Player" && active){ active = false; StartCoroutine(Reactivate()); collision.gameObject.GetComponent<RagdollCharacter>().ActivateRagdoll(); Vector3 explosionPos = transform.position; Collider[] colliders = Physics.OverlapSphere(explosionPos, range); foreach (Collider hit in colliders) { if (hit.GetComponent<Rigidbody>()) hit.GetComponent<Rigidbody>().AddExplosionForce(force, explosionPos, range, up); } } } IEnumerator Reactivate(){ yield return new WaitForSeconds(2); active = true; } } Save and close the script. Attach the script to the Landmine Game Object. Play the scene. Using the WASD keyboard control scheme, direct the character to the Landmine Game Object. Colliding with it will activate the character's Ragdoll physics and apply an explosion force to it. As a result, the character will be thrown away to a considerable distance and will no longer be in the control of its body movements, akin to a ragdoll. How it works... Unity's Ragdoll Wizard assigns, to selected transforms, the components Collider, Rigidbody, and Character Joint. In conjunction, those components make ragdoll physics possible. However, those components must be disabled whenever we want our character to be animated and controlled by the player. In our case, we switch those components on and off using the RagdollCharacter script and its two functions: ActivateRagdoll() and DeactivateRagdoll(), the latter including instructions to re-spawn our character in the appropriate place. For the testing purposes, we have also created the Landmine script, which calls RagdollCharacter script's function named ActivateRagdoll(). It also applies an explosion force to our ragdoll character, throwing it outside the explosion site. There's more... Instead of resetting the character's transform settings, you could have destroyed its gameObject and instantiated a new one over the respawn point using Tags. For more information on this subject, check Unity's documentation at: http://docs.unity3d.com/ScriptReference/GameObject.FindGameObjectsWithTag.html. Summary In this article we learned how to apply Ragdoll physics to a character. We also learned how to setup Ragdoll for the character of the game. To learn more please refer to the following books: Learning Unity 2D Game Development by Examplehttps://www.packtpub.com/game-development/learning-unity-2d-game-development-example. Unity Game Development Blueprintshttps://www.packtpub.com/game-development/unity-game-development-blueprints. Getting Started with Unityhttps://www.packtpub.com/game-development/getting-started-unity. Resources for Article: Further resources on this subject: Using a collider-based system [article] Looking Back, Looking Forward [article] The Vertex Functions [article]
Read more
  • 0
  • 0
  • 24995

article-image-amazon-joins-nsf-funding-fairness-ai-public-outcry-big-tech-ethicswashing
Sugandha Lahoti
27 Mar 2019
5 min read
Save for later

Amazon joins NSF in funding research exploring fairness in AI amidst public outcry over big tech #ethicswashing

Sugandha Lahoti
27 Mar 2019
5 min read
Behind the heels of Stanford’s HCAI Institute ( which, mind you, received public backlash for non-representative faculty makeup). Amazon is collaborating with the National Science Foundation (NSF) to develop systems based on fairness in AI. The company will be investing $10M each in artificial intelligence research grants over a three-year period. The official announcement was made by Prem Natarajan, VP of natural understanding in the Alexa AI group, who wrote in a blog post “With the increasing use of AI in everyday life, fairness in artificial intelligence is a topic of increasing importance across academia, government, and industry. Here at Amazon, the fairness of the machine learning systems we build to support our businesses is critical to establishing and maintaining our customers’ trust.” Per the blog post, Amazon will be collaborating with NSF to build trustworthy AI systems to address modern challenges. They will explore topics of transparency, explainability, accountability, potential adverse biases and effects, mitigation strategies, validation of fairness, and considerations of inclusivity. Proposals will be accepted from March 26 until May 10, to result in new open source tools, publicly available data sets, and publications. The two organizations plan to continue the program with calls for additional proposals in 2020 and 2021. There will be 6 to 9 awards of type Standard Grant or Continuing Grant. The award size will be $750,000 - up to a maximum of $1,250,000 for periods of up to 3 years. The anticipated funding amount is $7,600,000. “We are excited to announce this new collaboration with Amazon to fund research focused on fairness in AI,” said Jim Kurose, NSF's head for Computer and Information Science and Engineering. “This program will support research related to the development and implementation of trustworthy AI systems that incorporate transparency, fairness, and accountability into the design from the beginning.” The insidious nexus of private funding in public research: What does Amazon gain from collab with NSF? Amazon’s foray into fairness system looks more of a publicity stunt than eliminating AI bias. For starters, Amazon said that they will not be making the award determinations for this project. NSF would solely be awarding in accordance with its merit review process. However, Amazon said that Amazon researchers may be involved with the projects as an advisor only at the request of an awardee, or of NSF with the awardee's consent. As advisors, Amazon may host student interns who wish to gain further industry experience, which seems a bit dicey. Amazon will also not participate in the review process or receive proposal information. NSF will only be sharing with Amazon summary-level information that is necessary to evaluate the program, specifically the number of proposal submissions, number of submitting organizations, and numbers rated across various review categories. There was also the question of who exactly is funding since VII.B section of the proposal states: "Individual awards selected for joint funding by NSF and Amazon will be   funded through separate NSF and Amazon funding instruments." https://twitter.com/nniiicc/status/1110335108634951680 https://twitter.com/nniiicc/status/1110335004989521920 Nic Weber, the author of the above tweets and Assistant Professor at UW iSchool, also raises another important question: “Why does Amazon get to put its logo on a national solicitation (for a paltry $7.6 million dollars in basic research) when it profits in the multi-billions off of AI that is demonstrably unfair and harmful.” Twitter was abundant with tweets from those in working tech questioning Amazon’s collaboration. https://twitter.com/mer__edith/status/1110560653872373760 https://twitter.com/patrickshafto/status/1110748217887649793 https://twitter.com/smunson/status/1110657292549029888 https://twitter.com/haldaume3/status/1110697325251448833 Amazon has already been under the fire due to its controversial decisions in the recent past. In June last year, when the US Immigration and Customs Enforcement agency (ICE) began separating migrant children from their parents, Amazon came under fire as one of the tech companies that aided ICE with the software required to do so. Amazon has also faced constant criticisms since the news came that Amazon had sold its facial recognition product Rekognition to a number of law enforcement agencies in the U.S. in the first half of 2018. Amazon is also under backlash after a study by the Massachusetts Institute of Technology in January, found Amazon Rekognition incapable of reliably determining the sex of female and darker-skinned faces in certain scenarios. Amazon is yet to fix this AI-bias anomaly, and yet it has now started a new collaboration with NSF that ironically focusses on building bias-free AI systems. Amazon’s Ring (a smart doorbell company) also came under public scrutiny in January, after it gave access to its employees to watch live footage from cameras of the customers. In other news, yesterday, Google also formed an external AI advisory council to help advance the responsible development of AI. More details here. Amazon won’t be opening its HQ2 in New York due to public protests Amazon admits that facial recognition technology needs to be regulated Amazon’s Ring gave access to its employees to watch live footage of the customers, The Intercept reports
Read more
  • 0
  • 0
  • 24994

article-image-simple-pathfinding-algorithm-maze
Packt
23 Jul 2015
10 min read
Save for later

A Simple Pathfinding Algorithm for a Maze

Packt
23 Jul 2015
10 min read
In this article by Mário Kašuba, author of the book Lua Game Development Cookbook, explains that maze pathfinding can be used effectively in many types of games, such as side-scrolling platform games or top-down, gauntlet-like games. The point is to find the shortest viable path from one point on the map to another. This can be used for moving NPCs and players as well. (For more resources related to this topic, see here.) Getting ready This article will use a simple maze environment to find a path starting at the start point and ending at the exit point. You can either prepare one by yourself or let the computer create one for you. A map will be represented by a 2D-map structure where each cell will consist of a cell type and cell connections. The cell type values are as follows: 0 means a wall 1 means an empty cell 2 means the start point 3 means the exit point Cell connections will use a bitmask value to get information about which cells are connected to the current cell. The following diagram contains cell connection bitmask values with their respective positions: Now, the quite common problem in programming is how to implement an efficient data structure for 2D maps. Usually, this is done either with a relatively large one-dimensional array or with an array of arrays. All these arrays have a specified static size, so map dimensions are fixed. The problem arises when you use a simple 1D array and you need to change the map size during gameplay or the map size should be unlimited. This is where map cell indexing comes into place. Often you can use this formula to compute the cell index from 2D map coordinates: local index = x + y * map_width map[index] = value There's nothing wrong with this approach when the map size is definite. However, changing the map size would invalidate the whole data structure as the map_width variable would change its value. A solution to this is to use indexing that's independent from the map size. This way you can ensure consistent access to all elements even if you resize the 2D map. You can use some kind of hashing algorithm that packs map cell coordinates into one value that can be used as a unique key. Another way to accomplish this is to use the Cantor pairing function, which is defined for two input coordinates:   Index value distribution is shown in the following diagram: The Cantor pairing function ensures that there are no key collisions no matter what coordinates you use. What's more, it can be trivially extended to support three or more input coordinates. To illustrate the usage of the Cantor pairing function for more dimensions, its primitive form will be defined as a function cantor(k1, k2), where k1 and k2 are input coordinates. The pairing function for three dimensions will look like this: local function cantor3D(k1, k2, k3) return cantor(cantor(k1, k2), k3) end Keep in mind that the Cantor pairing function always returns one integer value. With higher number of dimensions, you'll soon get very large values in the results. This may pose a problem because the Lua language can offer 52 bits for integer values. For example, for 2D coordinates (83114015, 11792250) you'll get a value 0x000FFFFFFFFFFFFF that still can fit into 52-bit integer values without rounding errors. The larger coordinates will return inaccurate values and subsequently you'd get key collisions. Value overflow can be avoided by dividing large maps into smaller ones, where each one uses the full address space that Lua numbers can offer. You can use another coordinate to identify submaps. This article will use specialized data structures for a 2D map with the Cantor pairing function for internal cell indexing. You can use the following code to prepare this type of data structure: function map2D(defaultValue) local t = {} -- Cantor pair function local function cantorPair(k1, k2)    return 0.5 * (k1 + k2) * ((k1 + k2) + 1) + k2 end setmetatable(t, {    __index = function(_, k)      if type(k)=="table" then        local i = rawget(t, cantorPair(k[1] or 1, k[2] or 1))        return i or defaultValue      end    end,    __newindex = function(_, k, v)      if type(k)=="table" then        rawset(t, cantorPair(k[1] or 1, k[2] or 1), v)      else        rawset(t, k, v)      end    end, }) return t end The maze generator as well as the pathfinding algorithm will need a stack data structure. How to do it… This section is divided into two parts, where each one solves very similar problems from the perspective of the maze generator and the maze solver. Maze generation You can either load a maze from a file or generate a random one. The following steps will show you how to generate a unique maze. First, you'll need to grab a maze generator library from the GitHub repository with the following command: git clone https://github.com/soulik/maze_generator This maze generator uses the depth-first approach with backtracking. You can use this maze generator in the following steps. First, you'll need to set up maze parameters such as maze size, entry, and exit points. local mazeGenerator = require 'maze' local maze = mazeGenerator { width = 50, height = 25, entry = {x = 2, y = 2}, exit = {x = 30, y = 4}, finishOnExit = false, } The final step is to iteratively generate the maze map until it's finished or a certain step count is reached. The number of steps should always be one order of magnitude greater than the total number of maze cells mainly due to backtracking. Note that it's not necessary for each maze to connect entry and exit points in this case. for i=1,12500 do local result = maze.generate() if result == 1 then    break end end Now you can access each maze cell with the maze.map variable in the following manner: local cell = maze.map[{x, y}] local cellType = cell.type local cellConnections = cell.connections Maze solving This article will show you how to use a modified Trémaux's algorithm, which is based on depth-first search and path marking. This method guarantees finding the path to the exit point if there's one. It relies on using two keys in each step: current position and neighbors. This algorithm will use three state variables—the current position, a set of visited cells, and the current path from the starting point: local currentPosition = {maze.entry.x, maze.entry.y} local visistedCells = map2D(false) local path = stack() The whole maze solving process will be placed into one loop. This algorithm is always finite, so you can use the infinite while loop. -- A placeholder for neighbours function that will be defined later local neighbours   -- testing function for passable cells local cellTestFn = function(cell, position) return (cell.type >= 1) and (not visitedCells[position]) end   -- include starting point into path visitedCells[currentPosition] = true path.push(currentPosition)   while true do local currentCell = maze.map[currentPosition] -- is current cell an exit point? if currentCell and    (currentCell.type == 3 or currentCell.type == 4) then    break else    -- have a look around and find viable cells    local possibleCells = neighbours(currentPosition, cellTestFn)    if #possibleCells > 0 then      -- let's try the first available cell      currentPosition = possibleCells[1]      visitedCells[currentPosition] = true      path.push(currentPosition)    elseif not path.empty() then      -- get back one step      currentPosition = path.pop()    else      -- there's no solution      break    end end end This fairly simple algorithm uses the neighbours function to obtain a list of cells that haven't been visited yet: -- A shorthand for direction coordinates local neighbourLocations = { [0] = {0, 1}, [1] = {1, 0}, [2] = {0, -1}, [3] = {-1, 0}, }   local function neighbours(position0, fn) local neighbours = {} local currentCell = map[position0] if type(currentCell)=='table' then    local connections = currentCell.connections    for i=0,3 do      -- is this cell connected?      if bit.band(connections, 2^i) >= 1 then        local neighbourLocation = neighbourLocations[i]        local position1 = {position0[1] + neighbourLocation[1],         position0[2] + neighbourLocation[2]}        if (position1[1]>=1 and position1[1] <= maze.width and         position1[2]>=1 and position1[2] <= maze.height) then          if type(fn)=="function" then            if fn(map[position1], position1) then              table.insert(neighbours, position1)            end          else            table.insert(neighbours, position1)          end        end      end    end end return neighbours end When this algorithm finishes, a valid path between entry and exit points is stored in the path variable represented by the stack data structure. The path variable will contain an empty stack if there's no solution for the maze. How it works… This pathfinding algorithm uses two main steps. First, it looks around the current maze cell to find cells that are connected to the current maze cell with a passage. This will result in a list of possible cells that haven't been visited yet. In this case, the algorithm will always use the first available cell from this list. Each step is recorded in the stack structure, so in the end, you can reconstruct the whole path from the exit point to the entry point. If there are no maze cells to go, it will head back to the previous cell from the stack. The most important is the neighbours function, which determines where to go from the current point. It uses two input parameters: current position and a cell testing function. It looks around the current cell in four directions in clockwise order: up, right, down, and left. There must be a passage from the current cell to each surrounding cell; otherwise, it'll just skip to the next cell. Another step determines whether the cell is within the rectangular maze region. Finally, the cell is passed into the user-defined testing function, which will determine whether to include the current cell in a list of usable cells. The maze cell testing function consists of a simple Boolean expression. It returns true if the cell has a correct cell type (not a wall) and hasn't been visited yet. A positive result will lead to inclusion of this cell to a list of usable cells. Note that even if this pathfinding algorithm finds a path to the exit point, it doesn't guarantee that this path is the shortest possible. Summary We have learned how pathfinding works in games with a simple maze.With pathfinding algorithm, you can create intelligent game opponents that won't jump into a lava lake at the first opportunity. Resources for Article: Further resources on this subject: Mesh animation [article] Getting into the Store [article] Creating a Direct2D game window class [article]
Read more
  • 0
  • 0
  • 24989
article-image-unit-testing-and-end-end-testing
Packt
09 Mar 2017
4 min read
Save for later

Unit Testing and End-To-End Testing

Packt
09 Mar 2017
4 min read
In this article by Andrea Passaglia, the author of the book Vue.js 2 Cookbook, will cover stubbing external API calls with Sinon.JS. (For more resources related to this topic, see here.) Stub external API calls with Sinon.JS Normally when you do end-to-end testing and integration testing you would have the backend server running and ready to respond to you. I think there are many situations in which this is not desirable. As a frontend developer you take every opportunity to blame the backend guys. Getting started No particular skills are required to complete this recipe except that you should install Jasmine as a dependency. How to do it... First of all let's install some dependencies, starting with Jasmine as we are going to use it to run the whole thing. Also install Sinon.JS and axios before continuing, you just need to add the .js files. We are going to build an application that retrieves a post at the click of a button. In the HTML part write the following: <div id="app"> <button @click="retrieve">Retrieve Post</button> <p v-if="post">{{post}}</p> </div> The JavaScript part instead, is going to look like the following: const vm = new Vue({ el: '#app', data: { post: undefined }, methods: { retrieve () { axios .get('https://jsonplaceholder.typicode.com/posts/1') .then(response => { console.log('setting post') this.post = response.data.body }) } } }) If you launch your application now you should be able to see it working. Now we want to test the application but we don't like to connect to the real server. This would take additional time and it would not be reliable, instead we are going to take a sample, correct response from the server and use it instead. Sinon.JS has the concept of a sandbox. It means that whenever a test start, some dependencies, like axios are overwritten. After each test we can discard the sandbox and everything returns normal. An empty test with Sinon.JS looks like the following (add it after the Vue instance): describe('my app', () => { let sandbox beforeEach(() => sandbox = sinon.sandbox.create()) afterEach(() => sandbox.restore()) }) We want to stub the call to the get function for axios: describe('my app', () => { let sandbox beforeEach(() => sandbox = sinon.sandbox.create()) afterEach(() => sandbox.restore()) it('should save the returned post body', done => { const resolved = new Promise(resolve => r({ data: { body: 'Hello World' } }) ) sandbox.stub(axios, 'get').returns(resolved) ... done() }) }) We are overwriting axios here. We are saying that now the get method should return the resolved promise: describe('my app', () => { let sandbox beforeEach(() => sandbox = sinon.sandbox.create()) afterEach(() => sandbox.restore()) it('should save the returned post body', done => { const promise = new Promise(resolve => resolve({ data: { body: 'Hello World' } }) ) sandbox.stub(axios, 'get').returns(resolved) vm.retrieve() promise.then(() => { expect(vm.post).toEqual('Hello World') done() }) }) }) Since we are returning a promise (and we need to return a promise because the retrieve method is calling then on it) we need to wait until it resolves. We can launch the page and see that it works: How it works... In our case we used the sandbox to stub a method of one of our dependencies. This way the get method of axios never gets fired and we receive an object that is similar to what the backend would give us. Stubbing the API responses will get you isolated from the backend and its quirks. If something goes wrong you won't mind and moreover you can run your test without relying on the backend running and running correctly. There are many libraries and techniques to stub API calls in general, not only related to HTTP. Hopefully this recipe have given you a head start. Summary In this article we covered how we can stub an external API class with Sinon.JS. Resources for Article: Further resources on this subject: Installing and Using Vue.js [article] Introduction to JavaScript [article] JavaScript Execution with Selenium [article]
Read more
  • 0
  • 0
  • 24989

article-image-getting-started-with-gemini-ai
Packt
07 Sep 2023
2 min read
Save for later

Getting Started with Gemini AI

Packt
07 Sep 2023
2 min read
Introduction Gemini AI is a large language model (LLM) being developed by Google DeepMind. It is still under development, but it is expected to be more powerful than ChatGPT, the current state-of-the-art LLM. Gemini AI is being built on the technology and techniques used in AlphaGo, an early AI system developed by DeepMind in 2016. This means that Gemini AI is expected to have strong capabilities in planning and problem-solving. Gemini AI is a powerful tool that has the potential to be used in a wide variety of applications. Some of the potential use cases for Gemini AI include: Chatbots: Gemini AI could be used to create more realistic and engaging chatbots. Virtual assistants: Gemini AI could be used to create virtual assistants that can help users with tasks such as scheduling appointments, making reservations, and finding information. Content generation: Gemini AI could be used to generate creative content such as articles, blog posts, and scripts. Data analysis: Gemini AI could be used to analyze large datasets and identify patterns and trends. Medical diagnosis: Gemini AI could be used to assist doctors in diagnosing diseases. Financial trading: Gemini AI could be used to make trading decisions. How Gemini AI works Gemini AI is a neural network that has been trained on a massive dataset of text and code. This dataset includes books, articles, code repositories, and other forms of text. The neural network is able to learn the patterns and relationships between words and phrases in this dataset. This allows Gemini AI to generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way. How to use Gemini AI Gemini AI is not yet available to the public, but it is expected to be released in the future. When it is released, it will likely be available through a cloud-based API. This means that developers will be able to use Gemini AI in their own applications. To use Gemini AI, developers will need to first create an account and obtain an API key. Once they have an API key, they can use it to call the Gemini AI API. The API will allow them to interact with Gemini AI and use its capabilities. Here are some steps on how to install or get started with Gemini AI: Go to the Gemini AI website and create an account: Once you have created an account, you will be given an API key. Install the Gemini AI client library for your programming language. In your code, import the Gemini AI client library and initialize it with your API key. Call the Gemini AI API to generate text, translate languages, write different kinds of creative content, or answer your questions in an informative way. For more detailed instructions on how to install and use Gemini AI, please refer to the Gemini AI documentation. The future of Gemini AI Gemini AI is still under development, but it has the potential to revolutionize the way we interact with computers. In the future, Gemini AI could be used to create more realistic and engaging chatbots, virtual assistants, and other forms of AI-powered software. Gemini AI could also be used to improve our understanding of the world around us by analyzing large datasets and identifying patterns and trends. Conclusion Gemini AI is a powerful tool that has the potential to be used in a wide variety of applications. It is still under development, but it has the potential to revolutionize the way we interact with computers. In the future, Gemini AI could be used to create more realistic and engaging chatbots, virtual assistants, and other forms of AI-powered software. Gemini AI could also be used to improve our understanding of the world around us by analyzing large datasets and identifying patterns and trends.  
Read more
  • 0
  • 0
  • 24929

article-image-creating-effective-dashboards-using-splunk-tutorial
Sunith Shetty
28 Jul 2018
10 min read
Save for later

Creating effective dashboards using Splunk [Tutorial]

Sunith Shetty
28 Jul 2018
10 min read
Splunk is easy to use for developing a powerful analytical dashboard with multiple panels. A dashboard with too many panels, however, will require scrolling down the page and can cause the viewer to miss crucial information. An effective dashboard should generally meet the following conditions: Single screen view: The dashboard fits in a single window or page, with no scrolling Multiple data points: Charts and visualizations should display a number of data points Crucial information highlighted: The dashboard points out the most important information, using appropriate titles, labels, legends, markers, and conditional formatting as required Created with the user in mind: Data is presented in a way that is meaningful to the user Loads quickly: The dashboard returns results in 10 seconds or less Avoid redundancy: The display does not repeat information in multiple places In this tutorial, we learn to create different types of dashboards using Splunk. We will also discuss how to gather business requirements for your dashboards. Types of Splunk dashboards There are three kinds of dashboards typically created with Splunk: Dynamic form-based dashboards Real-time dashboards Dashboards as scheduled reports Dynamic form-based dashboards allow Splunk users to modify the dashboard data without leaving the page. This is accomplished by adding data-driven input fields (such as time, radio button, textbox, checkbox, dropdown, and so on) to the dashboard. Updating these inputs changes the data based on the selections. Dynamic form-based dashboards have existed in traditional business intelligence tools for decades now, so users who frequently use them will be familiar with changing prompt values on the fly to update the dashboard data. Real-time dashboards are often kept on a big panel screen for constant viewing, simply because they are so useful. You see these dashboards in data centers, network operations centers (NOCs), or security operations centers (SOCs) with constant format and data changing in real time. The dashboard will also have indicators and alerts for operators to easily identify and act on a problem. Dashboards like this typically show the current state of security, network, or business systems, using indicators for web performance and traffic, revenue flow, login failures, and other important measures. Dashboards as scheduled reports may not be exposed for viewing; however, the dashboard view will generally be saved as a PDF file and sent to email recipients at scheduled times. This format is ideal when you need to send information updates to multiple recipients at regular intervals, and don't want to force them to log in to Splunk to capture the information themselves. We will create the first two types of dashboards, and you will learn how to use the Splunk dashboard editor to develop advanced visualizations along the way. Gathering business requirements As a Splunk administrator, one of the most important responsibilities is to be responsible for the data. As a custodian of data, a Splunk admin has significant influence over how to interpret and present information to users. It is common for the administrator to create the first few dashboards. A more mature implementation, however, requires collaboration to create an output that is beneficial to a variety of user requirements and may be completed by a Splunk development resource with limited administrative rights. Make it a habit to consistently request users input regarding the Splunk delivered dashboards and reports and what makes them useful. Sit down with day-to-day users and layout, on a drawing board, for example, the business process flows or system diagrams to understand how the underlying processes and systems you're trying to measure really work. Look for key phrases like these, which signify what data is most important to the business: If this is broken, we lose tons of revenue... This is a constant point of failure... We don't know what's going on here... If only I can see the trend, it will make my work easier... This is what my boss wants to see... Splunk dashboard users may come from many areas of the business. You want to talk to all the different users, no matter where they are on the organizational chart. When you make friends with the architects, developers, business analysts, and management, you will end up building dashboards that benefit the organization, not just individuals. With an initial dashboard version, ask for users thoughts as you observe them using it in their work and ask what can be improved upon, added, or changed. We hope that at this point, you realize the importance of dashboards and are ready to get started creating some, as we will do in the following sections. Dynamic form-based dashboard In this section, we will create a dynamic form-based dashboard in our Destinations app to allow users to change input values and rerun the dashboard, presenting updated data. Here is a screenshot of the final output of this dynamic form-based dashboard: Let's begin by creating the dashboard itself and then generate the panels: Go the search bar in the Destinations app Run this search command: SPL> index=main status_type="*" http_uri="*" server_ip="*" | top status_type, status_description, http_uri, server_ip Be careful when copying commands with quotation marks. It is best to type in the entire search command to avoid problems. Go to Save As | Dashboard Panel Fill in the information based on the following screenshot: Click on Save Close the pop-up window that appears (indicating that the dashboard panel was created) by clicking on the X in the top-right corner of the window Creating a Status Distribution panel We will go to the after all the panel searches have been generated. Let's go ahead and create the second panel: In the search window, type in the following search command: SPL> index=main status_type="*" http_uri=* server_ip=* | top status_type You will save this as a dashboard panel in the newly created dashboard. In the Dashboard option, click on the Existing button and look for the new dashboard, as seen here. Don't forget to fill in the Panel Title as Status Distribution: Click on Save when you are done and again close the pop-up window, signaling the addition of the panel to the dashboard. Creating the Status Types Over Time panel Now, we'll move on to create the third panel: Type in the following search command and be sure to run it so that it is the active search: SPL> index=main status_type="*" http_uri=* server_ip=* | timechart count by http_status_code You will save this as a Dynamic Form-based Dashboard panel as well. Type in Status Types Over Time in the Panel Title field: Click on Save and close the pop-up window, signaling the addition of the panel to the dashboard. Creating the Hits vs Response Time panel Now, on to the final panel. Run the following search command: SPL> index=main status_type="*" http_uri=* server_ip=* | timechart count, avg(http_response_time) as response_time Save this dashboard panel as Hits vs Response Time: Arrange the dashboard We'll move on to look at the dashboard we've created and make a few changes: Click on the View Dashboard button. If you missed out on the View Dashboard button, you can find your dashboard by clicking on Dashboards in the main navigation bar. Let's edit the panel arrangement. Click on the Edit button. Move the Status Distribution panel to the upper-right row. Move the Hits vs Response Time panel to the lower-right row. Click on Save to save your layout changes. Look at the following screenshot. The dashboard framework you've created should now look much like this. The dashboard probably looks a little plainer than you expected it to. But don't worry; we will improve the dashboard visuals one panel at a time: Panel options in dashboards In this section, we will learn how to alter the look of our panels and create visualizations. Go to the edit dashboard mode by clicking on the Edit button. Each dashboard panel will have three setting options to work with: edit search, select visualization, and visualization format options. They are represented by three drop-down icons: The Edit Search window allows you to modify the search string, change the time modifier for the search, add auto-refresh and progress bar options, as well as convert the panel into a report: The Select Visualization dropdown allows you to change the type of visualization to use for the panel, as shown in the following screenshot: Finally, the Visualization Options dropdown will give you the ability to fine-tune your visualization. These options will change depending on the visualization you select. For a normal statistics table, this is how it will look: Pie chart – Status Distribution Go ahead and change the Status Distribution visualization panel to a pie chart. You do this by selecting the Select Visualization icon and selecting the Pie icon. Once done, the panel will look like the following screenshot: Stacked area chart – Status Types Over Time We will change the view of the Status Types Over Time panel to an area chart. However, by default, area charts will not be stacked. We will update this through adjusting the visualization options: Change the Status Types Over Time panel to an Area Chart using the same Select Visualization button as the prior pie chart exercise. Make the area chart stacked using the Format Visualization icon. In the Stack Mode section, click on Stacked. For Null Values, select Zero. Use the chart that follows for guidance: Click on Apply. The panel will change right away. Remove the _time label as it is already implied. You can do this in the X-Axis section by setting the Title to None. Close the Format Visualization window by clicking on the X in the upper-right corner: Here is the new stacked area chart panel: Column with overlay combination chart – Hits vs Response Time When representing two or more kinds of data with different ranges, using a combination chart—in this case combining a column and a line—can tell a bigger story than one metric and scale alone. We'll use the Hits vs Response Time panel to explore the combination charting options: In the Hits vs Response Time panel, change the chart panel visualization to Column In the Visualization Options window, click on Chart Overlay In the Overlay selection box, select response_time Turn on View as Axis Click on X-Axis from the list of options on the left of the window and change the Title to None Click on Legend from the list of options on the left Change the Legend Position to Bottom Click on the X in the upper-right-hand corner to close the Visualization Options window The new panel will now look similar to the following screenshot. From this and the prior screenshot, you can see there was clearly an outage in the overnight hours: Click on Done to save all the changes you made and exit the Edit mode The dashboard has now come to life. This is how it should look now: To summarize we saw how to create different types of dashboards. To know more about core Splunk functionalities to transform machine data into powerful insights, check out this book Splunk 7 Essentials, Third Edition. Splunk leverages AI in its monitoring tools Splunk Industrial Asset Intelligence (Splunk IAI) targets Industrial IoT marketplace Create a data model in Splunk to enable interactive reports and dashboards
Read more
  • 0
  • 0
  • 24904
article-image-web-application-information-gathering
Packt
05 Jun 2017
4 min read
Save for later

Web Application Information Gathering

Packt
05 Jun 2017
4 min read
In this article by Ishan Girdhar, author of the book, Kali Linux Intrusion and Exploitation Cookbook, we will cover the following recipes: Setup API keys for the recon-ng framework Use recon-ng for reconnaissance (For more resources related to this topic, see here.) Setting up API keys for recon-ng framework In this recipe, we will see how we need to set up API keys before we start using recon-ng. Recon-ng is one of the most powerful information gathering tools, if used appropriately, it can help pentesters locating good amount of information from public sources. With the latest version available, recon-ng provides the flexibility to set it up as your own app/client in various social networking websites. Getting ready For this recipe, you require an Internet connection and web browser. How to do it... To set up recon-ng API keys, open the terminal and launch recon-ng and type the commands shown in the following screenshot: Next, type keys list as shown in the following screenshot: Let's start by adding twitter_API & twitter_secret. Log in to Twitter, go to https://apps.twitter.com/, and create a new application as shown in the following screenshot: Click on Create Application once the application is created, navigate to Keys & Access tokens tabs, and copy the secret key and API key as shown in the following screenshot: Copy the API key and reopen the terminal window again run the following command to add the key: Keys add twitter_api <your-copied-api-key> Now, enter the following command to enter the twitter_secret name in recon-ng: keys add twitter_secret <you_twitter_secret> Once you added the keys, you can see the keys added in the recon-ng tool by entering the following command: keys list How it works... In this recipe, you learned how to add API keys to the recon-ng tool. To demonstrate the same, we have created a Twitter application and used Twitter_API and Twitter_Secret and added them to the recon-ng tool. The result is as shown in the following screenshot: Similarly, you will need to include all the API keys here in the recon-ng if you want to gather information from these sources. In next recipe, you will learn how to use recon-ng for information gathering. Use recon-ng for reconnaissance In this recipe, you will learn to use recon-ng for reconnaissance. Recon-ng is a full-featured Web Reconnaissance framework written in Python. Complete with independent modules, database interaction, built-in convenience functions, interactive help, and command completion, Recon-ng provides a powerful environment in which open source web-based reconnaissance can be conducted quickly and thoroughly. Getting ready To install Kali Linux, you will require an Internet connection. How to do it... Open a terminal and start the recon-ng framework, as shown in the following screenshot: Recon-ng has the look and feel like that of Metasploit. To see all the available modules, enter the following command: show modules Recon-ng will list all available modules, as shown in the following screenshot: Let's go ahead and use our first module for information gathering. Enter the following command: use recon/domains-vulnerabilities/punkspider Now, enter the commands shown in the following screenshot: As you can see, there are some vulnerabilities discovered and are available publically. Let's use another module that fetches any known and reported vulnerabilities from xssed.com. The XSSed project was created in early February 2007 by KF and DP. It provides information on all things related to cross-site scripting vulnerabilities and is the largest online archive of XSS vulnerable websites. It's a good repository of XSS to gather information. To begin with, enter the following command: Show module use recon/domains-vulnerabilities/xssed Show Options Set source Microsoft.com Show Options RUN You will see the following output: As you can see, recon-ng has aggregated the publically available vulnerabilities from XSSed, as shown in the following screenshot: Similarly, you can keep using the different modules until and unless you get your required information regarding your target. Summary In this article, you learned how to add API keys to the recon-ng tool. To demonstrate the same, we have created a Twitter application and used Twitter_API and Twitter_Secret and added them to the recon-ng tool. You also learned how to use recon-ng for reconnaissance. Resources for Article: Further resources on this subject: Getting Started with Metasploitable2 and Kali Linux [article] Wireless Attacks in Kali Linux [article] What is Kali Linux [article]
Read more
  • 0
  • 0
  • 24882

article-image-microsofts-metoo-reckoning-female-employees-speak-out-against-workplace-harassment-and-discrimination
Sugandha Lahoti
05 Apr 2019
7 min read
Save for later

Microsoft’s #MeToo reckoning: female employees speak out against workplace harassment and discrimination

Sugandha Lahoti
05 Apr 2019
7 min read
Microsoft was founded by Bill Gates and Paul Allen 44 years ago today on April 4, 1975. A lot has changed for the company since then in terms of its incredible growth story and inclusive business practices but some ghosts from its Silicon Valley bro culture past continue to linger. In 1978, the company started with eleven employees out of which only two were women; Andrea Lewis, technical writer, and Maria Wood bookkeeper; making women’s representation at 18 percent but with zero core engineering roles. Even with just two women, Microsoft was in troubles for its sexual conduct policies; Maria Wood left the company in 1983, suing it for sexual discrimination. She then disappeared from professional life to raise her children and volunteer for good causes. Andrea Lewis also left at almost the same time, eventually becoming a freelance journalist and fiction writer. Content note: this piece contains references to sexual harassment and abusive behaviors. As of June 30, 2018, the combined percentage of women who work at Microsoft and LinkedIn stood at 28 percent. For Microsoft alone, the percentage of women stood at 26.6 percent. The representation of women in technical roles is 19.9 percent and in leadership roles is 19.7 percent. The percentage of female interns at Microsoft stands at 42.5 percent in the past year. These numbers do not include the temp and contract workers at Microsoft. Source: Microsoft Blog Recently, Microsoft women employees shared their experiences on sexual harassment and discrimination they faced in the company in an email chain. According to Quartz, who first reported the news after reviewing more than 90 pages of emails, this chain has gained notice from the company’s senior leadership team. This chain, which started on March 20, was instigated Kathleen Hogan, Microsoft’s head of human resources. It was intended to be a way of organizing workers and raising the problems with CEO, Satya Nadella. In this regard it was successful. The company turned the weekly all-hands meeting on Thursday into a Q&A session where employees could discuss the toxic work culture and address the Microsoft leadership directly about the accusations that emerged in the thread. According to Wired, roughly 100 to 150 employees attended the Q&A in person, with many others watching via a live stream. Some female and male employees at the event wore all white, inspired by the congresswomen who wore "suffragette white" to the State of the Union in February. Responding to the concerns raised in the meeting, Nadella was  apparently empathetic and expressed sadness and disappointment. Hogan responded to the email chain on March 29 writing, “I would like to offer to anyone who has had such demeaning experiences including those who felt were dismissed by management or HR to email me directly, I will personally look into the situation with my team. I understand the devastating impact of such experiences, and [Nadella] wants to be made aware of any such behavior, and we will do everything we can to stop it.” What allegations of sexual harassment and abuse emerged in the Microsoft employees' email thread? One Microsoft Partner employee reportedly wrote in the email that she “was asked to sit on someone’s lap twice in one meeting in front of HR and other executives,” stating that they didn’t do anything in response to this violation of the company’s policy. “The person said that he did not have to listen and repeat the request a second time,” she wrote, according to Quartz. “No one said anything.” Another female Microsoft employee said that an employee of a Microsoft partner company threatened to kill her during a work trip if she didn’t engage in sexual acts, according to Quartz and Wired reports. “I raised immediate attention to HR and management,” she wrote, according to Quartz. “My male manager told me that ‘it sounded like he was just flirting’ and I should ‘get over it’. HR basically said that since there was no evidence, and this man worked for a partner company and not Microsoft, there was nothing they could do.” Another ex-Microsoft employee shared her story on Twitter. https://twitter.com/CindyGross/status/1113893229013995520 Another employee who had worked on the Xbox core team reportedly said in the email chain that being called a “bitch” was common within the company. She said the word had been used against her on more than one occasion, and even during roundtables where female members of the Xbox core team were in attendance. "Every woman, except for 1, had been called a bitch at work." “This thread has pulled the scab off a festering wound. The collective anger and frustration is palpable. A wide audience is now listening. And you know what? I’m good with that,” one Microsoft employee in the email chain wrote, according to Quartz. The problem is far bigger than Microsoft - it's the whole tech industry Sadly, reports of discriminatory and abusive behavior towards women are common across the tech industry. It would be wrong to see this as a Microsoft issue alone. For example, according to a 2016 survey, sixty percent of women working in Silicon Valley have experienced unwanted sexual advances. Two-thirds of these respondents said that these advances were from superiors - a clear abuse of power. Even a couple of years later, the report is a useful document that throws light on sexism and misogyny in an industry that remains dominated by men. According to court filings made public on Monday, this week, Women at Microsoft Corp working in U.S.-based technical jobs filed 238 internal complaints about gender discrimination or sexual harassment between 2010 and 2016. In response to these allegations, Kathleen Hogan sent an email to all Microsoft employees. Kathleen Hogan criticised In a medium blog post, Mitchel Lewis, criticised Hogan's email. He wrote "an embarrassing 10% of [Microsoft's] gender discrimination claims and 50% of their harassment claims, each of which had almost 90 or so instances last year, were found to lack merit by ERIT, which is a team comprised almost exclusively of lawyers on Microsoft’s payroll." He adds, "But as a staunch feminist, Kathleen did not address the fact that such a low rate of dignified claims can also serve as a correlate of an environment that discourages people to step forward with claims of abuse as it could be the result of an environment that is ripe for predation, corruption, and oppression." April Wensel, the founder of Compassionate Coding, shared her views on Twitter, both before and after the story broke out. Earlier in March, she had been at the receiving end of (arguably gendered) criticism from a senior male Microsoft employee who set up a Twitter poll to disprove Wensel’s observation that many tech workers are unhappy. Wensel noted, “The unchecked privilege is remarkable. Imagine trying to create positive change trapped in an organization that supports this kind of behavior." https://twitter.com/aprilwensel/status/1103178700130926592?s=20 https://twitter.com/aprilwensel/status/1113940085227933696 Microsoft Workers 4 good, a coalition of Microsoft employees, also tweeted showing support: https://twitter.com/MsWorkers4/status/1113934565737783296 https://twitter.com/MsWorkers4/status/1113934825931415552 The group had previously posted an open letter to Microsoft CEO in protest of the company’s $480 million deal with the U.S. Army to provide them with Hololens2. Other people also joined in solidarity with Microsoft's female employees: https://twitter.com/rod3000/status/1113924177428271104 https://twitter.com/OliviaGoldhill/status/1113836770871980034 https://twitter.com/cindygallop/status/1113889617755951104 Moving forward: keep the pressure on leadership How Microsoft chooses to move forward remains to be seen. Indeed, this is only a part of a broader story about an industry finally having to reckon with decades of sexism and marginalization. And while tackling it is clearly the right thing to do, whether that's possible at the moment is a huge question. One thing is for sure - it probably won't be tackled by leadership teams alone. The work done by grassroots-level organizations and figures like April Wensel is essential when it comes to effecting real change. BuzzFeed Report: Google’s sexual misconduct policy “does not apply retroactively to claims already compelled to arbitration”. Uber’s Head of corporate development, Cameron Poetzscher, resigns following a report on a 2017 investigation into sexual misconduct. Following Google, Facebook changes its forced arbitration policy for sexual harassment claims.
Read more
  • 0
  • 0
  • 24847
Modal Close icon
Modal Close icon