Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7014 Articles
article-image-intro-canvas-animations
Dylan Frankland
21 Nov 2016
11 min read
Save for later

Intro to Canvas Animations

Dylan Frankland
21 Nov 2016
11 min read
I think that most developers who have not had much experience with Canvas will generally believe that it is unnecessary, non-performant, or difficult to work with. Canvas's flexibility lends itself to be experimented with and compared against other solutions in the browser. You find comparison articles all over the web comparing things like animations, filter effects, GPU rendering, you name it with Canvas. The way that I look at it is that Canvas is best suited to creating custom shapes and effects that hook into events or actions in the DOM. Doing anything else should be left up to CSS considering it can handle most animations, transitions, and normal static shapes. Mastering Canvas opens a world of possibilities to create anything such as video games, smartwatch interfaces, and even more complex custom 3D graphics. The first step is understanding canvas in 2D and animating some elements before getting into more complex logic. A simple idea that might demonstrate the abilities of Canvas is to create a Cortana- or HAL-like interface that changes shape according to audio. First we will create the interface, a simple glowing circle, then we will practice animating it. Before getting started, I assume you will be using a newer version of Chrome. If you are using a non-WebKit browser, some of the APIs used may be slightly different. To get started, let us create a simple index.html page with the following markup: <!doctype html> <html> <head> <title>Demo</title> <style> body { padding: 0; margin: 0; background-color: #000; } </style> </head> <body> <canvas></canvas> <script></script> </body> </html> Obviously, this is nothing special yet, it just holds the simple HTML for us to begin coding our JavaScript. Next we will start to fill in the <script> block below the <canvas> element inside of the body. The first part of your script will have to get the reference to the <canvas> element, then set its size; for this experiment, let us just set the size to the entire window: (() => { const canvas = document.querySelector('canvas'); canvas.width = window.innerWidth; canvas.height = window.innerHeight; })(); Again, refreshing the page will not show much. This time let's render a small circle to begin with: (() => { const canvas = document.querySelector('canvas'); canvas.width = window.innerWidth; canvas.height = window.innerHeight; const context = canvas.getContext('2d'); // Set the context to create a 2D graphic context.translate(canvas.width / 2, canvas.height / 2); // The starting point for the graphic to be in the middle context.beginPath(); // Start to create a shape context.arc(0, 0, 50, 0, 2 * Math.PI, false); // Draw a circle, at point [0, 0] with a radius of 50, and make it 360 degrees (aka 2PI) context.fillStyle = '#8ED6FF'; // Set the color of the circle, make it a cool AI blue context.fill(); // Actually fill the shape in and close it })(); Alright! Now we have something that we can really start to work with. If you refresh that HTML page you should now be staring at a blue dot in the center of your screen. I do not know about you, but blue dots are sort of boring, nothing that interesting. Let's switch it up and create a more movie-esque AI-looking circle. (() => { const canvas = document.querySelector('canvas'); canvas.width = window.innerWidth; canvas.height = window.innerHeight; const context = canvas.getContext('2d'); // Set the context to create a 2D graphic const color = '#8ED6FF'; context.translate(canvas.width / 2, canvas.height / 2); // The starting point for the graphic to be in the middle context.beginPath(); // Start to create a shape context.arc(0, 0, 50, 0, 2 * Math.PI, false); // Draw a circle, at point [0, 0] with a radius of 50, and make it 360 degrees (aka 2PI) context.strokeStyle = color; // Set the stroke color context.shadowColor = color; // Set the "glow" color context.lineWidth = 10; // Set the circle/ring width context.shadowBlur = 60; // Set the amount of "glow" context.stroke(); // Draw the shape and close it })(); That is looking way better now. Next, let's start to animate this without hooking it up to anything. The standard way to do this is to create a function that alters the canvas each time that the DOM is able to render itself. We do that by creating a function which passes a reference to itself to window.requestAnimationFrame. requestAnimationFrame takes any function and executes it once the window is focused and has processing power to render another frame. This is a little confusing, but it is the best way to create smooth animations. Check out the example below: (() => { const canvas = document.querySelector('canvas'); canvas.width = window.innerWidth; canvas.height = window.innerHeight; const context = canvas.getContext('2d'); // Set the context to create a 2D graphic const color = '#8ED6FF'; const drawCircle = () => { context.translate(canvas.width / 2, canvas.height / 2); // The starting point for the graphic to be in the middle context.beginPath(); // Start to create a shape context.arc(0, 0, 50, 0, 2 * Math.PI, false); // Draw a circle, at point [0, 0] with a radius of 50, and make it 360 degrees (aka 2PI) context.strokeStyle = color; // Set the stroke color context.shadowColor = color; // Set the "glow" color context.lineWidth = 10; // Set the circle/ring width context.shadowBlur = 60; // Set the amount of "glow" context.stroke(); // Draw the shape and close it window.requestAnimationFrame(drawCircle); // Continue drawing circle }; window.requestAnimationFrame(drawCircle); // Start animation })(); This will not create any type of animation yet, but if you put a console.log inside the drawCircle function you would see a bunch of logs in the console, probably close to 60 times per second. The next step is to add some state to the this function and make it change size. We can do that by creating a size variable with an integer that we change up and down, plus another boolean variable called grow to keep track of the direction that we will change size. (() => { const canvas = document.querySelector('canvas'); canvas.width = window.innerWidth; canvas.height = window.innerHeight; const context = canvas.getContext('2d'); // Set the context to create a 2D graphic const color = '#8ED6FF'; let size = 50; // Default size is the minimum size let grow = true; const drawCircle = () => { context.translate(canvas.width / 2, canvas.height / 2); // The starting point for the graphic to be in the middle context.beginPath(); // Start to create a shape context.arc(0, 0, size, 0, 2 * Math.PI, false); // Draw a circle, at point [0, 0] with a radius of 50, and make it 360 degrees (aka 2PI) context.strokeStyle = color; // Set the stroke color context.shadowColor = color; // Set the "glow" color context.lineWidth = 10; // Set the circle/ring width context.shadowBlur = size + 10; // Set the amount of "glow" context.stroke(); // Draw the shape and close it // Check if the size needs to grow or shrink if (size <= 50) { // Minimum size grow = true; } else if (size >= 75) { // Maximum size grow = false; } // Grow or shrink the size size = size + (grow ? 1 : -1); window.requestAnimationFrame(drawCircle); // Continue drawing circle }; window.requestAnimationFrame(drawCircle); // Start animation })(); Refreshing the page and seeing nothing happen might be a little disheartening, but do not worry you are not the only one! Canvases require being cleared before being drawn upon again. Now we have created a way to change and update the canvas, but we need to introduce a way to clear it. (() => { const canvas = document.querySelector('canvas'); canvas.width = window.innerWidth; canvas.height = window.innerHeight; const context = canvas.getContext('2d'); // Set the context to create a 2D graphic const color = '#8ED6FF'; let size = 50; // Default size is the minimum size let grow = true; const drawCircle = () => { context.clearRect(0, 0, canvas.width, canvas.height); // Clear the contents of the canvas starting from [0, 0] all the way to the [totalWidth, totalHeight] context.translate(canvas.width / 2, canvas.height / 2); // The starting point for the graphic to be in the middle context.beginPath(); // Start to create a shape context.arc(0, 0, size, 0, 2 * Math.PI, false); // Draw a circle, at point [0, 0] with a radius of 50, and make it 360 degrees (aka 2PI) context.strokeStyle = color; // Set the stroke color context.shadowColor = color; // Set the "glow" color context.lineWidth = 10; // Set the circle/ring width context.shadowBlur = size + 10; // Set the amount of "glow" context.stroke(); // Draw the shape and close it // Check if the size needs to grow or shrink if (size <= 50) { // Minimum size grow = true; } else if (size >= 75) { // Maximum size grow = false; } // Grow or shrink the size size = size + (grow ? 1 : -1); window.requestAnimationFrame(drawCircle); // Continue drawing circle }; window.requestAnimationFrame(drawCircle); // Start animation })(); Now things may seems even more broken. Remember that context.translate function? That is setting the origin of all the commands to the middle of the canvas. To prevent that from messing up our canvas when we try to clear it, we will need to save the state of the canvas and restore before hand. (() => { const canvas = document.querySelector('canvas'); canvas.width = window.innerWidth; canvas.height = window.innerHeight; const context = canvas.getContext('2d'); // Set the context to create a 2D graphic const color = '#8ED6FF'; let size = 50; // Default size is the minimum size let grow = true; const drawCircle = () => { context.restore(); // Restore previous canvas state, does nothing if it wasn't saved before context.save(); // Saves it until the next time `drawCircle` is called context.clearRect(0, 0, canvas.width, canvas.height); // Clear the contents of the canvas starting from [0, 0] all the way to the [totalWidth, totalHeight] context.translate(canvas.width / 2, canvas.height / 2); // The starting point for the graphic to be in the middle context.beginPath(); // Start to create a shape context.arc(0, 0, size, 0, 2 * Math.PI, false); // Draw a circle, at point [0, 0] with a radius of 50, and make it 360 degrees (aka 2PI) context.strokeStyle = color; // Set the stroke color context.shadowColor = color; // Set the "glow" color context.lineWidth = 10; // Set the circle/ring width context.shadowBlur = size + 10; // Set the amount of "glow" context.stroke(); // Draw the shape and close it // Check if the size needs to grow or shrink if (size <= 50) { // Minimum size grow = true; } else if (size >= 75) { // Maximum size grow = false; } // Grow or shrink the size size = size + (grow ? 1 : -1); window.requestAnimationFrame(drawCircle); // Continue drawing circle }; window.requestAnimationFrame(drawCircle); // Start animation })(); Oh snap! Now you have got some seriously cool canvas animation going on. The final code should look like this: <!doctype html> <html> <head> <title>Demo</title> <style> body { padding: 0; margin: 0; background-color: #000; } </style> </head> <body> <canvas></canvas> <script> (() => { const canvas = document.querySelector('canvas'); canvas.width = window.innerWidth; canvas.height = window.innerHeight; const context = canvas.getContext('2d'); // Set the context to create a 2D graphic const color = '#8ED6FF'; let size = 50; // Default size is the minimum size let grow = true; const drawCircle = () => { context.restore(); context.save(); context.clearRect(0, 0, canvas.width, canvas.height); // Clear the contents of the canvas starting from [0, 0] all the way to the [totalWidth, totalHeight] context.translate(canvas.width / 2, canvas.height / 2); // The starting point for the graphic to be in the middle context.beginPath(); // Start to create a shape context.arc(0, 0, size, 0, 2 * Math.PI, false); // Draw a circle, at point [0, 0] with a radius of 50, and make it 360 degrees (aka 2PI) context.strokeStyle = color; // Set the stroke color context.shadowColor = color; // Set the "glow" color context.lineWidth = 10; // Set the circle/ring width context.shadowBlur = size + 10; // Set the amount of "glow" context.stroke(); // Draw the shape and close it // Check if the size needs to grow or shrink if (size <= 50) { // Minimum size grow = true; } else if (size >= 75) { // Maximum size grow = false; } // Grow or shrink the size size = size + (grow ? 1 : -1); window.requestAnimationFrame(drawCircle); // Continue drawing circle }; window.requestAnimationFrame(drawCircle); // Start animation })(); </script> </body> </html> Hopefully, this should give you a great idea on how to get started and create some really cool animations. The next logical step to this example is to hook it up to an audio so that it can react to voice like your own personal AI. Author Dylan Frankland is a frontend engineer at Narvar. He is an agile web developer with over 4 years of experience in developing and designing for start-ups and medium-sized businesses to create functional, fast, and beautiful web experiences.
Read more
  • 0
  • 0
  • 1933

article-image-create-dynamic-tilemaps-duality-c-part-ii
Lőrinc Serfőző
21 Nov 2016
7 min read
Save for later

Create Dynamic Tilemaps in Duality (C#) – Part II

Lőrinc Serfőző
21 Nov 2016
7 min read
Introduction In the first part of this tutorial, we learned how static tilemaps are created in the Duality game engine. Let's now take it to a higher level. In this article, the process of implementing a custom component is described. The new component modifies the tilemap at runtime, controlled by mouse clicks. Although it is a simple example, the method enables more advanced uses, such as procedural generation of game levels or destructible terrain. The repository containing the finished project is available on GitHub. Creating Custom Components It has been already mentioned that all functionality in Duality is implemented via plugins. To create a custom component, you have to compile a new Core Plugin. Although it might seem convoluted at first, Duality does most of the work and setup, so let's dive in! First, open the Visual Studio Solution by clicking on the 'Open Sourcecode' icon button, which is the second one on the toolbar. Alternatively, just open the ProjectPlugins.sln file located in '{Duality Project Dir}'. Once the IDE is open, inspect the sole loaded project called 'CorePlugin'. It has two source files (not counting AssemblyInfo.cs), and references to the Duality assembly. CorePlugin.cs contains a class inherited from CorePlugin. It's necessary to have this class in the solution to identify the assembly as a plugin, but usually it does not need to be modified, because game logic is implemented via custom components in most of the time. Let's have a look at the other class located in 'YourCustomComponentType.cs': using System; using System.Collections.Generic; using System.Linq; using Duality; namespace Duality_ { public class YourCustomComponentType : Component { } } There are a few important things to notice here. The custom component must be a subclass of Component. It has to be declared public; otherwise the new component wouldn't appear in the editor. Don't modify the code for now, but hit F7 to compile the assembly. Behind the scenes, the output assembly named 'GamePlugin.core.dll' (along with the debug symbol file and the xml documentation) is copied to '{Duality Project Dir}', and Dualitor loads them. The new component is available for being added to the game, like any other component: Adding the custom component At this point, the new component could be added to GameObjects, but it would not do anything yet. The next sections are about how to implement the game logic in the component. Laying the structure of ChangeTilesetCmp Adding a reference of the Tilemap Plugin to the VS project To access tilemap-related classes and information in the component, the Visual Studio project must have a reference to the assembly containing the Tilemap Plugin. Bring up the context menu on the 'CorePlugin' project's 'References' item in the Solution Explorer, and select the 'Add Reference' item. A dialog should appear; browse 'Tilemaps.core.dll' from the '{Duality Project Dir}'. Adding the Tilemap Plugin as a reference Defining the internal structure of ChangeTilemapCmp Rename the YourCustomComponentType class to ChangeTilemapCmp, and do the same with the container .cs file to stay consistent. First describe the function signatures used in the component: using Duality; using Duality.Components; using Duality.Editor; using Duality.Input; using Duality.Plugins.Tilemaps; namespace Duality_ [EditorHintCategory ("Tilemaps Tutorial")] // [1] public class ChangeTilemapCmp : Component, ICmpUpdatable { // [2] private Tilemap TilemapInScene { get; } private TilemapRenderer TilemapRendererInScene { get; } private Camera MainCamera { get; } void ICmpUpdatable.OnUpdate () // [3] { } private Vector2 GetWorldCoordOfMouse () // [4] { } private void ChangeTilemap (Vector2 worldPos) // [5] { } } } The EditorHintCategory attribute describes in which folder the component should appear when adding it. Several get-only properties are used to access specific components in the scene. See their implementation below. The OnUpdate function implements the ICmpUpdatable interface. It's called upon every update of the game loop. GetWorldCoordOfMouse returns the current position of the mouse transformed into the game world coordinates. ChangeTilemap, as its name suggests, changes the tilemap at a specified world location. It should not do anything if the location is not on the tilemap. After designing our little class, let's implement the details! Implementing the game logic in ChangeTilemapCmp Implementing the get-only properties The Tilemap and TilemapRenderer components are obviously needed by our logic. Since there is only one instance of them in the scene, it's easy to find them by type: private Tilemap TilemapInScene => this.GameObj.ParentScene.FindComponent<Tilemap> (); private TilemapRenderer TilemapRendererInScene => this.GameObj.ParentScene.FindComponent<TilemapRenderer>(); private Camera MainCamera => this.GameObj.ParentScene.FindComponent<Camera> (); Implementing the OnUpdate method This method is called for every frame, usually 60 times a second. We check if the left mouse button was pressed in that frame, and if yes, act accordingly: void ICmpUpdatable.OnUpdate () { if (DualityApp.Mouse.ButtonHit (MouseButton.Left)) ChangeTilemap (GetWorldCoordOfMouse ()); } Implementing the GetWorldCoordOfMouse method Here a simple transformation is needed. The DualityApp.Mouse.Pos property returns the mouse position on the screen. After a null-check, get the mouse position on the screen; then convert it to world position using the Camera's GetSpaceCoord method. private Vector2 GetWorldCoordOfMouse () { if (MainCamera == null) return Vector2.Zero; Vector2 mouseScreenPos = DualityApp.Mouse.Pos; return MainCamera.GetSpaceCoord (mouseScreenPos).Xy; } Implementing the ChangeTilemap method The main logic of ChangeTilemapCmp is implemented in this method: private void ChangeTilemap (Vector2 worldPos) { // [1] Tilemap tilemap = TilemapInScene; TilemapRenderer tilemapRenderer = TilemapRendererInScene; if (tilemap == null || tilemapRenderer == null) { Log.Game.WriteError("There are no tilemaps in the current scene!"); return; } // [2] Vector2 localPos = worldPos - tilemapRenderer.GameObj.Transform.Pos.Xy; // [3] Point2 tilePos = tilemapRenderer.GetTileAtLocalPos (localPos, TilePickMode.Reject); if (tilePos.X < 0 || tilePos.Y < 0) return; // [4] Tile clickedTile = tilemap.Tiles[tilePos.X, tilePos.Y]; int newTileIndex = clickedTile.BaseIndex == 0 ? 1 : 0; clickedTile.BaseIndex = newTileIndex; tilemap.SetTile(tilePos.X, tilePos.Y, clickedTile); } First check if there is any Tilemap and TilemapRenderer in the Scene. Transform the world position to the tilemap's local coordinate system by subtracting the tilemap's position from it. Acquire the indexed location of the clicked tile by the GetTileAtLocalPos method of the renderer. It returns a Point2, an integer-based vector, which represents the row and column of the tile in the tilemap. Because TilePickMode.Reject is passed as the second argument, it returns {-1, -1} when the 'player' clicks next to the tilemap. We should check that too, and return if that is the case. Get the Tile struct at the clicked position, and flip its tile index. Index 0 means the green tile, while index 1 refers to the red one. Afterwards, the tile has to be assigned back to the Tilemap, because in C#, structs are copied by value, not by reference. After finishing this, do not forget to compile the source again. Adding and testing ChangeTilemapCmp Create a new GameObject with ChangeTilemapCmp on it in the Scene View: Adding ChangeTilemapCmp Then save the scene with the floppy icon of the Scene View, and start the game by clicking the 'Run Game' icon button located on the toolbar. Test the game by clicking on the tilemap; it should change color on the tile you click. Running the game Summary Thank you for following along with this guide! Hopefully it helped you with the Duality game engine and its Tilemap Plugin. Remember that the above was a very simple example, but more sophisticated logic is achievable using these tools. If you want some inspiration, have a look at the entries of the Duality Tilemaps Jam, which took place in September, 2016. Author Lőrinc Serfőző is a software engineer at Graphisoft, the company behind the BIM solution ArchiCAD. He is studying mechatronics engineering at the Budapest University of Technology and Economics, an interdisciplinary field between the more traditional mechanical engineering, electrical engineering and informatics, and has quickly grown a passion toward software development. He is a supporter of open source and contributes to the C# and OpenGL-based Duality game engine, creating free plugins and tools for users.
Read more
  • 0
  • 0
  • 2275

article-image-how-create-2d-navigation-godot-engine-0
George Marques
18 Nov 2016
6 min read
Save for later

How to Create 2D Navigation with the Godot Engine

George Marques
18 Nov 2016
6 min read
The Godot Engine has built-in functionalities that makes it easy to create navigation in the game world. This post will cover how to make an object follow a fixed path and how to go between two points avoiding the obstacles in the way. Following a fixed path Godot has a couple of nodes that help you create a path that can be followed by another node. One use of this is to make an NPC follow a fixed path in the map. Assuming you have a new project, create a Path2D node. You can then use the controls on the toolbar to make a curve representing the path you will need to follow. Curve buttons After adding the points and adjusting the curves, you will have something like the following: Path curve Now you need to add a PathFollow2D node as a child of Path2D. This will do the actual movement based on the Offset property. Then add an AnimationPlayer node as child of the PathFollow2D. Create a new animation in the player. Set the length to five seconds. Add a keyframe on the start with value 0 for the Unit Offset property of PathFollow2D. You can do that by clicking on the key icon next to the property in the Inspector dock. Then go to the end of the animation and add a frame with the value of 1. This will make Unit Offset go from 0 to 1 in the period of the animation (five seconds in this case). Set the animation to loop and autoplay. To see the effect in practice, add a Sprite node as child of PathFollow2D. You can use the default Godot icon as the texture for it. Enable the Visible Navigation under the Debug Options menu (last button in the top center bar) to make it easier to see. Save the scene and play it to see the Godot robot run around the screen: Sprite following path That's it! Making an object follow a fixed path is quite easy with the built-in resources of Godot. Not even scripting is needed for this example. Navigation and Avoiding Obstacles Sometimes you don't have a fixed path to follow. It might change dynamically or your AI must determine the path and avoid the obstacles and walls that might be in the way. Don't worry because Godot will also help you in this regard. Create a new scene and add a Node2D as the root. Then add a Navigation2D as its child. This will be responsible for creating the paths for you. You now need to add a NavigationPolygonInstance node as child of the Navigation2D. This will hold the polygon used for navigation, to determine what the passable areas are. To create the polygon itself, click on the pencil button on the toolbar (it will appear only if the NavigationPolygonInstance node is selected). The first time you try to add a point, the editor will warn you that there's no NavigationPolygon resource and will offer you to create one. Click on the Create button and all will be set. Navitation resource warning First you need to create the outer boundaries of the navigable area. The polygon can have as many points as you need, but it does need to be a closed polygon. Note that you can right-click on points to remove them and hold Ctrl while clicking on lines to add points. Once you finish the boundaries, click the pencil button again and create polygons inside it to make the impassable areas, such as the walls. You will end up with something like the following: Navigation polygon Add a Sprite node as child of the root Node2D and set the texture of it (you can use the default Godot icon). This will be the object navigating through the space. Now add the following script to the root node. The most important detail here is the get_simple_path function, which returns a list of points to travel from start to end without passing through the walls. extends Node2D # Global variables var start = Vector2() var end = Vector2() var path = [] var speed = 1 var transition = 0 var path_points = 0 func _ready(): # Enable the processing of user input set_process_input(true) # Enable the general process callback set_process(true) func _input(event): # If the user press a mouse button if event.type == InputEvent.MOUSE_BUTTON and event.pressed: if event.button_index == BUTTON_LEFT: # If it's the left button, set the starting point start = event.global_pos elif event.button_index == BUTTON_RIGHT: # If it's the right button, set the ending point end = event.global_pos # Reset the sprite position get_node("Sprite").set_global_pos(start) transition = 0 func _process(delta): # Get the list of points that compose the path path = get_node("Navigation2D").get_simple_path(start, end) # If the path has less points than it did before, reset the transition point if path.size() < path_points: transition = 0 # Update the current amount of points path_points = path.size() # If there's less than 2 points, nothing can be done if path_points < 2: return var sprite = get_node("Sprite") # This uses the linear interpolation function from Vector2 to move the sprite in a constant # rate through the points of the path. Transition is a value from 0 to 1 to be used as a ratio. sprite.set_global_pos(sprite.get_global_pos().linear_interpolate(path[1], transition)) start = sprite.get_global_pos() transition += speed * delta # Reset the transition when it gets to the point. if transition > 1: transition = 0 # Update the node so the _draw() function is called update() func _draw(): # This draw a white circle with radius of 10px for each point in the path for p in path: draw_circle(p, 10, Color(1, 1, 1)) Enable the Visible Navigation in the Debug Options button to help you visualize the effect. Save and run the scene. You can then left-click somewhere to define a starting point, and right-click to define the ending point. The points will be marked as white circles and the Sprite will follow the path, clearing the intermediate points as it travels along. Navigating Godot bot Conclusion The Godot Engine has many features to ease the development of all kinds of games. The navigation functions have many utilities in top-down games, be it an RPG or an RTS. Tilesets also embed navigation polygons that can be used in a similar fashion. About the Author: George Marques is a Brazilian software developer who has been playing with programming in a variety of environments since he was a kid. He works as a freelancer programmer for web technologies based on open source solutions such as WordPress and Open Journal Systems. He's also one of the regular contributors of the Godot Engine, helping solving bugs and adding new features to the software, while also giving solutions to the community for the questions they have.
Read more
  • 0
  • 1
  • 28876

article-image-suggesters-improving-user-search-experience
Packt
18 Nov 2016
11 min read
Save for later

Suggesters for Improving User Search Experience

Packt
18 Nov 2016
11 min read
In this article by Bharvi Dixit, the author of the book Mastering ElasticSearch 5.0 - Third Edition, we will focus on the topics for improving the user search experience using suggesters, which allows you to correct user query spelling mistakes and build efficient autocomplete mechanisms. First, let's look on the query possibilities and the responses returned by Elasticsearch. We will try to show you the general principles, and then we will get into more details about each of the available suggesters. (For more resources related to this topic, see here.) Using the suggester under search Before Elasticsearch 5.0, there was a possibility to get suggestions for a given text by using a dedicated _suggest REST endpoint. But in Elasticsearch 5.0, this dedicated _suggest endpoint has been deprecated in favor of using suggest API. In this release, the suggest only search requests have been optimized for performance reasons and we can execute the suggetions _search endpoint. Similar to query object, we can use a suggest object and what we need to provide inside suggest object is the text to analyze and the type of used suggester (term or phrase). So if we would like to get suggestions for the words chrimes in wordl (note that we've misspelled the word on purpose), we would run the following query: curl -XPOST "http://localhost:9200/wikinews/_search?pretty" -d' { "suggest": { "first_suggestion": { "text": "chrimes in wordl", "term": { "field": "title" } } } }' The dedicated endpoint _suggest has been deprecated in Elasticsearch version 5.0 and might be removed in future releases, so be advised to use suggestion request under _search endpoint. All the examples covered in this article usage the same _search endpoint for suggest request. As you can see, the suggestion request wrapped inside suggest object and is send to Elasticsearch in its own object with the name we chose (in the preceding case, it is first_suggestion). Next, we specify the text for which we want the suggestion to be returned using the text parameter. Finally, we add the suggester object, which is either term or phrase. The suggester object contains its configuration, which for the term suggester used in the preceding command, is the field we want to use for suggestions (the field property). We can also send more than one suggestion at a time by adding multiple suggestion names. For example, if in addition to the preceding suggestion, we would also include a suggestion for the word arest, we would use the following command: curl -XPOST "http://localhost:9200/wikinews/_search?pretty" -d' { "suggest": { "first_suggestion": { "text": "chrimes in wordl", "term": { "field": "title" } }, "second_suggestion": { "text": "arest", "term": { "field": "text" } } } }' Understanding the suggester response Let's now look at the example response for the suggestion query we have executed. Although the response will differ for each suggester type, let's look at the response returned by Elasticsearch for the first command we've sent in the preceding code that used the term suggester: { "took" : 5, "timed_out" : false, "_shards" : { "total" : 5, "successful" : 5, "failed" : 0 }, "hits" : { "total" : 0, "max_score" : 0.0, "hits" : [ ] }, "suggest" : { "first_suggestion" : [ { "text" : "chrimes", "offset" : 0, "length" : 7, "options" : [ { "text" : "crimes", "score" : 0.8333333, "freq" : 36 }, { "text" : "choices", "score" : 0.71428573, "freq" : 2 }, { "text" : "chrome", "score" : 0.6666666, "freq" : 2 }, { "text" : "chimps", "score" : 0.6666666, "freq" : 1 }, { "text" : "crimea", "score" : 0.6666666, "freq" : 1 } ] }, { "text" : "in", "offset" : 8, "length" : 2, "options" : [ ] }, { "text" : "wordl", "offset" : 11, "length" : 5, "options" : [ { "text" : "world", "score" : 0.8, "freq" : 436 }, { "text" : "words", "score" : 0.8, "freq" : 6 }, { "text" : "word", "score" : 0.75, "freq" : 9 }, { "text" : "worth", "score" : 0.6, "freq" : 21 }, { "text" : "worst", "score" : 0.6, "freq" : 16 } ] } ] } } As you can see in the preceding response, the term suggester returns a list of possible suggestions for each term that was present in the text parameter of our first_suggestion section. For each term, the term suggester will return an array of possible suggestions with additional information. Looking at the data returned for the wordl term, we can see the original word (the text parameter), its offset in the original text parameter (the offset parameter), and its length (the length parameter). The options array contains suggestions for the given word and will be empty if Elasticsearch doesn't find any suggestions. Each entry in this array is a suggestion and is characterized by the following properties: text: This is the text of the suggestion. score: This is the suggestion score; the higher the score, the better the suggestion will be. freq: This is the frequency of the suggestion. The frequency represents how many times the word appears in documents in the index we are running the suggestion query against. The higher the frequency, the more documents will have the suggested word in its fields and the higher the chance that the suggestion is the one we are looking for. Please remember that the phrase suggester response will differ from the one returned by the terms suggester, The term suggester The term suggester works on the basis of the edit distance, which means that the suggestion with fewer characters that needs to be changed or removed to make the suggestion look like the original word is the best one. For example, let's take the words worl and work. In order to change the worl term to work, we need to change the l letter to k, so it means a distance of one. Of course, the text provided to the suggester is analyzed and then terms are chosen to be suggested. The phrase suggester The term suggester provides a great way to correct user spelling mistakes on a per-term basis. However, if we would like to get back phrases, it is not possible to do that when using this suggester. This is why the phrase suggester was introduced. It is built on top of the term suggester and adds additional phrase calculation logic to it so that whole phrases can be returned instead of individual terms. It uses N-gram based language models to calculate how good the suggestion is and will probably be a better choice to suggest whole phrases instead of the term suggester. The N-gram approach divides terms in the index into grams—word fragments built of one or more letters. For example, if we would like to divide the word mastering into bi-grams (a two letter N-gram), it would look like this: ma as st te er ri in ng. The completion suggester Till now we read about term suggester and phrase suggester which are used for providing suggestions but completion suggester is completely different and it is used for as a prefix-based suggester for allowing us to create the autocomplete (search as you type) functionality in a very performance-effective way because of storing complicated structures in the index instead of calculating them during query time. This suggester is not about correcting user spelling mistakes. In Elasticsearch 5.0, Completion suggester has gone through complete rewrite. Both the syntax and data structures of completion type field have been changed and so is the response structure. There are many new exciting features and speed optimizations have been introduced in the completion suggester. One of these features is making completion suggester near real time which allows deleted suggestions to omit from suggestion results as soon as they are deleted. The logic behind the completion suggester The prefix suggester is based on the data structure called Finite State Transducer (FST) ( For more information refer, http://en.wikipedia.org/wiki/Finite_state_transducer). Although it is highly efficient, it may require significant resources to build on systems with large amounts of data in them: systems that Elasticsearch is perfectly suitable for. If we would like to build such a structure on the nodes after each restart or cluster state change, we may lose performance. Because of this, the Elasticsearch creators decided to use an FST-like structure during index time and store it in the index so that it can be loaded into the memory when needed. Using the completion suggester To use a prefix-based suggester we need to properly index our data with a dedicated field type called completion. It stores the FST-like structure in the index. In order to illustrate how to use this suggester, let's assume that we want to create an autocomplete feature to allow us to show book authors, which we store in an additional index. In addition to author's names, we want to return the identifiers of the books they wrote in order to search for them with an additional query. We start with creating the authors index by running the following command: curl -XPUT "http://localhost:9200/authors" -d' { "mappings": { "author": { "properties": { "name": { "type": "keyword" }, "suggest": { "type": "completion" } } } } }' Our index will contain a single type called author. Each document will have two fields: the name field, which is the name of the author, and the suggest field, which is the field we will use for autocomplete. The suggest field is the one we are interested in; we've defined it using the completion type, which will result in storing the FST-like structure in the index. Implementing your own autocompletion Completion suggester has been designed to be a powerful and easily implemented solution for autocomplete but it supports only prefix query. Most of the time autocomplete need only work as a prefix query for example, If I type elastic then I expect elasticsearch as a suggestion, not nonelastic. There are some use cases, when one wants to implement more general partial word completion. Completion suggester fails to fulfill this requirement. The second limitation of completion suggester is, it does not allow advance queries and filters searched. To get rid of both these limitations we are going to implement a custom autocomplete feature based on N-gram, which works in almost all the scenarios. Creating index Lets create an index location-suggestion with following settings and mappings: curl -XPUT "http://localhost:9200/location-suggestion" -d' { "settings": { "index": { "analysis": { "filter": { "nGram_filter": { "token_chars": [ "letter", "digit", "punctuation", "symbol", "whitespace" ], "min_gram": "2", "type": "nGram", "max_gram": "20" } }, "analyzer": { "nGram_analyzer": { "filter": [ "lowercase", "asciifolding", "nGram_filter" ], "type": "custom", "tokenizer": "whitespace" }, "whitespace_analyzer": { "filter": [ "lowercase", "asciifolding" ], "type": "custom", "tokenizer": "whitespace" } } } } }, "mappings": { "locations": { "properties": { "name": { "type": "text", "analyzer": "nGram_analyzer", "search_analyzer": "whitespace_analyzer" }, "country": { "type": "keyword" } } } } }' Understanding the parameters If you look carefully in preceding curl request for creating the index, it contains both settings and the mappings. We will see them now in detail one by one. Configuring settings Our settings contains two custom analyzers: nGram_analyzer and whitespace_analyzer. We have made custom whitespace_analyzer using whitespace tokenizer just for making due that all the tokens are indexed in lowercase and ascifolded form. Our main interest is nGram_analyzer, which contains a custom filter nGram_filter consisting following parameters: type: Specifies type of token filters which is nGram in our case. token_chars: Specifies what kind of characters are allowed in the generated tokens. Punctuations and special characters are generally removed from the token streams but in our example, we have intended to keep them. We have kept whitespace also so that a text which contains United States and a user searches for u s, United States still appears in the suggestion. min_gram and max_gram: These two attributes set the minimum and maximum length of substrings that will generated and added to the lookup table. For example, according to our settings for the index, the token India will generate following tokens: [ "di", "dia", "ia", "in", "ind", "indi", "india", "nd", "ndi", "ndia" ] Configuring mappings The document type of our index is locations and it has two fields, name and country. The most important thing to see is the way analyzers has been defined for name field which will be used for autosuggestion. For this field we have set index analyzer to our custom nGram_analyzer where the search analyzer is set to whitespace_analyzer. The index_analyzer parameter is no more supported from Elasticsearch version 5.0 onward. Also, if you want to configure search_analyzer property for a field, then you must configure analyzer property too the way we have shown in the preceding example. Summary In this article we focused on improving user search experience. We started with term and phrase suggesters and then covered search as you type that is, autocompletion feature which is implemented using completion suggester. We also saw the limitations of completion suggester in handling advanced queries and partial matching which further solved by implementing our custom completion using N-gram. Resources for Article: Further resources on this subject: Searching Your Data [article] Understanding Mesos Internals [article] Big Data Analysis (R and Hadoop) [article]
Read more
  • 0
  • 0
  • 2764

article-image-introducing-algorithm-design-paradigms
Packt
18 Nov 2016
10 min read
Save for later

Introducing Algorithm Design Paradigms

Packt
18 Nov 2016
10 min read
In this article by David Julian and Benjamin Baka, author of the book Python Data Structures and Algorithm, we will discern three broad approaches to algorithm design. They are as follows: Divide and conquer Greedy algorithms Dynamic programming   (For more resources related to this topic, see here.) As the name suggests, the divide and conquer paradigm involves breaking a problem into smaller subproblems, and then in some way combining the results to obtain a global solution. This is a very common and natural problem solving technique and is, arguably, the most used approach to algorithm design. Greedy algorithms often involve optimization and combinatorial problems; the classic example is applying it to the traveling salesperson problem, where a greedy approach always chooses the closest destination first. This shortest path strategy involves finding the best solution to a local problem in the hope that this will lead to a global solution. The dynamic programming approach is useful when our subproblems overlap. This is different from divide and conquer. Rather than breaking our problem into independent subproblems, with dynamic programming, intermediate results are cached and can be used in subsequent operations. Like divide and conquer, it uses recursion. However, dynamic programing allows us to compare results at different stages. This can have a performance advantage over divide and conquer for some problems because it is often quicker to retrieve a previously calculated result from memory rather than having to recalculate it. Recursion and backtracking Recursion is particularly useful for divide and conquer problems; however, it can be difficult to understand exactly what is happening, since each recursive call is itself spinning off other recursive calls. At the core of a recursive function are two types of cases. Base cases, which tell the recursion when to terminate and recursive cases that call the function they are in. A simple problem that naturally lends itself to a recursive solution is calculating factorials. The recursive factorial algorithm defines two cases—the base case, when n is zero, and the recursive case, when n is greater than zero. A typical implementation is shown in the following code: def factorial(n): #test for a base case if n==0: return 1 # make a calculation and a recursive call f= n*factorial(n-1) print(f) return(f) factorial(4) This code prints out the digits 1, 2, 4, 24. To calculate 4!, we require four recursive calls plus the initial parent call. On each recursion, a copy of the methods variables is stored in memory. Once the method returns, it is removed from memory. Here is a way to visualize this process: It may not necessarily be clear if recursion or iteration is a better solution to a particular problem, after all, they both repeat a series of operations and both are very well suited to divide and conquer approaches to algorithm design. An iteration churns away until the problem is done. Recursion breaks the problem down into smaller chunks and then combines the results. Iteration is often easier for programmers because the control stays local to a loop, whereas recursion can more closely represent mathematical concepts such as factorials. Recursive calls are stored in memory, whereas iterations are not. This creates a tradeoff between processor cycles and memory usage, so choosing which one to use may depend on whether the task is processor or memory intensive. The following table outlines the key differences between recursion and iteration. Recursion Iteration Terminates when a base case is reached Terminates when a defined condition is met Each recursive call requires space in memory Each iteration is not stored in memory An infinite recursion results in a stack overflow error An infinite iteration will run while the hardware is powered Some problems are naturally better suited to recursive solutions Iterative solutions may not always be obvious Backtracking Backtracking is a form of recursion that is particularly useful for types of problems such as traversing tree structures where we are presented with a number of options at each node, from which we must choose one. Subsequently, we are presented with a different set of options, and depending on the series of choices made, either a goal state or a dead end is reached. If it is the latter, we mast backtrack to a previous node and traverse a different branch. Backtracking is a divide and conquer method for exhaustive search. Importantly, backtracking prunes branches that cannot give a result. An example of back tracking is given by the following. Here, we have used a recursive approach to generating all the possible permutations of a given string, s, of a given length n: def bitStr(n, s): if n == 1: return s return [ digit + bits for digit in bitStr(1,s)for bits in bitStr(n - 1,s)] print (bitStr(3,'abc')) This generates the following output: Note the double list compression and the two recursive calls within this comprehension. This recursively concatenates each element of the initial sequence, returned when n = 1, with each element of the string generated in the previous recursive call. In this sense, it is backtracking to uncover previously ungenerated combinations. The final string that is returned is all n letter combinations of the initial string. Divide and conquer – long multiplication For recursion to be more than just a clever trick, we need to understand how to compare it to other approaches, such as iteration, and to understand when it is use will lead to a faster algorithm. An iterative algorithm that we are all familiar with is the procedure you learned in primary math classes, which was used to multiply two large numbers, that is, long multiplication. If you remember, long multiplication involved iterative multiplying and carry operations followed by a shifting and addition operation. Our aim here is to examine ways to measure how efficient this procedure is and attempt to answer the question, is this the most efficient procedure we can use for multiplying two large numbers together? In the following figure, we can see that multiplying two 4-digit numbers together requires 16 multiplication operations, and we can generalize to say that an n digit number requires, approximately, n2 multiplication operations: This method of analyzing algorithms, in terms of number of computational primitives such as multiplication and addition, is important because it can give a way to understand the relationship between the time it takes to complete a certain computation and the size of the input to that computation. In particular, we want to know what happens when the input, the number of digits, n, is very large. Can we do better? A recursive approach It turns out that in the case of long multiplication, the answer is yes, there are in fact several algorithms for multiplying large numbers that require fewer operations. One of the most well-known alternatives to long multiplication is the Karatsuba algorithm, published in 1962. This takes a fundamentally different approach: rather than iteratively multiplying single digit numbers, it recursively carries out multiplication operation on progressively smaller inputs. Recursive programs call themselves on smaller subset of the input. The first step in building a recursive algorithm is to decompose a large number into several smaller numbers. The most natural way to do this is to simply split the number into halves: the first half comprising the most significant digits and the second half comprising the least significant digits. For example, our four-digit number, 2345, becomes a pair of two digit numbers, 23 and 45. We can write a more general decomposition of any two n-digit numbers x and y using the following, where m is any positive integer less than n. For x-digit number: For y-digit number: So, we can now rewrite our multiplication problem x and y as follows: When we expand and gather like terms we get the following: More conveniently, we can write it like this (equation 1): Here, It should be pointed out that this suggests a recursive approach to multiplying two numbers since this procedure itself involves multiplication. Specifically, the products ac, ad, bc, and bd all involve numbers smaller than the input number, and so it is conceivable that we could apply the same operation as a partial solution to the overall problem. This algorithm, so far consists of four recursive multiplication steps and it is not immediately clear if it will be faster than the classic long multiplication approach. What we have discussed so far in regards to the recursive approach to multiplication was well known to mathematicians since the late 19th century. The Karatsuba algorithm improves on this is by making the following observation. We really only need to know three quantities, z2 = ac, z1=ad +bc, and z0 = bd to solve equation 1. We need to know the values of a, b, c, and d as they contribute to the overall sum and products involved in calculating the quantities z2, z1, and z0. This suggests the possibility that perhaps we can reduce the number of recursive steps. It turns out that this is indeed the situation. Since the products ac and bd are already in their simplest form, it seems unlikely that we can eliminate these calculations. We can, however, make the following observation: When we subtract the quantities ac and bd, which we have calculated in the previous recursive step, we get the quantity we need, namely ad + bc: This shows that we can indeed compute the sum of ad and bc without separately computing each of the individual quantities. In summary, we can improve on equation 1 by reducing from four recursive steps to three. These three steps are as follows: Recursively calculate ac. Recursively calculate bd. Recursively calculate (a +b)(c + d) and subtract ac and bd. The following code shows a Python implementation of the Karatsuba algorithm: from math import log10 def karatsuba(x,y): # The base case for recursion if x < 10 or y < 10: return x*y #sets n, the number of digits in the highest input number n = max(int(log10(x)+1), int(log10(y)+1)) # rounds up n/2 n_2 = int(math.ceil(n / 2.0)) #adds 1 if n is uneven n = n if n % 2 == 0 else n + 1 #splits the input numbers a, b = divmod(x, 10**n_2) c, d = divmod(y, 10**n_2) #applies the three recursive steps ac = karatsuba(a,c) bd = karatsuba(b,d) ad_bc = karatsuba((a+b),(c+d)) - ac - bd #performs the multiplication return (((10**n)*ac) + bd + ((10**n_2)*(ad_bc))) To satisfy ourselves that this does indeed work, we can run the following test function: import random def test(): for i in range(1000): x = random.randint(1,10**5) y = random.randint(1,10**5) expected = x * y result = karatsuba(x, y) if result != expected: return("failed") return('ok') Summary In this article, we looked at a way to recursively multiply large numbers and also a recursive approach for merge sort. We saw how to use backtracking for exhaustive search and generating strings. Resources for Article: Further resources on this subject: Python Data Structures [article] How is Python code organized [article] Algorithm Analysis [article]
Read more
  • 0
  • 0
  • 24738

article-image-create-dynamic-tilemaps-duality-c-part-i
Lőrinc Serfőző
18 Nov 2016
6 min read
Save for later

Create Dynamic Tilemaps in Duality (C#) – Part I

Lőrinc Serfőző
18 Nov 2016
6 min read
Introduction This article guides you through the process of creating tilemaps and changing them at runtime in the Duality game engine. A Few Words about the Duality Game Engine Duality is an open source, Windows-only 2D game engine based on OpenGL and written entirely in C#. Despite being a non-commercial project, it is a rather mature product because it has been around since 2011. It has also proved to be capable of delivering professional games. Obviously, the feature bucket is somewhat limited compared to the major proprietary engines. On the other hand, this fact makes Duality an excellent learning tool, because the user often needs to implement logic on a lower level and also has the opportunity to inspect the source code of the engine. Thus I would recommend this engine to game developers with an intermediate skill level, who'd like to have a take on open source and are not afraid of learning by doing. This article does not describe the inner workings of Duality in detail because these are well-explained on the GitHub wiki of the project. If you are not familiar with the engine, please consider reviewing some of the key concepts described there. The repository containing the finished project is available on GitHub. Required tools First things first: Duality editor only supports Windows at the moment. You will need a C# compiler and a text editor. Visual Studio 2013 or higher is recommended, but Xamarin Studio/MonoDevelop also works. Duality itself is written in C# 4, but this tutorial uses language features introduced in C# 6. Creating a new project and initializing the Tilemap plugin Downloading the Duality binaries The latest binary package of the engine is available on its main site. Unlike most game engines, it does not need a setup process. In fact, every single project is a self-contained application, including the editor. This might seem strange at first, but this architecture makes version control, migration, and having different engine versions for different projects very easy. On top of this, it is also a self-building application. When you unzip the downloaded file, it contains only one executable: DualityEditor.exe. At its first run, it downloads the required packages from the main repository. It is based on the industry-standard NuGet format, and the packages are actually stored on NuGet.org. After accepting the license agreement and proceeding through the automated package install, the following screen should show up: Dualitor The usage of the editor application named Dualitor is intuitive; however, if you are not familiar with it yet, the quick start guide might be worth checking out. Adding the Tilemaps plugin to the project Duality has a modular architecture. Thus, every piece of distinct functionality is organized in modules, or, using the official terminology, plugins. Practically, every plugin is a .NET assembly that Duality loads and uses. There are two types of plugins: core plugins provide in-game functionality and services and are distributed with the game. On the other hand, editor plugins are used by Dualitor, and usually provide some sort of editing or content management functionality which aids the developer in creating the game. Now let's install the Tilemaps plugin. It's done via the Package Management window, which is available from the File menu. The following steps describe the installation process: Select the 'Online Repository' item from the 'View' dropdown. Select the Tilemaps (Editor) entry from the list. Since it depends on the Tilemaps (Core) package, the latter will be installed as well. Click on 'Install'. This downloads the two packages from the repository. Click on 'Apply'. Dualitor restarts and loads the plugins. Installing the Tilemaps plugin Tileset resource and Tilemap GameObject Creating the Tileset resource Resources are the asset holder objects in Duality. They can be simple imported types such as bitmap information, but some, like the Tileset resource we are going to use now, are generated by the editor. It contains information about rendering, collision, and autotiling properties, as well as a reference to the actual pixel data to be rendered. This pixel data is contained in another resource: the Pixmap. Let's start with a very simple tileset that consists of two solid color tiles, sized 32 x 32. Tileset Download this .png file, and drop it in the Project View. A new Pixmap is created. Next, create a new Tileset from the context menu of the Project View: Add Tileset At this point, the Tileset Resource does not know which Pixmap it should use. First, open the Tileset Editor window from the View menu on the menu bar. The same behavior is invoked when the Tileset is double-clicked in the Project View. The following steps describe how to establish this link: Select the newly created Tileset in the Tileset Editor. Add a new Visual Layer by clicking on the little green plus button. Select the 'Main Texture' layer, if it's not already selected. Drag and drop the Pixmap named 'tileset' into the 'SourceData' area. Since the default value of the tile size is already 32 x 32, there is no need to modify that. Click on 'Apply' to recompile the Resource. A quicker way to do this is by just selecting the 'Create Tileset' option from the Pixmap's context menu, but it is always nice to know what happens in the background. Linking the Pixmap and the Tileset resource Instantiating a Tilemap Duality, similar to many engines, uses a Scene-GameObject-Component model of object hierarchy. To display a tilemap in the scene, a GameObject is needed, with two components attached to it: a Tilemap and a TilemapRenderer. The former contains the actual tilemap information, that is, which tile is located at a specific position. Creating this GameObject is very easy; just drag and drop the Tileset Resource from the Project View to the Scene View. Switch back to the Camera #0 window from the Tileset Editor, and you should see a large green rectangle, which is the tilemap itself. Notice that a new GameObject called 'TilemapRenderer' appears in the Scene View, and it has 3 components attached: a Transform, a Tilemap, and a TilemapRenderer. To edit the Tilemap, the Camera #0 window mode has to be set to 'Tilemap Editor'. After doing that, a new window named 'Tile Palette' appears. The Tileset can be edited with multiple tools—feel free to experiment with them! Editing the Tilemap To be continued In the second part of this tutorial, some C# coding is involved. We will create a custom component to change the tilemap dynamically. Author Lőrinc Serfőző is a software engineer at Graphisoft, the company behind the BIM solution ArchiCAD. He is studying mechatronics engineering at the Budapest University of Technology and Economics, an interdisciplinary field between the more traditional mechanical engineering, electrical engineering, and informatics, and has quickly grown a passion toward software development. He is a supporter of open source and contributes to the C# and OpenGL-based Duality game engine, creating free plugins and tools for users.
Read more
  • 0
  • 0
  • 3129
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-r-statistical-package-interfacing-python
Janu Verma
17 Nov 2016
8 min read
Save for later

The R Statistical Package Interfacing with Python

Janu Verma
17 Nov 2016
8 min read
One of my coding hobbies is to explore different Python packages and libraries. In this post, I'll talk about the package rpy2, which is used to call R inside python. Being an avid user of R and a huge supporter of R graphical packages, I had always desired to call R inside my Python code to be able to produce beautiful visualizations. The R framework offers machinery for a variety of statistical and data mining tasks. Let's review the basics of R before we delve into R-Python interfacing. R is a statistical language which is free, is open source, and has comprehensive support for various statistical, data mining, and visualization tasks. Quick-R describes it as: "R is an elegant and comprehensive statistical and graphical programming language." R is one of the fastest growing languages, mainly due to the surge in interest in statistical learning and data science. The Data Science Specialization on Coursera has all courses taught in R. There are R packages for machine learning, graphics, text mining, bioinformatics, topics modeling, interactive visualizations, markdown, and many others. In this post, I'll give a quick introduction to R. The motivation is to acquire some knowledge of R to be able to follow the discussion on R-Python interfacing. Installing R R can be downloaded from one of the Comprehensive R Archive Network (CRAN) mirror sites. Running R To run R interactively on the command line, type r. Launch the standard GUI (which should have been included in the download) and type R code in it. RStudio is the most popular IDE for R. It is recommended, though not required, to install RStudio and run R on it. To write a file with R code, create a file with the .r extension (for example, myFirstCode.r). And run the code by typing the following on the terminal: Rscript file.r Basics of R The most fundamental data structure in R is a vector; actually everything in R is a vector (even numbers are 1-dimensional vectors). This is one of the strangest things about R. Vectors contain elements of the same type. A vector is created by using the c() function. a = c(1,2,5,9,11) a [1] 1 2 5 9 11 strings = c("aa", "apple", "beta", "down") strings [1] "aa" "apple" "beta" "down" The elements in a vector are indexed, but the indexing starts at 1 instead of 0, as in most major languages (for example, python). strings[1] [1] "aa" The fact that everything in R is a vector and that the indexing starts at 1 are the main reasons for people's initial frustration with R (I forget this all the time). Data Frames A lot of R packages expect data as a data frame, which are essentially matrices but the columns can be accessed by names. The columns can be of different types. Data frames are useful outside of R also. The Python package Pandas was written primarily to implement data frames and to do analysis on them. In R, data frames are created (from vectors) as follows: students = c("Anne", "Bret", "Carl", "Daron", "Emily") scores = c(7,3,4,9,8) grades = c('B', 'D', 'C', 'A', 'A') results = data.frame(students, scores, grades) results students scores grades 1 Anne 7 B 2 Bret 3 D 3 Carl 4 C 4 Daron 9 A 5 Emily 8 A The elements of a data frame can be accessed as: results$students [1] Anne Bret Carl Daron Emily Levels: Anne Bret Carl Daron Emily This gives a vector, the elements of which can be called by indexing. results$students[1] [1] Anne Levels: Anne Bret Carl Daron Emily Reading Files Most of the times the data is given as a comma-separated values (csv) file or a tab-separated values (tsv) file. We will see how to read a csv/tsv file in R and create a data frame from it. (Aside: The datasets in most Kaggle competitions are given as csv files and we are required to do machine learning on them. In Python, one creates a pandas data frame or a numpy array from this csv file.) In R, we use a read.csv or read.table command to load a csv file into memory, for example, for the Titanic competition on Kaggle: training_data <- read.csv("train.csv", header=TRUE) train <- data.frame(survived=train_all$Survived, age=train_all$Age, fare=train_all$Fare, pclass=train_all$Pclass) Similarly, a tsv file can be loaded as: data <- read.csv("file.tsv";, header=TRUE, delimiter="t") Thus given a csv/tsv file with or without headers, we can read it using the read.csv function and create a data frame using: data.frame(vector_1, vector_2, ... vector_n). This should be enough to start exploring R packages. Another command that is very useful in R is head(), which is similar to the less command on Unix. rpy2 First things first, we need to have both Python and R installed. Then install rpy2 from the Python package index (Pypi). To do this, simply type the following on the command line: pip install rpy2 We will use the high-level interface to R, the robjects subpackage of rpy2. import rpy2.robjects as ro We can pass commands to the R session by putting the R commands in the ro.r() method as strings. Recall that everything in R is a vector. Let's create a vector using robjects: ro.r('x=c(2,4,6,8)') print(ro.r('x')) [1] 2 4 6 8 Keep in mind that though x is an R object (vector), ro.r('x') is a Python object (rpy2 object). This can be checked as follows: type(ro.r('x')) <class 'rpy2.robjects.vectors.FloatVector'> The most important data types in R are data frames, which are essentially matrices. We can create a data frame using rpy2: ro.r('x=c(2,4,6,8)') ro.r('y=c(4,8,12,16)') ro.r('rdf=data.frame(x,y)') This created an R data frame, rdf. If we want to manipulate this data frame using Python, we need to convert it to a python object. We will convert the R data frame to a pandas data frame. The Python package pandas contains efficient implementations of data frame objects in python. import pandas.rpy.common as com df = com.load_data('rdf') print type(df) <class 'pandas.core.frame.DataFrame'> df.x = 2*df.x Here we have doubled each of the elements of the x vector in the data frame df. But df is a Python object, which we can convert back to an R data frame using pandas as: rdf = com.convert_to_r_dataframe(df) print type(rdf) <class 'rpy2.robjects.vectors.DataFrame'> Let's use the plotting machinery of R, which is the main purpose of studying rpy2: ro.r('plot(x,y)') Not only R data types, but rpy2 lets us import R packages as well (given that these packages are installed on R) and use them for analysis. Here we will build a linear model on x and y using the R package stats: from rpy2.robjects.packages import importr stats = importr('stats') base = importr('base') fit = stats.lm('y ~ x', data=rdf) print(base.summary(fit)) We get the following results: Residuals: 1 2 3 4 0 0 0 0 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 0 0 NA NA x 2 0 Inf <2e-16 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 0 on 2 degrees of freedom Multiple R-squared: 1, Adjusted R-squared: 1 F-statistic: Inf on 1 and 2 DF, p-value: < 2.2e-16 R programmers will immediately recognize the output as coming from applying linear model function lm() on data. I'll end this discussion with an example using my favorite R package ggplot2. I have written a lot of posts on data visualization using ggplot2. The following example is borrowed from the official documentation of rpy2. import math, datetime import rpy2.robjects.lib.ggplot2 as ggplot2 import rpy2.robjects as ro from rpy2.robjects.packages import importr base = importr('base') datasets = importr('datasets') mtcars = datasets.data.fetch('mtcars')['mtcars'] pp = ggplot2.ggplot(mtcars) + ggplot2.aes_string(x='wt', y='mpg', col='factor(cyl)') + ggplot2.geom_point() + ggplot2.geom_smooth(ggplot2.aes_string(group = 'cyl'), method = 'lm') pp.plot() Author: Janu Verma is a researcher in the IBM T.J. Watson Research Center, New York. His research interests are in mathematics, machine learning, information visualization, computational biology, and healthcare analytics. He has held research positions at Cornell University, Kansas State University, Tata Institute of Fundamental Research, Indian Institute of Science, and the Indian Statistical Institute. He has written papers for IEEE Vis, KDD, International Conference on HealthCare Informatics, Computer Graphics and Applications, Nature Genetics, IEEE Sensors Journals and so on. His current focus is on the development of visual analytics systems for prediction and understanding. He advises start-ups and other companies on data science and machine learning in the Delhi-NCR area. He can be found at Here.
Read more
  • 0
  • 0
  • 16961

article-image-how-build-facebook-messenger-bot
Amit Kothari
17 Nov 2016
9 min read
Save for later

How to build a Facebook Messenger Bot

Amit Kothari
17 Nov 2016
9 min read
One of the trending topics in the tech world this year is conversational interfaces. Everyone from big tech companies to start-ups are putting their bets on it. There is an explosion of virtual assistants and chat bots, and the number is increasing every week. What is conversational interface and why are companies investing in it? In this post we will learn about it and build a simple bot using the Facebook Messenger Platform. Conversational Interface A conversational interface is an interface that allows the users to interact with computers using natural language, instead of commands or clicks and taps. It can be voice-based like Siri, Ok Google, or Microsoft's Cortana, or it can be text-based like many Slack or Facebook Messenger bots. Instead of building apps for different platforms, device types, and sizes, companies can now provide a way to interact with their services using voice or text, without the need to download an app or learn how to use it. Facebook Messenger Bot Many apps like Slack, Telegraph, and Skype now offer platforms to build virtual assistants or bots. But with more than 1 billion users, Facebook Messenger is definitely the most popular choice. A Facebook Messenger bot works in the following way: The user sends a message to the bot using their Facebook Messenger account. The Facebook Messenger Platform will receive and post the message to the bot's server using a webhook. The bot server will receive the message, process it, and send a response back to the Facebook Messenger Platform using HTTP POST. Facebook will forward the message to the user. Let's build a simple Facebook Messenger Bot. Creating a Facebook App and Page First we need to create a Facebook Page and a Facebook app: Create a Facebook account, if you do not have one. Create a Facebook Page. This page will work as the identity of your bot. Create a Facebook App. Enter a display name and contact e-mail, and select a category for your bot. After creating the app, click on 'Add product' and then select the 'Messenger' option. Click on the 'Get started' button, and under the 'Token Generation' section, select the page you just created. This will generate a Page Access Token, which the bot server will need to communicate with the Facebook Messenger Platform. Copy this token for later use. You also need to set up webhook so that Facebook can forward the message to the bot server, but before that, let's build the server. Building the Bot Server You can use any language or framework to build the bot server. For this example, I am using Node.js. Let's start by creating an Express.js hello world app. Before we start, we need Node.js and npm installed. Follow the instructions on the Node.js website if you do not have these installed already: Create a new directory for your application, and inside of the app directory, create the package.json file by using the npm init command. Install Express and add it as a dependency using the npm install --save express command. Create a file called index.js with the following code: const app = require('express')(); app.set('port', (process.env.PORT || 5000)); app.get('/', function (req, res) { res.send('Helle World'); }); app.listen(app.get('port'), function () { console.log('App running on port', app.get('port')) }); Our hello world app is done. Run the node index.js command and open http://localhost:5000 in your browser. If all is good, you will see 'Hello World' text on the page. Now that we have the basic app set up, let's add the code to be able to communicate with the Facebook Messenger Platform: Install body-parser and request npm modules using the npm install --save body-parser request command. Body-parser is a middleware to parse incoming request bodies and request is an HTTP client. Now update index.js with the following code. Replace <fb-page-access-token> with the page access token generated before. You will need the validation token while setting up the webhook. Feel free to change it to anything you like: 'use strict' const app = require('express')(); const bodyParser = require('body-parser'); const request = require('request'); const VALIDATION_TOKEN = 'super_secret_validation_token'; const PAGE_ACCESS_TOKEN = '<fb-page-access-token>'; // replace <fb-page-access-token> with your page access token app.set('port', (process.env.PORT || 5000)); app.use(bodyParser.urlencoded({ extended: false })); app.use(bodyParser.json()); app.get('/', function (req, res) { res.send('Helle World'); }); app.get('/webhook', function (req, res) { if (req.query['hub.mode'] === 'subscribe' && req.query['hub.verify_token'] === VALIDATION_TOKEN) { res.status(200).send(req.query['hub.challenge']); } else { res.sendStatus(403); } }); app.post('/webhook/', function (req, res) { // Iterate over each entry req.body.entry.forEach(function (entry) { // Iterate over each messaging event entry.messaging.forEach(function (event) { let sender = event.sender.id if (event.message && event.message.text) { callSendAPI(sender, `Message received ${event.message.text}`); } }); res.sendStatus(200); // Respond with 200 to notify the message is received by the webhook successfully }); }); function callSendAPI(sender, text) { request({ url: 'https://graph.facebook.com/v2.6/me/messages', qs: { access_token: PAGE_ACCESS_TOKEN }, method: 'POST', json: { recipient: { id: sender }, message: { text: text }, } }, function (error, response, body) { if (error) { console.error('Unable to send message', error); } }); } app.listen(app.get('port'), function () { console.log('App running on port', app.get('port')) }); Here we have defined two new service endpoints. The first one responds to the GET request and will be used for verification when we set up the webhook. The second endpoint will respond to the POST request. This will process the incoming message and send a response using the callSendAPI method. For now, the server will simply send back the message received from the user. You can deploy the server anywhere you like. Because of its simplicity, I am going to use Heroku for this tutorial. Deploying the App to Heroku If you do not have an account, sign up on heroku.com and then install Heroku CLI. Initialize a git repository and commit the code to easily push it to Heroku. Use git init to initialise and git add . followed by git commit -m "Initial commit" to commit the code. Create a Procfile file with the following content. Procfile declares what commands to run on heroku. web: node index.js. Run the heroku create command to create and set up a new app on Heroku. Now you can run the app locally using the heroku local command. To deploy the app on Heroku, use git push heroku master and use heroku open to open the deployed app in the browser. Setting up the Messenger Webhook Now that the server is ready, we can set up the webhook: Go back to the app page; under the Messenger settings page, click on the 'Setup Webhooks' button. Enter the bot server webhook url (https:///webhook), and the validation token, and select all the options under subscription fields. We are ready to test our bot. Go to the Facebook page you created, click on message, and send a message to the bot. If everything is good, you will see a response from the bot. The bot is working, but we can update it to send a different response based on the user message. Let's create a simple bot for Game of Thrones fans, where if a user types in one of the great house's names, the bot will respond with their house words. Update index.js with the following code: 'use strict' const app = require('express')(); const bodyParser = require('body-parser'); const request = require('request'); const VALIDATION_TOKEN = 'super_secret_validation_token'; const PAGE_ACCESS_TOKEN = '<fb-page-access-token>'; // replace <fb-page-access-token> with your page access token const HOUSE_WORDS = { STARK: 'Winter is Coming', LANNISTER: 'Hear Me Roar!', BARATHEON: 'Ours is the Fury', TULLY: 'Family, Duty, Honor', GREYJOY: 'We Do Not Sow', MARTELL: 'Unbowed, Unbent, Unbroken', TYRELL: 'Growing Strong', FREY: 'We Stand Together', } app.set('port', (process.env.PORT || 5000)); app.use(bodyParser.urlencoded({ extended: false })); app.use(bodyParser.json()); app.get('/', function (req, res) { res.send('Helle World'); }); app.get('/webhook', function (req, res) { if (req.query['hub.mode'] === 'subscribe' && req.query['hub.verify_token'] === VALIDATION_TOKEN) { res.status(200).send(req.query['hub.challenge']); } else { res.sendStatus(403); } }); app.post('/webhook/', function (req, res) { // Iterate over each entry req.body.entry.forEach(function (entry) { // Iterate over each messaging event entry.messaging.forEach(function (event) { let sender = event.sender.id if (event.message && event.message.text) { callSendAPI(sender, getHouseWord(event.message.text)); } }); res.sendStatus(200); // Respond with 200 to notify the message is received by the webhook successfully }); }); function callSendAPI(sender, text) { request({ url: 'https://graph.facebook.com/v2.6/me/messages', qs: { access_token: PAGE_ACCESS_TOKEN }, method: 'POST', json: { recipient: { id: sender }, message: { text: text }, } }, function (error, response, body) { if (error) { console.error('Unable to send message', error); } }); } function getHouseWord(text) { const house = Object.keys(HOUSE_WORDS) .find(function (houseName) { return text.toUpperCase().indexOf(houseName) !== -1; }); return house ? HOUSE_WORDS[house] : 'No house word found :('; } app.listen(app.get('port'), function () { console.log('App running on port', app.get('port')) }); Now if you type one of the family names defined in our code, the bot will respond with the family words. So if you type 'stark', the bot will respond 'Winter is Coming'. App Review While you and other page admins can chat with the bot, to make it publicly available, you have to submit the bot for review. This can be done from the app page's 'App Review' option. Fill out the form, and once approved, your bot will be accessible by all Facebook Messenger users. Although our bot is working, it is not truly conversational. It only responds to a fix set of commands. To build a smarter bot, we need natural language processing. Natural language processors convert natural language text or audio to a format that computers can understand. There are many services that allow us to write smarter bot engines, and one of them is Wit. Wit was acquired by Facebook in 2015. You can read more about Wit and how to integrate it with your bot here. I hope you enjoyed this post. If you have built any bot on Facebook or any other platform, please share it with us in the comment section below. Author: Amit Kothari is a full-stack software developer based in Melbourne, Australia. He has 10+ years experience in designing and implementing software, mainly in Java/JEE. His recent experience is in building web applications using JavaScript frameworks like React and AngularJS, and backend micro services / REST API in Java. He is passionate about lean software development and continuous delivery.
Read more
  • 0
  • 0
  • 5500

article-image-setting-development-environment-android-wear-applications
Packt
16 Nov 2016
8 min read
Save for later

Setting up Development Environment for Android Wear Applications

Packt
16 Nov 2016
8 min read
"Give me six hours to chop down a tree and I will spend the first four sharpening the axe." -Abraham Lincoln In this article by Siddique Hameed, author of the book, Mastering Android Wear Application Development, they have discussed the steps, topics and process involved in setting up the development environment using Android Studio. If you have been doing Android application development using Android Studio, some of the items discussed here might already be familiar to you. However, there are some Android Wear platform specific items that may be of interest to you. (For more resources related to this topic, see here.) Android Studio Android Studio Integrated Development Environment (IDE) is based on IntelliJ IDEA platform. If you had done Java development using IntelliJ IDEA platform, you'll be feeling at home working with Android Studio IDE. Android Studio platform comes bundled with all the necessary tools and libraries needed for Android application development. If this is the first time you are setting up Android Studio on your development system, please make sure that you have satisfied all the requirements before installation. Please refer developer site (http://developer.android.com/sdk/index.html#Requirements) for checking on the items needed for the operating system of your choice. Please note that you need at least JDK version 7 installed on your machine for Android Studio to work. You can verify your JDK's version that by typing following commands shown in the following terminal window: If your system does not meet that requirement, please upgrade it using the method that is specific to your operating system. Installation Android Studio platform includes Android Studio IDE, SDK Tools, Google API Libraries and systems images needed for Android application development. Visit the http://developer.android.com/sdk/index.html URL for downloading Android Studio for your corresponding operating system and following the installation instruction. Git and GitHub Git is a distributed version control system that is used widely for open-source projects. We'll be using Git for sample code and sample projects as we go along the way. Please make sure that you have Git installed on your system by typing the command as shown in the following a terminal window. If you don't have it installed, please download and install it using this link for your corresponding operating system by visting https://git-scm.com/downloads. If you are working on Apple's Macintosh OS like El Capitan or Yosemite or Linux distributions like Ubuntu, Kubuntu or Mint, chances are you already have Git installed on it. GitHub (http://github.com) is a free and popular hosting service for Git based open-source projects. They make checking out and contributing to open-source projects easier than ever. Sign up with GitHub for a free account if you don't have an account already. We don't need to be an expert on Git for doing Android application development. But, we do need to be familiar with basic usages of Git commands for working with the project. Android Studio comes by default with Git and GitHub integration. It helps importing sample code from Google's GitHub repository and helps you learn by checking out various application code samples. Gradle Android application development uses Gradle (http://gradle.org/)as the build system. It is used to build, test, run and package the apps for running and testing Android applications. Gradle is declarative and uses convention over configuration for build settings and configurations. It manages all the library dependencies for compiling and building the code artifacts. Fortunately, Android Studio abstracts most of the common Gradle tasks and operations needed for development. However, there may be some cases where having some extra knowledge on Gradle would be very helpful. We won't be digging into Gradle now, we'll be discussing about it as and when needed during the course of our journey. Android SDK packages When you install Android Studio, it doesn't include all the Android SDK packages that are needed for development. The Android SDK separates tools, platforms and other components & libraries into packages that can be downloaded, as needed using the Android SDK Manager. Before we start creating an application, we need to add some required packages into the Android SDK. Launch SDK Manager from Android Studio Tools | Android | SDK Manager. Let's quickly go over a few items from what's displayed in the preceding screenshot. As you can see, the Android SDK's location is /opt/android-sdk on my machine, it may very well be different on your machine depending on what you selected during Android Studio installation setup. The important point to note is that the Android SDK is installed on a different location than the Android Studio's path (/Applications/Android Studio.app/). This is considered a good practice because the Android SDK installation can be unaffected depending on a new installation or upgrade of Android Studio or vice versa. On the SDK Platforms tab, select some recent Android SDK versions like Android 6.0, 5.1.1, and 5.0.1. Depending on the Android versions you are planning on supporting in your wearable apps, you can select other older Android versions. Checking on Show Package Details option on the bottom right, the SDK Manager will show all the packages that will be installed for a given Android SDK version. To be on the safer side, select all the packages. As you may have noticed already Android Wear ARM and Intel system images are included in the package selection. Now when you click on SDK Tools tab, please make sure the following items are selected: Android SDK Build Tools Android SDK Tools 24.4.1 (Latest version) Android SDK Platform-Tools Android Support Repository, rev 25 (Latest version) Android Support Library, rev 23.1.1 (Latest version) Google Play services, rev 29 (Latest version) Google Repository, rev 24 (Latest version) Intel X86 Emulator Accelerator (HAXM installer), rev 6.0.1 (Latest version) Documentation for Android SDK (Optional) Please do not change anything on SDK Update Sites. Keep the update sites as it was configured by default. Clicking on OK will take some time downloading and installing all the components and packages selected. Android Virtual Device Android Virtual Devices will enable us to test the code using Android Emulators. It lets us pick and choose various Android system target versions and form factors needed for testing. Launch Android Virtual Device Manager from Tools | Android | AVD Manager From Android Virtual Device Manager window, click on Create New Virtual Device button on the bottom left and proceed to the next screen and select Wear category Select Marshmallow API Level 23 on x86 and everything else as default, as shown in the following screenshot: Note that the current latest Android version is Marshmallow of API level 23 at the time of this writing. It may or may not be the latest version while you are reading this article. Feel free to select the latest version that is available during that time. Also, if you'd like to support or test in earlier Android versions, please feel free to do so in that screen. After the virtual device is selected successfully, you should see that listed on the Android Virtual Devices list as show in the following screenshot: Although it's not required to use real Android Wear device during development, sometimes it may be convenient and faster developing it in a real physical device. Let's build a skeleton App Since we have all the components and configurations needed for building wearable app, let's build a skeleton app and test out what we have so far. From Android Studio's Quick Start menu, click on Import an Android code sample tab: Select the Skeleton Wearable App from Wearable category as shown in following screenshot: Click Next and select your preferred project location. As you can see the skeleton project is cloned from Google's sample code repository from GitHub. Clicking on Finish button will pull the source code and Android Studio will compile and build the code and get it ready for execution. The following screenshot indicates that the Gradle build finished successfully without any errors. Click on Run configuration to run the app: When the app starts running, Android Studio will prompt us to select the deployment targets, we can select the emulator we created earlier and click OK. After the code compiles and uploaded to the emulator, the main activity of the Skeleton App will be launched as shown below: Clicking on SHOW NOTIFICATION will show the notification as below: Clicking on START TIMER will start the timer and run for five seconds and clicking on Finish Activity will close the activity take the emulator to the home screen. Summary We discussed the process involved in setting up the Android Studio development environment by covering the installation instruction, requirements and SDK tools, packages and other components needed for Android Wear development. We also checked out source code for Skeleton Wearable App from Google's sample code repository and successfully ran and tested it on Android device emulator. Resources for Article: Further resources on this subject: Building your first Android Wear Application [Article] Getting started with Android Development [Article] The Art of Android Development Using Android Studio [Article]
Read more
  • 0
  • 0
  • 7841

article-image-commonly-used-data-structures
Packt
16 Nov 2016
23 min read
Save for later

Commonly Used Data Structures

Packt
16 Nov 2016
23 min read
In this article by Erik Azar and Mario Eguiluz Alebicto, authors of the book Swift Data Structure and Algorithms, we will learn that the Swift language is truly powerful, but a powerful language is nothing if it doesn't have a powerful standard library to accompany it. The Swift standard library defines a base layer of functionality that you can use for writing your applications, including fundamental data types, collection types, functions and methods, and a large number of protocols. (For more resources related to this topic, see here.) We're going to take a close look at the Swift standard library, specifically support for collection types, with a very low level examination of Arrays, Dictionaries, Sets, and Tuples. The topics covered in this article are: Using the Swift standard library Implementing subscripting Understanding immutability Interoperability between Swift and Objective-C Swift Protocol-Oriented Programming Using the Swift standard library Users often treat a standard library as part of the language. In reality, philosophies on standard library design very widely, often with opposing views. For example, the C and C++ standard libraries are relatively small, containing only the functionality that "every developer" might reasonably require to develop an application. Conversely, languages such as Python, Java, and .NET have large standard libraries that include features, which tend to be separate in other languages, such as XML, JSON, localization, and e-mail processing. In the Swift programming language, the Swift standard library is separate from the language itself, and is a collection of classes, structures, enumerations, functions, and protocols, which are written in the core language. The Swift standard library is currently very small, even compared to C and C++. It provides a base layer of functionality through a series of generic structures and enums, which also adopt various protocols that are also defined in the library. You'll use these as the building blocks for applications that you develop. The Swift standard library also places specific performance characteristics on functions and methods in the library. These characteristics are guaranteed only when all participating protocols performance expectations are satisfied. An example is the Array.append(), its algorithm complexity is amortized O(1), unless the self's storage is shared with another live array; O(count) if self does not wrap a bridged NSArray; otherwise the efficiency is unspecified. We're going to take a detailed look at the implementations for Arrays, Dictionaries, Sets, and Tuples. You'll want to make sure you have at least Xcode 8.1 installed to work with code in this section Why Structures? If you're coming from a background working in languages such as Objective-C, C++, Java, Ruby, Python, or other object-oriented languages, you traditionally use classes to define the structure of your types. This is not the case in the Swift standard library; structures are used when defining the majority of the types in the library. If you're coming from an Objective-C or C++ background this might seem especially odd and feel wrong, because classes are much more powerful than structures. So why does Swift use structures, which are value types, when it also supports classes, which are reference types, that support inheritance, deinitializers, and reference counting? It's exactly because of the limited functionality of structures over classes why Swift uses structures instead of classes for the building blocks of the standard library. Because structures are value types it means they can have only one owner and are always copied when assigned to a new variable or passed to a function. This can make your code inherently safer because changes to the structure will not affect other parts of your application. The preceding description refers to the "copying" of value types. The behavior you see in your code will always be as if a copy took place. However, Swift only performs an actual copy behind the scenes when it is absolutely necessary to do so. Swift manages all value copying to ensure optimal performance, and you should not avoid assignment to try to preempt this optimization. Structures in Swift are far more powerful than in other C-based languages, they are very similar to classes. Swift structures support the same basic features as C-based structures, but Swift also adds support, which makes them feel more like classes. Features of Swift structures: In addition to an automatically generated memberwise initializer, they can have custom initializers Can have methods Can implement protocols So this may leave you asking, when should I use a class over a structure? Apple has published guidelines you can follow when considering to create a structure. If one or more of the following conditions apply, consider creating a structure: Its primary purpose is to encapsulate a few simple data values You expect the encapsulated values to be copied rather than referenced when you pass around or assign an instance of the structure Any properties stored by the structure are value types, which would be expected to be copied instead of referenced The structure doesn't need to inherit properties or behavior from another existing type In all other cases, create a class which will call instances of that class to be passed by the reference. Apple Guidelines for choosing between classes and structures: https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/ClassesAndStructures.html Declaring arrays in Swift An array stores values of the same type in an ordered list. Arrays in Swift have a few important differences between arrays in Objective-C. The first is elements in Swift arrays must be of the same type. If you're used to working with Objective-C NSArrays prior to Xcode 7 you may feel this is a disadvantage, but it actually is a significant advantage as it allows you to know exactly what type you get back when working with an array, allowing you to write more efficient code that leads to less defects. However, if you find you must have dissimilar types in an array, you can define the array to be of a protocol that is common to the other types, or defines the array of type AnyObject. Another difference is Objective-C array elements have to be a class type. In Swift, arrays are generic type collections, there are no limitations on what that type can be. You can create an array that holds any value type, such as Int, Float, String, or Enums, as well as classes. The last difference is, unlike arrays in Objective-C, Swift arrays are defined as a structure instead of a class. We know the basic constructs for declaring an integer array and performing operations such as adding, removing, and deleting elements from an array. Let's take a closer, more detailed examination of how arrays are implemented in the standard library. There are three Array types available in Swift: Array ContiguousArray ArraySlice Every Array class maintains a region of memory that stores the elements contained in the array. For array element types that are not a class or @objc protocol type, the arrays memory region is stored in contiguous blocks. Otherwise, when an arrays element type is a class or @objc protocol type, the memory region can be a contiguous block of memory, an instance of NSArray, or an instance of an NSArray subclass. If may be more efficient to use a ContiguousArray if you're going to store elements that are a class or an @objc protocol type. The ContiguousArray class shares many of the protocols that Array implements, so most of the same properties are supported. The key differentiator between Array and ContiguousArray is that ContiguousArray' do not provide support for bridging to Objective-C. The ArraySlice class represents a sub-sequence of an Array, ContiguousArray, or another ArraySlice. Like ContiguousArray, ArraySlice instances also use contiguous memory to store elements, and they do not bridge to Objective-C. An ArraySlice represents a sub-sequence from a larger, existing Array type. Because of this, you need to be aware of the side effects if you try storing an ArraySlice after the original Array's lifetime has ended and the elements are no longer accessible. This could lead to memory or object leaks thus Apple recommends that long-term storage of ArraySlice instances is discouraged. When you create an instance of Array, ContiguousArray, or ArraySlice, an extra amount of space is reserved for storage of its elements. The amount of storage reserved is referred to as an array's capacity, which represents the potential amount of storage available without having to reallocate the array. Since Swift arrays share an exponential growth strategy, as elements are appended to an array, the array will automatically resize when it runs out of capacity. When you amortize the append operations over many iterations, the append operations are performed in constant time. If you know ahead of time an array will contain a large number of elements, it may be more efficient to allocate additional reserve capacity at creation time. This will avoid constant reallocation of the array as you add new elements. The following code snippet shows an example of declaring an initial capacity of 500 Integer elements for the array intArray: // Create an array using full array syntax var intArray = Array<Int>() // Create an array using shorthand syntax intArray = [Int]() intArray.capacity // contains 0 intArray.reserveCapacity(500) intArray.capacity // contains 508 You can notice from the preceding example that our reserve capacity is actually larger than 500 integer elements. For performance reasons, Swift may allocate more elements than you requests. But you can be guaranteed that at least the number of elements specified will be created. When you make a copy of an array a separate physical copy is not made during the assignment. Swift implements a feature called copy-on-write, which means that array elements are not copied until a mutating operation is performed when more than one array instances is sharing the same buffer. The first mutating operation may cost O(n) in time and space, where n is the length of the array. Initializing Array The initialization phase prepares a struct, class, or enum for use through a method called init. If you're familiar with other languages such as C++, Java, or C# this type of initialization is performed in their class constructor, which is defined using the class's name. If you're coming from Objective-C, the Swift initializers will behave a little differently from how you're used to. In Objective-C, the init methods will directly return the object they initialize, callers will then check the return value when initializing a class and check for nil to see if the initialization process failed. In Swift, this type of behavior is implemented as a feature called failable initialization, which we'll discuss shortly. There are four initializers provided for the three types of Swift arrays implemented in the standard library. Additionally, you can use a dictionary literal to define a collection of one or more elements to initialize the array, the elements are separated by a comma (,): // Create an array using full array syntax var intArray = Array<Int>() // Create an array using shorthand syntax intArray = [Int]() // Use array literal declaration var intLiteralArray: [Int] = [1, 2, 3] // [1, 2, 3] //: Use shorthand literal declaration intLiteralArray = [1, 2, 3] // [1, 2, 3] //: Create an array with a default value intLiteralArray = [Int](count: 5, repeatedValue: 2) // [2, 2, 2, 2, 2] Adding and Updating Elements in an Array To add a new element to an array you can use the append(_:) method. This will add newElement to the end of the array: var intArray = [Int]() intArray.append(50) // [50] If you have an existing collection type you can use the append(_:) method. This will append elements from newElements to the end of the array: intArray.append([60, 65, 70, 75]) // [50, 60, 65, 70, 75] If you want to add elements at a specific index you can use the insert(newElement:at:) method. This will add newElement at index i: intArray.insert(newElement: 55, at: 1)// [50, 55, 60, 65, 70, 75] Requires i <= count otherwise you will receive a fatal error: Array index out of range message and execution will stop. To replace an element at a specific index you can use the subscript notation, providing the index of the element you want to replace: intArray[2] = 63 // [50, 55, 63, 65, 70, 75] Retrieving and Removing Elements from an Array There are several methods you can use to retrieve elements from an array. If you know the specific array index or sub-range of indexes you can use array subscripting: // Initial intArray elements // [50, 55, 63, 65, 70, 75] // Retrieve an element by index intArray[5] // returns 75 // Retrieve an ArraySlice of elements by subRange intArray[2..<5] // Returns elements between index 2 and less than index 5 // [63, 65, 70] // Retrieve an ArraySlice of elements by subRange intArray[2…5] // Returns elements between index 2 and index 5 // [63, 65, 70, 75] You can also iterate over the array, examining each element in the collection: for element in intArray { print(element) } // 50 // 55 // 63 // 65 // 70 // 75 You can also check if an array has a specific element or pass a closure that will allow you to evaluate each element: intArray.contains(55) // returns true See Apples Swift documentation for a complete list of methods available for arrays:https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/CollectionTypes.html Retrieving and Initializing Dictionaries A dictionary is an unordered collection that stores associations between keys and values of the same type with no defined ordering. Each value is associated with a unique key that acts as an identifier for the value in the dictionary. A dictionary data structure works just like a real world dictionary; to retrieve a definition, reference, translation, or some other type of information for a specific word you open the dictionary and locate the word to find the information you're looking for. With a dictionary data structure, you store and retrieve values by associating them with a key. A dictionary key type must conform to the Hashtable protocol. Initializing a Dictionary Just like arrays, there are two ways you can declare a dictionary using either the full or literal syntax: // Full syntax declaration var myDict = Dictionary<Int, String>() // Shorthand syntax declaration var myDict = [Int: String]() Additionally, you can use a dictionary literal to define a collection of one or more key-value pairs to initialize the dictionary. The key and value are separated by a colon (:) and the pairs are separated by a comma (,). If the keys and values have consistent types you do not need to declare the type of dictionary in the declaration, Swift will infer it based on the key-value pairs used during initialization, which saves a few keystrokes allowing you to use the shorter form: // Use dictionary literal declaration var myDict: [Int: String] = [1: "One", 2: "Two", 3: "Three"] // [2: "Two", 3: "Three", 1: "One"] // Use shorthand literal declaration var myDict = [1: "One", 2: "Two", 3: "Three"] // [2: "Two", 3: "Three", 1: "One"] Adding/Modifying/Removing a Key-Value Pair To add a new key-value pair or to update an existing pair you can use the updateValue(_:forKey) method or subscript notation. If the key is does not exist a new pair will be added, otherwise the existing pair is updated with the new value: // Add a new pair to the dictionary myDict.updateValue("Four", forKey: 4) $R0: String? = nil // [2: "Two", 3: "Three", 1: "One", 4: "Four"] // Add a new pair using subscript notation myDict[5] = "Five" // [5: "Five", 2: "Two", 3: "Three", 1: "One", 4: "Four"] Unlike the subscript method, the updateValue(_:forKey:) method will return the value that was replaced, or nil if a new key-value pair was added. To remove a key-value pair you can use the removeValue(forKey:) method, providing the key to delete or setting the key to nil using subscript notation: // Remove a pair from the dictionary – returns the removed pair let removedPair = myDict.removeValue(forKey: 1) removedPair: String? = "One" // [5: "Five", 2: "Two", 3: "Three", 4: "Four"] // Remove a pair using subscript notation myDict[2] = nil // [5: "Five", 3: "Three", 4: "Four"] Unlike the subscript method, the removeValue(forKey:) method will return the value that was removed, or nil if the key doesn't exist. Retrieving Values from a Dictionary You can retrieve specific key-value pairs from a dictionary using subscript notation. The key is passed to the square bracket subscript notation; it's possible the key may not exist so subscripts will return an Optional. You can use either optional binding or forced unwrapping to retrieve the pair or determine if it doesn't exist. Do not use forced unwrapping though unless you're absolutely sure the key exists or a runtime exception will be thrown: // Example using Optional Binding var myDict = [1: "One", 2: "Two", 3: "Three"] if let optResult = myDict[4] { print(optResult) } else { print("Key Not Found") } // Example using Forced Unwrapping – only use if you know the key will exist let result = myDict[3]! print(result) If instead of getting a specific value, you can iterate over the sequence of a dictionary and return a (key, value) tuple, which can be decomposed into explicitly named constants. In this example, (key, value) are decomposed into (stateAbbr, stateName): // Dictionary of state abbreviations to state names let states = [ "AL" : "Alabama", "CA" : "California", "AK" : "Alaska", "AZ" : "Arizona", "AR" : "Arkansas"] for (stateAbbr, stateName) in states { print("The state abbreviation for (stateName) is (stateAbbr)") } // Output of for...in The state abbreviation for Alabama is AL The state abbreviation for California is CA The state abbreviation for Alaska is AK The state abbreviation for Arizona is AZ The state abbreviation for Arkansas is AR You can see from the output that items in the dictionary are not output in the order they were inserted. Recall that dictionaries are unordered collections, so there is no guarantee that the order pairs will be retrieved when iterating over them. If you want to retrieve only the keys or values independently you can use the keys or values properties on a dictionary. These properties will return a LazyMapCollection instance over the collection. The elements of the result will be computed lazily each time they are read by calling the transform closure function on the base element. The key and value will appear in the same order as they would as a key-value pair, .0 member and .1 member, respectively: for (stateAbbr) in states.keys { print("State abbreviation: (stateAbbr)") } //Output of for...in State abbreviation: AL State abbreviation: CA State abbreviation: AK State abbreviation: AZ State abbreviation: AR for (stateName) in states.values { print("State name: (stateName)") } //Output of for...in State name: Alabama State name: California State name: Alaska State name: Arizona State name: Arkansas You can read more about the LazyMapCollection structure on Apple's developer site at https://developer.apple.com/library/prerelease/ios/documentation/Swift/Reference/Swift_LazyMapCollection_Structure/ There may be occasions when you want to iterate over a dictionary in an ordered manner. For those cases, you can make use of the global sort(_:) method. This will return an array containing the sorted elements of a dictionary as an array: // Sort the dictionary by the value of the key let sortedArrayFromDictionary = states.sort({ $0.0 < $1.0 }) // sortedArrayFromDictionary contains... // [("AK", "Alaska"), ("AL", "Alabama"), ("AR", "Arkansas"), ("AZ", "Arizona"), ("CA", "California")] for (key) in sortedArrayFromDictionary.map({ $0.0}) { print("The key: (key)") } //Output of for...in The key: AK The key: AL The key: AR The key: AZ The key: CA for (value) in sortedArrayFromDictionary.map({ $0.1}) { print("The value: (value)") } //Output of for...in The value: Alaska The value: Alabama The value: Arkansas The value: Arizona The value: California Let's walk through what is happening here. For the sort method, we are passing a closure that will compare if the first arguments key from the key-value pair, $0.0 against the second arguments key from the key-value pair, $1.0, if the first argument is less than the second it will be added to the new array. When the sort method has iterated over and sorted all of the elements, a new array of [(String, String)] containing the key-value pairings is returned. Next, we want to retrieve the list of keys from the sorted array. On the sortedArrayFromDictionary variable we call map({ $0.0}), the transform passed to the map method will add the .0 element of each array element in sortedArrayFromDictionary to the new array returned by the map method. The last step is to retrieve a list of values from the sorted array. We are performing the same call to the map method as we did to retrieve the list of keys, this time we want the .1 element of each array element in sortedArrayFromDictionary, though. Like the preceding example, these values will be added to the new array returned by the map method.     But what if you wanted to base your sorting order on the dictionary value instead of the key? This is simple to do; you would just change the parameter syntax that is passed to the sort method. Changing it to states.sort({ $0.1 < $1.1 }) will now compare the first arguments value from the key-value pair, $0.1, against the second arguments value from its key-value pair, $1.1, and adding the lesser of the two to the new array that will be returned. Declaring Sets A Set is an unordered collection of unique, non-nil elements with no defined ordering. A Set type must conform to the Hashtable protocol to be stored in a set. All of Swift's basic types are hashable by default. Enumeration case values that do not use associate values are also hashable by default. You can store a custom type in a Set, you'll just need to ensure that it conforms to the Hashable protocol, as well as the Equatable protocol since Hashtable conforms to it. Sets can be used anywhere you would use an array when ordering is not important and you want to ensure only unique elements are stored. Additionally, access time has the potential of being more efficient over arrays. With an array, the worst case scenario when searching for an element is O(n) where n is the size of the array. Whereas accessing an element in a Set is always constant time O(1), regardless of its size. Initializing a Set Unlike the other collection types, Sets cannot be inferred from an array literal alone and must be explicitly declared by specifying the Set type: // Full syntax declaration var stringSet = Set<String>() Because of Swift's type inference, you do not have to specify the type of the Set that you're initializing, it will infer the type based on the array literal it is initialized with. Remember though that the array literal must contain the same types: // Initialize a Set from an array literal var stringSet: Set = ["Mary", "John", "Sally"] print(stringSet.debugDescription) // Out of debugDescription shows stringSet is indeed a Set type "Set(["Mary", "John", "Sally"])" Modifying and Retrieving Elements of a Set To add a new element to a Set use the insert(_:) method. You can check if an element is already stored in a Set using the contains(_:) method. Swift provides several methods for removing elements from a Set, if you have an instance of the element you want to remove you can use the remove(_:) method, which takes an element instance. If you know the index of an element in the Set you can use the remove(at:) method, which takes an instance of SetIndex<Element>. If the Set count is greater than 0 you can use the removeFirst() method to remove the element and the starting index. Lastly, if you want to remove all elements from a Set, use the removeAll() method or removeAll(keepCapacity) method, if keepCapacity is true the current capacity will not decrease: var stringSet: Set = ["Erik", "Mary", "Michael", "John", "Sally"] // ["Mary", "Michael", "Sally", "John", "Erik"] stringSet.insert("Patrick") // ["Mary", "Michael", "Sally", "Patrick", "John", "Erik"] if stringSet.contains("Erik") { print("Found element") } else { print("Element not found") } // Found element stringSet.remove("Erik") // ["Mary", "Sally", "Patrick", "John", "Michael"] if let idx = stringSet.index(of: "John") { stringSet.remove(at: idx) } // ["Mary", "Sally", "Patrick", "Michael"] stringSet.removeFirst() // ["Sally", "Patrick", "Michael"] stringSet.removeAll() // [] You can iterate over a Set the same as you would the other collection types by using the for…in loop. The Swift Set type is unordered, so you can use the sort method like we did for the Dictionary type if you want to iterate over elements in a specific order: var stringSet: Set = ["Erik", "Mary", "Michael", "John", "Sally"] // ["Mary", "Michael", "Sally", "John", "Erik"] for name in stringSet { print("name = (name)") } // name = Mary // name = Michael // name = Sally // name = John // name = Erik for name in stringSet.sorted() { print("name = (name)") } // name = Erik // name = John // name = Mary // name = Michael // name = Sally Set Operations The Set type is modeled after the mathematical Set Theory and it implements methods that support basic Set operations for comparing two Sets, as well as operations that perform membership and equality comparisons between two Sets. Comparison Operations The Swift Set type contains four methods for performing common operations on two sets. The operations can be performed either by returning a new set, or using the operations alternate InPlace method to perform the operation in place on the source Set. The union(_:) and formUnion(_:) methods create a new Set or update the source Set with all the values from both Sets, respectively. The intersection(_:) and formIntersection(_:) methods create a new Set or update the source Set with values only common to both Sets, respectively. The symmetricDifference(_:) or formSymmetricDifference(_:) methods create a new Set or update the source Set with values in either Set, but not both, respectively. The subtracting(_:) or subtract(_:) methods create a new Set or update the source Set with values not in the specified Set: let adminRole: Set = [ "READ", "EDIT", "DELETE", "CREATE", "SETTINGS", "PUBLISH_ANY", "ADD_USER", "EDIT_USER", "DELETE_USER"] let editorRole: Set = ["READ", "EDIT", "DELETE", "CREATE", "PUBLISH_ANY"] let authorRole: Set = ["READ", "EDIT_OWN", "DELETE_OWN", "PUBLISH_OWN", "CREATE"] let contributorRole: Set = [ "CREATE", "EDIT_OWN"] let subscriberRole: Set = ["READ"] // Contains values from both Sets let fooResource = subscriberRole.union(contributorRole) // "READ", "EDIT_OWN", "CREATE" // Contains values common to both Sets let commonPermissions = authorRole.intersection(contributorRole) // "EDIT_OWN", "CREATE" // Contains values in either Set but not both let exclusivePermissions = authorRole.symmetricDifference(contributorRole) // "PUBLISH_OWN", "READ", "DELETE_OWN" Membership and Equality Operations Two sets are said to be equal if they contain precisely the same values, and since Sets are unordered, the ordering of the values between Sets does not matter. Use the == operator, which is the "is equal" operator, to determine if two Sets contain all of the same values: // Note ordering of the sets does not matter var sourceSet: Set = [1, 2, 3] var destSet: Set = [2, 1, 3] var isequal = sourceSet == destSet // isequal is true Use the isSubset(of:) method to determine if all of the values of a Set are contained in a specified Set. Use the isStrictSubset(of:) method to determine if a Set is a subset, but not equal to the specified Set. Use the isSuperset(of:) method to determine if a Set contains all of the values of the specified Set. Use the isStrictSuperset(of:) method to determine if a Set is a superset, but not equal to the specified Set. Use the isDisjoint(with:) method to determine if two Sets have the same values in common: let contactResource = authorRole // "EDIT_OWN", "PUBLISH_OWN", "READ", "DELETE_OWN", "CREATE" let userBob = subscriberRole // "READ" let userSally = authorRole // "EDIT_OWN", "PUBLISH_OWN", "READ", "DELETE_OWN", "CREATE" if userBob.isSuperset(of: fooResource){ print("Access granted") } else { print("Access denied") } // "Access denied" if userSally.isSuperset(of: fooResource){ print("Access granted") } else { print("Access denied") } // Access granted authorRole.isDisjoint(with: editorRole) // false editorRole.isSubset(of: adminRole) // true Summary In this article, we've learned about the difference between classes and structures and when you would use one type over another, as well as characteristics of value types and reference types and how each type is allocated at runtime. We went into some of the implementation details for the Array, Dictionary, and Set collection types that are implemented in the Swift standard library. Resources for Article: Further resources on this subject: Python Data Structures [Article] The pandas Data Structures [Article] Python Data Persistence using MySQL Part III: Building Python Data Structures Upon the Underlying Database Data [Article]
Read more
  • 0
  • 0
  • 2750
article-image-encrypt-and-hash-passwords
Pedro Narciso
16 Nov 2016
7 min read
Save for later

Encrypt and hash passwords

Pedro Narciso
16 Nov 2016
7 min read
A couple of days ago, Dropbox posted an explanation on how they store your passwords. In this post, we are going to implement a library that encrypts and hashes a password in a similar way. Disclaimer: Security is a complex AND important topic. It is YOUR responsability to educate yourself on it. As you may have read on the Dropbox blog post, they: Get the hash product of the SHA512 function over the password. Apply the bcrypt function over the previous result. Encrypt the bcrypt product with a symmetric encryption algorithm (AES256). The above transformation can be written in pseudocode as: hashed_password_to_store = encrypt_aes256(bcrypt(sha512('my password'))) Remember: You should not keep the AES256 key in your database nor hardcoded in the application code. Okay, let's start with it! First, we are going to create a project: $ mkdir password-encryption-tool $ cd password-encryption-tool $ npm init // init our node.js project Edit index.js and write the skeleton of the function. 'use strict'; function hashAndEncryptPassword(input){ let sha512hash = sha512(input); let bcryptHash = bcryptHash(sha512hash); let encryptedPassword = encrypt(bcryptHash); return encryptedPassword; } Good! Of course, that function does not work yet. The functions sha512, bcryptHash and encrypt are not yet implemented, but it gives us a starting point. The #sha512(input) function recives a password as input and returns its SHA512 product. To implement that function, we are going to use the Hash class from the crypto module. Docs here. function sha512(input){ let hasher = crypto.createHash('SHA512'); hasher.update(input); return hasher.digest('base64'); //Returns the hash in base64 format } Done! The bcrypt function requires a third party library. There are many options but here are my personal recommendations – bcrypt.js and bcrypt. The former is a pure JavaScript implementation with 0 dependencies. The latter is a native one and therefore faster (2.7 times), which is what I use personally. But, for this example, we are going to use bcrypt.js. Why? Because if you do not have the build environment needed for the native version, I don't want you to stop following this tutorial and start pulling out your hair because there is some dependency not compiling on your machine. Anyway, your code will work with both of them because bcrypt.js API is compatible with bcrypt. Okay, that was a long explanation. Let's install bcryptjs: $ npm install bcryptjs --save This installs bcryptjs and updates our package.json file with the new dependency. Bcrypt implementations expose both synchronous and asynchronous APIs. Given that the bcrypt algorithm was made to be slow, I am going to use the asynchronous API. After all, you don't want to block the event loop, do you? This decision forces us to refactor our code a little bit. The asynchronous version of #hashAndEncryptPassword(input) looks like: //... const crypto = require('crypto'), bcrypt = require('bcryptjs'); const BCRYPT_SALT_ROUNDS=10; function hashAndEncryptPassword(key, input, callback){ let sha512hash ; try{ sha512hash = sha512(input); }catch(err){ return callback(err); } bcrypt.hash(sha512hash, BCRYPT_SALT_ROUNDS, function(err, result){ var encryptedHash; if(err){ return callback(err); } try{ encryptedHash = encrypt(key, result); }catch(err){ return callback(err); } callback( null, encryptedHash); }); } //... Note the function signature change with the inclusion of the callback parameter, plus the bcrypt integration. Also, the SHA512 is wrapped in a try/catch block. Any error happening here will be properly handled and returned to the callback function. Time to implement the encrypt function. Node.js Crypto API exposes the Cipher class. Its usage is similar to the Hash class. We just need to provide the algorithm, key and IV. The IV is used for randomization at the encryption stage; for that reason, it is very important that you use a new random IV for each different value you encrypt. Later, when we want to decrypt a value, you need to provide the same key and IV. Because they do not need to be secret, we are going to append them along the encrypted hash. IVs are just a bunch of random bytes and their generation is quite simple. const IV = new Buffer(crypto.randomBytes(16)); // A buffer with 16 random bytes And here is our encrypt function: ... const IV_LEN = 16, ALGORITHM = 'AES-256-CTR'; function encrypt (key, input) { const IV = new Buffer(crypto.randomBytes(IV_LEN)); let cipher = crypto.createCipheriv(ALGORITHM, key, IV); cipher.setEncoding('base64'); cipher.write(input); cipher.end(); const cipherText = cipher.read(); // IV is not a secret. We can store it along the password return cipherText + '$' + IV.toString('base64'); } ... The encrypt function accepts key and input as arguments, and: Generates an IV.. Creates a cipher using the AES-256-CTR algorithm, our key and the IV. Sets cipherText constant with the encrypted version of input. Returns the cipherText plus $ plus the IV. We are almost done. We now have a function that allows us to safely store a user password in our database. Now, as you've probably guessed, when we want to check if a given password is valid, we need to compare it with our stored password. But, you can just simple compare them. You need to apply some transformations in order to get comparable versions. A simple approach could be to just apply the same transformation to a plain password and then they will be comparable. AES256(bcrypt(SHA512(plain text))) === hashed-and-encrypted But we are going to make it slightly different. Here is our recipe: Get the SHA512 of the password we want to validate. Decrypt our stored password. Use bcrypt.compare to compare them. The SHA512 part was already implemented as the sha512 function. To decrypt the stored password we need to create a decipher with crypto.createDecipheriv with the previously used algorithm, key, and IV. function decrypt(key, input){ var result; let [cipherText, IV] = input.split('$'); let buffIV = new Buffer(IV, 'base64'); let decipher = crypto.createDecipheriv(ALGORITHM, key, buffIV); result = decipher.update(cipherText, 'base64', 'utf8'); result += decipher.final('utf8'); return result; } Now we can implement our compare function. function compare(key, clearPassword, encryptedPassword, callback){ var hash; try{ hash = decrypt(key, encryptedPassword); }catch(err){ return callback(err); } bcrypt.compare(sha512(clearPassword), hash, callback); } And do not forget to export compare and hashAndEncryptPassword functions. exports.hashAndEncryptPassword = hashAndEncryptPassword; exports.compare = compare; We can now use our module to hash and encrypt passwords: var passwordTool = require('.'); // Remember to not include your key in your code! // better load it from an environment variable var key = 'BKYHAT11zlXUiXE3iZfzSEWfvwjdbfPK'; var password = 'super secret'; passwordTool.hashAndEncryptPassword(key, password, function(err, hash){ if(err){ throw err; } console.log('Password ready to store', hash); }); // prints: Password ready to store ZGmzcPy29oYjZj5+P/wg4nS0Bs64... And to match a user provided password with our stored copy: passwordTool.compare(key, password, storedPassword, function (err, result){ if(err){ throw err; } console.log('match?', result); }); You can find the complete source code of this article at https://github.com/revington/password-tool. About the author Pedro Narciso García Revington is a senior full-stack developer with 10+ years of experience in high scalability and availability, microservices, automated deployments, data processing, CI, (T,B,D)DD, and polyglot persistence. He is a self-taught, highly motivated problem solver with a focus on delivering quick and elegant solutions.
Read more
  • 0
  • 0
  • 3147

article-image-multithreading-qt
Packt
16 Nov 2016
13 min read
Save for later

Multithreading with Qt

Packt
16 Nov 2016
13 min read
Qt has its own cross-platform implementation of threading. In this article by Guillaume Lazar and Robin Penea, authors of the book Mastering Qt 5, we will study how to use Qt and the available tools provided by the Qt folks. (For more resources related to this topic, see here.) More specifically, we will cover the following: Understanding the QThread framework in depth The worker model and how you can offload a process from the main thread An overview of all the available threading technologies in Qt Discovering QThread Qt provides a sophisticated threading system. We assume that you already know threading basics and the associated issues (deadlocks, threads synchronization, resource sharing, and so on) and we will focus on how Qt implements it. The QThread is the central class for of the Qt threading system. A QThread instance manages one thread of execution within the program. You can subclass QThread to override the run() function, which will be executed in the QThread class. Here is how you can create and start a QThread: QThread thread; thread.start(); The start() function calling will automatically call the run() function of thread and emit the started() signal. Only at this point, the new thread of execution will be created. When run() is completed, thread will emit the finished() signal. This brings us to a fundamental aspect of QThread: it works seamlessly with the signal/slot mechanism. Qt is an event-driven framework, where a main event loop (or the GUI loop) processes events (user input, graphical, and so on) to refresh the UI. Each QThread comes with its own event loop that can process events outside the main loop. If not overridden, run() calls the QThread::exec() function, which starts the thread's event loop. You can also override QThread and call exec(), as follows: class Thread : public QThread { Q_OBJECT protected: void run() { Object* myObject = new Object(); connect(myObject, &Object::started, this, &Thread::doWork); exec(); } private slots: void doWork(); }; The started()signal will be processed by the Thread event loop only upon the exec() call. It will block and wait until QThread::exit() is called. A crucial thing to note is that a thread event loop delivers events for all QObject classes that are living in that thread. This includes all objects created in that thread or moved to that thread. This is referred to as the thread affinity of an object. Here's an example: class Thread : public QThread { Thread() : mObject(new QObject()) { } private : QObject* myObject; }; // Somewhere in MainWindow Thread thread; thread.start(); In this snippet, myObject is constructed in the Thread constructor, which is created in turn in MainWindow. At this point, thread is living in the GUI thread. Hence, myObject is also living in the GUI thread. An object created before a QCoreApplication object has no thread affinity. As a consequence, no event will be dispatched to it. It is great to be able to handle signals and slots in our own QThread, but how can we control signals across multiple threads? A classic example is a long running process that is executed in a separate thread that has to notify the UI to update some state: class Thread : public QThread { Q_OBJECT void run() { // long running operation emit result("I <3 threads"); } signals: void result(QString data); }; // Somewhere in MainWindow Thread* thread = new Thread(this); connect(thread, &Thread::result, this, &MainWindow::handleResult); connect(thread, &Thread::finished, thread, &QObject::deleteLater); thread->start(); Intuitively, we assume that the first connect function sends the signal across multiple threads (to have a result available in MainWindow::handleResult), whereas the second connect function should work on thread's event loop only. Fortunately, this is the case due to a default argument in the connect() function signature: the connection type. Let's see the complete signature: QObject::connect( const QObject *sender, const char *signal, const QObject *receiver, const char *method, Qt::ConnectionType type = Qt::AutoConnection) The type variable takes Qt::AutoConnection as a default value. Let's review the possible values of Qt::ConectionType enum as the official Qt documentation states: Qt::AutoConnection: If the receiver lives in the thread that emits the signal, Qt::DirectConnection is used. Otherwise, Qt::QueuedConnection is used. The connection type is determined when the signal is emitted. Qt::DirectConnection: This slot is invoked immediately when the signal is emitted. The slot is executed in the signaling thread. Qt::QueuedConnection: The slot is invoked when control returns to the event loop of the receiver's thread. The slot is executed in the receiver's thread. Qt::BlockingQueuedConnection: This is the same as Qt::QueuedConnection, except that the signaling thread blocks until the slot returns. This connection must not be used if the receiver lives in the signaling thread or else the application will deadlock. Qt::UniqueConnection: This is a flag that can be combined with any one of the preceding connection types, using a bitwise OR element. When Qt::UniqueConnection is set, QObject::connect() will fail if the connection already exists (that is, if the same signal is already connected to the same slot for the same pair of objects). When using Qt::AutoConnection, the final ConnectionType is resolved only when the signal is effectively emitted. If you look again at our example, the first connect(): connect(thread, &Thread::result, this, &MainWindow::handleResult); When the result() signal will be emitted, Qt will look at the handleResult() thread affinity, which is different from the thread affinity of the result() signal. The thread object is living in MainWindow (remember that it has been created in MainWindow), but the result() signal has been emitted in the run() function, which is running in a different thread of execution. As a result, a Qt::QueuedConnection function will be used. We will now take a look at the second connect(): connect(thread, &Thread::finished, thread, &QObject::deleteLater); Here, deleteLater() and finished() live in the same thread, therefore, a Qt::DirectConnection will be used. It is crucial that you understand that Qt does not care about the emitting object thread affinity, it looks only at the signal's "context of execution." Loaded with this knowledge, we can take another look at our first QThread example to have a complete understanding of this system: class Thread : public QThread { Q_OBJECT protected: void run() { Object* myObject = new Object(); connect(myObject, &Object::started, this, &Thread::doWork); exec(); } private slots: void doWork(); }; When Object::started() is emitted, a Qt::QueuedConnection function will be used. his is where your brain freezes. The Thread::doWork() function lives in another thread than Object::started(), which has been created in run(). If the Thread has been instantiated in the UI Thread, this is where doWork() would have belonged. This system is powerful but complex. To make things more simple, Qt favors the worker model. It splits the threading plumbing from the real processing. Here is an example: class Worker : public QObject { Q_OBJECT public slots: void doWork() { emit result("workers are the best"); } signals: void result(QString data); }; // Somewhere in MainWindow QThread* thread = new Thread(this); Worker* worker = new Worker(); worker->moveToThread(thread); connect(thread, &QThread::finished, worker, &QObject::deleteLater); connect(this, &MainWindow::startWork, worker, &Worker::doWork); connect(worker, &Worker::resultReady, this, handleResult); thread->start(); // later on, to stop the thread thread->quit(); thread->wait(); We start by creating a Worker class that has the following: A doWork()slot that will have the content of our old QThread::run() function A result()signal that will emit the resulting data Next, in MainWindow, we create a simple thread and an instance of Worker. The worker->moveToThread(thread) function is where the magic happens. It changes the affinity of the worker object. The worker now lives in the thread object. You can only push an object from your current thread to another thread. Conversely, you cannot pull an object that lives in another thread. You cannot change the thread affinity of an object if the object does not live in your thread. Once thread->start() is executed, we cannot call worker->moveToThread(this) unless we are doing it from this new thread. After that, we will use three connect() functions: We handle worker life cycle by reaping it when the thread is finished. This signal will use a Qt::DirectConnection function. We start the Worker::doWork() upon a possible UI event. This signal will use a Qt::QueuedConnection. We process the resulting data in the UI thread with handleResult(). This signal will use a Qt::QueuedConnection. To sum up, QThread can be either subclassed or used in conjunction with a worker class. Generally, the worker approach is favored because it separates more cleanly the threading affinity plumbing from the actual operation you want to execute in parallel. Flying over Qt multithreading technologies Built upon QThread, several threading technologies are available in Qt. First, to synchronize threads, the usual approach is to use a mutual exclusion (mutex) for a given resource. Qt provides it by the mean of the QMutex class. Its usage is straightforward: QMutex mutex; int number = 1; mutex.lock(); number *= 2; mutex.unlock(); From the mutex.lock() instruction, any other thread trying to lock the mutex object will wait until mutex.unlock() has been called. The locking/unlocking mechanism is error prone in complex code. You can easily forget to unlock a mutex in a specific exit condition, causing a deadlock. To simplify this situation, Qt provides a QMutexLocker that should be used where the QMutex needs to be locked: QMutex mutex; QMutexLocker locker(&mutex); int number = 1; number *= 2; if (overlyComplicatedCondition) { return; } else if (notSoSimple) { return; } The mutex is locked when the locker object is created, and it will be unlocked when locker is destroyed, for example, when it goes out of scope. This is the case for every condition we stated where the return statement appears. It makes the code simpler and more readable. If you need to create and destroy threads frequently, managing QThread instances by hand can become cumbersome. For this, you can use the QThreadPool class, which manages a pool of reusable QThreads. To execute code within threads managed by a QThreadPool, you will use a pattern very close to the worker we covered earlier. The main difference is that the processing class has to extend the QRunnable class. Here is how it looks: class Job : public QRunnable { void run() { // long running operation } } Job* job = new Job(); QThreadPool::globalInstance()->start(job); Just override the run() function and ask QThreadPool to execute your job in a separate thread. The QThreadPool::globalInstance() function is a static helper function that gives you access to an application global instance. You can create your own QThreadPool class if you need to have a finer control over the QThreadPool life cycle. Note that QThreadPool::start() takes the ownership of the job object and will automatically delete it when run() finishes. Watch out, this does not change the thread affinity like QObject::moveToThread() does with workers! A QRunnable class cannot be reused, it has to be a freshly baked instance. If you fire up several jobs, QThreadPool automatically allocates the ideal number of threads based on the core count of your CPU. The maximum number of threads that the QThreadPool class can start can be retrieved with QThreadPool::maxThreadCount(). If you need to manage threads by hand, but you want to base it on the number of cores of your CPU, you can use the handy static function, QThreadPool::idealThreadCount(). Another approach to multithreaded development is available with the Qt Concurrent framework. It is a higher level API that avoids the use of mutexes/locks/wait conditions and promotes the distribution of the processing among CPU cores. Qt Concurrent relies of the QFuture class to execute a function and expect a result later on: void longRunningFunction(); QFuture<void> future = QtConcurrent::run(longRunningFunction); The longRunningFunction() will be executed in a separated thread obtained from the default QThreadPool class. To pass parameters to a QFuture class and retrieve the result of the operation, use the following code: QImage processGrayscale(QImage& image); QImage lenna; QFuture<QImage> future = QtConcurrent::run(processGrayscale, lenna); QImage grayscaleLenna = future.result(); Here, we pass lenna as a parameter to the processGrayscale() function. Because we want a QImage as a result, we declare QFuture with the template type QImage. After that, future.result() blocks the current thread and waits for the operation to be completed to return the final QImage template type. To avoid blocking, QFutureWatcher comes to the rescue: QFutureWatcher<QImage> watcher; connect(&watcher, &QFutureWatcher::finished, this, &QObject::handleGrayscale); QImage processGrayscale(QImage& image); QImage lenna; QFuture<QImage> future = QtConcurrent::run(processImage, lenna); watcher.setFuture(future); We start by declaring a QFutureWatcher with the template argument matching the one used for QFuture. Then, simply connect the QFutureWatcher::finished signal to the slot you want to be called when the operation has been completed. The last step is to the tell the watcher to watch the future object with watcher.setFuture(future). This statement looks almost like it's coming from a science fiction movie. Qt Concurrent also provides a MapReduce and FilterReduce implementation. MapReduce is a programming model that basically does two things: Map or distribute the processing of datasets among multiple cores of the CPU Reduce or aggregate the results to provide it to the caller check styleThis technique has been first promoted by Google to be able to process huge datasets within a cluster of CPU. Here is an example of a simple Map operation: QList images = ...; QImage processGrayscale(QImage& image); QFuture<void> future = QtConcurrent::mapped( images, processGrayscale); Instead of QtConcurrent::run(), we use the mapped function that takes a list and the function to apply to each element in a different thread each time. The images list is modified in place, so there is no need to declare QFuture with a template type. The operation can be made a blocking operation using QtConcurrent::blockingMapped() instead of QtConcurrent::mapped(). Finally, a MapReduce operation looks like this: QList images = ...; QImage processGrayscale(QImage& image); void combineImage(QImage& finalImage, const QImage& inputImage); QFuture<void> future = QtConcurrent::mappedReduced( images, processGrayscale, combineImage); Here, we added a combineImage() that will be called for each result returned by the map function, processGrayscale(). It will merge the intermediate data, inputImage, into the finalImage. This function is called only once at a time per thread, so there is no need to use a mutex object to lock the result variable. The FilterReduce reduce follows exactly the same pattern, the filter function simply allows to filter the input list instead of transforming it. Summary In this article, we discovered how a QThread works and you learned how to efficiently use tools provided by Qt to create a powerful multi-threaded application. Resources for Article: Further resources on this subject: QT Style Sheets [article] GUI Components in Qt 5 [article] DOM and QTP [article]
Read more
  • 0
  • 1
  • 42447

article-image-getting-started-sorting-algorithms-java
Packt
16 Nov 2016
9 min read
Save for later

Getting Started with Sorting Algorithms in Java

Packt
16 Nov 2016
9 min read
In this article by Peter Verhas author of the book Java 9 Programming By Example, we will develop a simple sort program. Using this code as an example, we will look at different build tools, which are frequently used for Java projects, and learn the basic features of the Java language. (For more resources related to this topic, see here.) The problem we will solve The sorting problem is one of the oldest programming tasks that an engineer solves. We will have a set of records and we know that we will want to find a specific one sometime later, and we will want to find that one fast. To find it, we will sort the records in a specific order that helps finding the record we want fast. As an example, we can have the names of the students with some marks on cards. When students will come to the office asking for the result, we can turn all pages one after the other to find the name of the enquiring student. However, it is better if we sort the papers by the name of the students lexicographically. When a student comes, we can search the mark attached to the name much faster. We can look at the middle card; if it shows the name of the student, then we are happy to have found the name and the mark. If the card precedes the name of the student lexicographically, then we will continue searching in the second half, otherwise the first half. Following that approach, we can find the name of the student in no more steps than as many times the pack of cards can be halved. If we have two cards, then it is two steps at most. If it is four, then we will need three steps at most. If there are eight cards, then we may need four steps, but not more. If there are 1000 cards, then we may need at most 11 steps, while the original, non-sorted set will need 1000 steps, worst case. That is, approximately, it speeds up the search 100 times, so this is worth sorting the cards, unless the sorting itself takes too much time. In many cases, it is worth sorting the dataset and there are many sorting algorithms to do that. There are simpler and more complex algorithms, and as in many cases, more complex algorithms are the one that run faster. As we are focusing on the Java programming part and not the algorithm forging, in this article, we will develop a Java code that implements a simple and not-that-fast algorithm. Bubble sort The algorithm that we will implement in this article is well known as bubble sort. The approach is very simple. Begin at the start of the cards and compare the first and the second card. If the first card is later in lexicographic order than the second one, then swap the two cards. Then, repeat this for the card that is at the second place now, then the third, and so on. There is a card that is lexicographically the latest, say Wilson, and sometime later, we will get to this card as we go on swapping, the cards going from start to end. When we get this card and start to compare it with the next one, we will always swap them; this way, Wilson's card will travel to the last place where it has to be after the sort. All we have to do is repeat this travelling from the start and the occasional swapping of cards again, but this time only to the last but one element. This time, the second latest element will get to its place—say Wilkinson will be right before Wilson. If we have n cards, and we repeat this n-1 times, all cards will get to their place. Project structure and build tools When a project is more complex than a single class, and it usually is, then it is wise to define a project structure. We will have to decide where we store the source files, where the resource files (those that contain some resource for the program, but are not Java source) are, where should the .class files be written by the compiler, and so on. Generally, the structure is mainly the directory setup and configuring the tools that perform the build that use these tools. The compilation of complex programs cannot be feasibly done using the command line issuing javac commands. If we have a 100 Java source files, the compilation will require that many javac commands to be issued. We can write a simple bash script that does that. First, it will be just 100 lines, each compiling one source Java file to class file. Then, we will realize that this is only time, CPU, and power consuming to compile the files that are not changed since the last compilation. So, we can add some bash programming that checks the time stamp on the source and generated files. Then, we will probably realize that… whatever. At the end, we will end up with a tool that is essentially a build tool. And, this is already done. Instead of creating one, we will use a build tool that is ready. There are a few of them that can be found at https://en.wikipedia.org/wiki/List_of_build_automation_software Make The Make program was originally created in April 1976, so this is not a new tool. It is included in the Unix system so this tool is available without any extra installation on Linux, Mac OS X, or any other Unix-based system. Additionally, there are numerous ports of the tool on Windows and some version is/was included in the Visual C compiler toolset. The Make is not tied to Java. It was created when the major programming language was C, but it is not tied to C or any other language. Make is a dependency description language that has a very simple syntax. The Make, just like any other build tool, works controlled by a project description file. In case of make, this file contains a rule set. The description file is usually named Makefile, but in case the name of the description file is different, it can be specified as a command-line option to the make command. Rules in Makefile follow each other and a it is one or more lines. The first line starts at the first position (there is no tab or space at the start of the line) and the following lines start with a tab character. Thus, Makefile may look something like the following code: run : hello.jar java -cp hello.jar HelloWorld hello.jar : HelloWorld.class jar -cf hello.jar HelloWorld.class HelloWorld.class : HelloWorld.java javac HelloWorld.java The file defines three so-called targets: run, hello.jar, and HelloWorld.class. To create HelloWorld.class, type the following line at the Command Prompt: make HelloWorld.class The make will look at the rule and see that it depends on HelloWorld.java. If the HelloWorld.class file does not exist, or HelloWorld.java is newer than the Java source file, make will execute the command that is written on the next line and it will compile the Java source file. If the class file was created following the last modification of HelloWorld.java, then make knows that there is no need to run the command. In case of creating HelloWorld.class,the make program has an easy task. The source file was already there. If you issue the make hello.jar command, the procedure is more complex. The make command sees that in order to create hello.jar, it needs HelloWorld.class, which itself is also a target on another rule. Thus, it may need to be created. First, it starts the problem the same way as before. If HelloWorld.class is there, and is older than hello.jar, there is nothing to do. If it is not there, or is newer than hello.jar, then the jar -cf hello.jar HelloWorld.class command needs to be executed, but not yet. It remembers that this command has to be executed sometime in the future when all the commands that are needed to create HelloWorld.class are already executed successfully. Thus, it continues to create the class file exactly the same way as I already described earlier. In general, a rule can have the following format: target : dependencies command The make command can create any target using the make target command by first calculating which commands to execute and then executing them one by one. The commands are shell commands executing in a different process and may pose problems under Windows, which may render the Makefile files operating system dependent. Note that the run target is not an actual file that make creates. A target can be a file name or just a name for the target. In the latter case, make will never consider the readily available target. As we do not use make for Java project, there is no room to get into more details. Additionally, I cheated a bit by making the description of a rule simpler than it should be. The make tool has many powerful features out of the scope of this book. There are also several implementations that differ a little from each other. You will most probably meet the one made by the Free Software Foundation—the GNU make. And, of course, just in case of any Unix command-line tool, man is your friend. The man make command will display the documentation of the tool on the screen. The main points that you should remember about make are as follows: It defines the dependencies of the individual artifacts (targets) in a declarative way It defines the actions to create the missing artifacts in an imperative way. Summary In this article, we have developed a very basic sort algorithm. It was made purposefully simple so that we could reiterate on the basic and most important Java language elements, classes, packages, variables, methods, and so on. Resources for Article: Further resources on this subject: Algorithm Analysis [article] Introduction to C# and .NET [article] Parallel Computing [article]
Read more
  • 0
  • 0
  • 4613
article-image-kotlin-basics
Packt
16 Nov 2016
7 min read
Save for later

Kotlin Basics

Packt
16 Nov 2016
7 min read
In this article by Stephen Samuel and Stefan Bocutiu, the authors of the book Programming Kotlin, it’s time to discover the fundamental building blocks of Kotlin. This article will cover the basic constructs of the language, such as defining variables, control flow syntax, type inference, and smart casting, and its basic types and their hierarchy. (For more resources related to this topic, see here.) For those coming from a Java background, this article will also highlight some of the key differences between Kotlin and Java and how Kotlin’s language features are able to exist on the JVM. For those who are not existing Java programmers, then those differences can be safely skipped. vals and vars Kotlin has two keywords for declaring variables: val and var. A var is a mutable variable—a variable that can be changed to another value by reassigning it. This is equivalent to declaring a variable in Java. val name = “kotlin” Alternatively, the var can be initialized later: var name: String name = “kotlin” Variables defined with var can be reassigned since they are mutable: var name = “kotlin” name = “more kotlin” The val keyword is used to declare a read-only variable. This is equivalent to declaring a final variable in Java. A val must be initialized when created since it cannot be changed later: val name = “kotlin” A read-only variable does not mean the instance itself is automatically immutable. The instance may still allow its member variables to be changed via functions or properties. But the variable itself cannot change its value or be reassigned to another value. Type inference Did you notice in the previous section that the type of the variable was not included when it was initialized? This is different to Java, where the type of the variable must always accompany its declaration. Even though Kotlin is a strongly typed language, we don’t always need to declare types explicitly. The compiler can attempt to figure out the type of an expression from the information included in the expression. A simple val is an easy case for the compiler because the type is clear from the right-hand side. This mechanism is called type inference. This reduces boilerplate while keeping the type safety we expect of a modern language. Values and variables are not the only places where type inference can be used. It can also be used in closures where the type of the parameter(s) can be inferred from the function signature. It can also be used in single-line functions, where the return value can be inferred from the expression in the function, as this example demonstrates: fun plusOne(x: Int) = x + 1 Sometimes, it is helpful to add type inference if the type inferred by the compiler is not exactly what you want: val explicitType: Number = 12.3 Basic types One of the big differences between Kotlin and Java is that in Kotlin, everything is an object. If you come from a Java background, then you will already be aware that in Java, there are special primitive types, which are treated differently from objects. They cannot be used as generic types, do not support method/function calls, and cannot be assigned null. An example is the boolean primitive type. Java introduced wrapper objects to offer a workaround in which primitive types are wrapped in objects so that java.lang. Boolean wraps a boolean in order to smooth over the distinctions. Kotlin removes this necessity entirely from the language by promoting the primitives to full objects. Whenever possible, the Kotlin compiler will map basic types back to JVM primitives for performance reasons. However, the values must sometimes be boxed, such as when the type is nullable or when it is used in generics. Two different values that are boxed might not use the same instance, so referential equality is not guaranteed on boxed values. Numbers The built-in number types are as follows: Type Width long 64 int 32 short 16 byte 8 double 64 float 32 To create a number literal, use one of the following forms: val int = 123 val long = 123456L val double = 12.34 val float = 12.34F val hexadecimal = 0xAB val binary = 0b01010101 You will notice that a long value requires the suffix L and a float, F. The double type is used as the default for floating point numbers, and int for integral numbers. The hexadecimal and binary use the prefixes 0x and 0b respectively. Kotlin does not support the automatic widening of numbers, so conversion must be invoked explicitly. Each number has a function that will convert the value to one of the other number types: val int = 123 val long = int.toLong() val float = 12.34F val double = float.toDouble() The full set of methods for conversions between types is as follows: toByte() toShort() toInt() toLong() toFloat() toDouble() toChar() Unlike Java, there are no built-in bitwise operators, but named functions instead. This can be invoked like operators (except inverse): val leftShift = 1 shl 2 val rightShift = 1 shr 2 val unsignedRightShift = 1 ushr 2 val and = 1 and 0x00001111 val or = 1 and 0x00001111 val xor = 1 xor 0x00001111 val inv = 1.inv() Booleans Booleans are rather standard and support the usual negation, conjunction and disjunction operations. Conjunction and disjunction are lazily evaluated. So if the left-hand side satisfies the clause, then the right-hand side will not be evaluated: val x = 1 val y = 2 val z = 2 val isTrue = x < y && x < z val alsoTrue = x == y || y == z Chars Chars represent a single character. Character literals use single quotes, such as a or Z. Chars also support escaping for the following characters: t, b, n, r, , , \, $. All Unicode characters can be represented using the respective Unicode number, like so: u1234. Note that the char type is not treated as a number, unlike Java. Strings Just as in Java, strings are immutable. String literals can be created using double or triple quotes. Double quotes create an escaped string. In an escaped string, special characters such as newline must be escaped: val string = “string with n new line” Triple quotes create a raw string. In a raw string, no escaping is necessarily, and all characters can be included. val rawString = “““ raw string is super useful for strings that span many lines “““ Strings also provide an iterator function, so they can be used in a for loop. Arrays In Kotlin, we can create an array using the arrayOf() library function: val array = arrayOf(1, 2, 3) Alternatively, we can create an array from an initial size and a function that is used to generate each element: val perfectSquares = Array(10, { k -> k * k }) Unlike Java, arrays are not treated specially by the language and are regular collection classes. Instances of Array provide an iterator function and a size function as well as a get and set function. The get and set functions are also available through bracket syntax like many C style languages: val element1 = array[0] val element2 = array[1] array[2] = 5 To avoid boxing types that will ultimately be represented as primitives in the JVM, Kotlin provides alternative array classes that are specialized for each of the primitive types. This allows performance-critical code to use arrays as efficiently as they would do in plain Java. The provided classes are ByteArray, CharArray, ShortArray, IntArray, LongArray, BooleanArray, FloatArray, and DoubleArray. Comments Comments in Kotlin will come as no surprise to most programmers as they are the same as Java, Javascript, and C, among other languages. Block comments and line comments are supported: // line comment /* A block comment can span many lines */ Packages Packages allow us to split code into namespaces. Any file may begin with a package declaration: package com.packt.myproject class Foo fun bar(): String = “bar” The package name is used to give us the fully-qualified name (FQN) for a class, object, interface, or function. In the previous example, the Foo class has the FQN com.packt.myproject.Foo, and the top-level function bar has the FQN com.packt.myproject.bar. Summary In Kotlin, everything is an object in the sense that we can call member functions and properties on any variable. Some types are built in because their implementation is optimized, but to the user, they look like ordinary classes. In this article, we described most of these types: numbers, characters, booleans, and arrays. Resources for Article: Further resources on this subject: Responsive Applications with Asynchronous Programming [Article] Asynchronous Programming in F# [Article] Go Programming Control Flow [Article]
Read more
  • 0
  • 0
  • 6944

article-image-vector-representation-words
Janu Verma
15 Nov 2016
6 min read
Save for later

Vector Representation of Words

Janu Verma
15 Nov 2016
6 min read
In natural language processing (NLP) tasks, the first step is to represent a document as an element in a vector space. Standard machine learning and data mining algorithms expect a data instance as a vector; in fact, when we say data, we mean a matrix (a row/vector for each data point). There are various ways to express a textual document as a vector, depending on the problem and the assumptions of the model. In traditional Vector Space Models (VSMs), a word is represented as a vector of dimension equal to the size of vocabulary, with each word in the vocabulary corresponding to an entry in the vector. For example, if the text is "Friends work and play together", then our vocabulary has 5 words. We can represent the words as: Friends = [1,0,0,0,0] Work = [0,1,0,0,0] And = [0,0,1,0,0] Play = [0,0,0,1,0] Together = [0,0,0,0,1] Such a representation, called one-hot encoding, is very useful because we can merge these encodings to achieve a vector representation of the entire textual document, which is very central to modern search engines. The above vectors are binary, and there are other encodings possible, such as employing frequency or some other variant. If you are more curious, you can read about TF-IDF. This type of representation has obvious limitations, most importantly, it treats a word as atomic and provides useful information about the relationships that may exist between individual words. We can't perform any meaningful comparision between words other than equality. Furthermore, such a representation results in word vectors that are extremely sparse. Distributed Representations To overcome some of the limitations of the one-hot scheme, a distributed assumption is adapted, which states that words that appear in the same context are semantically closer than the words that do not share the same context. Using this principle, a word can be represented as points in a continous vector space, where semantically similar words correspond to nearby points. This represenation is also called word embeddings, since we are embedding word vectors in the distributed vector space. Essentially, the weight of each word in the vector is distributed across many dimensions. So, instead of a one-to-one mapping between a word and a basic vector (dimension), the word contribution is spread across all of the dimensions of the vector. The dimensions are believed to capture the semantic properties of the words. For example, for our text "Friends work and play together", each word can be represented as something like: Friends = [0.73,0.34,0.52,0.01] Work = [0.65,0.79,0.22,0.1] And = [0.87,0.94,0.14,0.7] Play = [0.73, 0.69, 0.89, 0.4] Together = [0.87,0.79,0.22,0.09] You can see that the words 'Friends' and 'Together' are closer to each other, and the words 'Work' and 'Play' have a higher similarity. Note that these vectors are chosen arbitrarily and do not show an actual representation. The sole purpose here is to give an example. Learning Distributed Representations: Word2Vec Distributed representations of words can be learned by training a model on a corpus of textual data. Mikolov, et. al. proposed an efficient method to learn these embeddings, making it feasible to learn high-quality word vectors on a huge corpus of data. The model is called word2vec, which uses a neural network to learn representations. Two architectures for neural network were proposed – Continuous Bag-of-Words (CBOW) and Skip-Gram. CBOW predicts the current word from a window of neighboring words, while skip-gram uses the current word to predict the surrounding words. Word2Vec models use a sliding window to quantify context. In each sliding window, there is a central word that is under attention, with a few words preceding and following the central word. One of the important parameters of the word2vec models is the length of the sliding window. Consider a textual document: "The goal of the manifold learning techniques is to 'learn' the low dimensional manifold." If we consider 4 words preceding and following the central word, the context of 'manifold' is: ['the', 'goal', 'of', 'the', 'learning', 'techniques', 'is', 'to']. These context words form the input layer of the neural network, and each word is represented as a vector using the one-hot schema. There is one hidden layer and one output layer. The output layer is formed by the central words ,that is, each element in the vocabulary. This way, we learn a representation for each word in terms of the context words. The actual ordering of the context words is irrelevant, which is called a bag-of-words assumption. The skip-gram method is completely opposite of the CBOW method. Here the central word is the input layer, and the context words are now at the output layer. CBOW is faster, but skip-gram does a better job for not-so-frequent words. The implementation of both the architectures can be found at Google Code. Google has also made public pre-trained word vectors that are trained on about 100 billion words from Google News dataset. Several other data, such as Wikipedia, have been used to compute word vectors. And modern neural network packages like Tensorflow have word2vec support. Refer to the word2vec tutorial of Tensorflow. It is important to understand that word2vec is not deep learning; in fact, both the CBOW and skip-gram architectures are shallow neural models with only one hidden layer. Applications Distributed representation of words has been successfully applied in many applications. Machine Translation has been shown to achieve much higher accuracy using distributed representations, so you can make the following assertions: - ```Distance(France, Germany) < Distance(France, Spain)``` - ```Vector('Paris') - Vector('France') + Vector('Italy') ~ Vector(Rome)``` - ```Vector('king') - Vector('man') + Vector('woman') ~ Vector('queen')``` - The odd one in [staple, hammer, saw, drill] is staple. Item2vec: word2vec for collborative filtering and recommendation system, so you can infer: Vector(David Guetta) - Vector(Avicii) + Vector(Beyonce) -> Vector(Rihanna) BioVectors: Word vectors for Bioinformations. BioVectors can characterize biological sequences in terms of biochemical and biophysical interpretations of the underlying patterns References Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of NAACL HLT, 2013 Word2Vec Implementation Tensorflow Example Python Implementation Tomas Mikolov, Quoc V. Le and Ilya Sutskever. Exploiting Similarities among Languages for Machine Translation. We show how the word vectors can be applied to machine translation. Barkan, O; Koenigstein, N (2016).Item2Vec: Neural Item Embedding for Collaborative Filtering Asgari, Ehsaneddin; Mofrad, Mohammad R.K. (2015). Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics. PloS one. 10 (11): e0141287.
Read more
  • 0
  • 0
  • 14953
Modal Close icon
Modal Close icon