Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-visualizations-made-easy-gnuplot
Packt
01 Mar 2012
12 min read
Save for later

Visualizations made easy with gnuplot

Packt
01 Mar 2012
12 min read
  This article, written by Lee Philips the author of gnuplot Cookbook contains the following :   Making a surface plot Using coordinate mappings Coloring the surface Making a contour plot Making a vector plot Making an image plot or heat map Combining contours and images Combining surfaces with images Plotting a path in 3D Drawing parametric surfaces Making a surface plot A surface plot represents the dependent quantity z, which depends on the two independent variables x and y, as a surface whose height indicates the value of z.   The previous figure is a perspective drawing of a surface representing the Bessel function J0(r), where r is the distance from (x=0, y=0). The height of the surface shows the value of J0, given on the vertical axis (unlabeled in this figure, but usually called z). The other two (unlabeled) axes defning the plane above which the surface is drawn are the x and y axes.     How to do it… The following code listing is the script that coaxed gnuplot into making the previous figure:   set isosamples 40unset keyset title "J_0(r^2)"set xrange [-4:4]set yrange [-4:4]set ztics 1splot besj0(x**2+y**2)set view 29,53 #Done implicitly by mousing.set term pngcairo mono enhancedset out 'bessel.png'replot   How it works… There are several new commands in this recipe. The set isosamples command sets the isoline density. This is analogous to the set samples command when making 2D plots, but it sets the number of lines used in forming the surface. The number of isosamples can be set independently in each direction; if one number is specifed, it is used for both directions. The default of 10 usually creates surfaces that are far too coarse to be useful.   Turning to the second of the highlighted commands, the splot command is the 3D version of our old friend the plot command (it probably initially stood for "surface plot", but now can do several things besides plot surfaces, as we shall see in the rest of this article). It expects a function of x and y rather than x alone. Although we are interested in plotting something that has the type of symmetry that would be most conveniently expressed in polar (spherical or cylindrical) coordinates, these geometries are not available for function plots in 3D in gnuplot. (They are available through the set mapping command for data plots, as we shall see later in this article.) Therefore in such cases, we are required to convert our expressions to the rectangular coordinate system. Instead of what we would call r in a cylindrical coordinate system, here we use the equivalent x**2 + y**2.   In this recipe, we would like to illustrate, as far as possible, the interactive approach to creating a fnal 3D plot. The next highlighted line, beginning with set view, can be entered on the command line or included in a script. The view is the orientation in degrees of the perspective drawing of the 3D plot. Naturally, it does arise in 2D. It is diffcult to determine what is the most useful view for a particular plot without looking at it and experimenting with it; therefore, even if our fnal product is intended to be a fle, a common workfow is to frst create the plot using an interactive terminal (x11 or wxt). Then we rotate the plot with the mouse, and possibly scale and zoom it using the middle mouse button, until we arrive at the desired appearance. This is what we mean by the comment in the set view command. Now we can reset the terminal to the fnal output device that we need, specify the output fle, and simply say replot. The view and scaling at which we left the interactive plot is retained as a set of global settings and will be refected in our fnal output fle. These settings are also displayed at the bottom of the interactive plot window, so we can record them if we are going to make similar plots in the future, or want a set of plots to be drawn with the same settings. Note that we also redefned the ztics value. This is because when the plot is tilted to the fnal view angle that we chose, the perspective causes the tic labels on the vertical axis to be crowded together; this is a common problem with 3D plots, and taking manual control of the tics on the z-axis is the solution.   There's more… Following is the same plot with one setting changed (aside from a slight adjustment in the view angle):   While the frst plot was essentially a wireframe that we could see through, this version has the appearance of a solid, opaque surface. All we need to do is to say set hidden3d. This, which only works when lines or linespoints are being used, makes the surface appear opaque by removing from the plot any part of the surface, other surfaces, and other plot elements such as the axes and tic labels, that are behind the surface from our point of view. The underside of the surface is shown in a contrasting color with a color output device, but the two sides of the surface are not distinguished in monochrome. The name of the setting refers to the technique of hidden line removal; gnuplot is justly famed for the quality of its hidden line removal algorithm, and is one reason this program is so well regarded for its 3D plotting ability.   Using coordinate mappings It is possible, when making 3D plots from data fles, for the data to be interpreted in spherical or cylindrical coordinates rather than the default Cartesian system. For details, type help set mapping. We will give an example of using the cylindrical mapping to conveniently draw a shape with cylindrical symmetry.   The previous figure is a perspective view of a surface that somewhat resembles a Christmas tree ornament. The relevant feature of this surface is that it has rotational symmetry around the z (vertical) axis, which means that it is most naturally expressed in cylindrical coordinates.   How to do it… Try the following script:   set mapping cylindricalunset ticsunset borderset hiddenset xrange [-pi : pi]set yrange [-pi : pi]set zrange [0 : pi]set iso 60unset keysplot '++' using 1:2:(sin($2)) with lines   How it works… There are several new ideas used in this recipe. Breaking it down, these are:   The set mapping command The frst, highlighted line contains the new command that is the subject of this recipe. When the default Cartesian (x-y-z) coordinate system is changed to cylindrical then the columns of data read in during a data plot are interpreted as θ-z-r, where θ is the angular coordinate, z is the vertical coordinate, and r is the radius. A spherical mapping is also available and explained in the gnuplot online help (help set mapping). If the data fle only has two columns, then the plot is drawn with r = 1.   In our example we don't want to plot from a data fle, however. We want to plot a function given directly in the script. This presents us with a problem, as gnuplot does not support cylindrical or spherical plots of functions in 3D. The solution is to use one of gnuplot's pseudofles.   The ++ pseudofle The "++" pseudofle creates rows of imaginary data with three columns x-y-z unless we change the coordinate mapping, which of course in this example we have. Setting the mapping to cylindrical means that the fctitious data columns will be interpreted as θ-z-r.   Now to plot a function, we use the using notation applied to the imaginary columns of data. We've done this in the fnal line of the script, where we plot the sine of the second column (z).   To clarify the use of "++" when plotting surfaces, note that, in Cartesian coordinates, the two commands "splot sin(x)+cos(y)" and "splot '++' using 1:2:(sin($1)+cos($2)) with lines" produce exactly the same plot.   Coordinate ranges We have also established ranges for all variables in the set xrange and two other commands following it. The ranges for the polar coordinates are taken from the corresponding Cartesian coordinates, that is, when we set the xrange, we are setting both the range of the x-axis displayed on the plot and the range of the variable θ in the cylindrical coordinate system. It is mandatory to set xrange and yrange when using the "++" flename.   This mixing of the coordinate system in which the function is calculated and the Cartesian system in which it is displayed can be confusing, but the example shows a strategy, which should make it possible to get predictable results. Setting the xrange and yrange as we've done puts the r = 0 point in the middle of the graph and prevents part of the plot from being cut off. It also sets up a full rotation of the angular coordinate over a range of 2 p.   If we wanted to plot, say, our shape with half of it sliced off by a vertical plane, the easiest way to do this is not to fddle with the coordinate ranges, but to apply a transformation to one of the fctitious data columns: splot '++' using ($1/2):2:(sin($2)) with lines, will do the trick without any surprising side effects. In this example the underlying angular coordinate (column 1) still passes through a full rotation, but we've divided it in half without changing the way the figure is projected onto the Cartesian display. Note that the 60 isolines will still be used in our reduced angular range, so we might want to set iso to a smaller value.   Completing the picture We've eliminated all of the graph adornments (unset tics, unset border, unset key) so we will be left with only the surface. The isosamples are set to give a suffciently smooth surface drawing that is nevertheless not too crowded with isosurface lines (see the previous recipe). set hidden ensures that we shall see only the outer surface of the shape.   Coloring the surface The wireframe splot with hidden line removal that we covered in the frst recipe of this article, Making a surface plot, gives the visual impression of a solid surface. The numerical value encoded into the surface's height can be visually estimated, roughly, by the perspective provided by the isolines in conjunction with the tics on the vertical axis. But gnuplot also has a way to draw real solid surfaces whose height is indicated by color or shade.   The previous figure shows the same mathematical function plotted in the frst recipe in this article (Making a surface plot). Now the numerical value of the function at any point is indicated by both the height of the surface and its shade; the surface is now drawn as an opaque membrane rather than as a network of curves.   How to do it…   To produce the previous figure, run the following in gnuplot:   set isosamples 100set samples 100unset keyset title "J_0(r^2)"set xrange [-4:4]set yrange [-4:4]set ztics 1unset surfaceset pm3dsplot besj0(x**2+y**2) The surface will be drawn with a palette of colors when a color output device is being used and with a grayscale palette when using a monochrome terminal.     How it works… If you compare the previous script with the one in the Making a surface plot recipe at the beginning of this article, you will see that the only signifcant difference is the highlighted line. The pm3d mode colors the imaginary surface being plotted according to its height or z-value at every point, with the mapping between the height and the color or shade determined by the palette, which we shall discuss in some more detail shortly.   The other modifcations are to increase the number of isolines, in order to get a smoother surface, and to turn off the drawing of the individual isolines themselves with the command unset surface. We also need to set the sample frequency; generally we want this to be equal to the isosample frequency. In pm3d mode, the two orthogonal sets of isolines are drawn with two different spacings given by the two parameters. Although the gnuplot manual claims that the global hidden3d setting does not affect pm3d surface plots, it in fact seems to, and should not be turned on, as it appears to slightly degrade the drawing quality.   There's more… Sometimes we want both a colored surface and a set of isolines; in fact, this can often be the clearest type of quantitative 3D plot. The way to achieve the highest quality in this type of graph is to use the hidden3d linestyle option to pm3d, as we do in the following script:   set iso 30set samp 30unset keyset title "J_0(r^2)"set xrange [-4:4]set yrange [-4:4]set ztics 1unset surfset style line 1 lt 4 lw .5set pm3d at s hidden3d 1splot besj0(x**2+y**2) This requires us to defne a user linestyle. Then the linestyle is referred to in an option to the set pm3d command. This will cause the isolines to be drawn using lines in this style, which allows us to have them in any color, thickness, or pattern supported by our terminal. Further, the isolines will be drawn with hidden line removal, so they will appear to be embedded in the opaque surface. As before, the global hidden3d option should not be turned on.   Note that we've also reduced the sample and isoline frequency, to keep our plot from being too crowded with isolines. (The at s component of the set pm3d command means at surface.)  
Read more
  • 0
  • 0
  • 13522

article-image-using-llamaindex-for-ai-assisted-knowledge-management
Andrei Gheorghiu
08 Jun 2023
10 min read
Save for later

Using LlamaIndex for AI-Assisted Knowledge Management

Andrei Gheorghiu
08 Jun 2023
10 min read
IntroductionOne of the hottest questions of the moment for a lot of strategy and decision makers across most industries worldwide is:How can AI help my business?Afterall, with great disruption also comes great opportunity. A sound business strategy should not ignore emerging changes in the market. We’re still at the early stages of understanding AI and I’m not going to provide a definitive answer to this question in this article, but the good news is that this article should provide a part of the answer.Knowledge is power, right?And yet we all know how it is to struggle trying to retain and efficiently re-use the knowledge we gather.We strive to learn from our successes and mistakes and we invest a lot of time and money in building fancy knowledge bases just to discover later that unfortunately we keep repeating the same mistakes and reinventing the wheel.In my experience as a consultant, the biggest issue (especially for medium and large companies) is not a lack of knowledge but on the contrary, it’s too much knowledge and an inability to use it in a timely and effective manner. The solutionThis article presents a very simple yet effective way of indexing large quantities of pre-existing knowledge that can later be retrieved by natural language queries or integrated with chatbot systems.As usual, take it as a starting point. The code example is trivial and lacks any error handling, but provides the building blocks to work from.My example builds on your existing knowledge base and leverages LlamaIndex and the power of Large Language Models (in this case GPT 3.5 Turbo from OpenAI).Why LlamaIndex? Well, created by Jerry Liu, LlamaIndex is a robust open-source resource that empowers you to organize and search your data for a variety of applications, including answering questions, summarizing information or serving as a part of a chatbot system. It provides data connectors to ingest your existing data sources in many different formats (such as text files, PDF, docs, SQL, etc.). It then allows you to structure your data (via indices or graphs) so that this data can be easily used with LLMs. In many ways, it is similar to Langchain but more focused on data storage and retrieval instead of automated AI agents.In short, this article will show you how, with just a few lines of code, you can index your enterprise's knowledge base and then have the ability to query and retrieve information from GPT 3.5 Turbo with your own knowledge base on top of that in the most natural way: plain English. Logic diagramCreating the IndexRetrieving the knowledge PrerequisitesMake sure you check these points before you start writing the code:Make sure you store your OpenAI API key in a local environment variable for secure and efficient access. The code works on the assumption that the API key is stored on your local environment (OPENAI_API_KEY).I’ve used Python v3.11. If you’re running an older version, an update is recommended to make sure you don’t run into any compatibility issues.Install the requirements:pip install openaipip install llama-index Create a subfolder in your .PY file’s location (in my example the name of the subfolder is ‘stories’). You will store your knowledge base in .TXT files in that location. If your knowledge articles are in different formats (e.g., PDF or DOCX) you will have to:Change the code to use a different LlamaIndex data connector (https://gpt-index.readthedocs.io/en/latest/reference/readers.html) – this is my recommended solution – or:Convert all your documents in .TXT format and use the code as it is.For my demo, I have created (with the help of GPT-4) three fictional stories that will represent our proprietary ‘knowledge’ base: Your ‘stories’ folder should now look like this: The CodeFor the sake of simplicity, I’ve split the functionality into two different scripts:Index_stories.py (responsible for reading the ‘stories’ folder, creating an index and saving it for later queries)Query_stories.py (demonstrating how to query GPT 3.5 and then filter the AI response through our own knowledge base)Let’s begin with Index_stories.py:from llama_index import GPTVectorStoreIndex, SimpleDirectoryReader # Loading from a directory documents = SimpleDirectoryReader('stories').load_data() # Construct a vector store index index = GPTVectorStoreIndex.from_documents(documents) # Save your index to a .json file index.storage_context.persist()As you can see, the code is using SimpleDirectoryReader from LlamaIndex to read all .TXT files from the ‘stories’ folder. It then creates a simple vector index that can later be used to run queries over the content of these documents.In case you’re wondering what a vector index represents, imagine you're in a library with thousands of books, and you're looking for a specific book. Instead of having to go through each book one by one, this index acts in a similar way to a library catalog.  It helps you find the book you're looking for quickly.In the context of this code, GPTVectorStoreIndex is like that library catalog. It's a tool that helps organize and find specific pieces of information (like documents or stories) quickly and efficiently. When you ask a question, it looks through all the information it has and finds the most relevant answer for you. It's like a super-efficient librarian that knows exactly where everything is.The last line of the code saves the index in a sub-folder called ‘storage’ so that we do not have to recreate it every time and we are able to reuse it in the future. Now, for the querying part. Here’s the second script: Query_stories.py:from llama_index import GPTVectorStoreIndex, StorageContext, load_index_from_storage import openai import os openai.api_key = os.getenv('OPENAI_API_KEY') def prompt_chatGPT(task): response = openai.ChatCompletion.create( model="gpt-3.5-turbo", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": task} ] ) AI_response = response['choices'][0]['message']['content'].strip() return AI_response # rebuild storage context storage_context = StorageContext.from_defaults(persist_dir="storage") # load index index = load_index_from_storage(storage_context) # Querying GPT 3.5 Turbo prompt = "Tell me how Tortellini Macaroni's brother managed to conquer Rome." answer = prompt_chatGPT(prompt) print('Original AI answer: ' + answer +'\n\n') # Refining the answer in the context of our knowledge base query_engine = index.as_query_engine() response = query_engine.query(f'The answer to the following prompt: "{prompt}" is :"answer". If the answer is aligned to our knowledge, return the answer. Otherwise return a corrected answer') print('Custom knowledge answer: ' + str(response))How it worksAfter indexing the ‘stories’ folder, once you run the query_stories.py script the code will first load the index from the ‘storage’ sub-folder. It then prompts the GPT 3.5 Turbo model with a hard-coded question: “Tell me how Tortellini Macaroni’s brother managed to conquer Rome”. After the response is received, it queries our ‘stories’ to see if the answer aligns with our ‘knowledge’. Then, you’ll receive two answers.The first one is the original answer from GPT 3.5 Turbo:As expected, the AI model identified Mr. Spaghetti as a potentially fictional character and could not find any historical references of him conquering Rome.The second answer though, checks with our ‘knowledge’ and, because we have different information in our ‘stories’, it modifies the answer into:If you’ve read the three GPT-4-created stories you’ve noticed that Story1.txt mentions Biscotti as a fictional conqueror of Rome but not his brother and Story2.txt mentions Tortellini and his farm adventures but does not mention any relationship with Biscotti. Only the third story (Story3.txt) describes the nature of their relationship.This shows not only that the vector index managed to correctly record the knowledge from the individual stories but also proves the query function managed to provide a contextual response to our question.In addition to the Vector Store Index, there are several other types of indexes that can be used depending on the specific needs of your project.For instance, the List Index simply stores Nodes as a sequential chain, making it a straightforward and efficient choice for certain applications, such as where the order of data matters and where you frequently need to access all the data in the order it was added. An example might be a timeline of events or a log of transactions, where you often want to retrieve all entries in chronological order.Another option is the Tree Index which builds a hierarchical tree from a set of Nodes, which can be particularly useful when dealing with complex, nested data. For instance, if you're building a file system explorer, a tree index would be a good choice because files and directories naturally form a tree-like structure.There’s also the Keyword Table Index, which extracts keywords from each Node and builds a mapping from each keyword to the corresponding Nodes. This can be a powerful tool for text-based queries, allowing for quick and precise retrieval of relevant information.Each of these indexes offers unique advantages, and the choice between them would depend on the nature of your data and the specific requirements of your use case. ConclusionNow, think about the possibilities. Instead of fictional stories, we could have a collection of our standard operating procedures, process descriptions, knowledge articles, disaster recovery plans, change schedules and so on and so forth. Or, as another example we could build a chatbot that can solve generic users requests simply using GPT3.5 knowledge but forwards more specific issues (indexed from our knowledge base) to a support team.This brings unlimited potential in automation of our business processes and improvement of the decision-making process. You get the best of both worlds: the power of Large Language Models combined with the value of your own knowledge base. Security considerationsWorking on this article made me realize that we cannot really trust our interactions with an AI model unless we are in full control of the entire technology stack. Just because the interface might look familiar, it doesn’t necessary mean that bad actors cannot compromise the integrity of the data by injecting false or censored responses to our queries. But that’s a story for another time! Final noteThis article barely scratches the surface of the full capabilities of LlamaIndex. It is not meant to be a comprehensive guide into this topic but rather serve as an example start point for integrating AI technologies in our day-to-day business processes. I encourage you to have an in-depth study of LlamaIndex’s capabilities (https://gpt-index.readthedocs.io/en/latest/) if you want to take advantage of its full capabilities.About the AuthorAndrei Gheorghiu is an experienced trainer with a passion for helping learners achieve their maximum potential. He always strives to bring a high level of expertise and empathy to his teaching.With a background in IT audit, information security, and IT service management, Andrei has delivered training to over 10,000 students across different industries and countries. He is also a Certified Information Systems Security Professional and Certified Information Systems Auditor, with a keen interest in digital domains like Security Management and Artificial Intelligence.In his free time, Andrei enjoys trail running, photography, video editing and exploring the latest developments in technology.You can connect with Andrei on:LinkedIn: https://www.linkedin.com/in/gheorghiu/Twitter: https://twitter.com/aqg8017 
Read more
  • 0
  • 0
  • 13519

article-image-fundamental-selinux-concepts
Packt
14 Nov 2016
41 min read
Save for later

Fundamental SELinux Concepts

Packt
14 Nov 2016
41 min read
In this article by Sven Vermeulen, the author of the book SELinux System Administration Second Edition, we will see how Security Enhanced Linux (SELinux) brings additional security measures for your Linux system to further protect the resources on the system. This article explains why SELinux has opted to use labels to identify resources, the way SELinux differentiates itself from regular Linux access controls through the enforcement of security rules, how the access control rules enforced by SELinux are provided through policy files, the different SELinux implementations between Linux distributions. (For more resources related to this topic, see here.) Providing more security to Linux Seasoned Linux administrators and security engineers already know that they need to put some trust in the users and processes on their system in order for the system to remain secure. This is partially because users can attempt to exploit vulnerabilities found in the software running on the system, but a large contribution to this trust level is because the secure state of the system depends on the behavior of the users. A Linux user with access to sensitive information could easily leak that out to the public, manipulate the behavior of the applications he or she launches, and do many other things that affect the security of the system. The default access controls that are active on a regular Linux system are discretionary; it is up to the user's how the access controls should behave. The Linux discretionary access control (DAC) mechanism is based on the user and/or group information of the process and is matched against the user and/or group information of the file, directory, or other resource being manipulated. Consider the /etc/shadow file, which contains the password and account information of the local Linux accounts: $ ls -l /etc/shadow -rw------- 1 root root 1010 Apr 25 22:05 /etc/shadow Without additional access control mechanisms in place, this file is readable and writable by any process that is owned by the root user, regardless of the purpose of the process on the system. The shadow file is a typical example of a sensitive file that we don't want to see leaked or abused in any other fashion. Yet, the moment someone has access to the file, they can copy it elsewhere, for example to a home directory, or even mail it to a different computer and attempt to attack the password hashes stored within. Another example of how Linux DAC requires trust from its users is when a database is hosted on the system. Database files themselves are (hopefully) only accessible to runtime users of the database management system (DBMS) and the Linux root user. Properly secured systems will only grant trusted users access to these files (for instance, through sudo) by allowing them to change their effective user ID from their personal user to database runtime user or even root account for a well-defined set of commands. These users too can analyze the database files and gain access to potentially confidential information in the database without going through the DBMS. However, regular users are not the only reason for securing a system. Lots of software daemons run as the Linux root user or have significant privileges on the system. Errors within those daemons can easily lead to information leakage or might even lead to exploitable remote command execution vulnerabilities. Backup software, monitoring software, change management software, scheduling software, and so on: they all often run with the highest privileged account possible on a regular Linux system. Even when the administrator does not allow privileged users, their interaction with daemons induces a potential security risk. As such, the users are still trusted to correctly interact with these applications in order for the system to function properly. Through this, the administrator leaves the security of the system to the discretion of its (many) users. Enter SELinux, which provides an additional access control layer on top of the standard Linux DAC mechanism. SELinux provides a mandatory access control (MAC) system that, unlike its DAC counterpart, gives the administrator full control over what is allowed on the system and what isn't. It accomplishes this by supporting a policy-driven approach over what processes are and aren't allowed to do and by enforcing this policy through the Linux kernel. Mandatory means that access control is enforced by the operating system and defined solely by the administrator. Users and processes do not have permission to change the security rules, so they cannot work around the access controls; security is not left to their discretion anymore. The word mandatory here, just like the word discretionary before, was not chosen by accident to describe the abilities of the access control system: both are known terms in the security research field and have been described in many other publications, including the Trusted Computer System Evaluation Criteria (TCSEC) (http://csrc.nist.gov/publications/history/dod85.pdf) standard (also known as the Orange Book) by the Department of Defense in the United States of America in 1985. This publication has led to the common criteria standard for computer security certification (ISO/IEC 15408), available at http://www.commoncriteriaportal.org/cc/. Using Linux security modules Consider the example of the shadow file again. A MAC system can be configured to only allow a limited number of processes to read and write to the file. A user logged on as root cannot directly access the file or even move it around. He can't even change the attributes of the file: # id uid=0(root) gid=0(root) # cat /etc/shadow cat: /etc/shadow: Permission denied # chmod a+r /etc/shadow chmod: changing permissions of '/etc/shadow': Permission denied This is enforced through rules that describe when the contents of a file can be read. With SELinux, these rules are defined in the SELinux policy and are loaded when the system boots. It is the Linux kernel itself that is responsible for enforcing the rules. Mandatory access control systems such as SELinux can be easily integrated into the Linux kernel through its support for Linux Security Modules (LSM): High-level overview of how LSM is integrated into the Linux kernel LSM has been available in the Linux kernel since version 2.6, sometime in December 2003. It is a framework that provides hooks inside the Linux kernel to various locations, including the system call entry points, and allows a security implementation such as SELinux to provide functions to be called when a hook is triggered. These functions can then do their magic (for instance, checking the policy and other information) and give a go/no-go back to allow the call to go through or not. LSM by itself does not provide any security functionality; instead, it relies on security implementations that do the heavy lifting. SELinux is one of the implementations that use LSM, but there are several others: AppArmor, Smack, TOMOYO Linux, and Yama, to name a few. At the time of writing this, only one main security implementation can be active through the LSM hooks. Work is underway to enable stacking multiple security implementations, allowing system administrators to have more than one implementation active. Recent work has already allowed multiple implementations to be defined (but not simultaneously active). When supported, this will allow administrators to pick the best features of a number of implementations and enforce smaller LSM-implemented security controls on top of the more complete security model implementations, such as SELinux, TOMOYO, Smack, or AppArmor. Extending regular DAC with SELinux SELinux does not change the Linux DAC implementation nor can it override denials made by the Linux DAC permissions. If a regular system (without SELinux) prevents a particular access, there is nothing SELinux can do to override this decision. This is because the LSM hooks are triggered after the regular DAC permission checks have been executed. For instance, if you need to allow an additional user access to a file, you cannot add an SELinux policy to do that for you. Instead, you will need to look into other features of Linux, such as the use of POSIX access control lists. Through the setfacl and getfacl commands (provided by the acl package) the user can set additional permissions on files and directories, opening up the selected resource to additional users or groups. As an example, let's grant user lisa read-write access to a file using setfacl: $ setfacl -m u:lisa:rw /path/to/file Similarly, to view the current POSIX ACLs applied to the file, use this command: $ getfacl /path/to/file # file: file # owner: swift # group: swift user::rw- user:lisa:rw- group::r-- mask::r-- other::r-- Restricting root privileges The regular Linux DAC allows for an all-powerful user: root. Unlike most other users on the system, the logged-on root user has all the rights needed to fully manage the entire system, ranging from overriding access controls to controlling audits, changing user IDs, managing the network, and much more. This is supported through a security concept called capabilities (for an overview of Linux capabilities, check out the capabilities manual page: man capabilities). SELinux is also able to restrict access to these capabilities in a fine-grained manner. Due to this fine-grained authorization aspect of SELinux, even the root user can be confined without impacting the operations on the system. The aforementioned example of accessing /etc/shadow is just one example of a restriction that a powerful user as root still might not be able to make due to the SELinux access controls being in place. When SELinux was added to the mainstream Linux kernel, some security projects even went as far as providing public root shell access to an SELinux-protected system, asking hackers and other security researchers to compromise the box. The ability to restrict root was welcomed by system administrators who sometimes need to pass on the root password or root shell to other users (for example, database administrators) who needed root privileges when their software went haywire. Thanks to SELinux, the administrator can now pass on a root shell while resting assured that the user only has those rights he needs, and not full system-administration rights. Reducing the impact of vulnerabilities If there is one benefit of SELinux that needs to be stressed, while often also being misunderstood, it is its ability to reduce the impact of vulnerabilities. A properly written SELinux policy confines applications so that their allowed activities are reduced to a minimum set. This least-privilege model ensures that abnormal application behavior is not only detected and audited but also prevented. Many application vulnerabilities can be exploited to execute tasks that an application is not meant to do. When this happens, SELinux will prevent this. However, there are two misconceptions about SELinux state and its ability to thwart exploits, namely, the impact of the policy and the exploitation itself. If the policy is not written in a least-privilege model, then SELinux might consider this nonstandard behavior as normal and allow the actions to continue. For policy writers, this means that their policy code has to be very fine-grained. Sadly, that makes writing policies very time-consuming; there are more than 80 classes and over 200 permissions known to SELinux, and policy rules need to take into account all these classes and permissions for each interaction between two objects or resources. As a result, policies tend to become convoluted and harder to maintain. Some policy writers make the policies more permissive than is absolutely necessary, which might result in exploits becoming successful even though the action is not expected behavior from an application point of view. Some application policies are explicitly marked as unconfined (which is discussed later in this article), showing that they are very liberal in their allowed permissions. Red Hat Enterprise Linux even has several application policies as completely permissive, and it starts enforcing access controls for those applications only after a few releases. The second misconception is the exploit itself. If an application's vulnerability allows an unauthenticated user to use the application services as if he were authorized, SELinux will not play a role in reducing the impact of the vulnerability; it only notices the behavior of the application itself and not of the sessions internal to the application. As long as the application itself behaves as expected (accessing its own files and not poking around in other filesystems), SELinux will happily allow the actions to take place. It is only when the application starts behaving erratically that SELinux stops the exploit from continuing. Exploits such as remote command execution (RCE) against applications that should not be executing random commands (such as database management systems or web servers, excluding CGI-like functionality) will be prevented, whereas session hijacking or SQL injection attacks are not controllable through SELinux policies. Enabling SELinux support Enabling SELinux on a Linux system is not just a matter of enabling the SELinux LSM module within the Linux kernel. An SELinux implementation comprises the following: The SELinux kernel subsystem, implemented in the Linux kernel through LSM Libraries, used by applications that need to interact with SELinux Utilities, used by administrators to interact with SELinux Policies, which define the access controls themselves The libraries and utilities are bundled by the SELinux user space project (https://github.com/SELinuxProject/selinux/wiki). Next to the user space applications and libraries, various components on a Linux system are updated with SELinux-specific code, including the init system and several core utilities. Because SELinux isn't just a switch that needs to be toggled, Linux distributions that support SELinux usually come with SELinux predefined and loaded: Fedora and Red Hat Enterprise Linux (with its derivatives, such as CentOS and Oracle Linux) are well-known examples. Other supporting distributions might not automatically have SELinux enabled but can easily support it through the installation of additional packages (which is the case with Debian and Ubuntu), and others have a well-documented approach on how to convert a system to SELinux (for example, Gentoo and Arch Linux). Labeling all resources and objects When SELinux has to decide whether it has to allow or deny a particular action, it makes a decision based on the context of both the subject (which is initiating the action) and the object (which is the target of the action). These contexts (or parts of the context) are mentioned in the policy rules that SELinux enforces. The context of a process is what identifies the process to SELinux. SELinux has no notion of Linux process ownership and, frankly, does not care how the process is called, which process ID it has, and what account the process runs as. All it wants to know is what the context of that process is, which is represented to users and administrators as a label. Label and context are often used interchangeably, and although there is a  technical distinction (one is a representation of the other), we will not dwell on that much. Let's look at an example label: the context of the current user (try it out yourself if you are on an SELinux-enabled system): $ id -Z unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 The id command, which returns information about the current user, is executed here with the -z switch (a commonly agreed-upon switch for displaying SELinux information). It shows us the context of the current user (actually the context of the id process itself when it was executing). As we can see, the context has a string representation and looks as if it has five fields (it doesn't; it has four fields—the last field just happens to contain a :). SELinux developers decided to use labels instead of real process and file (or other resource) metadata for its access controls. This is different to MAC systems such as AppArmor, which use the path of the binary (and thus the process name) and the paths of the resources to handle permission checks. The decision to make SELinux a label-based mandatory access control was taken for various reasons, which are as follows: Using paths might be easier to comprehend for administrators, but this doesn't allow us to keep the context information close to the resource. If a file or directory is moved or remounted or a process has a different namespace view on the files, then the access controls might behave differently. With label-based contexts, this information is retained and the system keeps controlling the resource properly. Contexts reveal the purpose of the process very well. The same binary application can be launched in different contexts depending on how it got started. The context value (such as the one shown in the id -Z output earlier) is exactly what the administrator needs. With it, he knows what the rights are of each of the running instances, but he can also deduce from it how the process might have been launched and what its purpose is. Contexts also make abstractions of the object itself. We are used to talking about processes and files, but contexts are also applicable to less tangible resources such as pipes (interprocess communication) or database objects. Path-based identification only works as long as you can write a path. As an example, consider the following policies: Allow the httpd processes to bind to TCP port 80 Allow the processes labeled with httpd_t to bind to TCP ports labeled with http_port_t In the first example, we cannot easily reuse this policy when the web server process isn't using the httpd binary (perhaps because it was renamed or it isn't Apache but another web server) or when we want to have HTTP access on a different port. With the labeled approach, the binary can be called apache2 or MyWebServer.py; as long as the process is labeled httpd_t, the policy applies. The same happens with the port definition: you can label port 8080 with http_port_t and thus allow the web servers to bind to that port as well. Dissecting the SELinux context To come to a context, SELinux uses at least three, and sometimes four, values. Let's look at the context of an Apache web server as an example: $ ps -eZ | grep httpd system_u:system_r:httpd_t:s0 511 ? 00:00:00 httpd As we can see, the process is assigned a context that contains following fields: system_u: This represents the SELinux user system_r: This represents the SELinux role httpd_t: This represents the SELinux type (also known as the domain in the case of a process) s0: This represents the sensitivity level This structure can be depicted as follows: The structure of a SELinux context, using the id -Z output as an example When we work with SELinux, contexts are all we need. In the majority of cases, it is the third field (called the domain or type) that is most important since the majority of SELinux policy rules (over 99 percent) consist of rules related to the interaction between two types (without mentioning roles, users, or sensitivity levels). SELinux contexts are aligned with LSM security attributes and exposed to the user space, allowing end users and applications to easily query the contexts. An interesting place where these attributes are presented is within the /proc pseudo filesystem. Inside each process's /proc/<pid> location, we find a subdirectory called attr, inside of which the following files can be found: $ ls /proc/$$/attr current fscreate prev exec keycreate sockcreate All these files, if read, display either nothing or an SELinux context. If it is empty, then that means the application has not explicitly set a context for that particular purpose, and the SELinux context will be deduced either from the policy or inherited from its parent. The meaning of the files are as follows: The current file displays the current SELinux context of the process. The exec file displays the SELinux context that will be assigned by the next application execution done through this application. It is usually empty. The fscreate file displays the SELinux context that will be assigned to the next file that is written by the application. It is usually empty. The keycreate file displays the SELinux context that will be assigned to the keys cached in the kernel by this application. It is usually empty. The prev file displays the previous SELinux context for this particular process. This is usually the context of its parent application. The sockcreate file displays the SELinux context that will be assigned to the next socket created by the application. It is usually empty. If an application has multiple subtasks, then the same information is available in each subtask directory at /proc/<pid>/task/<taskid>/attr. Enforcing access through types The SELinux type (the third part of an SELinux context) of a process (called the domain) is the basis of the fine-grained access controls of that process with respect to itself and other types (which can be processes, files, sockets, network interfaces, and more). In most SELinux literature, the SELinux label-based access control mechanism is fine-tuned to say that SELinux is a type enforcement mandatory access control system: when some actions are denied, the fine-grained access controls on the type level are most likely to blame. With type enforcement, SELinux is able to control what an application is allowed to do based on how it got executed in the first place: a web server that is launched interactively by a user will run with a different type than a web server executed through the init system, even though the process binary and path are the same. The web server launched from the init system is most likely trusted (and thus allowed to do whatever web servers are supposed to do), whereas a manually launched web server is less likely to be considered normal behavior and as such will have different privileges. The majority of SELinux resources will focus on types. Even though the SELinux type is just the third part of an SELinux context, it is the most important one for most administrators. Most documentation will even just talk about a type such as httpd_t rather than a full SELinux context. Take a look at the following dbus-daemon processes: # ps -eZ | grep dbus-daemon system_u:system_r:system_dbusd_t 4531 ? 00:00:00 dbus-daemon staff_u:staff_r:staff_dbusd_t 5266 ? 00:00:00 dbus-daemon In this example, one dbus-daemon process is the system D-Bus daemon running with the aptly named system_dbusd_t type, whereas another one is running with the staff_dbusd_t type assigned to it. Even though their binaries are completely the same, they both serve a different purpose on the system and as such have a different type assigned. SELinux then uses this type to govern the actions allowed by the process towards other types, including how system_dbusd_t can interact with staff_dbusd_t. SELinux types are by convention suffixed with _t, although this is not mandatory. Granting domain access through roles SELinux roles (the second part of an SELinux context) allow SELinux to support role-based access controls. Although type enforcement is the most used (and known) part of SELinux, role-based access control is an important method to keep a system secure, especially from malicious user attempts. SELinux roles are used to define which process types (domains) user processes can be in. As such, they help define what a user can and cannot do. By convention, SELinux roles are defined with an _r suffix. On most SELinux-enabled systems, the following roles are made available to be assigned to users: user_r This role is meant for restricted users: the user_r SELinux role is only allowed to have processes with types specific to end-user applications. Privileged types, including those used to switch to another Linux user, are not allowed for this role. staff_r This role is meant for non-critical operations: the SELinux staff_r role is generally restricted to the same applications as the restricted user, but it has the ability to switch roles. It is the default role for operators to be in (so as to keep those users in the least privileged role as long as possible). sysadm_r This role is meant for system administrators: the sysadm_r SELinux role is very privileged, enabling various system-administration tasks. However, certain end-user application types might not be supported (especially if those types are used for potentially vulnerable or untrusted software) to keep the system free from infections. system_r This role is meant for daemons and background processes: the system_r SELinux role is quite privileged, supporting the various daemon and system process types. However, end-user application types and other administrative types are not allowed in this role. unconfined_r This role is meant for end users: the unconfined_r role is allowed a limited number of types, but those types are very privileged as it is meant for running any application launched by a user in a more or less unconfined manner (not restricted by SELinux rules). This role as such is only available if the system administrator wants to protect certain processes (mostly daemons) while keeping the rest of the system operations almost untouched by SELinux. Other roles might be supported as well, such as guest_r and xguest_r, depending on the distribution. It is wise to consult the distribution documentation for more information about the supported roles. An overview of available roles can be obtained through the seinfo command (part of setools-console in RHEL or app-admin/setools in Gentoo): # seinfo --role Roles: 14 auditadm_r dbadm_r ... unconfined_r Limiting roles through users An SELinux user (the first part of an SELinux context) is different from a Linux user. Unlike Linux user information, which can change while the user is working on the system (through tools such as sudo or su), the SELinux policy can (and generally will) enforce that the SELinux user remain the same even when the Linux user itself has changed. Because of the immutable state of the SELinux user, specific access controls can be implemented to ensure that users cannot work around the set of permissions granted to them, even when they get privileged access. An example of such an access control is the user-based access control (UBAC) feature that some Linux distributions (optionally) enable, which prevents users from accessing files of different SELinux users even when those users try to use the Linux DAC controls to open up access to each other's files. The most important feature of SELinux users, however, is that SELinux user definitions restrict which roles the (Linux) user is allowed to be in. A Linux user is first assigned to an SELinux user—multiple Linux users can be assigned to the same SELinux user. Once set, that user cannot switch to an SELinux role he isn't meant to be in. This is the role-based access control implementation of SELinux: Mapping Linux accounts to SELinux users SELinux users are, by convention, defined with a _u suffix, although this is not mandatory. The SELinux users that most distributions have available are named after the role they represent, but instead of ending with _r, they end with _u. For instance, for the sysadm_r role, there is a sysadm_u SELinux user. Controlling information flow through sensitivities The fourth part of an SELinux context, the sensitivity, is not always present (some Linux distributions by default do not enable sensitivity labels). If they are present though, then this part of the label is needed for the multi-level security (MLS) support within SELinux. Sensitivity labels allow classification of resources and restriction of access to those resources based on a security clearance. These labels consist of two parts: a confidentiality value (prefixed with s) and a category value (prefixed with c). In many larger organizations and companies, documents are labeled internal, confidential, or strictly confidential. SELinux can assign processes a certain clearance level towards these resources. With MLS, SELinux can be configured to follow the Bell-LaPadula model, a security model that can be characterized by no read up and no write down: based on a process clearance level, that process cannot read anything with a higher confidentiality level nor write to (or communicate otherwise with) any resource with a lower confidentiality level. SELinux does not use the internal, confidential, and other labels. Instead, it uses numbers from 0 (lowest confidentiality) to whatever the system administrator has defined as the highest value (this is configurable and set when the SELinux policy is built). Categories allow resources to be tagged with one or more categories, on which access controls are also possible. The idea behind categories is to support multitenancy (for example, systems hosting applications for multiple customers) within a Linux system, by having processes and resources belonging to one tenant to be assigned a particular set of categories, whereas the processes and resources of another tenant get a different set of categories. When a process does not have proper categories assigned, it cannot do anything with the resources (or other processes) that have other categories assigned. An unwritten convention in the SELinux world is that (at least) two categories are used to differentiate between tenants. By having services randomly pick two categories for a tenant out of a predefined set of categories, while ensuring each tenant has a unique combination, these services receive proper isolation. The use of two categories is not mandatory but is implemented by services such as sVirt and Docker. In that sense, categories can be seen as tags, allowing access to be granted only when the tags of the process and the target resource match. As multilevel security is not often used, the benefits of only using categories is persisted in what is called multi-category security (MCS). This is a special MLS case, where only a single confidentiality level is supported (s0). Defining and distributing policies Enabling SELinux does not automatically start the enforcement of access. If SELinux is enabled and it cannot find a policy, it will refuse to start. That is because the policy defines the behavior of the system (what SELinux should allow). SELinux policies are generally distributed in a compiled form (just like with software) as policy modules. These modules are then aggregated into a single policy store and loaded in memory to allow SELinux to enforce the policy rules on the system. Gentoo, being a source-based meta-distribution, distributes the SELinux policies as (source) code as well, which is compiled and built at install time, just like it does with other software. The following diagram shows the relationship between policy rules, policy modules, and a policy package (which is often a one-to-one mapping towards a policy store): Relationship between policy rules, policy modules and policy store Writing SELinux policies A SELinux policy writer can write down the policy rules in (currently) three possible languages: In standard SELinux source format—a human-readable and well-established language for writing SELinux policies In reference policy style—this extends the standard SELinux source format with M4 macros to facilitate the development of policies. In the SELinux Common Intermediate Language (CIL)—a computer-readable (and, with some effort, human-readable) format for SELinux policies. Most SELinux supporting distributions base their policy on the reference policy (https://github.com/TresysTechnology/refpolicy/wiki), a fully functional SELinux policy set managed as a free software project. This allows distributions to ship with a functional policy set rather than having to write one themselves. Many project contributors are distribution developers, trying to push changes of their distribution to the reference policy project itself, where the changes are peer-reviewed to make sure no rules are brought into the project that might jeopardize the security of any platform. It easily becomes very troublesome to write reusable policy modules without the extensive set of M4 macros offered by the reference policy project. The SELinux CIL format is quite recent (RHEL 7.2 does not support it yet), and although it is very much in use already (the recent SELinux user space converts everything in CIL in the background), it is not that common yet for policy writers to use it directly. As an example, consider the web server rule we discussed earlier, repeated here for your convenience: Allow the processes labeled with httpd_t to bind to TCP ports labeled with http_port_t. In the standard SELinux source format, this is written down as follows: allow httpd_t http_port_t : tcp_socket { name_bind }; Using reference policy style, this rule is part of the following macro call: corenet_tcp_bind_http_port(httpd_t) In CIL language, the rule would be expressed as follows: (allow httpd_t http_port_t (tcp_socket (name_bind))) In most representations, we can see what the rule is about: The subject (who is taking the action): In this case, it is a processes labeled with the httpd_t type. The target resource or object (the target for the action): In this case, it is a TCP socket (tcp_socket) labeled with the http_port_t type. In reference policy style, this is implied by the function name. The action or permission: In this case, it is binding to a port (name_bind). In reference policy style, this is implied by the function name. The result that the policy will enforce: In this case, it is that the action is allowed (allow). In reference policy style, this is implied by the function name. A policy is generally written for an application or set of applications. So the preceding example will be part of the policy written for web servers. Policy writers will generally create three files per application or application set: A .te file, which contains the type enforcement rules. An .if file, which contains interface and template definitions, allowing policy writers to easily use the newly generated policy rules to enhance other policies with. You can compare this to header files in other programming languages. An .fc file, which contains file context expressions. These are rules that assign labels to resources on the filesystem. A finished policy will then be packaged into an SELinux policy module. Distributing policies through modules Initially, SELinux used a single, monolithic policy approach: all possible access control rules are maintained in a single policy file. It quickly became clear that this is not manageable in the long term, and the idea of developing a modular policy approach was born. Within the modular approach, policy developers can write isolated policy sets for a particular application (or set of applications), roles, and so on. These policies then get built and distributed as policy modules. Platforms that need access controls for a particular application load the SELinux policy module that defines the access rules for that application. The process of building policy modules is shown in the next diagram. It also shows where CIL comes into play, even when the policy rules themselves are not written in CIL. For distributions that do not yet support CIL, semodule will directly go from the .pp file to the policy.## file. Build process from policy rule to policy store With the recent SELinux user space, the *.pp files (which are the SELinux policy modules) are considered to be written in a high-level language (HLL). Do not assume that this means they are human readable: these files are binary files. The consideration here is that SELinux wants to support writing SELinux policies in a number of formats, which it calls high-level languages, as long as it has a parser that can convert the files into CIL. Marking the binary module formats as high-level allowed the SELinux project to introduce the distinction between high-level languages and CIL in a backward-compatible manner. When distributing SELinux policy modules, most Linux distributions place the *.pp SELinux policy modules inside /usr/share/selinux, usually within a subdirectory named after the policy store (such as targeted). There, these modules are ready for administrators to activate them. When activating a module, the semodule command (part of the policycoreutils package) will copy those modules into a dedicated directory: /etc/selinux/targeted/modules/active/modules (RHEL) or /var/lib/selinux/mcs/active/modules (Gentoo). This location is defined by the version of the SELinux user space—more recent versions use the /var/lib location. When all modules are aggregated in a single location, the final policy binary is compiled, resulting in /etc/selinux/targeted/policy/policy.30 (or some other number) and loaded in memory. On RHEL, the SELinux policies are provided by the selinux-policy-targeted (or -minimum or -mls) package. On Gentoo, they are provided by the various sec-policy/selinux-* packages (Gentoo uses separate packages for each module, reducing the number of SELinux policies that are loaded on an average system). Bundling modules in a policy store A policy store contains a single comprehensive policy, and only a single policy can be active on a system at any point in time. Administrators can switch policy stores, although this often requires the system to be rebooted and might even require relabeling the entire system (relabeling is the act of resetting the contexts on all files and resources available on that system). The active policy on the system can be queried using sestatus (SELinux status, provided through the policycoreutils package), as follows: # sestatus | grep Loaded policy Loaded policy name: targeted In this example, the currently loaded policy (store) is named targeted. The policy name that SELinux will use upon its next reboot is defined in the /etc/selinux/config configuration file as the SELINUXTYPE parameter. It is the init system of systems (be it a SysV-compatible init system or systemd) that is generally responsible for loading the SELinux policy, effectively activating SELinux support on the system. The init system reads the configuration, locates the policy store, and loads the policy file in memory. If the init system does not support this (in other words, it is not SELinux-aware) then the policy can be loaded through the load_policy command, part of the policycoreutils package. Distinguishing between policies The most common SELinux policy store names are strict, targeted, mcs, and mls. None of the names assigned to policy stores are fixed, though, so it is a matter of convention. Hence, it is recommended to consult the distribution documentation to verify what should be the proper name of the policy. Still, the name often provides some information about the SELinux options that are enabled through the policy. Supporting MLS One of the options that can be enabled is MLS support. If it's disabled, the SELinux context will not have a fourth field with sensitivity information in it, making the contexts of processes and files look as follows: staff_u:sysadm_r:sysadm_t To check whether MLS is enabled, it is sufficient to see whether the context indeed doesn't contain such a fourth field, but it can also be acquired from the Policy MLS status line in the output of sestatus: # sestatus | grep MLS Policy MLS Status: disabled Another method would be to look into the pseudo file, /sys/fs/selinux/mls. A value of 0 means disabled, whereas a value of 1 means enabled: # cat /sys/fs/selinux/mls 0 Policy stores that have MLS enabled are generally targeted, mcs and mls, whereas strict generally has MLS disabled. Dealing with unknown permissions Permissions (such as read, open, and lock) are defined both in the Linux kernel and in the policy itself. However, sometimes, newer Linux kernels support permissions that the current policy does not yet understand. Take the block_suspend permission (to be able to block system suspension) as an example. If the Linux kernel supports (and checks) this permission but the loaded SELinux policy does not understand that permission yet, then SELinux has to decide how it should deal with the permission. SELinux can be configured to do one of the following actions: allow: assume everything that is not understood is allowed deny: assume no one is allowed to perform this action reject: stop and halt the system This is configured through the deny_unknown value. To see the state for unknown permissions, look for the Policy deny_unknown status line in sestatus: # sestatus | grep deny_unknown Policy deny_unknown status: denied Administrators can set this for themselves in the /etc/selinux/semanage.conf file through the handle-unknown variable (with allow, deny, or reject). RHEL by default allows unknown permissions, whereas Gentoo by default denies them. Supporting unconfined domains An SELinux policy can be very strict, limiting applications as close as possible to their actual behavior, but it can also be very liberal in what applications are allowed to do. One of the concepts available in many SELinux policies is the idea of unconfined domains. When enabled, it means that certain SELinux domains (process contexts) are allowed to do almost anything they want (of course, within the boundaries of the regular Linux DAC permissions, which still hold) and only a select number of domains are truly confined (restricted) in their actions. Unconfined domains have been brought forward to allow SELinux to be active on desktops and servers where administrators do not want to fully restrict the entire system, but only a few of the applications running on it. Generally, these implementations focus on constraining network-facing services (such as web servers and database management systems) while allowing end users and administrators to roam around unrestricted. With other MAC systems, such as AppArmor, unconfinement is inherently part of the design of the system as they only restrict actions for well-defined applications or users. However, SELinux was designed to be a full mandatory access control system and thus needs to provide access control rules even for those applications that shouldn't need any. By marking these applications as unconfined, almost no additional restrictions are imposed by SELinux. We can see whether unconfined domains are enabled on the system through seinfo, which we use to query the policy for the unconfined_t SELinux type. On a system where unconfined domains are supported, this type will be available: # seinfo -tunconfined_t unconfined_t For a system where unconfined domains are not supported, the type will not be part of the policy: # seinfo -tunconfined_t ERROR: could not find datum for type unconfined_t Most distributions that enable unconfined domains call their policy targeted, but this is just a convention that is not always followed. Hence, it is always best to consult the policy using seinfo. RHEL enables unconfined domains, whereas with Gentoo, this is a configurable setting through the unconfined USE flag. Limiting cross-user sharing When UBAC is enabled, certain SELinux types will be protected by additional constraints. This will ensure that one SELinux user cannot access files (or other specific resources) of another user, even when those users are sharing their data through the regular Linux permissions. UBAC provides some additional control over information flow between resources, but it is far from perfect. In its essence, it is made to isolate SELinux users from one another. A constraint in SELinux is an access control rule that uses all parts of a context to make its decision. Unlike type-enforcement rules, which are purely based on the type, constraints can take the SELinux user, SELinux role, or sensitivity label into account. Constraints are generally developed once and then left untouched—most policy writers will not touch constraints during their development efforts. Many Linux distributions, including RHEL, disable UBAC. Gentoo allows users to select whether or not they want UBAC through the Gentoo ubac USE flag (which is enabled by default). Incrementing policy versions While checking the output of sestatus, we see that there is also a notion of policy versions: # sestatus | grep version Max kernel policy version: 28 This version has nothing to do with the versioning of policy rules but with the SELinux features that the currently running kernel supports. In the preceding output, 28 is the highest policy version the kernel supports. Every time a new feature is added to SELinux, the version number is increased. The policy file itself (which contains all the SELinux rules loaded at boot time by the system) can be found in /etc/selinux/targeted/policy (where targeted refers to the policy store used, so if the system uses a policy store named strict, then the path would be /etc/selinux/strict/policy). If multiple policy files exist, we can use the output of seinfo to find out which policy file is used: # seinfo Statistics for policy file: /etc/selinux/targeted/policy/policy.30 Policy Version & Type: v.30 (binary, mls) ... The next table provides the current list of policy feature enhancements and the Linux kernel version in which that feature is introduced. Many of the features are only of concern to the policy developers, but knowing the evolution of the features gives us a good idea about the evolution of SELinux. Version Linux kernel Description 12   The old API for SELinux, now deprecated. 15 2.6.0 Introduced the new API for SELinux. 16 2.6.5 Added support for conditional policy extensions. 17 2.6.6 Added support for IPv6. 18 2.6.8 Added support for fine-grained netlink socket permissions. 19 2.6.12 Added support for MLS. 20 2.6.14 Reduced the size of the access vector table. 21 2.6.19 Added support for MLS range transitions. 22 2.6.25 Introduced policy capabilities. 23 2.6.26 Added support for per-domain permissive mode. 24 2.6.28 Added support for explicit hierarchy (type bounds). 25 2.6.39 Added support for filename-based transitions. 26 3.0 Added support for role transitions for non-process classes. Added support for role attributes. 27 3.5 Added support for flexible inheritance of user and role for newly created objects. 28 3.5 Added support for flexible inheritance of type for newly created objects. 29 3.14 Added support for attributes within SELinux constraints. 30 4.3 Added support for extended permissions and implemented first on IOCTL controls. Enhanced SELinux XEN support. History of SELinux feature evolution By default, when an SELinux policy is built, the highest supported version as defined by the Linux kernel and libsepol (the library responsible for building the SELinux policy binary) is used. Administrators can force a version to be lower using the policy-version parameter in /etc/selinux/semanage.conf. Different policy content Besides the aforementioned policy capabilities, the main difference between policies (and distributions) is the policy content itself. We already covered that most distributions base their policy on the reference policy project. But although that project is considered the master for most distributions, each distribution has its own deviation from the main policy set. Many distributions make extensive additions to the policy without directly passing the policies to the upstream reference policy project. There are several possible reasons why this is not directly done: The policy enhancements or additions are still immature: Red Hat initially starts with policies being active but permissive, meaning the policies are not enforced. Instead, SELinux logs what it would have prevented and, based on those, logs the policies that are enhanced. This ensures that a policy is only ready after a few releases. The policy enhancements or additions are too specific to the distribution: If a policy set is not reusable for other distributions, then some distributions will opt to keep those policies to themselves as the act of pushing changes to upstream projects takes quite some effort. The policy enhancements or additions haven't followed the upstream rules and guidelines: The reference policy has a set of guidelines that policies need to adhere to. If a policy set does not comply with these rules, then it will not be accepted. The policy enhancements or additions are not implementing the same security model as the reference policy project wants: As SELinux is a very extensive mandatory access control system, it is possible to write completely different policies. The distribution does not have the time or resources to push changes upstream. This ensures that SELinux policies between distributions (and even releases of the same distribution) can, content-wise, be quite different. Gentoo for instance aims to follow the reference policy project closely, with changes being merged within a matter of weeks. Summary In this article, we saw that SELinux offers a more fine-grained access control mechanism on top of the Linux access controls. SELinux is implemented through Linux Security Modules and uses labels to identify its resources and processes based on ownership (user), role, type, and even the security sensitivity and categorization of the resource. We covered how SELinux policies are handled within an SELinux-enabled system and briefly touched upon how policy writers structure policies. Linux distributions implement SELinux policies, which might be a bit different from each other based on supporting features, such as sensitivity labels, default behavior for unknown permissions, support for confinement levels, or specific constraints put in place such as UBAC. However, most of the policy rules themselves are similar and are even based on the same upstream reference policy project. Resources for Article: Further resources on this subject: SELinux - Highly Secured Web Hosting for Python-based Web Applications [article] Introduction to Docker [article] Booting the System [article]
Read more
  • 0
  • 0
  • 13507

article-image-rounding
Packt
25 Nov 2013
3 min read
Save for later

Rounding up...

Packt
25 Nov 2013
3 min read
(For more resources related to this topic, see here.) We have now successfully learned how to secure our users' passwords using hashes; however, we should take a look at the big picture, just in case. The following figure shows what a very basic web application looks like: Note the https transmission tag: HTTPS is a secure transfer protocol, which allows us to transport information in a secure way. When we transport sensitive data such as passwords in a Web Application, anyone who intercepts the connection can easily get the password in plain text, and our users' data would be compromised. In order to avoid this, we should always use HTTPS when there's sensitive data involved. HTTPS is fairly easy to setup, you just need to buy an SSL certificate and configure it with your hosting provider. Configuration varies depending on the provider, but usually they provide an easy way to do it. It is strongly suggested to use HTTPS for authentication, sign up, sign in, and other sensitive data processes. As a general rule, most (if not all) of the data exchange that requires the user to be logged in should be protected. Keep in mind that HTTPS comes at a cost, so try to avoid using HTTPS on static pages that have public information. Always keep in mind that to protect the password, we need ensure secure transport (with HTTPS) and secure storage (with strong hashes) as well. Both are critical phases and we need to be very careful with them. Now that our passwords and other sensitive data are being transferred in a secure way, we can get into the application workflow. Consider the following steps for an authentication process: The application receives an Authentication Request. The Web Layer takes care of it as it gets the parameters (username and password), and passes them to the Authentication Service. The Authentication Service calls the Database Access Layer to retrieve the user from the database. The Database Access Layer queries the database, gets the user, and returns it to the Authentication Service. The Authentication Service gets the stored hash from the users' data retrieved from the database, extracts the salt and the amount of iterations, and calls the Hashing Utility passing the password from the authentication request, the salt, and the iterations. The Hashing Utility generates the hash and returns it to the Authentication Service. The Authentication Service performs a constant-time comparison between the stored hash and the generated hash, and we inform the Web Layer if the user is authenticated or not. The Web Layer returns the corresponding view to the user depending on whether they are authenticated or not. The following figure can help us understand how this works, please consider that flows 1, 2, 3, and 4 are bidirectional: The Authentication Service and the Hashing Utility components are the ones we have been working with so far. We already know how to create hashes, this workflow is an example to understand when we should it. Summary In this article we learned how to create hashes and have now successfully learned how to secure our users' passwords using hashes. We have also learned that we need to ensure secure transport (with HTTPS) and secure storage (with strong hashes) as well. Resources for Article: Further resources on this subject: FreeRADIUS Authentication: Storing Passwords [Article] EJB 3.1: Controlling Security Programmatically Using JAAS [Article] So, what is Spring for Android? [Article]
Read more
  • 0
  • 0
  • 13507

article-image-how-to-create-observables-in-rxjs-tutorial
Sugandha Lahoti
10 Apr 2019
7 min read
Save for later

How to create observables in RxJS [Tutorial]

Sugandha Lahoti
10 Apr 2019
7 min read
Reactive programming requires us to change the way that we think about events in an application. Reactive programming requires us to think about events as a stream of values. For example, a mouse click event can be represented as a stream of data. Every click event generates a new value in the data stream. In reactive programming, we can use the stream of data to query and manipulate the values in the stream. Observables are streams of data, and this explains why it is easy to imagine that we can represent an event such as an onClick event using an observable. However, the use cases for observables are much more diverse than that. In this article, we are going to explore how to create an observable given different types. This article is taken from the book Hands-On Functional Programming with TypeScript by Remo H. Jansen. In this book, you will discover the power of functional programming, lazy evaluation, monads, concurrency, and immutability to create succinct and expressive implementations. Creating observables from a value We can create an observable given a value using the of function. In the old versions of RxJS, the function of was a static method of the Observable class, which was available as Observable.of. This should remind us to use the of method of the Applicative type in category theory because observables take some inspiration from category theory. However, in RxJS 6.0, the of method is available as a standalone factory function: import { of } from "rxjs"; const observable = of(1); const subscription = observable.subscribe( (value) => console.log(value), (error: any) => console.log(error), () => console.log("Done!") ); subscription.unsubscribe(); The preceding code snippet declares an observable with one unique value using the of function. The code snippet also showcases how we can subscribe to an observable using the subscribe method. The subscribe method takes three function arguments: Item handler: Invoked once for each item in the sequence. Error handler: Invoked if there is an error in the sequence. This argument is optional. Done handler: Invoked when there are no more items in the sequence. This argument is optional. The following diagram is known as a marble diagram and is used to represent observables in a visual manner. The arrow represents the time and the circles are values. In this case, we have only one value: As we can see, the circle also has a small vertical line in the middle. This line is used to represent the last element in an observable. In this case, the item handler in the subscription will only be invoked once. Creating observables from arrays We can create an observable given an existing array using the from function: import { from } from "rxjs"; const observable = from([10, 20, 30]); const subscription = observable.subscribe( (value) => console.log(value), (error: any) => console.log(error), () => console.log("Done!") ); subscription.unsubscribe(); The preceding code snippet declares an observable with three values using the from function. The code snippet also showcases how we can subscribe once more. The following marble diagram represents the preceding example in a visual manner. The generated observable has three values (10, 20, and 30) and 30 is the last element in the observable: We can alternatively use the interval function to generate an array with a given number of elements: import { interval } from "rxjs"; const observable = interval(10); const subscription = observable.subscribe( (value) => console.log(value), (error: any) => console.log(error), () => console.log("Done!") ); subscription.unsubscribe(); The preceding code snippet declares an observable with ten values using the interval function. The code snippet also showcases how we can subscribe once more. In this case, the item handler in the subscription will be invoked ten times. The following marble diagram represents the preceding example in a visual manner. The generating observable has ten values, and 9 is the last item contained by it:  In this case, the item handler in the subscription will be invoked ten times. Creating observables from events It is also possible to create an observable using an event as the source of the items in the stream. We can do this using the fromEvent function: import { fromEvent } from "rxjs"; const observable = fromEvent(document, "click"); const subscription = observable.subscribe( (value) => console.log(value) ); subscription.unsubscribe(); In this case, the item handler in the subscription will be invoked as many times as the click event takes place. Please note that the preceding example can only be executed in a web browser. To execute the preceding code in a web browser, you will need to use a module bundler, such as Webpack. Creating observables from callbacks It is also possible to create an observable that will iterate the arguments of a callback using the bindCallback function: import { bindCallback } from "rxjs"; import fetch from "node-fetch"; function getJSON(url: string, cb: (response: unknown|null) => void) { fetch(url) .then(response => response.json()) .then(json => cb(json)) .catch(_ => cb(null)); } const uri = "https://jsonplaceholder.typicode.com/todos/1"; const observableFactory = bindCallback(getJSON); const observable = observableFactory(uri); const subscription = observable.subscribe( (value) => console.log(value) ); subscription.unsubscribe(); The preceding example uses the node-fetch module because the fetch function is not available in Node.js. You can install the node-fetch module using the following npm command: npm install node-fetch @types/node-fetch The getJSON function takes a URL and a callback as its arguments. When we pass it to the bindCallback function, a new function is returned. The new function takes a URL as its only argument and returns an observable instead of taking a callback. In Node.js, callbacks follow a well-defined pattern. The Node.js callbacks take two arguments, error and result, and don't throw exceptions. We must use the error argument to check whether something went wrong instead of a try/catch statement. RxJS also defines a function named bindNodeCallback that allows us to work with the callbacks: import { bindNodeCallback } from "rxjs"; import * as fs from "fs"; const observableFactory = bindNodeCallback(fs.readFile); const observable = observableFactory("./roadNames.txt"); const subscription = observable.subscribe( (value) => console.log(value.toString()) ); subscription.unsubscribe(); The helpers, bindCallback and bindNodeCallback, have very similar behavior, but the second has been specially designed to work with Node.js callbacks. Creating observables from promises Another potential source of items for an observable sequence is a Promise. RxJS also allows us to handle this use case with the from function. We must pass a Promise instance to the from function. In the following example, we use the fetch function to send an HTTP request. The fetch function returns a promise that is passed to the from function: import { bindCallback } from "rxjs"; import fetch from "node-fetch"; const uri = "https://jsonplaceholder.typicode.com/todos/1"; const observable = from(fetch(uri)).pipe(map(x => x.json())); const subscription = observable.subscribe( (value) => console.log(value.toString()) ); subscription.unsubscribe(); The generated observable will contain the result of the promise as its only item. Cold and hot observables The official RxJS documentation explores the differences between cold and hot observables as follows: "Cold observables start running upon subscription, that is, the observable sequence only starts pushing values to the observers when Subscribe is called. Values are also not shared among subscribers. This is different from hot observables, such as mouse move events or stock tickers, which are already producing values even before a subscription is active. When an observer subscribes to a hot observable sequence, it will get all values in the stream that are emitted after it subscribes. The hot observable sequence is shared among all subscribers, and each subscriber is pushed to the next value in the sequence." It is important to understand these differences if we want to have control over the execution flow of our components. The key point to remember is that cold observables are lazily evaluated. In this article, we learned what observables are and how we can create them and work with them. To know more about working with observables, and other aspects of functional programming, read our book Hands-On Functional Programming with TypeScript. What makes functional programming a viable choice for artificial intelligence projects? Why functional programming in Python matters: Interview with best selling author, Steven Lott Introducing Coconut for making functional programming in Python simpler
Read more
  • 0
  • 0
  • 13498

article-image-installing-tensorflow-in-windows-ubuntu-and-mac-os
Amarabha Banerjee
21 Feb 2018
7 min read
Save for later

Installing TensorFlow in Windows, Ubuntu and Mac OS

Amarabha Banerjee
21 Feb 2018
7 min read
[box type="note" align="" class="" width=""]This article is taken from the book Machine Learning with Tensorflow 1.x, written by Quan Hua, Shams Ul Azeem and Saif Ahmed. This book will help tackle common commercial machine learning problems with Google’s TensorFlow 1.x library.[/box] Today, we shall explore the basics of getting started with TensorFlow, its installation and configuration process. The proliferation of large public datasets, inexpensive GPUs, and open-minded developer culture has revolutionized machine learning efforts in recent years. Training data, the lifeblood of machine learning, has become widely available and easily consumable in recent years. Computing power has made the required horsepower available to small businesses and even individuals. The current decade is incredibly exciting for data scientists. Some of the top platforms used in the industry include Caffe, Theano, and Torch. While the underlying platforms are actively developed and openly shared, usage is limited largely to machine learning practitioners due to difficult installations, non-obvious configurations, and difficulty with productionizing solutions. TensorFlow has one of the easiest installations of any platform, bringing machine learning capabilities squarely into the realm of casual tinkerers and novice programmers. Meanwhile, high-performance features, such as—multiGPU support, make the platform exciting for experienced data scientists and industrial use as well. TensorFlow also provides a reimagined process and multiple user-friendly utilities, such as TensorBoard, to manage machine learning efforts. Finally, the platform has significant backing and community support from the world's largest machine learning powerhouse--Google. All this is before even considering the compelling underlying technical advantages, which we'll dive into later. Installing TensorFlow TensorFlow conveniently offers several types of installation and operates on multiple operating systems. The basic installation is CPU-only, while more advanced installations unleash serious horsepower by pushing calculations onto the graphics card, or even to multiple graphics cards. We recommend starting with a basic CPU installation at first. More complex GPU and CUDA installations will be discussed in Appendix, Advanced Installation. Even with just a basic CPU installation, TensorFlow offers multiple options, which are as follows: A basic Python pip installation A segregated Python installation via Virtualenv A fully segregated container-based installation via Docker Ubuntu installation Ubuntu is one of the best Linux distributions for working with Tensorflow. We highly recommend that you use an Ubuntu machine, especially if you want to work with GPU. We will do most of our work on the Ubuntu terminal. We will begin with installing pythonpip and python-dev via the following command: sudo apt-get install python-pip python-dev A successful installation will appear as follows: If you find missing packages, you can correct them via the following command: sudo apt-get update --fix-missing Then, you can continue the python and pip installation. We are now ready to install TensorFlow. The CPU installation is initiated via the following command: sudo pip install tensorflow A successful installation will appear as follows: macOS installation If you use Python, you will probably already have the Python package installer, pip. However, if not, you can easily install it using the easy_install pip command. You'll note that we actually executed sudo easy_install pip—the sudo prefix was required because the installation requires administrative rights. We will make the fair assumption that you already have the basic package installer, easy_install, available; if not, you can install it from https://pypi.python.org/pypi/setuptools. A successful installation will appear as shown in the following screenshot: Next, we will install the six package: sudo easy_install --upgrade six A successful installation will appear as shown in the following screenshot: Surprisingly, those are the only two prerequisites for TensorFlow, and we can now install the core platform. We will use the pip package installer mentioned earlier and install TensorFlow directly from Google's site. The most recent version at the time of writing this book is v1.3, but you should change this to the latest version you wish to use: sudo pip install tensorflow The pip installer will automatically gather all the other required dependencies. You will see each individual download and installation until the software is fully installed. A successful installation will appear as shown in the following screenshot: That's it! If you were able to get to this point, you can start to train and run your first model. Skip to Chapter 2, Your First Classifier, to train your first model. macOS X users wishing to completely segregate their installation can use a VM instead, as described in the Windows installation. Windows installation As we mentioned earlier, TensorFlow with Python 2.7 does not function natively on Windows. In this section, we will guide you through installing TensorFlow with Python 3.5 and set up a VM with Linux if you want to use TensorFlow with Python 2.7. First, we need to install Python 3.5.x or 3.6.x 64-bit from the following links: https://www.python.org/downloads/release/python-352/ https://www.python.org/downloads/release/python-362/ Make sure that you download the 64-bit version of Python where the name of the installation has amd64, such as python-3.6.2-amd64.exe. The Python 3.6.2 installation looks like this: We will select Add Python 3.6 to PATH and click Install Now. The installation process will complete with the following screen: We will click the Disable path length limit and then click Close to finish the Python installation. Now, let's open the Windows PowerShell application under the Windows menu. We will install the CPU-only version of Tensorflow with the following command: pip3 install tensorflow. The result of the installation will look like this: Congratulations, you can now use TensorFlow on Windows with Python 3.5.x or 3.6.x support. In the next section, we will show you how to set up a VM to use TensorFlow with Python 2.7. However, you can skip to the Test installation section of Chapter 2, Your First Classifier, if you don't need Python 2.7. Now, we will show you how to set up a VM with Linux to use TensorFlow with Python 2.7. We recommend the free VirtualBox system available at https://www.virtualbox.org/wiki/Downloads. The latest stable version at the time of writing is v5.0.14, available at the following URL: http:/ / download. virtualbox. org/ virtualbox/ 5. 1. 28/ VirtualBox- 5. 1. 28- 117968- Win. exe A successful installation will allow you to run the Oracle VM VirtualBox Manager dashboard, which looks like this: Testing the installation In this section, we will use TensorFlow to compute a simple math operation. First, open your terminal on Linux/macOS or Windows PowerShell in Windows. Now, we need to run python to use TensorFlow with the following command: python Enter the following program in the Python shell: import tensorflow as tf a = tf.constant(1.0) b = tf.constant(2.0) c = a + b sess = tf.Session() print(sess.run(c)) The result will look like the following screen where 3.0 is printed at the end: We covered TensorFlow installation on the three major operating systems, so that you are up and running with the platform. Windows users faced an extra challenge, as TensorFlow on Windows only supports Python 3.5.x or Python 3.6.x 64-bit version. However, even Windows users should now be up and running. Further get a detailed understanding of implementing Tensorflow with contextual examples in this post. If you liked this article, be sure to check out Machine Learning with Tensorflow 1.x which will help you take up any challenge you may face while implementing TensorFlow 1.x in your machine learning environment.  
Read more
  • 0
  • 0
  • 13492
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-writing-blog-application-nodejs-and-angularjs
Packt
16 Feb 2016
35 min read
Save for later

Writing a Blog Application with Node.js and AngularJS

Packt
16 Feb 2016
35 min read
In this article, we are going to build a blog application by using Node.js and AngularJS. Our system will support adding, editing, and removing articles, so there will be a control panel. The MongoDB or MySQL database will handle the storing of the information and the Express framework will be used as the site base. It will deliver the JavaScript, CSS, and the HTML to the end user, and will provide an API to access the database. We will use AngularJS to build the user interface and control the client-side logic in the administration page. (For more resources related to this topic, see here.) This article will cover the following topics: AngularJS fundamentals Choosing and initializing a database Implementing the client-side part of an application with AngularJS Exploring AngularJS AngularJS is an open source, client-side JavaScript framework developed by Google. It's full of features and is really well documented. It has almost become a standard framework in the development of single-page applications. The official site of AngularJS, http://angularjs.org, provides a well-structured documentation. As the framework is widely used, there is a lot of material in the form of articles and video tutorials. As a JavaScript library, it collaborates pretty well with Node.js. In this article, we will build a simple blog with a control panel. Before we start developing our application, let's first take a look at the framework. AngularJS gives us very good control over the data on our page. We don't have to think about selecting elements from the DOM and filling them with values. Thankfully, due to the available data-binding, we may update the data in the JavaScript part and see the change in the HTML part. This is also true for the reverse. Once we change something in the HTML part, we get the new values in the JavaScript part. The framework has a powerful dependency injector. There are predefined classes in order to perform AJAX requests and manage routes. You could also read Mastering Web Development with AngularJS by Peter Bacon Darwin and Pawel Kozlowski, published by Packt Publishing. Bootstrapping AngularJS applications To bootstrap an AngularJS application, we need to add the ng-app attribute to some of our HTML tags. It is important that we pick the right one. Having ng-app somewhere means that all the child nodes will be processed by the framework. It's common practice to put that attribute on the <html> tag. In the following code, we have a simple HTML page containing ng-app: <html ng-app> <head> <script src="angular.min.js"></script> </head> <body> ... </body> </html>   Very often, we will apply a value to the attribute. This will be a module name. We will do this while developing the control panel of our blog application. Having the freedom to place ng-app wherever we want means that we can decide which part of our markup will be controlled by AngularJS. That's good, because if we have a giant HTML file, we really don't want to spend resources parsing the whole document. Of course, we may bootstrap our logic manually, and this is needed when we have more than one AngularJS application on the page. Using directives and controllers In AngularJS, we can implement the Model-View-Controller pattern. The controller acts as glue between the data (model) and the user interface (view). In the context of the framework, the controller is just a simple function. For example, the following HTML code illustrates that a controller is just a simple function: <html ng-app> <head> <script src="angular.min.js"></script> <script src="HeaderController.js"></script> </head> <body> <header ng-controller="HeaderController"> <h1>{{title}}</h1> </header> </body> </html>   In <head> of the page, we are adding the minified version of the library and HeaderController.js; a file that will host the code of our controller. We also set an ng-controller attribute in the HTML markup. The definition of the controller is as follows: function HeaderController($scope) { $scope.title = "Hello world"; } Every controller has its own area of influence. That area is called the scope. In our case, HeaderController defines the {{title}} variable. AngularJS has a wonderful dependency-injection system. Thankfully, due to this mechanism, the $scope argument is automatically initialized and passed to our function. The ng-controller attribute is called the directive, that is, an attribute, which has meaning to AngularJS. There are a lot of directives that we can use. That's maybe one of the strongest points of the framework. We can implement complex logic directly inside our templates, for example, data binding, filtering, or modularity. Data binding Data binding is a process of automatically updating the view once the model is changed. As we mentioned earlier, we can change a variable in the JavaScript part of the application and the HTML part will be automatically updated. We don't have to create a reference to a DOM element or attach event listeners. Everything is handled by the framework. Let's continue and elaborate on the previous example, as follows: <header ng-controller="HeaderController"> <h1>{{title}}</h1> <a href="#" ng-click="updateTitle()">change title</a> </header>   A link is added and it contains the ng-click directive. The updateTitle function is a function defined in the controller, as seen in the following code snippet: function HeaderController($scope) { $scope.title = "Hello world"; $scope.updateTitle = function() { $scope.title = "That's a new title."; } }   We don't care about the DOM element and where the {{title}} variable is. We just change a property of $scope and everything works. There are, of course, situations where we will have the <input> fields and we want to bind their values. If that's the case, then the ng-model directive can be used. We can see this as follows: <header ng-controller="HeaderController"> <h1>{{title}}</h1> <a href="#" ng-click="updateTitle()">change title</a> <input type="text" ng-model="title" /> </header>   The data in the input field is bound to the same title variable. This time, we don't have to edit the controller. AngularJS automatically changes the content of the h1 tag. Encapsulating logic with modules It's great that we have controllers. However, it's not a good practice to place everything into globally defined functions. That's why it is good to use the module system. The following code shows how a module is defined: angular.module('HeaderModule', []); The first parameter is the name of the module and the second one is an array with the module's dependencies. By dependencies, we mean other modules, services, or something custom that we can use inside the module. It should also be set as a value of the ng-app directive. The code so far could be translated to the following code snippet: angular.module('HeaderModule', []) .controller('HeaderController', function($scope) { $scope.title = "Hello world"; $scope.updateTitle = function() { $scope.title = "That's a new title."; } });   So, the first line defines a module. We can chain the different methods of the module and one of them is the controller method. Following this approach, that is, putting our code inside a module, we will be encapsulating logic. This is a sign of good architecture. And of course, with a module, we have access to different features such as filters, custom directives, and custom services. Preparing data with filters The filters are very handy when we want to prepare our data, prior to be displayed to the user. Let's say, for example, that we need to mention our title in uppercase once it reaches a length of more than 20 characters: angular.module('HeaderModule', []) .filter('customuppercase', function() { return function(input) { if(input.length > 20) { return input.toUpperCase(); } else { return input; } }; }) .controller('HeaderController', function($scope) { $scope.title = "Hello world"; $scope.updateTitle = function() { $scope.title = "That's a new title."; } });   That's the definition of the custom filter called customuppercase. It receives the input and performs a simple check. What it returns, is what the user sees at the end. Here is how this filter could be used in HTML: <h1>{{title | customuppercase}}</h1> Of course, we may add more than one filter per variable. There are some predefined filters to limit the length, such as the JavaScript to JSON conversion or, for example, date formatting. Dependency injection Dependency management can be very tough sometimes. We may split everything into different modules/components. They have nicely written APIs and they are very well documented. However, very soon, we may realize that we need to create a lot of objects. Dependency injection solves this problem by providing what we need, on the fly. We already saw this in action. The $scope parameter passed to our controller, is actually created by the injector of AngularJS. To get something as a dependency, we need to define it somewhere and let the framework know about it. We do this as follows: angular.module('HeaderModule', []) .factory("Data", function() { return { getTitle: function() { return "A better title."; } } }) .controller('HeaderController', function($scope, Data) { $scope.title = Data.getTitle(); $scope.updateTitle = function() { $scope.title = "That's a new title."; } });   The Module class has a method called factory. It registers a new service that could later be used as a dependency. The function returns an object with only one method, getTitle. Of course, the name of the service should match the name of the controller's parameter. Otherwise, AngularJS will not be able to find the dependency's source. The model in the context of AngularJS In the well-known Model-View-Controller pattern, the model is the part that stores the data in the application. AngularJS doesn't have a specific workflow to define models. The $scope variable could be considered a model. We keep the data in properties attached to the current scope. Later, we can use the ng-model directive and bind a property to the DOM element. We already saw how this works in the previous sections. The framework may not provide the usual form of a model, but it's made like that so that we can write our own implementation. The fact that AngularJS works with plain JavaScript objects, makes this task easily doable. Final words on AngularJS AngularJS is one of the leading frameworks, not only because it is made by Google, but also because it's really flexible. We could use just a small piece of it or build a solid architecture using the giant collection of features. Selecting and initializing the database To build a blog application, we need a database that will store the published articles. In most cases, the choice of the database depends on the current project. There are factors such as performance and scalability and we should keep them in mind. In order to have a better look at the possible solutions, we will have a look at the two of the most popular databases: MongoDB and MySQL. The first one is a NoSQL type of database. According to the Wikipedia entry (http://en.wikipedia.org/wiki/ NoSQL) on NoSQL databases: "A NoSQL or Not Only SQL database provides a mechanism for storage and retrieval of data that is modeled in means other than the tabular relations used in relational databases." In other words, it's simpler than a SQL database, and very often stores information in the key value type. Usually, such solutions are used when handling and storing large amounts of data. It is also a very popular approach when we need flexible schema or when we want to use JSON. It really depends on what kind of system we are building. In some cases, MySQL could be a better choice, while in some other cases, MongoDB. In our example blog, we're going to use both. In order to do this, we will need a layer that connects to the database server and accepts queries. To make things a bit more interesting, we will create a module that has only one API, but can switch between the two database models. Using NoSQL with MongoDB Let's start with MongoDB. Before we start storing information, we need a MongoDB server running. It can be downloaded from the official page of the database https://www.mongodb.org/downloads. We are not going to handle the communication with the database manually. There is a driver specifically developed for Node.js. It's called mongodb and we should include it in our package.json file. After successful installation via npm install, the driver will be available in our scripts. We can check this as follows: "dependencies": { "mongodb": "1.3.20" }   We will stick to the Model-View-Controller architecture and the database-related operations in a model called Articles. We can see this as follows: var crypto = require("crypto"), type = "mongodb", client = require('mongodb').MongoClient, mongodb_host = "127.0.0.1", mongodb_port = "27017", collection; module.exports = function() { if(type == "mongodb") { return { add: function(data, callback) { ... }, update: function(data, callback) { ... }, get: function(callback) { ... }, remove: function(id, callback) { ... } } } else { return { add: function(data, callback) { ... }, update: function(data, callback) { ... }, get: function(callback) { ... }, remove: function(id, callback) { ... } } } }   It starts with defining a few dependencies and settings for the MongoDB connection. Line number one requires the crypto module. We will use it to generate unique IDs for every article. The type variable defines which database is currently accessed. The third line initializes the MongoDB driver. We will use it to communicate with the database server. After that, we set the host and port for the connection and at the end a global collection variable, which will keep a reference to the collection with the articles. In MongoDB, the collections are similar to the tables in MySQL. The next logical step is to establish a database connection and perform the needed operations, as follows: connection = 'mongodb://'; connection += mongodb_host + ':' + mongodb_port; connection += '/blog-application'; client.connect(connection, function(err, database) { if(err) { throw new Error("Can't connect"); } else { console.log("Connection to MongoDB server successful."); collection = database.collection('articles'); } });   We pass the host and the port, and the driver is doing everything else. Of course, it is a good practice to handle the error (if any) and throw an exception. In our case, this is especially needed because without the information in the database, the frontend has nothing to show. The rest of the module contains methods to add, edit, retrieve, and delete records: return { add: function(data, callback) { var date = new Date(); data.id = crypto.randomBytes(20).toString('hex'); data.date = date.getFullYear() + "-" + date.getMonth() + "-" + date.getDate(); collection.insert(data, {}, callback || function() {}); }, update: function(data, callback) { collection.update( {ID: data.id}, data, {}, callback || function(){ } ); }, get: function(callback) { collection.find({}).toArray(callback); }, remove: function(id, callback) { collection.findAndModify( {ID: id}, [], {}, {remove: true}, callback ); } }   The add and update methods accept the data parameter. That's a simple JavaScript object. For example, see the following code: { title: "Blog post title", text: "Article's text here ..." }   The records are identified by an automatically generated unique id. The update method needs it in order to find out which record to edit. All the methods also have a callback. That's important, because the module is meant to be used as a black box, that is, we should be able to create an instance of it, operate with the data, and at the end continue with the rest of the application's logic. Using MySQL We're going to use an SQL type of database with MySQL. We will add a few more lines of code to the already working Articles.js model. The idea is to have a class that supports the two databases like two different options. At the end, we should be able to switch from one to the other, by simply changing the value of a variable. Similar to MongoDB, we need to first install the database to be able use it. The official download page is http://www.mysql.com/downloads. MySQL requires another Node.js module. It should be added again to the package. json file. We can see the module as follows: "dependencies": { "mongodb": "1.3.20", "mysql": "2.0.0" }   Similar to the MongoDB solution, we need to firstly connect to the server. To do so, we need to know the values of the host, username, and password fields. And because the data is organized in databases, a name of the database. In MySQL, we put our data into different databases. So, the following code defines the needed variables: var mysql = require('mysql'), mysql_host = "127.0.0.1", mysql_user = "root", mysql_password = "", mysql_database = "blog_application", connection;   The previous example leaves the password field empty but we should set the proper value of our system. The MySQL database requires us to define a table and its fields before we start saving data. So, the following code is a short dump of the table used in this article: CREATE TABLE IF NOT EXISTS `articles` ( `id` int(11) NOT NULL AUTO_INCREMENT, `title` longtext NOT NULL, `text` longtext NOT NULL, `date` varchar(100) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 AUTO_INCREMENT=1;   Once we have a database and its table set, we can continue with the database connection, as follows: connection = mysql.createConnection({ host: mysql_host, user: mysql_user, password: mysql_password }); connection.connect(function(err) { if(err) { throw new Error("Can't connect to MySQL."); } else { connection.query("USE " + mysql_database, function(err, rows, fields) { if(err) { throw new Error("Missing database."); } else { console.log("Successfully selected database."); } }) } });   The driver provides a method to connect to the server and execute queries. The first executed query selects the database. If everything is ok, you should see Successfully selected database as an output in your console. Half of the job is done. What we should do now is replicate the methods returned in the first MongoDB implementation. We need to do this because when we switch to the MySQL usage, the code using the class will not work. And by replicating them we mean that they should have the same names and should accept the same arguments. If we do everything correctly, at the end our application will support two types of databases. And all we have to do is change the value of the type variable: return { add: function(data, callback) { var date = new Date(); var query = ""; query += "INSERT INTO articles (title, text, date) VALUES ("; query += connection.escape(data.title) + ", "; query += connection.escape(data.text) + ", "; query += "'" + date.getFullYear() + "-" + date.getMonth() + "-" + date.getDate() + "'"; query += ")"; connection.query(query, callback); }, update: function(data, callback) { var query = "UPDATE articles SET "; query += "title=" + connection.escape(data.title) + ", "; query += "text=" + connection.escape(data.text) + " "; query += "WHERE id='" + data.id + "'"; connection.query(query, callback); }, get: function(callback) { var query = "SELECT * FROM articles ORDER BY id DESC"; connection.query(query, function(err, rows, fields) { if (err) { throw new Error("Error getting."); } else { callback(rows); } }); }, remove: function(id, callback) { var query = "DELETE FROM articles WHERE id='" + id + "'"; connection.query(query, callback); } }   The code is a little longer than the one generated in the first MongoDB variant. That's because we needed to construct MySQL queries from the passed data. Keep in mind that we have to escape the information, which comes to the module. That's why we use connection.escape(). With these lines of code, our model is completed. Now we can add, edit, remove, or get data. Let's continue with the part that shows the articles to our users. Developing the client side with AngularJS Let's assume that there is some data in the database and we are ready to present it to the users. So far, we have only developed the model, which is the class that takes care of the access to the information. To simplify the process, we will Express here. We need to first update the package.json file and include that in the framework, as follows: "dependencies": { "express": "3.4.6", "jade": "0.35.0", "mongodb": "1.3.20", "mysql": "2.0.0" }   We are also adding Jade, because we are going to use it as a template language. The writing of markup in plain HTML is not very efficient nowadays. By using the template engine, we can split the data and the HTML markup, which makes our application much better structured. Jade's syntax is kind of similar to HTML. We can write tags without the need to close them: body p(class="paragraph", data-id="12"). Sample text here footer a(href="#"). my site   The preceding code snippet is transformed to the following code snippet: <body> <p data-id="12" class="paragraph">Sample text here</p> <footer><a href="#">my site</a></footer> </body>   Jade relies on the indentation in the content to distinguish the tags. Let's start with the project structure, as seen in the following screenshot: We placed our already written class, Articles.js, inside the models directory. The public directory will contain CSS styles, and all the necessary client-side JavaScript: the AngularJS library, the AngularJS router module, and our custom code. We will skip some of the explanations about the following code. Our index.js file looks as follows: var express = require('express'); var app = express(); var articles = require("./models/Articles")(); app.set('views', __dirname + '/views'); app.set('view engine', 'jade'); app.use(express.static(__dirname + '/public')); app.use(function(req, res, next) { req.articles = articles; next(); }); app.get('/api/get', require("./controllers/api/get")); app.get('/', require("./controllers/index")); app.listen(3000); console.log('Listening on port 3000');   At the beginning, we require the Express framework and our model. Maybe it's better to initialize the model inside the controller, but in our case this is not necessary. Just after that, we set up some basic options for Express and define our own middleware. It has only one job to do and that is to attach the model to the request object. We are doing this because the request object is passed to all the route handlers. In our case, these handlers are actually the controllers. So, Articles.js becomes accessible everywhere via the req.articles property. At the end of the script, we placed two routes. The second one catches the usual requests that come from the users. The first one, /api/get, is a bit more interesting. We want to build our frontend on top of AngularJS. So, the data that is stored in the database should not enter the Node.js part but on the client side where we use Google's framework. To make this possible, we will create routes/controllers to get, add, edit, and delete records. Everything will be controlled by HTTP requests performed by AngularJS. In other words, we need an API. Before we start using AngularJS, let's take a look at the /controllers/api/get.js controller: module.exports = function(req, res, next) { req.articles.get(function(rows) { res.send(rows); }); }   The main job is done by our model and the response is handled by Express. It's nice because if we pass a JavaScript object, as we did, (rows is actually an array of objects) the framework sets the response headers automatically. To test the result, we could run the application with node index.js and open http://localhost:3000/api/ get. If we don't have any records in the database, we will get an empty array. If not, the stored articles will be returned. So, that's the URL, which we should hit from within the AngularJS controller in order to get the information. The code of the /controller/index.js controller is also just a few lines. We can see the code as follows: module.exports = function(req, res, next) { res.render("list", { app: "" }); }   It simply renders the list view, which is stored in the list.jade file. That file should be saved in the /views directory. But before we see its code, we will check another file, which acts as a base for all the pages. Jade has a nice feature called blocks. We may define different partials and combine them into one template. The following is our layout.jade file: doctype html html(ng-app="#{app}") head title Blog link(rel='stylesheet', href='/style.css') script(src='/angular.min.js') script(src='/angular-route.min.js') body block content   There is only one variable passed to this template, which is #{app}. We will need it later to initialize the administration's module. The angular.min.js and angular-route.min.js files should be downloaded from the official AngularJS site, and placed in the /public directory. The body of the page contains a block placeholder called content, which we will later fill with the list of the articles. The following is the list.jade file: extends layout block content .container(ng-controller="BlogCtrl") section.articles article(ng-repeat="article in articles") h2 {{article.title}} br small published on {{article.date}} p {{article.text}} script(src='/blog.js')   The two lines in the beginning combine both the templates into one page. The Express framework transforms the Jade template into HTML and serves it to the browser of the user. From there, the client-side JavaScript takes control. We are using the ng-controller directive saying that the div element will be controlled by an AngularJS controller called BlogCtrl. The same class should have variable, articles, filled with the information from the database. ng-repeat goes through the array and displays the content to the users. The blog.js class holds the code of the controller: function BlogCtrl($scope, $http) { $scope.articles = [ { title: "", text: "Loading ..."} ]; $http({method: 'GET', url: '/api/get'}) .success(function(data, status, headers, config) { $scope.articles = data; }) .error(function(data, status, headers, config) { console.error("Error getting articles."); }); }   The controller has two dependencies. The first one, $scope, points to the current view. Whatever we assign as a property there is available as a variable in our HTML markup. Initially, we add only one element, which doesn't have a title, but has text. It is shown to indicate that we are still loading the articles from the database. The second dependency, $http, provides an API in order to make HTTP requests. So, all we have to do is query /api/get, fetch the data, and pass it to the $scope dependency. The rest is done by AngularJS and its magical two-way data binding. To make the application a little more interesting, we will add a search field, as follows: // views/list.jade header .search input(type="text", placeholder="type a filter here", ng-model="filterText") h1 Blog hr   The ng-model directive, binds the value of the input field to a variable inside our $scope dependency. However, this time, we don't have to edit our controller and can simply apply the same variable as a filter to the ng-repeat: article(ng-repeat="article in articles | filter:filterText") As a result, the articles shown will be filtered based on the user's input. Two simple additions, but something really valuable is on the page. The filters of AngularJS can be very powerful. Implementing a control panel The control panel is the place where we will manage the articles of the blog. Several things should be made in the backend before continuing with the user interface. They are as follows: app.set("username", "admin"); app.set("password", "pass"); app.use(express.cookieParser('blog-application')); app.use(express.session());   The previous lines of code should be added to /index.js. Our administration should be protected, so the first two lines define our credentials. We are using Express as data storage, simply creating key-value pairs. Later, if we need the username we can get it with app.get("username"). The next two lines enable session support. We need that because of the login process. We added a middleware, which attaches the articles to the request object. We will do the same with the current user's status, as follows: app.use(function(req, res, next) { if (( req.session && req.session.admin === true ) || ( req.body && req.body.username === app.get("username") && req.body.password === app.get("password") )) { req.logged = true; req.session.admin = true; }; next(); });   Our if statement is a little long, but it tells us whether the user is logged in or not. The first part checks whether there is a session created and the second one checks whether the user submitted a form with the correct username and password. If these expressions are true, then we attach a variable, logged, to the request object and create a session that will be valid during the following requests. There is only one thing that we need in the main application's file. A few routes that will handle the control panel operations. In the following code, we are defining them along with the needed route handler: var protect = function(req, res, next) { if (req.logged) { next(); } else { res.send(401, 'No Access.'); } } app.post('/api/add', protect, require("./controllers/api/add")); app.post('/api/edit', protect, require("./controllers/api/edit")); app.post('/api/delete', protect, require("./controllers/api/ delete")); app.all('/admin', require("./controllers/admin"));   The three routes, which start with /api, will use the model Articles.js to add, edit, and remove articles from the database. These operations should be protected. We will add a middleware function that takes care of this. If the req.logged variable is not available, it simply responds with a 401 - Unauthorized status code. The last route, /admin, is a little different because it shows a login form instead. The following is the controller to create new articles: module.exports = function(req, res, next) { req.articles.add(req.body, function() { res.send({success: true}); }); }   We transfer most of the logic to the frontend, so again, there are just a few lines. What is interesting here is that we pass req.body directly to the model. It actually contains the data submitted by the user. The following code, is how the req.articles.add method looks for the MongoDB implementation: add: function(data, callback) { data.ID = crypto.randomBytes(20).toString('hex'); collection.insert(data, {}, callback || function() {}); } And the MySQL implementation is as follows: add: function(data, callback) { var date = new Date(); var query = ""; query += "INSERT INTO articles (title, text, date) VALUES ("; query += connection.escape(data.title) + ", "; query += connection.escape(data.text) + ", "; query += "'" + date.getFullYear() + "-" + date.getMonth() + "-" + date.getDate() + "'"; query += ")"; connection.query(query, callback); } In both the cases, we need title and text in the passed data object. Thankfully, due to Express' bodyParser middleware, this is what we have in the req.body object. We can directly forward it to the model. The other route handlers are almost the same: // api/edit.js module.exports = function(req, res, next) { req.articles.update(req.body, function() { res.send({success: true}); }); } What we changed is the method of the Articles.js class. It is not add but update. The same technique is applied in the route to delete an article. We can see it as follows: // api/delete.js module.exports = function(req, res, next) { req.articles.remove(req.body.id, function() { res.send({success: true}); }); }   What we need for deletion is not the whole body of the request but only the unique ID of the record. Every API method sends {success: true} as a response. While we are dealing with API requests, we should always return a response. Even if something goes wrong. The last thing in the Node.js part, which we have to cover, is the controller responsible for the user interface of the administration panel, that is, the. / controllers/admin.js file: module.exports = function(req, res, next) { if(req.logged) { res.render("admin", { app: "admin" }); } else { res.render("login", { app: "" }); } }   There are two templates that are rendered: /views/admin.jade and /views/login. jade. Based on the variable, which we set in /index.js, the script decides which one to show. If the user is not logged in, then a login form is sent to the browser, as follows: extends layout block content .container header h1 Administration hr section.articles article form(method="post", action="/admin") span Username: br input(type="text", name="username") br span Password: br input(type="password", name="password") br br input(type="submit", value="login")   There is no AngularJS code here. All we have is the good old HTML form, which submits its data via POST to the same URL—/admin. If the username and password are correct, the .logged variable is set to true and the controller renders the other template: extends layout block content .container header h1 Administration hr a(href="/") Public span | a(href="#/") List span | a(href="#/add") Add section(ng-view) script(src='/admin.js')   The control panel needs several views to handle all the operations. AngularJS has a great router module, which works with hashtags-type URLs, that is, URLs such as / admin#/add. The same module requires a placeholder for the different partials. In our case, this is a section tag. The ng-view attribute tells the framework that this is the element prepared for that logic. At the end of the template, we are adding an external file, which keeps the whole client-side JavaScript code that is needed by the control panel. While the client-side part of the applications needs only loading of the articles, the control panel requires a lot more functionalities. It is good to use the modular system of AngularJS. We need the routes and views to change, so the ngRoute module is needed as a dependency. This module is not added in the main angular.min.js build. It is placed in the angular-route.min.js file. The following code shows how our module starts: var admin = angular.module('admin', ['ngRoute']); admin.config(['$routeProvider', function($routeProvider) { $routeProvider .when('/', {}) .when('/add', {}) .when('/edit/:id', {}) .when('/delete/:id', {}) .otherwise({ redirectTo: '/' }); } ]);   We configured the router by mapping URLs to specific routes. At the moment, the routes are just empty objects, but we will fix that shortly. Every controller will need to make HTTP requests to the Node.js part of the application. It will be nice if we have such a service and use it all over our code. We can see an example as follows: admin.factory('API', function($http) { var request = function(method, url) { return function(callback, data) { $http({ method: method, url: url, data: data }) .success(callback) .error(function(data, status, headers, config) { console.error("Error requesting '" + url + "'."); }); } } return { get: request('GET', '/api/get'), add: request('POST', '/api/add'), edit: request('POST', '/api/edit'), remove: request('POST', '/api/delete') } });   One of the best things about AngularJS is that it works with plain JavaScript objects. There are no unnecessary abstractions and no extending or inheriting special classes. We are using the .factory method to create a simple JavaScript object. It has four methods that can be called: get, add, edit, and remove. Each one of them calls a function, which is defined in the helper method request. The service has only one dependency, $http. We already know this module; it handles HTTP requests nicely. The URLs that we are going to query are the same ones that we defined in the Node.js part. Now, let's create a controller that will show the articles currently stored in the database. First, we should replace the empty route object .when('/', {}) with the following object: .when('/', { controller: 'ListCtrl', template: ' <article ng-repeat="article in articles"> <hr /> <strong>{{article.title}}</strong><br /> (<a href="#/edit/{{article.id}}">edit</a>) (<a href="#/delete/{{article.id}}">remove</a>) </article> ' })   The object has to contain a controller and a template. The template is nothing more than a few lines of HTML markup. It looks a bit like the template used to show the articles on the client side. The difference is the links used to edit and delete. JavaScript doesn't allow new lines in the string definitions. The backward slashes at the end of the lines prevent syntax errors, which will eventually be thrown by the browser. The following is the code for the controller. It is defined, again, in the module: admin.controller('ListCtrl', function($scope, API) { API.get(function(articles) { $scope.articles = articles; }); });   And here is the beauty of the AngularJS dependency injection. Our custom-defined service API is automatically initialized and passed to the controller. The .get method fetches the articles from the database. Later, we send the information to the current $scope dependency and the two-way data binding does the rest. The articles are shown on the page. The work with AngularJS is so easy that we could combine the controller to add and edit in one place. Let's store the route object in an external variable, as follows: var AddEditRoute = { controller: 'AddEditCtrl', template: ' <hr /> <article> <form> <span>Title</spna><br /> <input type="text" ng-model="article.title"/><br /> <span>Text</spna><br /> <textarea rows="7" ng-model="article.text"></textarea> <br /><br /> <button ng-click="save()">save</button> </form> </article> ' };   And later, assign it to the both the routes, as follows: .when('/add', AddEditRoute) .when('/edit/:id', AddEditRoute)   The template is just a form with the necessary fields and a button, which calls the save method in the controller. Notice that we bound the input field and the text area to variables inside the $scope dependency. This comes in handy because we don't need to access the DOM to get the values. We can see this as follows: admin.controller( 'AddEditCtrl', function($scope, API, $location, $routeParams) { var editMode = $routeParams.id ? true : false; if (editMode) { API.get(function(articles) { articles.forEach(function(article) { if (article.id == $routeParams.id) { $scope.article = article; } }); }); } $scope.save = function() { API[editMode ? 'edit' : 'add'](function() { $location.path('/'); }, $scope.article); } })   The controller receives four dependencies. We already know about $scope and API. The $location dependency is used when we want to change the current route, or, in other words, to forward the user to another view. The $routeParams dependency is needed to fetch parameters from the URL. In our case, /edit/:id is a route with a variable inside. Inside the code, the id is available in $routeParams.id. The adding and editing of articles uses the same form. So, with a simple check, we know what the user is currently doing. If the user is in the edit mode, then we fetch the article based on the provided id and fill the form. Otherwise, the fields are empty and new records will be created. The deletion of an article can be done by using a similar approach, which is adding a route object and defining a new controller. We can see the deletion as follows: .when('/delete/:id', { controller: 'RemoveCtrl', template: ' ' })   We don't need a template in this case. Once the article is deleted from the database, we will forward the user to the list page. We have to call the remove method of the API. Here is how the RemoveCtrl controller looks like: admin.controller( 'RemoveCtrl', function($scope, $location, $routeParams, API) { API.remove(function() { $location.path('/'); }, $routeParams); } );   The preceding code depicts same dependencies like in the previous controller. This time, we simply forward the $routeParams dependency to the API. And because it is a plain JavaScript object, everything works as expected. Summary In this article, we built a simple blog by writing the backend of the application in Node.js. The module for database communication, which we wrote, can work with the MongoDB or MySQL database and store articles. The client-side part and the control panel of the blog were developed with AngularJS. We then defined a custom service using the built-in HTTP and routing mechanisms. Node.js works well with AngularJS, mainly because both are written in JavaScript. We found out that AngularJS is built to support the developer. It removes all those boring tasks such as DOM element referencing, attaching event listeners, and so on. It's a great choice for the modern client-side coding stack. You can refer to the following books to learn more about Node.js: Node.js Essentials Learning Node.js for Mobile Application Development Node.js Design Patterns Resources for Article: Further resources on this subject: Node.js Fundamentals [Article] AngularJS Project [Article] Working with Live Data and AngularJS [Article]
Read more
  • 0
  • 2
  • 13484

article-image-creating-random-insults
Packt
28 Apr 2015
21 min read
Save for later

Creating Random Insults

Packt
28 Apr 2015
21 min read
In this article by Daniel Bates, the author of Raspberry Pi for Kids - Second edition, we're going to learn and use the Python programming language to generate random funny phrases such as, Alice has a smelly foot! (For more resources related to this topic, see here.) Python In this article, we are going to use the Python programming language. Almost all programming languages are capable of doing the same things, but they are usually designed with different specializations. Some languages are designed to perform one job particularly well, some are designed to run code as fast as possible, and some are designed to be easy to learn. Scratch was designed to develop animations and games, and to be easy to read and learn, but it can be difficult to manage large programs. Python is designed to be a good general-purpose language. It is easy to read and can run code much faster than Scratch. Python is a text-based language. Using it, we type the code rather than arrange building blocks. This makes it easier to go back and change the pieces of code that we have already written, and it allows us to write complex pieces of code more quickly. It does mean that we need to type our programs accurately, though—there are no limits to what we can type, but not all text will form a valid program. Even a simple spelling mistake can result in errors. Lots of tutorials and information about the available features are provided online at http://docs.python.org/2/. Learn Python the Hard Way, by Shaw Zed A., is another good learning resource, which is available at http://learnpythonthehardway.org. As an example, let's take a look at some Scratch and Python code, respectively, both of which do the same thing. Here's the Scratch code: The Python code that does the same job looks like: def count(maximum):    value = 0    while value < maximum:        value = value + 1        print "value =", value   count(5) Even if you've never seen any Python code before, you might be able to read it and tell what it does. Both the Scratch and Python code count from 0 to a maximum value, and display the value each time. The biggest difference is in the first line. Instead of waiting for a message, we define (or create) a function, and instead of sending a message, we call the function (more on how to run Python code, shortly). Notice that we include maximum as an argument to the count function. This tells Python the particular value we would like to keep as the maximum, so we can use the same code with different maximum values. The other main differences are that we have while instead of forever if, and we have print instead of say. These are just different ways of writing the same thing. Also, instead of having a block of code wrap around other blocks, we simply put an extra four spaces at the beginning of a line to show which code is contained within a particular block. Python programming To run a piece of Python code, open Python 2 from the Programming menu on the Raspberry Pi desktop and perform the following steps: Type the previous code into the window and you should notice that it can recognize how many spaces to start a line with. When you have finished the function block, press Enter a couple of times, until you see >>>. This shows that Python recognizes that your block of code has been completed, and that it is ready to receive a new command. Now, you can run your code by typing in count(5) and pressing Enter. You can change 5 to any number you like and press Enter again to count to a different number. We're now ready to create our program! The Raspberry Pi also supports Python 3, which is very similar but incompatible with Python 2. You can check out the differences between Python 2 and Python 3 at http://python-future.org/compatible_idioms.html. The program we're going to use to generate phrases As mentioned earlier, our program is going to generate random, possibly funny, phrases for us. To do this, we're going to give each phrase a common structure, and randomize the word that appears in each position. Each phrase will look like: <name> has a <adjective> <noun> Where <name> is replaced by a person's name, <adjective> is replaced by a descriptive word, and <noun> is replaced by the name of an object. This program is going to be a little larger than our previous code example, so we're going to want to save it and modify it easily. Navigate to File | New Window in Python 2. A second window will appear which starts off completely blank. We will write our code in this window, and when we run it, the results will appear in the first window. For the rest of the article, I will call the first window the Shell, and the new window the Code Editor. Remember to save your code regularly! Lists We're going to use a few different lists in our program. Lists are an important part of Python, and allow us to group together similar things. In our program, we want to have separate lists for all the possible names, adjectives, and nouns that can be used in our sentences. We can create a list in this manner: names = ["Alice", "Bob", "Carol"] Here, we have created a variable called names, which is a list. The list holds three items or elements: Alice, Bob, and Carol. We know that it is a list because the elements are surrounded by square brackets, and are separated by commas. The names need to be in quote marks to show that they are text, and not the names of variables elsewhere in the program. To access the elements in a list, we use the number which matches its position, but curiously, we start counting from zero. This is because if we know where the start of the list is stored, we know that its first element is stored at position start + 0, the second element is at position start + 1, and so on. So, Alice is at position 0 in the list, Bob is at position 1, and Carol is at position 2. We use the following code to display the first element (Alice) on the screen: print names[0] We've seen print before: it displays text on the screen. The rest of the code is the name of our list (names), and the position of the element in the list that we want surrounded by square brackets. Type these two lines of code into the Code Editor, and then navigate to Run | Run Module (or press F5). You should see Alice appear in the Shell. Feel free to play around with the names in the list or the position that is being accessed until you are comfortable with how lists work. You will need to rerun the code after each change. What happens if you choose a position that doesn't match any element in the list, such as 10? Adding randomness So far, we have complete control over which name is displayed. Let's now work on displaying a random name each time we run the program. Update your code in the Code Editor so it looks like: import random names = ["Alice", "Bob", "Carol"] position = random.randrange(3) print names[position] In the first line of the code, we import the random module. Python comes with a huge amount of code that other people have written for us, separated into different modules. Some of this code is simple, but makes life more convenient for us, and some of it is complex, allowing us to reuse other people's solutions for the challenges we face and concentrate on exactly what we want to do. In this case, we are making use of a collection of functions that deal with random behavior. We must import a module before we are able to access its contents. Information on the available modules available can be found online at www.python.org/doc/. After we've created the list of names, we then compute a random position in the list to access. The name random.randrange tells us that we are using a function called randrange, which can be found inside the random module that we imported earlier. The randrange function gives us a random whole number less than the number we provide. In this case, we provide 3 because the list has three elements and we store the random position in a new variable called position. Finally, instead of accessing a fixed element in the names list, we access the element that position refers to. If you run this code a few times, you should notice that different names are chosen randomly. Now, what happens if we want to add a fourth name, Dave, to our list? We need to update the list itself, but we also need to update the value we provide to randrange to let it know that it can give us larger numbers. Making multiple changes just to add one name can cause problems—if the program is much larger, we may forget which parts of the code need to be updated. Luckily, Python has a nice feature which allows us to make this simpler. Instead of a fixed number (such as 3), we can ask Python for the length of a list, and provide that to the randrange function. Then, whenever we update the list, Python knows exactly how long it is, and can generate suitable random numbers. Here is the code, which is updated to make it easier to change the length of the list: import random names = ["Alice", "Bob", "Carol"] length = len(names) position = random.randrange(length) print names[position] Here, we've created a new variable called length to hold the length of the list. We then use the len function (which is short for length) to compute the length of our list, and we give length to the randrange function. If you run this code, you should see that it works exactly as it did before, and it easily copes if you add or remove elements from the list. It turns out that this is such a common thing to do, that the writers of the random module have provided a function which does the same job. We can use this to simplify our code: import random names = ["Alice", "Bob", "Carol", "Dave"] print random.choice(names) As you can see, we no longer need to compute the length of the list or a random position in it: random.choice does all of this for us, and simply gives us a random element of any list we provide it with. As we will see in the next section, this is useful since we can reuse random.choice for all the different lists we want to include in our program. If you run this program, you will see that it works the same as it did before, despite being much shorter. Creating phrases Now that we can get a random element from a list, we've crossed the halfway mark to generating random sentences! Create two more lists in your program, one called adjectives, and the other called nouns. Put as many descriptive words as you like into the first one, and a selection of objects into the second. Here are the three lists I now have in my program: names = ["Alice", "Bob", "Carol", "Dave"] adjectives = ["fast", "slow", "pretty", "smelly"] nouns = ["dog", "car", "face", "foot"] Also, instead of printing our random elements immediately, let's store them in variables so that we can put them all together at the end. Remove the existing line of code with print in it, and add the following three lines after the lists have been created: name = random.choice(names) adjective = random.choice(adjectives) noun = random.choice(nouns) Now, we just need to put everything together to create a sentence. Add this line of code right at the end of the program: print name, "has a", adjective, noun Here, we've used commas to separate all of the things we want to display. The name, adjective, and noun are our variables holding the random elements of each of the lists, and "has a" is some extra text that completes the sentence. print will automatically put a space between each thing it displays (and start a new line at the end). If you ever want to prevent Python from adding a space between two items, separate them with + rather than a comma. That's it! If you run the program, you should see random phrases being displayed each time, such as Alice has a smelly foot or Carol has a fast car. Making mischief So, we have random phrases being displayed, but what if we now want to make them less random? What if you want to show your program to a friend, but make sure that it only ever says nice things about you, or bad things about them? In this section, we'll extend the program to do just that. Dictionaries The first thing we're going to do is replace one of our lists with a dictionary. A dictionary in Python uses one piece of information (a number, some text, or almost anything else) to search for another. This is a lot like the dictionaries you might be used to, where you use a word to search for its meaning. In Python, we say that we use a key to look for a value. We're going to turn our adjectives list into a dictionary. The keys will be the existing descriptive words, and the values will be tags that tell us what sort of descriptive words they are. Each adjective will be "good" or "bad". My adjectives list becomes the following dictionary. Make similar changes to yours. adjectives = {"fast":"good", "slow":"bad", "pretty":"good", "smelly":"bad"} As you can see, the square brackets from the list become curly braces when you create a dictionary. The elements are still separated by commas, but now each element is a key-value pair with the adjective first, then a colon, and then the type of adjective it is. To access a value in a dictionary, we no longer use the number which matches its position. Instead, we use the key with which it is paired. So, as an example, the following code will display "good" because "pretty" is paired with "good" in the adjectives dictionary: print adjectives["pretty"] If you try to run your program now, you'll get an error which mentions random.choice(adjectives). This is because random.choice expects to be given a list, but is now being given a dictionary. To get the code working as it was before, replace that line of code with this: adjective = random.choice(adjectives.keys()) The addition of .keys() means that we only look at the keys in the dictionary—these are the adjectives we were using before, so the code should work as it did previously. Test it out now to make sure. Loops You may remember the forever and repeat code blocks in Scratch. In this section, we're going to use Python's versions of these to repeatedly choose random items from our dictionary until we find one which is tagged as "good". A loop is the general programming term for this repetition—if you walk around a loop, you will repeat the same path over and over again, and it is the same with loops in programming languages. Here is some code, which finds an adjective and is tagged as "good". Replace your existing adjective = line of code with these lines: while True:    adjective = random.choice(adjectives.keys())    if adjectives[adjective] == "good":        break The first line creates our loop. It contains the while key word, and a test to see whether the code should be executed inside the loop. In this case, we make the test True, so it always passes, and we always execute the code inside. We end the line with a colon to show that this is the beginning of a block of code. While in Scratch we could drag code blocks inside of the forever or repeat blocks, in Python we need to show which code is inside the block in a different way. First, we put a colon at the end of the line, and then we indent any code which we want to repeat by four spaces. The second line is the code we had before: we choose a random adjective from our dictionary. The third line uses adjectives[adjective] to look into the (adjectives) dictionary for the tag of our chosen adjective. We compare the tag with "good" using the double = sign (a double = is needed to make the comparison different from the single = case, which stores a value in a variable). Finally, if the tag matches "good",we enter another block of code: we put a colon at the end of the line, and the following code is indented by another four spaces. This behaves the same way as the Scratch if block. The fourth line contains a single word: break. This is used to escape from loops, which is what we want to do now that we have found a "good" adjective. If you run your code a few times now, you should see that none of the bad adjectives ever appear. Conditionals In the preceding section, we saw a simple use of the if statement to control when some code was executed. Now, we're going to do something a little more complex. Let's say we want to give Alice a good adjective, but give Bob a bad adjective. For everyone else, we don't mind if their adjective is good or bad. The code we already have to choose an adjective is perfect for Alice: we always want a good adjective. We just need to make sure that it only runs if our random phrase generator has chosen Alice as its random person. To do this, we need to put all the code for choosing an adjective within another if statement, as shown here: if name == "Alice":    while True:        adjective = random.choice(adjectives.keys())        if adjectives[adjective] == "good":            break Remember to indent everything inside the if statement by an extra four spaces. Next, we want a very similar piece of code for Bob, but also want to make sure that the adjective is bad: elif name == "Bob":    while True:        adjective = random.choice(adjectives.keys())        if adjectives[adjective] == "bad":           break The only differences between this and Alice's code is that the name has changed to "Bob", the target tag has changed to "bad", and if has changed to elif. The word elif in the code is short for else if. We use this version because we only want to do this test if the first test (with Alice) fails. This makes a lot of sense if we look at the code as a whole: if our random person is Alice, do something, else if our random person is Bob, do something else. Finally, we want some code that can deal with everyone else. This time, we don't want to perform another test, so we don't need an if statement: we can just use else: else:    adjective = random.choice(adjectives.keys()) With this, our program does everything we wanted it to do. It generates random phrases, and we can even customize what sort of phrase each person gets. You can add as many extra elif blocks to your program as you like, so as to customize it for different people. Functions In this section, we're not going to change the behavior of our program at all; we're just going to tidy it up a bit. You may have noticed that when customizing the types of adjectives for different people, you created multiple sections of code, which were almost identical. This isn't a very good way of programming because if we ever want to change the way we choose adjectives, we will have to do it multiple times, and this makes it much easier to make mistakes or forget to make a change somewhere. What we want is a single piece of code, which does the job we want it to do, and then be able to use it multiple times. We call this piece of code a function. We saw an example of a function being created in the comparison with Scratch at the beginning of this article, and we've used a few functions from the random module already. A function can take some inputs (called arguments) and does some computation with them to produce a result, which it returns. Here is a function which chooses an adjective for us with a given tag: def chooseAdjective(tag):    while True:        item = random.choice(adjectives.keys())        if adjectives[item] == tag:            break    return item In the first line, we use def to say that we are defining a new function. We also give the function's name and the names of its arguments in brackets. We separate the arguments by commas if there is more than one of them. At the end of the line, we have a colon to show that we are entering a new code block, and the rest of the code in the function is indented by four spaces. The next four lines should look very familiar to you—they are almost identical to the code we had before. The only difference is that instead of comparing with "good" or "bad", we compare with the tag argument. When we use this function, we will set tag to an appropriate value. The final line returns the suitable adjective we've found. Pay attention to its indentation. The line of code is inside the function, but not inside the while loop (we don't want to return every item we check), so it is only indented by four spaces in total. Type the code for this function anywhere above the existing code, which chooses the adjective; the function needs to exist in the code prior to the place where we use it. In particular, in Python, we tend to place our code in the following order: Imports Functions Variables Rest of the code This allows us to use our functions when creating the variables. So, place your function just after the import statement, but before the lists. We can now use this function instead of the several lines of code that we were using before. The code I'm going to use to choose the adjective now becomes: if name == "Alice":    adjective = chooseAdjective("good") elif name == "Bob":    adjective = chooseAdjective("bad") else:    adjective = random.choice(adjectives.keys()) This looks much neater! Now, if we ever want to change how an adjective is chosen, we just need to change the chooseAdjective function, and the change will be seen in every part of the code where the function is used. Complete code listing Here is the final code you should have when you have completed this article. You can use this code listing to check that you have everything in the right order, or look for other problems in your code. Of course, you are free to change the contents of the lists and dictionaries to whatever you like; this is only an example: import random   def chooseAdjective(tag):    while True:        item = random.choice(adjectives.keys())        if adjectives[item] == tag:            break    return item   names = ["Alice", "Bob", "Carol", "Dave"] adjectives = {"fast":"good", "slow":"bad", "pretty":"good", "smelly":"bad"} nouns = ["dog", "car", "face", "foot"]   name = random.choice(names) #adjective = random.choice(adjectives) noun = random.choice(nouns)   if name == "Alice":    adjective = chooseAdjective("good") elif name == "Bob":    adjective = chooseAdjective("bad") else:    adjective = random.choice(adjectives.keys())   print name, "has a", adjective, noun Summary In this article, we learned about the Python programming language and how it can be used to create random phrases. We saw that it shared lots of features with Scratch, but is simply presented differently. Resources for Article: Further resources on this subject: Develop a Digital Clock [article] GPS-enabled Time-lapse Recorder [article] The Raspberry Pi and Raspbian [article]
Read more
  • 0
  • 0
  • 13476

article-image-ubuntu-910-how-upgrade
Packt
18 Nov 2009
5 min read
Save for later

Ubuntu 9.10: How To Upgrade

Packt
18 Nov 2009
5 min read
So the new Ubuntu is here and you’re just dying to upgrade and have a look at all the new features! With just a few simple steps you'll be up and running the new system in no time! Before you dive right in, there are a few things you should know, and a few ways to (hopefully) make your upgrade process more pleasant. This article is broken up into sections outlining the preparation, requirements and upgrade steps needed for each platform. It is important to follow the steps in order to ensure a full and painless upgrade. Also, please follow only one of the upgrade paths. In other words, there are different methods for a Desktop as compared to a Server. You only need to follow those steps applicable to you. A Note Regarding Upgrades vs Fresh Installations You may be wondering whether it is better to upgrade your current installation or do a fresh install from CD. There are benefits to doing a fresh installation to be sure, but there are also benefits to upgrading your system in place. I know people that swear by one method, and others that swear by another. In the end, both methods are supported and will give you the same Ubuntu experience. Fresh installations will require a complete wipe of your hard disk. This means that you'll need to backup any important documents, pictures or other files that you'll want to keep. Have you ever done a fresh installation before and realized only too late that you forgot to back something up? I have. It's easy to miss something. Using the in-place upgrade methods found in this article you won't need to worry about backups. With an in-place upgrade you can generally keep working on your machine while applications are upgraded in the background. This means you can continue to browse the web or send and receive email while the system is upgraded. Bottom line is that upgrades are thoroughly tested and just as well supported as fresh installations. Preparation When upgrading your system from one release to the next, there are certain requirements that you must meet in order to be successful. First of all, and most importantly in this instance, this upgrade path is only possible from Ubuntu 9.04 "Jaunty Jackalope" to Ubuntu 9.10 "Karmic Koala". If you are using a release previous to 9.04 (8.10 or earlier), stop now. This upgrade process will not work, is not supported and will likely cause problems. If you are unsure which version you have installed, you can run this command in your terminal to find out. (Applications > Accessories > Terminal) lsb_release -a If you find that you are on a release previous to Ubuntu 9.04, you will need to decide whether it is best to do a fresh installation or do an incremental upgrade leading up to 9.10. Incremental upgrades, as well as fresh installations are beyond the scope of this article, but there is detailed documentation on the matter found here: https://help.ubuntu.com/community/UpgradeNotes Updates Once you have verified that you are using Ubuntu 9.04 "Jaunty Jackalope" you will be able to begin the upgrade proccess. In order for the latest version to become available to you, you'll need to apply any pending updates to your current version. There are two ways to apply available updates pending a system upgrade. The first method applies to the graphical Desktop or Laptop platform. The second method applies to a server, or non-graphical installation. Remember, please only follow the steps applicable to you. Graphical Updates (Pre-Upgrade) If you are using the graphical environment you can check for and apply updates by way of the Update Manager tool. This can be found by navigating to: (System > Administration > Update Manager). This tool will automatically scan for and list any pending updates. Be sure to apply all available updates before moving to the next step. You can ensure that there are no more pending updates by clicking Check and verifying that it displays the message "Your system is up to date". Command Line Updates (Pre-Upgrade) For those more comfortable with the command line interface, or those running a non-graphical Server installation, you can run the following command to check for and apply any available system updates. sudo aptitude update && sudo aptitude safe-upgrade && sudo aptitude full-upgrade Apply any updates that are pending from the command above before you move to the next step. You can repeat this command until no more updates are offered to ensure you are ready. Now that you have applied the remainder of the updates for your current system, you can move to the next step. In the next step, Selecting a Mirror, you will learn how to use an alternate, often faster, package repository for your updates. This means that instead of using the default and often overwhelmed main Ubuntu servers for updates you can configure your system to use one closer to you. This often results in faster downloads and upgrades.
Read more
  • 0
  • 0
  • 13470

article-image-the-developer-tester-face-off-needs-to-end
Aaron Lazar
21 Jul 2018
6 min read
Save for later

The developer-tester face-off needs to end. It's putting our projects at risk.

Aaron Lazar
21 Jul 2018
6 min read
Penny and Leonard work at the same company as a tester and developer respectively. Penny arrives home late, to find Leonard on the couch with his legs up on the table, playing his favourite video game. Leonard: Oh hi sweety, it looks like you had a long day at work. Penny, throwing him a hostile, sideways glance, heads over to the refrigerator. Penny: Did you remember to take out the garbage? Leonard: Of course, sweety. I used 2 bags so Sheldon’s Szechuan sauce from Szechuan Palace doesn’t seep through. Penny: Did you buy new shampoo for the bathroom? Leonard: Yes, I picked up your regular one from the store on the way back. Penny: And did you slap on a last minute field on the SPA at work? Leonard, pausing his video game and answering in a soft, high pitched voice: Whaaaaaat? Source: giphy If you’re a developer or a tester, you’ve probably been in this situation at least once, if not more. Even if your husband or wife might not be on the other side of the source code. The war goes on... The funny thing is that this isn’t something that’s happened in the recent past. The war between Developers and Testers is a long standing, unresolved battle, that is usually brought up in bouts of unnecessary humor. The truth is that this battle is the cause of several projects slipping deadlines, teams not respecting each other’s views, etc. Here we’ll discuss some of the main reasons for this disconnect and try and address them in hope of making the office a better place. #1 You talkin’ to me? One of the main reasons that developers and testers are not on the same page is because neither bother to communicate effectively with the other. Each individual considers informing the other about the strategy/techniques used, a waste of effort. Obviously there are bound to be issues arising with such a disjointed team. The only way to resolve this problem is to toss egos out of the window, sit down and resolve problems like professionals. While tickets might be the most professional and efficient way to resolve things, walking up to the person (if possible), and discussing the best way forward lets you build a relationship, and resolve things more effectively. Moreover, the person on the receiving end will not consider the move offensive, or demeaning. #2 Is it ‘team’ or ‘teams’? You know the answer to this one, but you’re still not willing to accept it. IT managers and team leads need to create an environment in which developers and testers are not two separate teams. Rather, consider them all as engineers working in the same team, towards the same goal! There’s no better recipe to meet success. Use modern methods like Mob or Pair Programming, where both developers and testers work together closely. The ideal scenario would be to possibly have both team members work on the same machine, addressing and strategising to achieve the goal with continuous, real-time feedback. A good pairing station, Source: ministry of testing #3 On the same page? Which book you got there? If you’re a developer, this one’s especially for you, so listen carefully! Most developers aren’t aware of what tools the testers in their teams use, which is a sin. Being aware of testing tools, methodologies and processes, goes a long way in enabling a smooth and speedy testing process. A developer will be able to understand which parts of their code can probably be a tester’s target, what changes would give testers a tough time and on the other hand, what makes it easy. #4 One goal, two paths to achieve it Well, this is true a lot of times. Developers aim to “build” an application. Testers on the other hand aim to “break” the application. Now while this is not wrong, it’s the vision with which the tester is actually planning on breaking things. Testers, you should always keep the customer’s or end user’s requirements clear, while approaching the application. I may not be an actual tester, and you might wonder how I can empathise with other testers. Honestly, I don’t build software, neither do I test any. But I’ve been in a very similar role, earlier on in my Publishing career. As a Commissioning Gatekeeper, I was responsible for validating book and video ideas from the Commissioning Editors. Like a tester, my job was to identify why or how something wouldn’t work in the market. Now, I could easily approach a particular book idea from the perspective of ‘trashing’ it. But when I learned to approach it from the customer’s point of view, my perspective changed, and I was able to give better constructive feedback to the editor. Don’t aim to destroy, aim to improve. If you must kill off an idea or a feature, do it firmly but with kindness. #5 Trust the Developer’s testing skills Yes! Lack of trust is one of the main reasons why there’s so much friction between developers and testers. A tester needs to understand the developer and believe that they can also write tests with a clear goal in mind. Test Driven Development is a great approach to follow. Here, the developer will know better, what angles to test from, and this can help the tester write a mutually defined test case for the developer to run. At the same time, the tester can also provide insight into how to address bugs that might creep up while running the tests. With this combined knowledge, the developers will be able to minimize the number of bugs at the first go itself! Toss in a Business-Driven Development approach, and you’ve got yourself a team that delivers user stories that are more aligned to the business requirement than ever before! In the end, developers and testers, both need to set their egos aside and make peace with each other. If you really look at it, it’s not that hard at all. It’s all about how the two collaborate to create better software, rather than working in silos. IT managers can play an important role here, and they need to understand the advantages and limitations of their team. They need to ensure the unity of the team by encouraging more engaging ways of working as well as introducing modern methodologies that would assist a peaceful, collaborative effort. Why does more than half the IT industry suffer from Burnout? Abandoning Agile Unit testing with Java frameworks: JUnit and TestNG [Tutorial]
Read more
  • 0
  • 0
  • 13465
article-image-unleashing-the-potential-of-gpus-for-training-llms
Shankar Narayanan
22 Sep 2023
8 min read
Save for later

Unleashing the Potential of GPUs for Training LLMs

Shankar Narayanan
22 Sep 2023
8 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!IntroductionThere is no doubt about Language Models being the true marvels in the arena of artificial intelligence. These sophisticated systems have the power to manipulate human language, understand, and even generate with astonishing accuracy.However, one can often complain about the immense computational challenges beyond these medical abilities. For instance, LLM training requires the incorporation of complex mathematical operations along with the processing of vast data. This is where the Graphics Processing Units (GPU) come into play. It serves as the engine that helps to power the language magic.Let me take you through the GPU advancement and innovations to support the Language Model. Parallely, we will explore how Nvidia helps revolutionize the enterprise LLM use cases.Role of GPUs in LLMs To understand the significance of GPU, let us first understand the concept of LLM.What is LLM?LLM or Large Language Models are AI systems that help generate human language. They have various applications, including translation services, sentiment analysis, chatbots, and content generation. Generative Pre-trained Transformer or GPT models, including BERT and GPT3, are popular among every LLM.These models require training, including vast data sets with billions of phrases and words. The model learns to predict while mastering the nuances and structure of language. It is like an intricate puzzle that requires enormous computational power.The need for GPUsThe Graphics Processing Units are specifically designed to undergo parallel processing. This characteristic makes them applicable to train the LLMs. The GPU can tackle thousands of tasks simultaneously, unlike the Central Processing Unit or CPU, which excels at handling sequential tasks.The training of a Large Language Model is like a massive jigsaw puzzle. Each puzzle piece represents a smaller portion of the model's language understanding. Using a CPU could only help one to work on one of these pieces at a simple time. But with GPU, one could work on various pieces parallelly while speeding up the whole process.Besides, GPU offers high computational throughput that one requires for complex mathematical operations. Their competency lies in metric multiplication, one of the fundamentals of neural network training. All these attributes make GPU indispensable for deep learning tasks like LLMs.Here is one of the practical example of how GPU works in LLM training: (Python)import time import torch # Create a large random dataset data = torch.randn(100000, 1000) # Training with CPU start_time = time.time() for _ in range(100):    model_output = data.matmul(data) cpu_training_time = time.time() - start_time print(f"CPU Training Time: {cpu_training_time:.2f} seconds") # Training with GPU if torch.cuda.is_available():    data = data.cuda()    start_time = time.time()    for _ in range(100):        model_output = data.matmul(data)    gpu_training_time = time.time() - start_time    print(f"GPU Training Time: {gpu_training_time:.2f} seconds") else:    print("GPU not available.")GPU Advancements and LLMDue to the rising demands of LLMs and AI, GPU technology is evolving rapidly. These advancements, however, play a significant role in constituting the development of sophisticated language models.One such advancement is the increase in GPU memory capacity. Technically, the larger model requires more excellent memory to process massive data sets. Hence, modern GPUs offer substantial memory capacity, allowing researchers to build and train more substantial large language models.One of the critical aspects of training a Large Language Model is its speed. Sometimes, it can take months to prepare and train a large language model. But with the advent of faster GPU, things have changed dramatically. The quicker GPU reduces the training time and accelerates research and development. Apart from that, it also reduces the energy consumption that is often associated with training these large models.Let us explore the memory capacity of the GPU using a code snippet.(Python)import torch # Check GPU memory capacity if torch.cuda.is_available():    gpu_memory = torch.cuda.get_device_properties(0).total_memory    print(f"GPU Memory Capacity: {gpu_memory / (1024**3):.2f} GB") else:    print("GPU not available.")For the record, Nvidia's Tensor Core technology has been one of the game changers in this aspect. It accelerates one of the core operations in deep learning, i.e., the matrix computation process, allowing the LLMs to train faster and more efficiently.Using matrix Python and PYTorh, you can showcase the speedup with GPU processing.import time import torch # Create large random matrices matrix_size = 1000 cpu_matrix = torch.randn(matrix_size, matrix_size) gpu_matrix = torch.randn(matrix_size, matrix_size).cuda()  # Move to GPU # Perform matrix multiplication with CPU start_time = time.time() result_cpu = torch.matmul(cpu_matrix, cpu_matrix) cpu_time = time.time() - start_time # Perform matrix multiplication with GPU start_time = time.time() result_gpu = torch.matmul(gpu_matrix, gpu_matrix) gpu_time = time.time() - start_time print(f"CPU Matrix Multiplication Time: {cpu_time:.4f} seconds") print(f"GPU Matrix Multiplication Time: {gpu_time:.4f} seconds")Nvidia's Contribution to GPU InnovationRegarding GPU innovation, the presence of Nvidia cannot be denied. It has a long-standing commitment to Machine Learning and advancing AI. Hence, it is a natural ally for the large language model community.Here is how Tensor Cores can be utilized with PYTorch.import torch # Enable Tensor Cores (requires a compatible GPU) if torch.cuda.is_available():    torch.backends.cuda.matmul.allow_tf32 = True # Create a tensor x = torch.randn(4096, 4096, device="cuda") # Perform matrix multiplication using Tensor Cores result = torch.matmul(x, x)It is interesting to know that Nvidia's graphics processing unit has powered several breakthroughs in LLM and AI models. BERT and GPT3 are known to harness the computational might of Nvidia's Graphics Processing Unit to achieve remarkable capabilities. Nvidia's dedication to the Artificial Intelligence world encompasses power and efficiency. The design of the graphics processing unit handles every AI workload with optimal performance per watt. It makes Nvidia one of the eco-friendly options for Large Language Model training procedures.As part of AI-focused hardware and architecture, the Tensor Core technology enables efficient and faster deep learning. This technology is instrumental in pushing the boundaries of LLM research.Supporting Enterprise LLM Use-caseThe application of LLM has a far-fetched reach, extending beyond research, labs, and academia. Indeed, they have entered the enterprise world with a bang. From analyzing massive datasets for insights to automating customer support through chatbots, large language models are transforming how businesses operate.Here, the Nvidia Graphics Processing Unit supports the enterprise LLM use cases. Enterprises often require LLM to handle vast amounts of data in real-time. With optimized AI performance and parallel processing power, Nvidia's GPU can provide the needed acceleration for these applications.Various companies across industries are harnessing the Nvidia GPU for developing LLM-based solutions to automate tasks, provide better customer experiences, and enhance productivity. From healthcare organizations analyzing medical records to financial institutions and predicting market trends, Nvidia drives enterprise LLM innovations.ConclusionNvidia continues to be the trailblazer in the captivating journey of training large language models. They are not only the hardware muscle for LLM but constantly innovate to make GPU capable and efficient with each generation.LLM is on the run to become integral to our daily lives. From business solutions to personal assistants, Nvidia's commitment to its GPU innovation ensures more power to the growth of language models. The synergy between AI and Nvidia GPU is constantly shaping the future of enterprise LLM use cases, helping organizations to achieve new heights in innovation and efficiency.Frequently Asked Questions1. How does the GPU accelerate the training process of large language models?The Graphics Processing Unit has parallel processing capabilities to allow the work of multiple tasks simultaneously. Such parallelism helps train Large Language Models by efficiently processing many components in understanding and generating human language.2. How does Nvidia contribute to GPU innovation for significant language and AI models?Nvidia has developed specialized hardware, including Tensor Core, optimized for AI workloads. The graphic processing unit of Nvidia powered numerous AI breakthroughs while providing efficient AI hardware to advance the development of Large Language Models.3. What are the expectations for the future of GPU innovation and launch language model?The future of GPU innovation promises efficient, specialized, and robust hardware tailored to the needs of AI applications and Large Language Models. It will continuously drive the development of sophisticated language models while opening up new possibilities for AI-power solutions.Author BioShankar Narayanan (aka Shanky) has worked on numerous different cloud and emerging technologies like Azure, AWS, Google Cloud, IoT, Industry 4.0, and DevOps to name a few. He has led the architecture design and implementation for many Enterprise customers and helped enable them to break the barrier and take the first step towards a long and successful cloud journey. He was one of the early adopters of Microsoft Azure and Snowflake Data Cloud. Shanky likes to contribute back to the community. He contributes to open source is a frequently sought-after speaker and has delivered numerous talks on Microsoft Technologies and Snowflake. He is recognized as a Data Superhero by Snowflake and SAP Community Topic leader by SAP.
Read more
  • 0
  • 0
  • 13461

article-image-moving-database-sql-server-2005-sql-server-2008-three-steps
Packt
23 Oct 2009
3 min read
Save for later

Moving a Database from SQL Server 2005 to SQL Server 2008 in Three Steps

Packt
23 Oct 2009
3 min read
(For more resources on Microsoft, see here.) Introduction There are several options if one wishes to move a database from a SQL Server 2005 to SQL 2008 Server. First of all there is a 'Copy Database Wizard' in SQL 2008 Server which is meant for transferring a database from any version of SQL Server 2000 and above to 2008 version. This Wizard can operate in two ways. In the first option it can attach a database (even one on the network) and uses the SQL 2008 SQL Server agent. The Copying of the database is implemented by an Integration Services package to run as a SQL Server Agent job that is scheduled to run immediately or according to some configurable schedule. This will therefore depend on correctly configuring the SQL Server Agent. In order to use the attach / detach process, the remote server will be stopped and if the database / log files are on a shared drive they are correctly brought in by the wizard. In the other option the database will be copied using the SQL Server Management Program for which the source database need not be stopped. However this is slower than the previous method and would also require the SQL Server Agent since a package has to be run. An option which works without too much hassles is manually detaching and attaching the database/log files. In this step-by-step (really two steps) tutorial, this simple procedure is described. If you are just interested in taking a small database from 2005 to 2008 server the author strongly recommends this procedure. Interested readers may also want to read my other popular article Moving Data from SQL Server 2000 to SQL Server 2005 Step 1: Detaching the Database Highlight the database you want to transfer in the Databases node in the SQL Server Management Studio as shown in the next figure. Right click this database as shown and click on Detach... Make sure the database is running (notice the green arrow for HodentekSQL Express which is a junior version of SQL 2005). This brings up the Detach Database window as shown. Place a check mark for 'Drop' as shown and click on OK. This removes the 'Pubs' node from the Databases folder in the SQL Server Management Studio (You may need to attach it again). With this accomplished you can physically move the files or do what you want with them. Step 2: Copy the DATA / LOG Files Copy the pubs.mdf and pubs.ldf files to a location on the C: drive of the machine on which SQL 2008 Server is installed.
Read more
  • 0
  • 0
  • 13456

article-image-build-an-ai-based-personal-financial-advisor-with-langchain
Louis Owen
09 Oct 2023
11 min read
Save for later

Build an AI-based Personal Financial Advisor with LangChain

Louis Owen
09 Oct 2023
11 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!IntroductionManaging personal finances is a cumbersome process. Receipts, bank statements, credit card bills, and expense records accumulate quickly. Despite our best intentions, many of us often struggle to maintain a clear overview of our financial health. It's not uncommon to overlook expenses or procrastinate on updating our budgets. Inevitably, this leads to financial surprises and missed opportunities for financial growth.Even when we diligently track our expenses, the data can be overwhelming. Analyzing every transaction to identify patterns, pinpoint areas for improvement, and set financial goals is no easy feat. It's challenging to answer questions like, "Am I spending too much on entertainment?" or "Is my investment portfolio well-balanced?" or "Should I cut back on dining out?" or "Do I need to limit socializing with friends?" and "Is it time to increase my investments?"Imagine having a personal assistant capable of automating these financial tasks, effortlessly transforming your transaction history into valuable insights. What if, at the end of each month, you received comprehensive financial analyses and actionable recommendations? Thanks to the rapid advancements in generative AI, this dream has become a reality, made possible by the incredible capabilities of LLM. No more endless hours of spreadsheet tinkering or agonizing over budgeting apps.In this article, we'll delve into the development of a personal financial advisor powered by LangChain. This virtual assistant will not only automate the tracking of your finances but also provide tailored recommendations based on your unique spending patterns and financial goals.Building an AI-powered personal financial advisor is an exciting endeavor. Here's an overview of how the personal financial advisor operates:Data Input: Users upload their personal transaction history, which includes details of income, expenses, investments, and savings.Data Processing: LangChain with LLM in the backend will process the data, categorize expenses, identify trends, and compare your financial activity with your goals and benchmarks.Financial Analysis: The advisor generates a detailed financial analysis report, highlighting key insights such as spending habits, saving potential, and investment performance.Actionable Recommendations: The advisor will also provide actionable recommendations for the user. It can suggest adjustments to your budget, recommend investment strategies, and even propose long-term financial plans.The benefits of having an AI-powered personal financial advisor are numerous:Time-Saving: No more tedious data entry and manual budget tracking. The advisor handles it all, giving you more time for what matters most in your life.Personalized Insights: The advisor tailors recommendations based on your unique financial situation, ensuring they align with your goals and aspirations.Financial Confidence: With regular updates and guidance, you gain a better understanding of your financial health and feel more in control of your money.Long-Term Planning: The advisor’s ability to provide insights into long-term financial planning ensures you're well-prepared for your future.Without wasting any more time, let’s take a deep breath, make yourselves comfortable, and be ready to learn how to build your AI-based personal financial advisor with LangChain!What is LangChain?LangChain is developed to harness the incredible potential of LLM, LangChain enables the creation of applications that are not only context-aware but also capable of reasoning, all while maintaining a user-friendly and modular approach.LangChain is more than just a framework; it's a paradigm shift in the world of language model-driven applications. Here's a closer look at what makes LangChain a transformative force:Context-Aware Applications: LangChain empowers applications to be context-aware. This means that these applications can connect to various sources of context, such as prompt instructions, few-shot examples, or existing content, to enhance the depth and relevance of their responses. Whether you're seeking answers or recommendations, LangChain ensures that responses are firmly grounded in context.Reasoning Abilities: One of LangChain's standout features is its ability to reason effectively. It leverages language models to not only understand context but also make informed decisions. These decisions can range from determining how to answer a given question based on the provided context to deciding what actions to take next. LangChain doesn't just provide answers; it understands the "why" behind them.Why LangChain?The power of LangChain lies in its value propositions, which make it an indispensable tool for developers and businesses looking to harness the potential of language models:Modular Components: LangChain offers a comprehensive set of abstractions for working with language models. These abstractions are not only powerful but also modular, allowing developers to work with them seamlessly, whether they're using the entire LangChain framework or not. This modularity simplifies the development process and promotes code reuse.Off-the-Shelf Chains: LangChain provides pre-built, structured assemblies of components known as "off-the-shelf chains." These chains are designed for specific high-level tasks, making it incredibly easy for developers to kickstart their projects. Whether you're a seasoned AI developer or a newcomer, these pre-configured chains save time and effort.Customization and Scalability: While off-the-shelf chains are fantastic for quick starts, LangChain doesn't restrict you. The framework allows for extensive customization, enabling developers to tailor existing chains to their unique requirements or even create entirely new ones. This flexibility ensures that LangChain can accommodate a wide range of applications, from simple chatbots to complex AI systems.LangChain isn't just a run-of-the-mill framework; it's a versatile toolkit designed to empower developers to create sophisticated language model-powered applications. At the heart of LangChain is a set of interconnected modules, each serving a unique purpose. These modules are the building blocks that make LangChain a powerhouse for AI application development.Model I/O: At the core of LangChain's capabilities is its ability to interact seamlessly with language models. This module facilitates communication with these models, enabling developers to leverage their natural language processing prowess effortlessly.Retrieval: LangChain recognizes that real-world applications require access to relevant data. The Retrieval module allows developers to integrate application-specific data sources into their projects, enhancing the context and richness of responses.Chains: Building upon the previous modules, Chains bring structure and order to the development process. Developers can create sequences of calls, orchestrating interactions with language models and data sources to achieve specific tasks or goals.Agents: Let chains choose which tools to use given high-level directives. Agents take the concept of automation to a new level. They allow chains to make intelligent decisions about which tools to employ based on high-level directives. This level of autonomy streamlines complex processes and enhances application efficiency.Memory: Memory is vital for continuity in applications. This module enables LangChain applications to remember and retain their state between runs of a chain, ensuring a seamless user experience and efficient data handling.Callbacks: Transparency and monitoring are critical aspects of application development. Callbacks provide a mechanism to log and stream intermediate steps of any chain, offering insights into the inner workings of the application and facilitating debugging.Building the Personal Financial AdvisorLet’s start building our personal financial advisor with LangChain! For the sake of simplicity, let’s consider only three data sources: monthly credit card statements, bank account statements, and cash expense logs. The following is an example of the data format for each of the sources. ## Monthly Credit Card Statement Date: 2023-09-01 Description: Grocery Store Amount: $150.00 Balance: $2,850.00 Date: 2023-09-03 Description: Restaurant Dinner Amount: $50.00 Balance: $2,800.00 Date: 2023-09-10 Description: Gas Station Amount: $40.00 Balance: $2,760.00 Date: 2023-09-15 Description: Utility Bill Payment Amount: $100.00 Balance: $2,660.00 Date: 2023-09-20 Description: Salary Deposit Amount: $3,000.00 Balance: $5,660.00 Date: 2023-09-25 Description: Online Shopping Amount: $200.00 Balance: $5,460.00 Date: 2023-09-30 Description: Investment Portfolio Contribution Amount: $500.00 Balance: $4,960.00 ## Bank Account Statement Date: 2023-08-01 Description: Rent Payment Amount: $1,200.00 Balance: $2,800.00 Date: 2023-08-05 Description: Grocery Store Amount: $200.00 Balance: $2,600.00 Date: 2023-08-12 Description: Internet and Cable Bill Amount: $80.00 Balance: $2,520.00 Date: 2023-08-15 Description: Freelance Gig Income Amount: $700.00 Balance: $3,220.00 Date: 2023-08-21 Description: Dinner with Friends Amount: $80.00 Balance: $3,140.00 Date: 2023-08-25 Description: Savings Account Transfer Amount: $300.00 Balance: $3,440.00 Date: 2023-08-29 Description: Online Shopping Amount: $150.00 Balance: $3,290.00 ## Cash Expense Log Date: 2023-07-03 Description: Coffee Shop Amount: $5.00 Balance: $95.00 Date: 2023-07-10 Description: Movie Tickets Amount: $20.00 Balance: $75.00 Date: 2023-07-18 Description: Gym Membership Amount: $50.00 Balance: $25.00 Date: 2023-07-22 Description: Taxi Fare Amount: $30.00 Balance: -$5.00 (Negative balance indicates a debt) Date: 2023-07-28 Description: Bookstore Amount: $40.00 Balance: -$45.00 Date: 2023-07-30 Description: Cash Withdrawal Amount: $100.00 Balance: -$145.00To create our personal financial advisor, we’ll use the chat model interface provided by LangChain. There are several important components to build a chatbot with LangChain:`chat model`: Chat models are essential for creating conversational chatbots. These models are designed to generate human-like responses in a conversation. You can choose between chat models and LLMs (Large Language Models) depending on the tone and style you want for your chatbot. Chat models are well-suited for natural, interactive conversations.`prompt template`: Prompt templates help you construct prompts for your chatbot. They allow you to combine default messages, user input, chat history, and additional context to create meaningful and dynamic conversations. Using prompt templates makes it easier to generate responses that flow naturally in a conversation.`memory`: Memory in a chatbot context refers to the ability of the bot to remember information from previous parts of the conversation. This can be crucial for maintaining context and providing relevant responses. Memory types can vary depending on your use case, and they can include short-term and long-term memory.`retriever` (optional): Retrievers are components that help chatbots access domain-specific knowledge or retrieve information from external sources. If your chatbot needs to provide detailed, domain-specific information, a retriever can be a valuable addition to your system.First, we need to set the API key for our LLM. We’ll use OpenAI in this example.import os os.environ["OPENAI_API_KEY"] = “your openai key”Then, we can simply load the necessary chat modules from LangChain. from langchain.schema import ( AIMessage,    HumanMessage,    SystemMessage ) from langchain.chat_models import ChatOpenAIThe ChatOpenAI is the main class that connects with the OpenAI LLM. We can pass `HumanMessage` and `SystemMessage` to this class and it will return the response from the LLM in the type of `AIMessage`.chat = ChatOpenAI(model_name=”gpt-3.5-turbo”) messages = [SystemMessage(content=prompt),                    HumanMessage(content=data)] chat(messages)Let’s see the following example where we pass the prompt along with the data and the LLM returns the response via the ChatOpenAI object. Boom! We just got our first analysis and recommendation from our personal financial advisor. This is a very simple example of how to create our personal financial advisor. Of course, there’s still a lot of room for improvement. For example, currently, we need to pass manually the relevant data sources as the HumanMessage. However, as mentioned before, LangChain provides a built-in class to perform retrieval. This means that we can just create another script to automatically dump all of the relevant data into some sort of document or even database, and then LangChain can directly read the data directly from there. Hence, we can get automated reports every month without needing to manually input the relevant data.ConclusionCongratulations on keeping up to this point! Throughout this article, you have learned what is LangChain, what it is capable of, and how to build a personal financial advisor with LangChain. Hope the best for your experiment in creating your personal financial advisor and see you in the next article!Author BioLouis Owen is a data scientist/AI engineer from Indonesia who is always hungry for new knowledge. Throughout his career journey, he has worked in various fields of industry, including NGOs, e-commerce, conversational AI, OTA, Smart City, and FinTech. Outside of work, he loves to spend his time helping data science enthusiasts to become data scientists, either through his articles or through mentoring sessions. He also loves to spend his spare time doing his hobbies: watching movies and conducting side projects. Currently, Louis is an NLP Research Engineer at Yellow.ai, the world’s leading CX automation platform. Check out Louis’ website to learn more about him! Lastly, if you have any queries or any topics to be discussed, please reach out to Louis via LinkedIn.
Read more
  • 0
  • 0
  • 13447
article-image-how-to-build-an-options-trading-web-app-using-q-learning
Sunith Shetty
13 Apr 2018
19 min read
Save for later

How to build an options trading web app using Q-learning

Sunith Shetty
13 Apr 2018
19 min read
Today we will learn to develop an options trading web app using Q-learning algorithm and will also evaluate the model. Developing an options trading web app using Q-learning The trading algorithm is the process of using computers programmed to follow a defined set of instructions for placing a trade in order to generate profits at a speed and frequency that is impossible for a human trader. The defined sets of rules are based on timing, price, quantity, or any mathematical model. Problem description Through this project, we will predict the price of an option on a security for N days in the future according to the current set of observed features derived from the time of expiration, the price of the security, and volatility. The question would be: what model should we use for such an option pricing model? The answer is that there are actually many; Black-Scholes stochastic partial differential equations (PDE) is one of the most recognized. In mathematical finance, the Black-Scholes equation is necessarily a PDE overriding the price evolution of a European call or a European put under the Black-Scholes model. For a European call or put on an underlying stock paying no dividends, the equation is: Where V is the price of the option as a function of stock price S and time t, r is the risk-free interest rate, and σ σ (displaystyle sigma) is the volatility of the stock. One of the key financial insights behind the equation is that anyone can perfectly hedge the option by buying and selling the underlying asset in just the right way without any risk. This hedge implies that there is only one right price for the option, as returned by the Black-Scholes formula. Consider a January maturity call option on an IBM with an exercise price of $95. You write a January IBM put option with an exercise price of $85. Let us consider and focus on the call options of a given security, IBM. The following chart plots the daily price of the IBM stock and its derivative call option for May 2014, with a strike price of $190: Figure 1: IBM stock and call $190 May 2014 pricing in May-Oct 2013 Now, what will be the profit and loss be for this position if IBM is selling at $87 on the option maturity date? Alternatively, what if IBM is selling at $100? Well, it is not easy to compute or predict the answer. However, in options trading, the price of an option depends on a few parameters, such as time decay, price, and volatility: Time to expiration of the option (time decay) The price of the underlying security The volatility of returns of the underlying asset A pricing model usually does not consider the variation in trading volume in terms of the underlying security. Therefore, some researchers have included it in the option trading model. As we have described, any RL-based algorithm should have an explicit state (or states), so let us define the state of an option using the following four normalized features: Time decay (timeToExp): This is the time to expiration once normalized in the range of (0, 1). Relative volatility (volatility): within a trading session, this is the relative variation of the price of the underlying security. It is different than the more complex volatility of returns defined in the Black-Scholes model, for example. Volatility relative to volume (vltyByVol): This is the relative volatility of the price of the security adjusted for its trading volume. Relative difference between the current price and the strike price (priceToStrike): This measures the ratio of the difference between the price and the strike price to the strike price. The following graph shows the four normalized features that can be used for the IBM option strategy: Figure 2: Normalized relative stock price volatility, volatility relative to trading volume, and price relative to strike price for the IBM stock Now let us look at the stock and the option price dataset. There are two files IBM.csv and IBM_O.csv contain the IBM stock prices and option prices, respectively. The stock price dataset has the date, the opening price, the high and low price, the closing price, the trade volume, and the adjusted closing price. A shot of the dataset is given in the following diagram: Figure 3: IBM stock data On the other hand, IBM_O.csv has 127 option prices for IBM Call 190 Oct 18, 2014. A few values are 1.41, 2.24, 2.42, 2.78, 3.46, 4.11, 4.51, 4.92, 5.41, 6.01, and so on. Up to this point, can we develop a predictive model using a Q-Learning, algorithm that can help us answer the previously mentioned question: Can it tell us the how IBM can make maximum profit by utilizing all the available features? Well, we know how to implement the Q-Learning, and we know what option trading is. Implementing an options trading web application The goal of this project is to create an options trading web application that creates a Q-Learning model from the IBM stock data. Then the app will extract the output from the model as a JSON object and show the result to the user. Figure 4, shows the overall workflow: Figure 4: Workflow of the options trading Scala web The compute API prepares the input for the Q-learning algorithm, and the algorithm starts by extracting the data from the files to build the option model. Then it performs operations on the data such as normalization and discretization. It passes all of this to the Q-learning algorithm to train the model. After that, the compute API gets the model from the algorithm, extracts the best policy data, and puts it onto JSON to be returned to the web browser. Well, the implementation of the options trading strategy using Q-learning consists of the following steps: Describing the property of an option Defining the function approximation Specifying the constraints on the state transition Creating an option property Considering the market volatility, we need to be a bit more realistic, because any longer- term prediction is quite unreliable. The reason is that it would fall outside the constraint of the discrete Markov model. So, suppose we want to predict the price for next two days—that is, N= 2. That means the price of the option two days in the future is the value of the reward profit or loss. So, let us encapsulate the following four parameters: timeToExp: Time left until expiration as a percentage of the overall duration of the option Volatility normalized Relative volatility of the underlying security for a given trading session vltyByVol: Volatility of the underlying security for a given trading session relative to a trading volume for the session priceToStrike: Price of the underlying security relative to the Strike price for a given trading session The OptionProperty class defines the property of a traded option on a security. The constructor creates the property for an option: class OptionProperty(timeToExp:  Double,volatility: Double,vltyByVol: Double,priceToStrike:  Double) {  nval toArray  = Array[Double](timeToExp,  volatility, vltyByVol,  priceToStrike)  require(timeToExp   > 0.01, s"OptionProperty  time to expiration  found  $timeToExp  required 0.01") } Creating an option model Now we need to create an OptionModel to act as the container and the factory for the properties of the option. It takes the following parameters and creates a list of option properties, propsList, by accessing the data source of the four features described earlier: The symbol of the security. The strike price for option, strikePrice. The source of the data, src. The minimum time decay or time to expiration, minTDecay. Out-of-the-money options expire worthlessly, and in-the-money options have a very different price behavior as they get closer to the expiration. Therefore, the last minTDecay trading sessions prior to the expiration date are not used in the training process. The number of steps (or buckets), nSteps, is used in approximating the values of each feature. For instance, an approximation of four steps creates four buckets: (0, 25), (25, 50), (50, 75), and (75, 100). Then it assembles OptionProperties and computes the normalized minimum time to the expiration of the option. Then it computes an approximation of the value of options by discretization of the actual value in multiple levels from an array of options prices; finally it returns a map of an array of levels for the option price and accuracy. Here is the constructor of the class: class OptionModel( symbol:  String, strikePrice: Double, src:  DataSource, minExpT: Int, nSteps:  Int ) Inside this class implementation, at first, a validation is done using the check() method, by checking the following: strikePrice: A positive price is required minExpT: This has to be between 2 and 16 nSteps: Requires a minimum of two steps Here's the invocation of this method: check(strikePrice,  minExpT, nSteps) The signature of the preceding method is shown in the following code: def check(strikePrice:  Double, minExpT: Int, nSteps:  Int): Unit = { require(strikePrice  > 0.0, s"OptionModel.check  price found $strikePrice required  > 0") require(minExpT  > 2 && minExpT  < 16,s"OptionModel.check  Minimum expiration time found  $minExpT required  ]2, 16[") require(nSteps   > 1,s"OptionModel.check,  number of steps found $nSteps required  > 1") } Once the preceding constraint is satisfied, the list of option properties, named propsList, is created as follows: val propsList  = (for { price  <- src.get(adjClose) volatility  <- src.get(volatility) nVolatility  <- normalize[Double](volatility) vltyByVol  <- src.get(volatilityByVol) nVltyByVol <- normalize[Double](vltyByVol) priceToStrike  <- normalize[Double](price.map(p  => 1.0 - strikePrice / p)) } yield { nVolatility.zipWithIndex./:(List[OptionProperty]())  { case (xs,  (v, n)) => val normDecay  = (n + minExpT).toDouble  / (price.size + minExpT) new OptionProperty(normDecay,  v, nVltyByVol(n), priceToStrike(n))  :: xs } .drop(2).reverse }).get In the preceding code block, the factory uses the zipWithIndex Scala method to represent the index of the trading sessions. All feature values are normalized over the interval (0, 1), including the time decay (or time to expiration) of the normDecay option. The quantize() method of the OptionModel class converts the normalized value of each option property of features into an array of bucket indices. It returns a map of profit and loss for each bucket keyed on the array of bucket indices: def quantize(o:  Array[Double]): Map[Array[Int],  Double] = { val mapper  = new mutable.HashMap[Int,  Array[Int]] val acc:  NumericAccumulator[Int]  = propsList.view.map(_.toArray) map(toArrayInt(_)).map(ar  => { val enc = encode(ar) mapper.put(enc,  ar) enc }) .zip(o)./:( new NumericAccumulator[Int])  { case (_acc,  (t, y)) => _acc  += (t, y); _acc } acc.map  { case (k,  (v, w)) =>  (k, v / w) } .map  { case (k,  v) => (mapper(k),  v) }.toMap } The method also creates a mapper instance to index the array of buckets. An accumulator, acc, of type NumericAccumulator extends the Map[Int,  (Int, Double)] and computes this tuple (number of occurrences of features on each bucket, sum of the increase or decrease of the option price). The toArrayInt method converts the value of each option property (timeToExp, volatility, and so on) into the index of the appropriate bucket. The array of indices is then encoded to generate the id or index of a state. The method updates the accumulator with the number of occurrences and the total profit and loss for a trading session for the option. It finally computes the reward on each action by averaging the profit and loss on each bucket. The signature of the encode(), toArrayInt() is given in the following code: private def encode(arr:  Array[Int]): Int = arr./:((1,  0)) { case ((s,  t), n) =>  (s * nSteps,  t + s * n) }._2 private def toArrayInt(feature:  Array[Double]): Array[Int] = feature.map(x  => (nSteps * x).floor.toInt) final class NumericAccumulator[T] extends mutable.HashMap[T,  (Int, Double)] { def +=(key:  T, x: Double):  Option[(Int, Double)]  = { val newValue  = if (contains(key))  (get(key).get._1 + 1,  get(key).get._2 + x) else (1,  x) super.put(key,  newValue) } } Finally, and most importantly, if the preceding constraints are satisfied (you can modify these constraints though) and once the instantiation of the OptionModel class generates a list of OptionProperty elements if the constructor succeeds; otherwise, it generates an empty list. Putting it altogether Because we have implemented the Q-learning algorithm, we can now develop the options trading application using Q-learning. However, at first, we need to load the data using the DataSource class (we will see its implementation later on). Then we can create an option model from the data for a given stock with default strike and minimum expiration time parameters, using OptionModel, which defines the model for a traded option, on a security. Then we have to create the model for the profit and loss on an option given the underlying security. The profit and loss are adjusted to produce positive values. It instantiates an instance of the Q-learning class, that is, a generic parameterized class that implements the Q-learning algorithm. The Q-learning model is initialized and trained during the instantiation of the class, so it can be in the correct state for the runtime prediction. Therefore, the class instances have only two states: successfully trained and failed training Q-learning value action. Then the model is returned to get processed and visualized. So, let us create a Scala object and name it QLearningMain. Then, inside the QLearningMain object, define and initialize the following parameters: Name: Used to indicate the reinforcement algorithm's name (for our case, it's Q- learning) STOCK_PRICES: File that contains the stock data OPTION_PRICES: File that contains the available option data STRIKE_PRICE: Option strike price MIN_TIME_EXPIRATION: Minimum expiration time for the option recorded QUANTIZATION_STEP: Steps used in discretization or approximation of the value of the security ALPHA: Learning rate for the Q-learning algorithm DISCOUNT (gamma): Discount rate for the Q-learning algorithm MAX_EPISODE_LEN:Maximum number of states visited per episode NUM_EPISODES: Number of episodes used during training MIN_COVERAGE: Minimum coverage allowed during the training of the Q- learning model NUM_NEIGHBOR_STATES: Number of states accessible from any other state REWARD_TYPE: Maximum reward or Random Tentative initializations for each parameter are given in the following code: val name: String = "Q-learning"// Files containing the historical prices for the stock and option val STOCK_PRICES = "/static/IBM.csv" val OPTION_PRICES = "/static/IBM_O.csv"// Run configuration parameters val STRIKE_PRICE = 190.0 // Option strike price val MIN_TIME_EXPIRATION = 6 // Min expiration time for option recorded val QUANTIZATION_STEP = 32 // Quantization step (Double => Int) val ALPHA = 0.2 // Learning rate val DISCOUNT = 0.6 // Discount rate used in Q-Value update equation val MAX_EPISODE_LEN = 128 // Max number of iteration for an episode val NUM_EPISODES = 20 // Number of episodes used for training. val NUM_NEIGHBHBOR_STATES = 3 // No. of states from any other state Now the run() method accepts as input the reward type (Maximum  reward in our case), quantized step (in our case, QUANTIZATION_STEP), alpha (the learning rate, ALPHA in our case) and gamma (in our case, it's DISCOUNT, the discount rate for the Q-learning algorithm). It displays the distribution of values in the model. Additionally, it displays the estimated Q-value for the best policy on a Scatter plot (we will see this later). Here is the workflow of the preceding method: First, it extracts the stock price from the IBM.csv file Then it creates an option model createOptionModel using the stock prices and quantization, quantizeR (see the quantize method for more and the main method invocation later) The option prices are extracted from the IBM_o.csv file After that, another model, model, is created using the option model to evaluate it on the option prices, oPrices Finally, the estimated Q-Value (that is, Q-value = value * probability) is displayed 0n a Scatter plot using the display method By amalgamating the preceding steps, here's the signature of the run() method: private def run(rewardType:  String,quantizeR: Int,alpha:  Double,gamma: Double): Int = { val sPath  = getClass.getResource(STOCK_PRICES).getPath val src  = DataSource(sPath,  false, false, 1).get val option  = createOptionModel(src,  quantizeR) val oPricesSrc  = DataSource(OPTION_PRICES,  false, false, 1).get val oPrices  = oPricesSrc.extract.get val model  = createModel(option,  oPrices, alpha, gamma)model.map(m  => {if (rewardType  != "Random") display(m.bestPolicy.EQ,m.toString,s"$rewardType  with quantization order $quantizeR")1}).getOrElse(-1) } Now here is the signature of the createOptionModel() method that creates an option model using (see the OptionModel class): private def createOptionModel(src:  DataSource, quantizeR: Int): OptionModel = new OptionModel("IBM",  STRIKE_PRICE, src, MIN_TIME_EXPIRATION, quantizeR) Then the createModel() method creates a model for the profit and loss on an option given the underlying security. Note that the option prices are quantized using the quantize() method defined earlier. Then the constraining method is used to limit the number of actions available to any given state. This simple implementation computes the list of all the states within a radius of this state. Then it identifies the neighboring states within a predefined radius. Finally, it uses the input data to train the Q-learning model to compute the minimum value for the profit, a loss so the maximum loss is converted to a null profit. Note that the profit and loss are adjusted to produce positive values. Now let us see the signature of this method: def createModel(ibmOption: OptionModel,oPrice: Seq[Double],alpha: Double,gamma: Double): Try[QLModel] = { val qPriceMap = ibmOption.quantize(oPrice.toArray) val numStates = qPriceMap.size val neighbors = (n: Int) => { def getProximity(idx: Int, radius: Int): List[Int] = { val idx_max = if (idx + radius >= numStates) numStates - 1 else idx + radius val idx_min = if (idx < radius) 0 else idx - radiusRange(idx_min, idx_max + 1).filter(_ != idx)./:(List[Int]())((xs, n) => n :: xs)}getProximity(n, NUM_NEIGHBHBOR_STATES) } val qPrice: DblVec = qPriceMap.values.toVector val profit: DblVec = normalize(zipWithShift(qPrice, 1).map { case (x, y) => y - x}).get val maxProfitIndex = profit.zipWithIndex.maxBy(_._1)._2 val reward = (x: Double, y: Double) => Math.exp(30.0 * (y - x)) val probabilities = (x: Double, y: Double) => if (y < 0.3 * x) 0.0 else 1.0println(s"$name Goal state index: $maxProfitIndex") if (!QLearning.validateConstraints(profit.size, neighbors)) thrownew IllegalStateException("QLearningEval Incorrect states transition constraint") val instances = qPriceMap.keySet.toSeq.drop(1) val config = QLConfig(alpha, gamma, MAX_EPISODE_LEN, NUM_EPISODES, 0.1) val qLearning = QLearning[Array[Int]](config,Array[Int](maxProfitIndex),profit,reward,proba bilities,instances,Some(neighbors)) val modelO = qLearning.getModel if (modelO.isDefined) { val numTransitions = numStates * (numStates - 1)println(s"$name Coverage ${modelO.get.coverage} for $numStates states and $numTransitions transitions") val profile = qLearning.dumpprintln(s"$name Execution profilen$profile")display(qLearning)Success(modelO.get)} else Failure(new IllegalStateException(s"$name model undefined")) } Note that if the preceding invocation cannot create an option model, the code fails to show a message that the model creation failed. Nonetheless, remember that the minCoverage used in the following line is important, considering the small dataset we used (because the algorithm will converge very quickly): val config  = QLConfig(alpha,  gamma, MAX_EPISODE_LEN,  NUM_EPISODES, 0.0) Although we've already stated that it is not assured that the model creation and training will be successful, a Naïve clue would be using a very small minCoverage value between 0.0 and 0.22. Now, if the preceding invocation is successful, then the model is trained and ready for making prediction. If so, then the display method is used to display the estimated Q-value = value * probability in a Scatter plot. Here is the signature of the method: private def display(eq:  Vector[DblPair],results: String,params:  String): Unit = { import org.scalaml.plots.{ScatterPlot,  BlackPlotTheme, Legend} val labels  = Legend(name,  s"Q-learning config:  $params", "States", "States")ScatterPlot.display(eq, labels,  new BlackPlotTheme) } Hang on and do not lose patience! We are finally ready to see a simple rn and inspect the result. So let us do it: def main(args: Array[String]): Unit = {run("Maximum reward", QUANTIZATION_STEP, ALPHA, DISCOUNT) Action: state 71 => state 74 Action: state 71 => state 73 Action: state 71 => state 72 Action: state 71 => state 70 Action: state 71 => state 69 Action: state 71 => state 68...Instance: [I@1f021e6c - state: 124 Action: state 124 => state 125 Action: state 124 => state 123 Action: state 124 => state 122 Action: state 124 => state 121Q-learning Coverage 0.1 for 126 states and 15750 transitions Q-learning Execution profile Q-Value -> 5.572310105096295, 0.013869013819834967, 4.5746487300071825, 0.4037703812585325, 0.17606260549479869, 0.09205272504875522, 0.023205692430068765, 0.06363082458984902, 50.405283888218435... 6.5530411130514015 Model: Success(Optimal policy: Reward - 1.00,204.28,115.57,6.05,637.58,71.99,12.34,0.10,4939.71,521.30,402.73, with coverage: 0.1) Evaluating the model The preceding output shows the transition from one state to another, and for the 0.1 coverage, the Q-Learning model had 15,750 transitions for 126 states to reach goal state 37 with optimal rewards. Therefore, the training set is quite small and only a few buckets have actual values. So we can understand that the size of the training set has an impact on the number of states. Q-Learning will converge too fast for a small training set (like what we have for this example). However, for a larger training set, Q-Learning will take time to converge; it will provide at least one value for each bucket created by the approximation. Also, by seeing those values, it is difficult to understand the relation between Q-values and states. So what if we can see the Q-values per state? Why not! We can see them on a scatter plot: Figure  5: Q-value  per state Now let us display the profile of the log of the Q-value (QLData.value) as the recursive search (or training) progress for different episodes or epochs. The test uses a learning rate α = 0.1 and a discount rate γ = 0.9 (see more in the deployment section): Figure 6: Profile of the logarithmic Q-Value for different  epochs during Q-learning training The preceding chart illustrates the fact that the Q-value for each profile is independent of the order of the epochs during training. However, the number of iterations to reach the goal state depends on the initial state selected randomly in this example. To get more insights, inspect the output on your editor or access the API endpoint at http://localhost:9000/api/compute (see following). Now, what if we display the distribution of values in the model and display the estimated Q-value for the best policy on a Scatter plot for the given configuration parameters? Figure 7: Maximum reward with quantization 32 with the QLearning The final evaluation consists of evaluating the impact of the learning rate and discount rate on the coverage of the training: Figure 7: Impact of the learning rate and discount rate on the coverage of the training The coverage decreases as the learning rate increases. This result confirms the general rule of using learning rate < 0.2. A similar test to evaluate the impact of the discount rate on the coverage is inconclusive. We learned to develop a real-life application for options trading using a reinforcement learning algorithm called Q-learning. You read an excerpt from a book written by Md. Rezaul Karim, titled Scala Machine Learning Projects. In this book, you will learn to develop, build, and deploy research or commercial machine learning projects in a production-ready environment. Check out other related posts: Getting started with Q-learning using TensorFlow How to implement Reinforcement Learning with TensorFlow How Reinforcement Learning works  
Read more
  • 0
  • 0
  • 13444

article-image-asynchronous-programming-f
Packt
12 Oct 2016
15 min read
Save for later

Asynchronous Programming in F#

Packt
12 Oct 2016
15 min read
In this article by Alfonso Garcia Caro Nunez and Suhaib Fahad, author of the book Mastering F#, sheds light on how writing applications that are non-blocking or reacting to events is increasingly becoming important in this cloud world we live in. A modern application needs to carry out a rich user interaction, communicate with web services, react to notifications, and so on; the execution of reactive applications is controlled by events. Asynchronous programming is characterized by many simultaneously pending reactions to internal or external events. These reactions may or may not be processed in parallel. (For more resources related to this topic, see here.) In .NET, both C# and F# provide asynchronous programming experience through keywords and syntaxes. In this article, we will go through the asynchronous programming model in F#, with a bit of cross-referencing or comparison drawn with the C# world. In this article, you will learn about asynchronous workflows in F# Asynchronous workflows in F# Asynchronous workflows are computation expressions that are setup to run asynchronously. It means that the system runs without blocking the current computation thread when a sleep, I/O, or other asynchronous process is performed. You may be wondering why do we need asynchronous programming and why can't we just use the threading concepts that we did for so long. The problem with threads is that the operation occupies the thread for the entire time that something happens or when a computation is done. On the other hand, asynchronous programming will enable a thread only when it is required, otherwise it will be normal code. There is also lot of marshalling and unmarshalling (junk) code that we will write around to overcome the issues that we face when directly dealing with threads. Thus, asynchronous model allows the code to execute efficiently whether we are downloading a page 50 or 100 times using a single thread or if we are doing some I/O operation over the network and there are a lot of incoming requests from the other endpoint. There is a list of functions that the Async module in F# exposes to create or use these asynchronous workflows to program. The asynchronous pattern allows writing code that looks like it is written for a single-threaded program, but in the internals, it uses async blocks to execute. There are various triggering functions that provide a wide variety of ways to create the asynchronous workflow, which is either a background thread, a .NET framework task object, or running the computation in the current thread itself. In this article, we will use the example of downloading the content of a webpage and modifying the data, which is as follows: let downloadPage (url: string) = async { let req = HttpWebRequest.Create(url) use! resp = req.AsyncGetResponse() use respStream = resp.GetResponseStream() use sr = new StreamReader(respStream) return sr.ReadToEnd() } downloadPage("https://www.google.com") |> Async.RunSynchronously The preceding function does the following: The async expression, { … }, generates an object of type Async<string> These values are not actual results; rather, they are specifications of tasks that need to run and return a string Async.RunSynchronously takes this object and runs synchronously We just wrote a simple function with asynchronous workflows with relative ease and reason about the code, which is much better than using code with Begin/End routines. One of the most important point here is that the code is never blocked during the execution of the asynchronous workflow. This means that we can, in principle, have thousands of outstanding web requests—the limit being the number supported by the machine, not the number of threads that host them. Using let! In asynchronous workflows, we will use let! binding to enable execution to continue on other computations or threads, while the computation is being performed. After the execution is complete, the rest of the asynchronous workflow is executed, thus simulating a sequential execution in an asynchronous way. In addition to let!, we can also use use! to perform asynchronous bindings; basically, with use!, the object gets disposed when it loses the current scope. In our previous example, we used use! to get the HttpWebResponse object. We can also do as follows: let! resp = req.AsyncGetResponse() // process response We are using let! to start an operation and bind the result to a value, do!, which is used when the return of the async expression is a unit. do! Async.Sleep(1000) Understanding asynchronous workflows As explained earlier, asynchronous workflows are nothing but computation expressions with asynchronous patterns. It basically implements the Bind/Return pattern to implement the inner workings. This means that the let! expression is translated into a call to async. The bind and async.Return function are defined in the Async module in the F# library. This is a compiler functionality to translate the let! expression into computation workflows and, you, as a developer, will never be required to understand this in detail. The purpose of explaining this piece is to understand the internal workings of an asynchronous workflow, which is nothing but a computation expression. The following listing shows the translated version of the downloadPage function we defined earlier: async.Delay(fun() -> let req = HttpWebRequest.Create(url) async.Bind(req.AsyncGetResponse(), fun resp -> async.Using(resp, fun resp -> let respStream = resp.GetResponseStream() async.Using(new StreamReader(respStream), fun sr " -> reader.ReadToEnd() ) ) ) ) The following things are happening in the workflow: The Delay function has a deferred lambda that executes later. The body of the lambda creates an HttpWebRequest and is forwarded in a variable req to the next segment in the workflow. The AsyncGetResponse function is called and a workflow is generated, where it knows how to execute the response and invoke when the operation is completed. This happens internally with the BeginGetResponse and EndGetResponse functions already present in the HttpWebRequest class; the AsyncGetResponse is just a wrapper extension present in the F# Async module. The Using function then creates a closure to dispose the object with the IDisposable interface once the workflow is complete. Async module The Async module has a list of functions that allows writing or consuming asynchronous code. We will go through each function in detail with an example to understand it better. Async.AsBeginEnd It is very useful to expose the F# workflow functionality out of F#, say if we want to use and consume the API's in C#. The Async.AsBeginEnd method gives the possibility of exposing the asynchronous workflows as a triple of methods—Begin/End/Cancel—following the .NET Asynchronous Programming Model (APM). Based on our downloadPage function, we can define the Begin, End, Cancel functions as follows: type Downloader() = let beginMethod, endMethod, cancelMethod = " Async.AsBeginEnd downloadPage member this.BeginDownload(url, callback, state : obj) = " beginMethod(url, callback, state) member this.EndDownload(ar) = endMethod ar member this.CancelDownload(ar) = cancelMethod(ar) Async.AwaitEvent The Async.AwaitEvent method creates an asynchronous computation that waits for a single invocation of a .NET framework event by adding a handler to the event. type MyEvent(v : string) = inherit EventArgs() member this.Value = v; let testAwaitEvent (evt : IEvent<MyEvent>) = async { printfn "Before waiting" let! r = Async.AwaitEvent evt printfn "After waiting: %O" r.Value do! Async.Sleep(1000) return () } let runAwaitEventTest () = let evt = new Event<Handler<MyEvent>, _>() Async.Start <| testAwaitEvent evt.Publish System.Threading.Thread.Sleep(3000) printfn "Before raising" evt.Trigger(null, new MyEvent("value")) printfn "After raising" > runAwaitEventTest();; > Before waiting > Before raising > After raising > After waiting : value The testAwaitEvent function listens to the event using Async.AwaitEvent and prints the value. As the Async.Start will take some time to start up the thread, we will simply call Thread.Sleep to wait on the main thread. This is for example purpose only. We can think of scenarios where a button-click event is awaited and used inside an async block. Async.AwaitIAsyncResult Creates a computation result and waits for the IAsyncResult to complete. IAsyncResult is the asynchronous programming model interface that allows us to write asynchronous programs. It returns true if IAsyncResult issues a signal within the given timeout. The timeout parameter is optional, and its default value is -1 of Timeout.Infinite. let testAwaitIAsyncResult (url: string) = async { let req = HttpWebRequest.Create(url) let aResp = req.BeginGetResponse(null, null) let! asyncResp = Async.AwaitIAsyncResult(aResp, 1000) if asyncResp then let resp = req.EndGetResponse(aResp) use respStream = resp.GetResponseStream() use sr = new StreamReader(respStream) return sr.ReadToEnd() else return "" } > Async.RunSynchronously (testAwaitIAsyncResult "https://www.google.com") We will modify the downloadPage example with AwaitIAsyncResult, which allows a bit more flexibility where we want to add timeouts as well. In the preceding example, the AwaitIAsyncResult handle will wait for 1000 milliseconds, and then it will execute the next steps. Async.AwaitWaitHandle Creates a computation that waits on a WaitHandle—wait handles are a mechanism to control the execution of threads. The following is an example with ManualResetEvent: let testAwaitWaitHandle waitHandle = async { printfn "Before waiting" let! r = Async.AwaitWaitHandle waitHandle printfn "After waiting" } let runTestAwaitWaitHandle () = let event = new System.Threading.ManualResetEvent(false) Async.Start <| testAwaitWaitHandle event System.Threading.Thread.Sleep(3000) printfn "Before raising" event.Set() |> ignore printfn "After raising" The preceding example uses ManualResetEvent to show how to use AwaitHandle, which is very similar to the event example that we saw in the previous topic. Async.AwaitTask Returns an asynchronous computation that waits for the given task to complete and returns its result. This helps in consuming C# APIs that exposes task based asynchronous operations. let downloadPageAsTask (url: string) = async { let req = HttpWebRequest.Create(url) use! resp = req.AsyncGetResponse() use respStream = resp.GetResponseStream() use sr = new StreamReader(respStream) return sr.ReadToEnd() } |> Async.StartAsTask let testAwaitTask (t: Task<string>) = async { let! r = Async.AwaitTask t return r } > downloadPageAsTask "https://www.google.com" |> testAwaitTask |> Async.RunSynchronously;; The preceding function is also downloading the web page as HTML content, but it starts the operation as a .NET task object. Async.FromBeginEnd The FromBeginEnd method acts as an adapter for the asynchronous workflow interface by wrapping the provided Begin/End method. Thus, it allows using large number of existing components that support an asynchronous mode of work. The IAsyncResult interface exposes the functions as Begin/End pattern for asynchronous programming. We will look at the same download page example using FromBeginEnd: let downloadPageBeginEnd (url: string) = async { let req = HttpWebRequest.Create(url) use! resp = Async.FromBeginEnd(req.BeginGetResponse, req.EndGetResponse) use respStream = resp.GetResponseStream() use sr = new StreamReader(respStream) return sr.ReadToEnd() } The function accepts two parameters and automatically identifies the return type; we will use BeginGetResponse and EndGetResponse as our functions to call. Internally, Async.FromBeginEnd delegates the asynchronous operation and gets back the handle once the EndGetResponse is called. Async.FromContinuations Creates an asynchronous computation that captures the current success, exception, and cancellation continuations. To understand these three operations, let's create a sleep function similar to Async.Sleep using timer: let sleep t = Async.FromContinuations(fun (cont, erFun, _) -> let rec timer = new Timer(TimerCallback(callback)) and callback state = timer.Dispose() cont(()) timer.Change(t, Timeout.Infinite) |> ignore ) let testSleep = async { printfn "Before" do! sleep 5000 printfn "After 5000 msecs" } Async.RunSynchronously testSleep The sleep function takes an integer and returns a unit; it uses Async.FromContinuations to allow the flow of the program to continue when a timer event is raised. It does so by calling the cont(()) function, which is a continuation to allow the next step in the asynchronous flow to execute. If there is any error, we can call erFun to throw the exception and it will be handled from the place we are calling this function. Using the FromContinuation function helps us wrap and expose functionality as async, which can be used inside asynchronous workflows. It also helps to control the execution of the programming with cancelling or throwing errors using simple APIs. Async.Start Starts the asynchronous computation in the thread pool. It accepts an Async<unit> function to start the asynchronous computation. The downloadPage function can be started as follows: let asyncDownloadPage(url) = async { let! result = downloadPage(url) printfn "%s" result"} asyncDownloadPage "http://www.google.com" |> Async.Start   We wrap the function to another async function that returns an Async<unit> function so that it can be called by Async.Start. Async.StartChild Starts a child computation within an asynchronous workflow. This allows multiple asynchronous computations to be executed simultaneously, as follows: let subTask v = async { print "Task %d started" v Thread.Sleep (v * 1000) print "Task %d finished" v return v } let mainTask = async { print "Main task started" let! childTask1 = Async.StartChild (subTask 1) let! childTask2 = Async.StartChild (subTask 5) print "Subtasks started" let! child1Result = childTask1 print "Subtask1 result: %d" child1Result let! child2Result = childTask2 print "Subtask2 result: %d" child2Result print "Subtasks completed" return () } Async.RunSynchronously mainTask Async.StartAsTask Executes a computation in the thread pool and returns a task that will be completed in the corresponding state once the computation terminates. We can use the same example of starting the downloadPage function as a task. let downloadPageAsTask (url: string) = async { let req = HttpWebRequest.Create(url) use! resp = req.AsyncGetResponse() use respStream = resp.GetResponseStream() use sr = new StreamReader(respStream) return sr.ReadToEnd() } |> Async.StartAsTask let task = downloadPageAsTask("http://www.google.com") prinfn "Do some work" task.Wait() printfn "done"   Async.StartChildAsTask Creates an asynchronous computation from within an asynchronous computation, which starts the given computation as a task. let testAwaitTask = async { print "Starting" let! child = Async.StartChildAsTask <| async { // " Async.StartChildAsTask shall be described later print "Child started" Thread.Sleep(5000) print "Child finished" return 100 } print "Waiting for the child task" let! result = Async.AwaitTask child print "Child result %d" result } Async.StartImmediate Runs an asynchronous computation, starting immediately on the current operating system thread. This is very similar to the Async.Start function we saw earlier: let asyncDownloadPage(url) = async { let! result = downloadPage(url) printfn "%s" result"} asyncDownloadPage "http://www.google.com" |> Async.StartImmediate Async.SwitchToNewThread Creates an asynchronous computation that creates a new thread and runs its continuation in it: let asyncDownloadPage(url) = async { do! Async.SwitchToNewThread() let! result = downloadPage(url) printfn "%s" result"} asyncDownloadPage "http://www.google.com" |> Async.Start   Async.SwitchToThreadPool Creates an asynchronous computation that queues a work item that runs its continuation, as follows: let asyncDownloadPage(url) = async { do! Async.SwitchToNewThread() let! result = downloadPage(url) do! Async.SwitchToThreadPool() printfn "%s" result"} asyncDownloadPage "http://www.google.com" |> Async.Start   Async.SwitchToContext Creates an asynchronous computation that runs its continuation in the Post method of the synchronization context. Let's assume that we set the text from the downloadPage function to a UI textbox, then we will do it as follows: let syncContext = System.Threading.SynchronizationContext() let asyncDownloadPage(url) = async { do! Async.SwitchToContext(syncContext) let! result = downloadPage(url) textbox.Text <- result"} asyncDownloadPage "http://www.google.com" |> Async.Start   Note that in the console applications, the context will be null. Async.Parallel The Parallel function allows you to execute individual asynchronous computations queued in the thread pool and uses the fork/join pattern. let parallel_download() = let sites = ["http://www.bing.com"; "http://www.google.com"; "http://www.yahoo.com"; "http://www.search.com"] let htmlOfSites = Async.Parallel [for site in sites -> downloadPage site ] |> Async.RunSynchronously printfn "%A" htmlOfSites   We will use the same example of downloading HTML content in a parallel way. The preceding example shows the essence of parallel I/O computation The async function, { … }, in the downloadPage function shows the asynchronous computation These are then composed in parallel using the fork/join combinator In this sample, the composition executed waits synchronously for overall result Async.OnCancel A cancellation interruption in the asynchronous computation when a cancellation occurs. It returns an asynchronous computation trigger before being disposed. // This is a simulated cancellable computation. It checks the " token source // to see whether the cancel signal was received. let computation " (tokenSource:System.Threading.CancellationTokenSource) = async { use! cancelHandler = Async.OnCancel(fun () -> printfn " "Canceling operation.") // Async.Sleep checks for cancellation at the end of " the sleep interval, // so loop over many short sleep intervals instead of " sleeping // for a long time. while true do do! Async.Sleep(100) } let tokenSource1 = new " System.Threading.CancellationTokenSource() let tokenSource2 = new " System.Threading.CancellationTokenSource() Async.Start(computation tokenSource1, tokenSource1.Token) Async.Start(computation tokenSource2, tokenSource2.Token) printfn "Started computations." System.Threading.Thread.Sleep(1000) printfn "Sending cancellation signal." tokenSource1.Cancel() tokenSource2.Cancel() The preceding example implements the Async.OnCancel method to catch or interrupt the process when CancellationTokenSource is cancelled. Summary In this article, we went through detail, explanations about different semantics in asynchronous programming with F#, used with asynchronous workflows. We saw a number of functions from the Async module. Resources for Article: Further resources on this subject: Creating an F# Project [article] Asynchronous Control Flow Patterns with ES2015 and beyond [article] Responsive Applications with Asynchronous Programming [article]
Read more
  • 0
  • 0
  • 13428
Modal Close icon
Modal Close icon