Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-understanding-hbase-ecosystem
Packt
24 Nov 2014
11 min read
Save for later

Understanding the HBase Ecosystem

Packt
24 Nov 2014
11 min read
This article by Shashwat Shriparv, author of the book, Learning HBase, will introduce you to the world of HBase. (For more resources related to this topic, see here.) HBase is a horizontally scalable, distributed, open source, and a sorted map database. It runs on top of Hadoop file system that is Hadoop Distributed File System (HDFS). HBase is a NoSQL nonrelational database that doesn't always require a predefined schema. It can be seen as a scaling flexible, multidimensional spreadsheet where any structure of data is fit with on-the-fly addition of new column fields, and fined column structure before data can be inserted or queried. In other words, HBase is a column-based database that runs on top of Hadoop distributed file system and supports features such as linear scalability (scale out), automatic failover, automatic sharding, and more flexible schema. HBase is modeled on Google BigTable. It was inspired by Google BigTable, which is a compressed, high-performance, proprietary data store built on the Google filesystem. HBase was developed as a Hadoop subproject to support storage of structural data, which can take advantage of most distributed files systems (typically, the Hadoop Distributed File System known as HDFS). The following table contains key information about HBase and its features: Features Description Developed by Apache Written in Java Type Column oriented License Apache License Lacking features of relational databases SQL support, relations, primary, foreign, and unique key constraints, normalization Website http://hbase.apache.org Distributions Apache, Cloudera Download link http://mirrors.advancedhosters.com/apache/hbase/ Mailing lists The user list: user-subscribe@hbase.apache.org The developer list: dev-subscribe@hbase.apache.org Blog http://blogs.apache.org/hbase/ HBase layout on top of Hadoop The following figure represents the layout information of HBase on top of Hadoop: There is more than one ZooKeeper in the setup, which provides high availability of master status; a RegionServer may contain multiple rations. The RegionServers run on the machines where DataNodes run. There can be as many RegionServers as DataNodes. RegionServers can have multiple HRegions; one HRegion can have one HLog and multiple HFiles with its associate's MemStore. HBase can be seen as a master-slave database where the master is called HMaster, which is responsible for coordination between client application and HRegionServer. It is also responsible for monitoring and recording metadata changes and management. Slaves are called HRegionServers, which serve the actual tables in form of regions. These regions are the basic building blocks of the HBase tables, which contain distribution of tables. So, HMaster and RegionServer work in coordination to serve the HBase tables and HBase cluster. Usually, HMaster is co-hosted with Hadoop NameNode daemon process on a server and communicates to DataNode daemon for reading and writing data on HDFS. The RegionServer runs or is co-hosted on the Hadoop DataNodes. Comparing architectural differences between RDBMs and HBase Let's list the major differences between relational databases and HBase: Relational databases HBase Uses tables as databases Uses regions as databases Filesystems supported are FAT, NTFS, and EXT Filesystem supported is HDFS The technique used to store logs is commit logs The technique used to store logs is Write-Ahead Logs (WAL) The reference system used is coordinate system The reference system used is ZooKeeper Uses the primary key Uses the row key Partitioning is supported Sharding is supported Use of rows, columns, and cells Use of rows, column families, columns, and cells HBase features Let's see the major features of HBase that make it one of the most useful databases for the current and future industry: Automatic failover and load balancing: HBase runs on top of HDFS, which is internally distributed and automatically recovered using multiple block allocation and replications. It works with multiple HMasters and region servers. This failover is also facilitated using HBase and RegionServer replication. Automatic sharding: An HBase table is made up of regions that are hosted by RegionServers and these regions are distributed throughout the RegionServers on different DataNodes. HBase provides automatic and manual splitting of these regions to smaller subregions, once it reaches a threshold size to reduce I/O time and overhead. Hadoop/HDFS integration: It's important to note that HBase can run on top of other filesystems as well. While HDFS is the most common choice as it supports data distribution and high availability using distributed Hadoop, for which we just need to set some configuration parameters and enable HBase to communicate to Hadoop, an out-of-the-box underlying distribution is provided by HDFS. Real-time, random big data access: HBase uses log-structured merge-tree (LSM-tree) as data storage architecture internally, which merges smaller files to larger files periodically to reduce disk seeks. MapReduce: HBase has a built-in support of Hadoop MapReduce framework for fast and parallel processing of data stored in HBase. You can search for the Package org.apache.hadoop.hbase.mapreduce for more details. Java API for client access: HBase has a solid Java API support (client/server) for easy development and programming. Thrift and a RESTtful web service: HBase not only provides a thrift and RESTful gateway but also web service gateways for integrating and accessing HBase besides Java code (HBase Java APIs) for accessing and working with HBase. Support for exporting metrics via the Hadoop metrics subsystem: HBase provides Java Management Extensions (JMX) and exporting matrix for monitoring purposes with tools such as Ganglia and Nagios. Distributed: HBase works when used with HDFS. It provides coordination with Hadoop so that distribution of tables, high availability, and consistency is supported by it. Linear scalability (scale out): Scaling of HBase is not scale up but scale out, which means that we don't need to make servers more powerful but we add more machines to its cluster. We can add more nodes to the cluster on the fly. As soon as a new RegionServer node is up, the cluster can begin rebalancing, start the RegionServer on the new node, and it is scaled up, it is as simple as that. Column oriented: HBase stores each column separately in contrast with most of the relational databases, which uses stores or are row-based storage. So in HBase, columns are stored contiguously and not the rows. More about row- and column-oriented databases will follow. HBase shell support: HBase provides a command-line tool to interact with HBase and perform simple operations such as creating tables, adding data, and scanning data. This also provides full-fledged command-line tool using which we can interact with HBase and perform operations such as creating table, adding data, removing data, and a few other administrative commands. Sparse, multidimensional, sorted map database: HBase is a sparse, multidimensional, sorted map-based database, which supports multiple versions of the same record. Snapshot support: HBase supports taking snapshots of metadata for getting the previous or correct state form of data. HBase in the Hadoop ecosystem Let's see where HBase sits in the Hadoop ecosystem. In the Hadoop ecosystem, HBase provides a persistent, structured, schema-based data store. The following figure illustrates the Hadoop ecosystem: HBase can work as a separate entity on the local filesystem (which is not really effective as no distribution is provided) as well as in coordination with Hadoop as a separate but connected entity. As we know, Hadoop provides two services, a distributed files system (HDFS) for storage and a MapReduce framework for processing in a parallel mode. When there was a need to store structured data (data in the form of tables, rows and columns), which most of the programmers are already familiar with, the programmers were finding it difficult to process the data that was stored on HDFS as an unstructured flat file format. This led to the evolution of HBase, which provided a way to store data in a structural way. Consider that we have got a CSV file stored on HDFS and we need to query from it. We would need to write a Java code for this, which wouldn't be a good option. It would be better if we could specify the data key and fetch the data from that file. So, what we can do here is create a schema or table with the same structure of CSV file to store the data of the CSV file in the HBase table and query using HBase APIs, or HBase shell using key. Data representation in HBase Let's look into the representation of rows and columns in HBase table: An HBase table is divided into rows, column families, columns, and cells. Row keys are unique keys to identify a row, column families are groups of columns, columns are fields of the table, and the cell contains the actual value or the data. So, we have been through the introduction of HBase; now, let's see what Hadoop and its components are in brief. It is assumed here that you are already familiar with Hadoop; if not, following a brief introduction about Hadoop will help you to understand it. Hadoop Hadoop is an underlying technology of HBase, providing high availability, fault tolerance, and distribution. It is an Apache-sponsored, free, open source, Java-based programming framework which supports large dataset storage. It provides distributed file system and MapReduce, which is a distributed programming framework. It provides a scalable, reliable, distributed storage and development environment. Hadoop makes it possible to run applications on a system with tens to tens of thousands of nodes. The underlying distributed file system provides large-scale storage, rapid data access. It has the following submodules: Hadoop Common: This is the core component that supports the other Hadoop modules. It is like the master components facilitating communication and coordination between different Hadoop modules. Hadoop distributed file system: This is the underlying distributed file system, which is abstracted on the top of the local filesystem that provides high throughput of read and write operations of data on Hadoop. Hadoop YARN: This is the new framework that is shipped with newer releases of Hadoop. It provides job scheduling and job and resource management. Hadoop MapReduce: This is the Hadoop-based processing system that provides parallel processing of large data and datasets. Other Hadoop subprojects are HBase, Hive, Ambari, Avro, Cassandra (Cassandra isn't a Hadoop subproject, it's a related project; they solve similar problems in different ways), Mahout, Pig, Spark, ZooKeeper (ZooKeeper isn't a Hadoop subproject. It's a dependency shared by many distributed systems), and so on. All of these have different usability and the combination of all these subprojects forms the Hadoop ecosystem. Core daemons of Hadoop The following are the core daemons of Hadoop: NameNode: This stores and manages all metadata about the data present on the cluster, so it is the single point of contact to Hadoop. In the new release of Hadoop, we have an option of more than one NameNode for high availability. JobTracker: This runs on the NameNode and performs the MapReduce of the jobs submitted to the cluster. SecondaryNameNode: This maintains the backup of metadata present on the NameNode, and also records the file system changes. DataNode: This will contain the actual data. TaskTracker: This will perform tasks on the local data assigned by the JobTracker. The preceding are the daemons in the case of Hadoop v1 or earlier. In newer versions of Hadoop, we have ResourceManager instead of JobTracker, the node manager instead of TaskTrackers, and the YARN framework instead of a simple MapReduce framework. The following is the comparison between daemons in Hadoop 1 and Hadoop 2: Hadoop 1 Hadoop 2 HDFS NameNode Secondary NameNode DataNode   NameNode (more than one active/standby) Checkpoint node DataNode Processing MapReduce v1 JobTracker TaskTracker   YARN (MRv2) ResourceManager NodeManager Application Master Comparing HBase with Hadoop As we now know what HBase and what Hadoop are, let's have a comparison between HDFS and HBase for better understanding: Hadoop/HDFS HBase This provide filesystem for distributed storage This provides tabular column-oriented data storage This is optimized for storage of huge-sized files with no random read/write of these files This is optimized for tabular data with random read/write facility This uses flat files This uses key-value pairs of data The data model is not flexible Provides a flexible data model This uses file system and processing framework This uses tabular storage with built-in Hadoop MapReduce support This is mostly optimized for write-once read-many This is optimized for both read/write many Summary So in this article, we discussed the introductory aspects of HBase and it's features. We have also discussed HBase's components and their place in the HBase ecosystem. Resources for Article: Further resources on this subject: The HBase's Data Storage [Article] HBase Administration, Performance Tuning [Article] Comparative Study of NoSQL Products [Article]
Read more
  • 0
  • 0
  • 15533

article-image-monitoring-additional-servers
Packt
19 Aug 2013
7 min read
Save for later

Monitoring additional servers

Packt
19 Aug 2013
7 min read
(For more resources related to this topic, see here.) Step 1 – Installing munin-node First we need to connect to the server we want to monitor and install munin-node. In our examples, we will be using the name muninnode as the name of our additional server. Your server will probably have a different name, so every time you see muninnode in an example, you should replace that with the name of the server you are using. Examples will also use the term username, which you should replace with your username. But first, let's install munin-node. For Debian or Ubuntu, use the following commands: ssh username@muninnodesudo apt-get install munin-node For Red Hat or Fedora, use: ssh username@muninnodesudo yum install munin-node Next, we will take a look at the generated configuration file. It is located at /etc/munin/ munin-node.conf. Please open it up in your favorite editor. The first thing we have to take care of is the fact that we want our master to be able to connect to this node. For security reasons, munin-node defaults to allowing only connections from the localhost to query its data. So, let's scroll down to the allow section and add a line beneath it. If your master has a static IP address, please enter it in the allow section in the following format: allow 10.0.0.200 This will grant the master at 10.0.0.200 access to the data of this node. If your server has a dynamic IP or you want to trust your entire network range, you can either add a single line for every possible IP addresses or use a cidr_allow section. Please note that you can only use this if you have the Net::CIDR Perl module installed. Most systems will have this by default, but if you are having problems, you should check that. cidr_allow 10.0.0.0/24 This will grant anyone connecting from any IP from 10.0.0.0 to 10.0.0.255 to fetch all the information available in this node. After you have done this, you need to save the file and restart the munin-node daemon. For older versions of Debian or Ubuntu, use the following command: sudo invoke-rc.d munin-node restart For Debian or Ubuntu and Red Hat or Fedora, use: sudo service munin-node restart Step 2 – Testing your munin-node installation Now that we have installed the node, it is a good idea to check if it functions correctly. We will do this by connecting to the node and fetching some information. ssh username@muninnodetelnet localhost 4949versionlistquit You should get the following output: ssh username@muninnodeWelcome to muninnodeusername@muninnode:~$ telnet muninnode 4949Trying 127.0.0.1...Connected to localhost.Escape character is '^]'.# munin node at muninnode.versionmunins node on muninnode. version: 2.0.9-2listcpu df df_inode entropy forks fw_packets http_loadtime if_err_eth0if_eth0 interrupts iostat iostat_ios irqstats load memorymunin_stats ntp_kernel_err ntp_kernel_pll_freq ntp_kernel_pll_offntp_offset open_files open_inodes proc_pri processes sensors swapthreads uptime users vmstatquitConnection closed by foreign host.username@muninnode:~$ Please note that the node might be a bit impatient with you. If you connect to it using Telnet and then give no further instructions for a few seconds, munin-node will automatically disconnect you, thinking you are just wasting it's time. If this happens, just go ahead and try again. Now that we know that munin-node is running, we want to make sure it is functioning correctly. Munin-node keeps its log file at /var/log/munin/munin-node.log. Let's take a look at that. ssh username@muninnodetail /var/log/munin/munin.log You should be able to see your connection attempt in the log; it should look something like the following: 2013/01/01-12:30:10 CONNECT TCP Peer: "127.0.0.1:44363" Local: "127.0.0.1:4949" If you have a node that is experiencing problems with connections or a plugin, make sure to look at this log file for exceptions or error messages. Step 3 – Installing additional plugins When munin-node was installed, it ran its autodetect script to enable plugins from its standard library if they were applicable to your system. If you have installed new software on this machine, you can easily re-run this script to see if Munin can help you monitor the new software. If you, for example, have installed MySQL or PostgreSQL, then this is what you do: ssh username@muninnodesudo munin-node-configure --suggestsudo munin-node-configure --shell The first command will show you all the plugins your munin-node has out of the box and whether they apply to your system. The second command will display the commands you will have to execute to create the symbolic links in order to enable those suggestions. Please note that not all plugins support this, and therefore, not all applicable plugins will automatically be enabled. Munin-node has to be restarted after you've added new plugins; otherwise, these changes will not take effect. Step 4 – Adding the new node to the master Now that we've completely configured the node and tested to see if it works, we are ready to add the node to our master. To do this, we have to go to our master and test whether we can connect back to our munin-node. ssh username@muninmastertelnet 4949 muninnodeversionlistquit This should display the version and the capabilities of the munin-node running on the muninnode server. If this does not work, make sure you have started the munin-node on the muninnode server and also check whether firewalls allow you to connect to it on port 4949. Also go ahead and recheck the allowed IP addresses in the munin-node configuration as mentioned in step 2. If this is working correctly, go ahead and open up the file at /etc/munin/munin.conf. Here, we scroll down until we see the following host tree: # a simple host tree[localhost.localdomain]address 127.0.0.1use_node_name yes We need to add our new munin-node to this host tree as follows: # the host tree of our local network[localhost.localdomain]address 127.0.0.1use_node_name yes[muninnode.localdomain]address 10.0.0.200use_node_name yes Now, we'll have to wait at least 10 minutes before we will be able to see our new node on the Munin master's website. Go ahead and point your browser to your Munin master at http: //localhost/munin or at http://your_munin_master/munin; you should see something like the following screenshot: After a couple of minutes, you should be able to see graphs for your node and even compare the nodes of your cluster side by side. Troubleshooting Now it could very well be possible that it isn't working for you. Here are the few steps you should check first: Check the Munin master log at /var/log/munin/munin.log for errors. Check the Munin node log at /var/log/munin/munin-node.log on the munin server for access calls and errors. Try to connect from your Munin master to your node using Telnet 4949. If you can connect, type nodes and check whether the name of your node is there. Still in Telnet, type list munninnode.localdomain and check whether you get a list of plugins. If not, add your hostname to /etc/munin/munin-node.conf (see the Munin node configuration section). Summary We looked at the first step in expanding your munin cluster. Once you know how to add one server, you will be able to add all of them! Resources for Article : Further resources on this subject: Device Management in Zenoss Core Network and System Monitoring: Part 1 [Article] HP Network Node Manager 9: Understanding Smart Plug-Ins [Article] An Introduction to Flash Builder 4-Network Monitor [Article]
Read more
  • 0
  • 0
  • 15527

article-image-introduction-cyber-extortion
Packt
19 Jun 2017
21 min read
Save for later

Introduction to Cyber Extortion

Packt
19 Jun 2017
21 min read
In this article by Dhanya Thakkar, the author of the book Preventing Digital Extortion, explains how often we make the mistake of relying on the past for predicting the future, and nowhere is this more relevant than in the sphere of the Internet and smart technology. People, processes, data, and things are tightly and increasingly connected, creating new, intelligent networks unlike anything else we have seen before. The growth is exponential and the consequences are far reaching for individuals, and progressively so for businesses. We are creating the Internet of Things and the Internet of Everything. (For more resources related to this topic, see here.) It has become unimaginable to run a business without using the Internet. It is not only an essential tool for current products and services, but an unfathomable well for innovation and fresh commercial breakthroughs. The transformative revolution is spillinginto the public sector, affecting companies like vanguards and diffusing to consumers, who are in a feedback loop with suppliers, constantly obtaining and demanding new goods. Advanced technologies that apply not only to machine-to-machine communication but also to smart sensors generate complex networks to which theoretically anything that can carry a sensor can be connected. Cloud computing and cloud-based applications provide immense yet affordable storage capacity for people and organizations and facilitate the spread of data in more ways than one. Keeping in mind the Internet’s nature, the physical boundaries of business become blurred, and virtual data protection must incorporate a new characteristic of security: encryption. In the middle of the storm of the IoT, major opportunities arise, and equally so, unprecedented risks lurk. People often think that what they put on the Internet is protected and closed information. It is hardly so. Sending an e-mail is not like sending a letter in a closed envelope. It is more like sending a postcard, where anyone who gets their hands on it can read what's written on it. Along with people who want to utilize the Internet as an open business platform, there are people who want to find ways of circumventing legal practices and misusing the wealth of data on computer networks by unlawfully gaining financial profits, assets, or authority that can be monetized. Being connected is now critical. As cyberspace is growing, so are attempts to violate vulnerable information gaining global scale. This newly discovered business dynamic is under persistent threat of criminals. Cyberspace, cyber crime, and cyber security are perceptibly being found in the same sentence. Cyber crime –under defined and under regulated A massive problem encouraging the perseverance and evolution of cyber crime is the lack of an adequate unanimous definition and the under regulation on a national, regional, and global level. Nothing is criminal unless stipulated by the law. Global law enforcement agencies, academia, and state policies have studied the constant development of the phenomenon since its first appearance in 1989, in the shape of the AIDS Trojan virus transferred from an infected floppy disk. Regardless of the bizarre beginnings, there is nothing entertaining about cybercrime. It is serious. It is dangerous. Significant efforts are made to define cybercrime on a conceptual level in academic research and in national and regional cybersecurity strategies. Still, as the nature of the phenomenon evolves, so must the definition. Research reports are still at a descriptive level, and underreporting is a major issue. On the other hand, businesses are more exposed due to ignorance of the fact that modern-day criminals increasingly rely on the Internet to enhance their criminal operations. Case in point: Aaushi Shah and Srinidhi Ravi from the Asian School of Cyber Laws have created a cybercrime list by compiling a set of 74 distinctive and creativelynamed actions emerging in the last three decades that can be interpreted as cybercrime. These actions target anything from e-mails to smartphones, personal computers, and business intranets: piggybacking, joe jobs, and easter eggs may sound like cartoons, but their true nature resembles a crime thriller. The concept of cybercrime Cyberspace is a giant community made out of connected computer users and data on a global level. As a concept, cybercrime involves any criminal act dealing withcomputers andnetworks, including traditional crimes in which the illegal activities are committed through the use of a computer and the Internet. As businesses become more open and widespread, the boundary between data freedom and restriction becomes more porous. Countless e-shopping transactions are made, hospitals keep record of patient histories, students pass exams, and around-the-clock payments are increasingly processed online. It is no wonder that criminals are relentlessly invading cyberspace trying to find a slipping crack. There are no recognizable border controls on the Internet, but a business that wants to evade harm needs to understand cybercrime's nature and apply means to restrict access to certain information. Instead of identifying it as a single phenomenon, Majid Jar proposes a common denominator approach for all ICT-related criminal activities. In his book Cybercrime and Society, Jar refers to Thomas and Loader’s working concept of cybercrime as follows: “Computer-mediated activities which are either illegal or considered illicit by certain parties and which can be conducted through global electronic network.” Jar elaborates the important distinction of this definition by emphasizing the difference between crime and deviance. Criminal activities are explicitly prohibited by formal regulations and bear sanctions, while deviances breach informal social norms. This is a key note to keep in mind. It encompasses the evolving definition of cybercrime, which keeps transforming after resourceful criminals who constantly think of new ways to gain illegal advantages. Law enforcement agencies on a global level make an essential distinction between two subcategories of cybercrime: Advanced cybercrime or high-tech crime Cyber-enabled crime The first subcategory, according to Interpol, includes newly emerged sophisticated attacks against computer hardware and software. On the other hand, the second category contains traditional crimes in modern clothes,for example crimes against children, such as exposing children to illegal content; financial crimes, such as payment card frauds, money laundering, and counterfeiting currency and security documents; social engineering frauds; and even terrorism. We are much beyond the limited impact of the 1989 cybercrime embryo. Intricate networks are created daily. They present new criminal opportunities, causing greater damage to businesses and individuals, and require a global response. Cybercrime is conceptualized as a service embracing a commercial component.Cybercriminals work as businessmen who look to sell a product or a service to the highest bidder. Critical attributes of cybercrime An abridged version of the cybercrime concept provides answers to three vital questions: Where are criminal activities committed and what technologies are used? What is the reason behind the violation? Who is the perpetrator of the activities? Where and how – realm Cybercrime can be an online, digitally committed, traditional offense. Even if the component of an online, digital, or virtual existence were not included in its nature, it would still have been considered crime in the traditional, real-world sense of the word. In this sense, as the nature of cybercrime advances, so mustthe spearheads of lawenforcement rely on laws written for the non-digital world to solve problems encountered online. Otherwise, the combat becomesstagnant and futile. Why – motivation The prefix "cyber" sometimes creates additional misperception when applied to the digital world. It is critical to differentiate cybercrime from other malevolent acts in the digital world by considering the reasoning behind the action. This is not only imperative for clarification purposes, but also for extending the definition of cybercrime over time to include previously indeterminate activities. Offenders commit a wide range of dishonest acts for selfish motives such as monetary gain, popularity, or gratification. When the intent behind the behavior is misinterpreted, confusion may arise and actions that should not have been classified as cybercrime could be charged with criminal prosecution. Who –the criminal deed component The action must be attributed to a perpetrator. Depending on the source, certain threats can be translated to the criminal domain only or expanded to endanger potential larger targets, representing an attack to national security or a terrorist attack. Undoubtedly, the concept of cybercrime needs additional refinement, and a comprehensive global definition is in progress. Along with global cybercrime initiatives, national regulators are continually working on implementing laws, policies, and strategies to exemplify cybercrime behaviors and thus strengthen combating efforts. Types of common cyber threats In their endeavors to raise cybercrime awareness, the United Kingdom'sNational Crime Agency (NCA) divided common and popular cybercrime activities by affiliating themwith the target under threat. While both individuals and organizations are targets of cyber criminals, it is the business-consumer networks that suffer irreparable damages due to the magnitude of harmful actions. Cybercrime targeting consumers Phishing The term encompasses behavior where illegitimate e-mails are sent to the receiver to collect security information and personal details Webcam manager A webcam manager is an instance of gross violating behavior in which criminals take over a person's webcam File hijacker Criminals hijack files and hold them "hostage" until the victim pays the demanded ransom Keylogging With keylogging, criminals have the means to record what the text behind the keysyou press on your keyboard is Screenshot manager A screenshot manager enables criminals to take screenshots of an individual’s computer screen Ad clicker Annoying but dangerous ad clickers direct victims’ computer to click on a specific harmful link Cybercrime targeting businesses Hacking Hacking is basically unauthorized access to computer data. Hackers inject specialist software with which they try to take administrative control of a computerized network or system. If the attack is successful, the stolen data can be sold on the dark web and compromise people’s integrity and safety by intruding and abusing the privacy of products as well as sensitive personal and business information. Hacking is particularly dangerous when it compromises the operation of systems that manage physical infrastructure, for example, public transportation. Distributed denial of service (DDoS) attacks When an online service is targeted by a DDoS attack, the communication links overflow with data from messages sent simultaneously by botnets. Botnets are a bunch of controlled computers that stop legitimate access to online services for users. The system is unable to provide normal access as it cannot handle the huge volume of incoming traffic. Cybercrime in relation to overall computer crime Many moons have passed since 2001, when the first international treatythat targeted Internet and computer crime—the Budapest Convention on Cybercrime—was adopted. The Convention’s intention was to harmonize national laws, improve investigative techniques, and increase cooperation among nations. It was drafted with the active participation of the Council of Europe's observer states Canada, Japan, South Africa, and the United States and drawn up by the Council of Europe in Strasbourg, France. Brazil and Russia, on the other hand, refused to sign the document on the basis of not being involved in the Convention's preparation. In The Understanding Cybercrime: A Guide to Developing Countries(Gercke, 2011), Marco Gercke makes an excellent final point: “Not all computer-related crimes come under the scope of cybercrime. Cybercrime is a narrower notion than all computer-related crime because it has to include a computer network. On the other hand, computer-related crime in general can also affect stand-alone computer systems.” Although progress has been made, consensus over the definition of cybercrime is not final. Keeping history in mind, a fluid and developing approach must be kept in mind when applying working and legal interpretations. In the end, international noncompliance must be overcome to establish a common and safe ground to tackle persistent threats. Cybercrime localized – what is the risk in your region? Europol’s heat map for the period between 2014 and 2015 reports on the geographical distribution of cybercrime on the basis of the United Nations geoscheme. The data in the report encompassed cyber-dependent crime and cyber-enabled fraud, but it did not include investigations into online child sexual abuse. North and South America Due to its overwhelming presence, it is not a great surprise that the North American region occupies several lead positions concerning cybercrime, both in terms of enabling malicious content and providing residency to victims in the regions that participate in the global cybercrime numbers. The United States hosted between 20% and nearly 40% of the total world's command-and-control servers during 2014. Additionally, the US currently hosts over 45% of the world's phishing domains and is in the pack of world-leading spam producers. Between 16% and 20% percent of all global bots are located in the United States, while almost a third of point-of-sale malware and over 40% of all ransomware incidents were detected there. Twenty EU member states have initiated criminal procedures in which the parties under suspicion were located in the United States. In addition, over 70 percent of the countries located in the Single European Payment Area have been subject to losses from skimmed payment cards because of the distinct way in which the US, under certain circumstances, processes card payments without chip-and-PIN technology. There are instances of cybercrime in South America, but the scope of participation by the southern continent is way smaller than that of its northern neighbor, both in industry reporting and in criminal investigations. Ecuador, Guatemala, Bolivia, Peru, and Brazil are constantly rated high on the malware infection scale, and the situation is not changing, while Argentina and Colombia remain among the top 10 spammer countries. Brazil has a critical role in point-of-sale malware, ATM malware, and skimming devices. Europe The key aspect making Europe a region with excellent cybercrime potential is the fast, modern, and reliable ICT infrastructure. According to The Internet Organised Crime Threat Assessment (IOCTA) 2015, Cybercriminals abuse Western European countries to host malicious content and launch attacks inside and outside the continent. EU countries host approximately 13 percent of the global malicious URLs, out of which Netherlands is the leading country, while Germany, the U.K., and Portugal come second, third, and fourth respectively. Germany, the U.K., the Netherlands, France, and Russia are important hosts for bot C&C infrastructure and phishing domains, while Italy, Germany, the Netherlands, Russia, and Spain are among the top sources of global spam. Scandinavian countries and Finland are famous for having the lowest malware infection rates. France, Germany, Italy, and to some extent the U.K. have the highest malware infection rates and the highest proportion of bots found within the EU. However, the findings are presumably the result of the high population of the aforementioned EU countries. A half of the EU member states identified criminal infrastructure or suspects in the Netherlands, Germany, Russia, or the United Kingdom. One third of the European law enforcement agencies confirmed connections to Austria, Belgium, Bulgaria, the Czech Republic, France, Hungary, Italy, Latvia, Poland, Romania, Spain, or Ukraine. Asia China is the United States' counterpart in Asia in terms of the top position concerning reported threats to Internet security. Fifty percent of the EU member states' investigations on cybercrime include offenders based in China. Moreover, certain authorities quote China as the source of one third of all global network attacks. In the company of India and South Korea, China is third among the top-10 countries hosting botnet C&C infrastructure, and it has one of the highest global malware infection rates. India, Indonesia, Malaysia, Taiwan, and Japan host serious bot numbers, too. Japan takes on a significant part both as a source country and as a victim of cybercrime. Apart from being an abundant spam source, Japan is included in the top three Asian countries where EU law enforcement agencies have identified cybercriminals. On the other hand, Japan, along with South Korea and the Philippines, is the most popular country in the East and Southeast region of Asia where organized crime groups run sextortion campaigns. Vietnam, India, and China are the top Asian countries featuring spamming sources. Alternatively, China and Hong Kong are the most prominent locations for hosting phishing domains. From another point of view, the country code top-level domains (ccTLDs) for Thailand and Pakistan are commonly used in phishing attacks. In this region, most SEPA members reported losses from the use of skimmed cards. In fact, five (Indonesia, Philippines, South Korea, Vietnam, and Malaysia) out of the top six countries are from this region. Africa Africa remains renowned for combined and sophisticated cybercrime practices. Data from the Europol heat map report indicates that the African region holds a ransomware-as-a-service presence equivalent to the one of the European black market. Cybercriminals from Africa make profits from the same products. Nigeria is on the list of the top 10 countries compiled by the EU law enforcement agents featuring identified cybercrime perpetrators and related infrastructure. In addition, four out of the top five top-level domains used for phishing are of African origin: .cf, .za, .ga, and .ml. Australia and Oceania Australia has two critical cybercrime claims on a global level: First, the country is present in several top-10 charts in the cybersecurity industry, including bot populations, ransomware detection, and network attack originators. Second, the country-code top-level domain for the Palau Islands in Micronesia is massively used by Chinese attackers as the TLD with the second highest proportion of domains used for phishing. Cybercrime in numbers Experts agree that the past couple of years have seen digital extortion flourishing. In 2015 and 2016, cybercrime reached epic proportions. Although there is agreement about the serious rise of the threat, putting each ransomware aspect into numbers is a complex issue. Underreporting is not an issue only in academic research but also in practical case scenarios. The threat to businesses around the world is growing, because businesses keep it quiet. The scope of extortion is obscured because companies avoid reporting and pay the ransom in order to settle the issue in a conducive way. As far as this goes for corporations, it is even more relevant for public enterprises or organizations that provide a public service of any kind. Government bodies, hospitals, transportation companies, and educational institutions are increasingly targeted with digital extortion. Cybercriminals estimate that these targets are likely to pay in order to protect drops in reputation and to enable uninterrupted execution of public services. When CEOs and CIOs keep their mouths shut, relying on reported cybercrime numbers can be a tricky question. The real picture is not only what is visible in the media or via professional networking, but also what remains hidden and is dealt with discreetly by the security experts. In the second quarter of 2015, Intel Security reported an increase in ransomware attacks by 58%. Just in the first 3 months of 2016, cybercriminals amassed $209 million from digital extortion. By making businesses and authorities pay the relatively small average ransom amount of $10,000 per incident, extortionists turn out to make smart business moves. Companies are not shaken to the core by this amount. Furthermore, they choose to pay and get back to business as usual, thus eliminating further financial damages that may arise due to being out of business and losing customers. Extortionists understand the nature of ransom payment and what it means for businesses and institutions. As sound entrepreneurs, they know their market. Instead of setting unreasonable skyrocketing prices that may cause major panic and draw severe law enforcement action, they keep it low profile. In this way, they maintain the dark business in flow, moving from one victim to the next and evading legal measures. A peculiar perspective – Cybercrime in absolute and normalized numbers “To get an accurate picture of the security of cyberspace, cybercrime statistics need to be expressed as a proportion of the growing size of the Internet similar to the routine practice of expressing crime as a proportion of a population, i.e., 15 murders per 1,000 people per year.” This statement by Eric Jardine from the Global Commission on Internet Governance (Jardine, 2015) launched a new perspective of cybercrime statistics, one that accounts for the changing nature and size of cyberspace. The approach assumes that viewing cybercrime findings isolated from the rest of the changes in cyberspace provides a distorted view of reality. The report aimed at normalizing crime statistics and thus avoiding negative, realistic cybercrime scenarios that emerge when drawing conclusions from the limited reliability of absolute numbers. In general, there are three ways in which absolute numbers can be misinterpreted: Absolute numbers can negatively distort the real picture, while normalized numbers show whether the situation is getting better Both numbers can show that things are getting better, but normalized numbers will show that the situation is improving more quickly Both numbers can indicate that things are deteriorating, but normalized numbers will indicate that the situation is deteriorating at a slower rate than absolute numbers Additionally, the GCIG (Global Commission on Internet Governance) report includes some excellent reasoning about the nature of empirical research undertaken in the age of the Internet. While almost everyone and anything is connected to the network and data can be easily collected, most of the information is fragmented across numerous private parties. Normally, this entangles the clarity of the findings of cybercrime presence in the digital world. When data is borrowed from multiple resources and missing slots are modified with hypothetical numbers, the end result can be skewed. Keeping in mind this observation, it is crucial to emphasize that the GCIG report measured the size of cyberspace by accounting for eight key aspects: The number of active mobile broadband subscriptions The number of smartphones sold to end users The number of domains and websites The volume of total data flow The volume of mobile data flow The annual number of Google searches The Internet’s contribution to GDP It has been illustrated several times during this introduction that as cyberspace grows, so does cybercrime. To fight the menace, businesses and individuals enhance security measures and put more money into their security budgets. A recent CIGI-Ipsos (Centre for International Governance Innovation - Ipsos) survey collected data from 23,376 Internet users in 24 countries, including Australia, Brazil, Canada, China, Egypt, France, Germany, Great Britain, Hong Kong, India, Indonesia, Italy, Japan, Kenya, Mexico, Nigeria, Pakistan, Poland, South Africa, South Korea, Sweden, Tunisia, Turkey, and the United States. Survey results showed that 64% of users were more concerned about their online privacy compared to the previous year, whereas 78% were concerned about having their banking credentials hacked. Additionally, 77% of users were worried about cyber criminals stealing private images and messages. These perceptions led to behavioral changes: 43% of users started avoiding certain sites and applications, some 39% regularly updated passwords, while about 10% used the Internet less (CIGI-Ipsos, 2014). GCIC report results are indicative of a heterogeneous cyber security picture. Although many cybersecurity aspects are deteriorating over time, there are some that are staying constant, and a surprising number are actually improving. Jardine compares cyberspace security to trends in crime rates in a specific country operationalizing cyber attacks via 13 measures presented in the following table, as seen in Table 2 of Summary Statistics for the Security of Cyberspace(E. Jardine, GCIC Report, p. 6):    Minimum Maximum Mean Standard Deviation New Vulnerabilities 4,814 6,787 5,749 781.880 Malicious Web Domains 29,927 74,000 53,317 13,769.99 Zero-day Vulnerabilities 8 24 14.85714 6.336 New Browser Vulnerabilities 232 891 513 240.570 Mobile Vulnerabilities 115 416 217.35 120.85 Botnets 1,900,000 9,437,536 4,485,843 2,724,254 Web-based Attacks 23,680,646 1,432,660,467 907,597,833 702,817,362 Average per Capita Cost 188 214 202.5 8.893818078 Organizational Cost 5,403,644 7,240,000 6,233,941 753,057 Detection and Escalation Costs 264,280 455,304 372,272 83,331 Response Costs 1,294,702 1,738,761 1,511,804 152,502.2526 Lost Business Costs 3,010,000 4,592,214 3,827,732 782,084 Victim Notification Costs 497,758 565,020 565,020 30,342   While reading the table results, an essential argument must be kept in mind. Statistics for cybercrime costs are not available worldwide. The author worked with the assumption that data about US costs of cybercrime indicate costs on a global level. For obvious reasons, however, this assumption may not be true, and many countries will have had significantly lower costs than the US. To mitigate the assumption's flaws, the author provides comparative levels of those measures. The organizational cost of data breaches in 2013 in the United States was a little less than six million US dollars, while the average number on the global level, which was drawn from the Ponemon Institute’s annual Cost of Data Breach Study (from 2011, 2013, and 2014 via Jardine, p.7) measured the overall cost of data breaches, including the US ones, as US$2,282,095. The conclusion is that US numbers will distort global cost findings by expanding the real costs and will work against the paper's suggestion, which is that normalized numbers paint a rosier picture than the one provided by absolute numbers. Summary In this article, we have covered the birth and concept of cyber crime and the challenges law enforcement, academia, and security professionals face when combating its threatening behavior. We also explored the impact of cyber crime by numbers on varied geographical regions, industries, and devices. Resources for Article:  Further resources on this subject: Interactive Crime Map Using Flask [article] Web Scraping with Python [article]
Read more
  • 0
  • 0
  • 15520

article-image-ai-distilled-25-openais-gpt-store-and-gpt-4-turbo-xais-grok-stability-ais-3d-model-generator-microsofts-phi-15-gen-ai-powered-vector-search-apps
Merlyn Shelley
10 Nov 2023
12 min read
Save for later

AI_Distilled #25: OpenAI’s GPT Store and GPT-4 Turbo, xAI’s Grok, Stability AI’s 3D Model Generator, Microsoft’s Phi 1.5, Gen AI-Powered Vector Search Apps

Merlyn Shelley
10 Nov 2023
12 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!The AI Product Manager's Handbook ($35.99 Value) FREE for a limited time! Gain expertise as an AI product manager to effectively oversee the design, development and deployment of AI products. Master the skills needed to bring tangible value to your organization through successful AI implementation.Seize this exclusive opportunity and grab your copy now before it slips away on November 16th!  👋 Hello ,Step into another edition of AI_Distilled, brimming with updates in AI/ML, LLMs, NLP, GPT, and Gen AI. Our aim is to help you enhance your AI skills and stay abreast of the ever-evolving trends in this domain. Let’s get started with our news and analysis with an industry expert’s opinion. “Unfortunately, we have biases that live in our data, and if we don’t acknowledge that and if we don’t take specific actions to address it then we’re just going to continue to perpetuate them or even make them worse.” - Kathy Baxter, Responsible AI Architect, Salesforce. Baxter made an important point, for data underlies ML models, and errors will simply result in a domino effect that can have drastic consequences. Equally important is how AI handles data privacy, especially when you consider how apps like ChatGPT have now crossed 100 million weekly users. The Apple CEO recently hinted at major investments in responsible AI, which will likely transform smart handheld devices in 2024 with major AI upgrades. In this issue, we’ll talk about OpenAI unveiling major upgrades and features including GPT-4 Turbo model and DALL-E 3 API, Microsoft’s new breakthrough with smaller AI model, Elon Musk unveiling xAI’s "Grok" competing with GPT, and OpenAI launching the GPT Store for user-created custom AI models. We’ve also got you your fresh dose of AI secret knowledge and tutorials on unlocking zero-shot adaptive prompting for LLMs, creating a Python chat web app with OpenAI's API and Reflex, and 9 open source tools to boost your AI app. 📥 Feedback on the Weekly EditionWhat do you think of this issue and our newsletter?Please consider taking the short survey below to share your thoughts and you will get a free PDF of the “The Applied Artificial Intelligence Workshop” eBook upon completion. Complete the Survey. Get a Packt eBook for Free!Writer’s Credit: Special shout-out to Vidhu Jain for their valuable contribution to this week’s newsletter content!  Cheers,  Merlyn Shelley  Editor-in-Chief, Packt  Ready to level up your coding game?  🚀 Dive into the Software Supply Chain Security Survey and let's talk vulnerabilities, security practices, and all things code!  🤓 Share your insights and stand a chance to snag some epic prizes, including the coveted MX Master 3S, Raspberry Pi 4 Model B 4GB, $5 Udemy gift credits, and more!  🌟 Your code-savvy opinions could be your ticket to tech greatness.  Don't miss out—join the conversation now! 👩‍💻  Interested? Tell us what you think! SignUp | Advertise | Archives⚡ TechWave: AI/GPT News & Analysis🔹 OpenAI Unveils Major Upgrades and Features Including GPT-4 Turbo Model, DALL-E 3 API, Crosses 100 million Weekly Users: At its DevDay event, OpenAI announced significant new capabilities and lower pricing for its AI platform. This includes a more advanced GPT-4 Turbo model with 128K context size and multimodal abilities. OpenAI also released new developer products like the Assistants API and DALL-E 3 integration. Additional updates include upgraded models, customization options, expanded rate limits, and the Copyright Shield protection. Together these represent major progress in features, accessibility and affordability. ChatGPT also achieved 100 million weekly users and over two million developers, marking a significant milestone in its growth.  🔹 Stability AI Launches AI-Powered 3D Model Generator: Stability AI debuts Stable 3D, empowering non-experts to craft 3D models through simple descriptions or image uploads. The tool generates editable .obj files, marking the company's entry into the AI-driven 3D modeling landscape. Questions about training data origin and prior copyright controversies arise, highlighting a strategic move amid financial struggles. 🔹 Apple CEO Hints at Generative AI Plans: Apple CEO Tim Cook hinted at significant investments in generative AI during the recent earnings call. While specifics were not disclosed, Cook emphasized responsible deployment over time. Apple's existing AI in iOS and Apple Watch showcases its commitment, with rumors suggesting major AI updates in 2024, solidifying Apple's leadership in the space. 🔹 Microsoft Unveils Breakthrough with Smaller AI Model: Microsoft researchers revealed a major new capability added to their small AI model Phi 1.5. It can now interpret images, a skill previously limited to much larger models like OpenAI's ChatGPT. Phi 1.5 has only 1.3 billion parameters compared to GPT-4's 1.7 trillion, making it exponentially more efficient. This shows less expensive AI can mimic bigger models. Smaller models need less computing power, saving costs and emissions. Microsoft sees small and large models as complementary, optimizing tasks between them. The breakthrough signals wider access to advanced AI as smaller models spread.🔹 OpenAI Unveils GPT Store for User-Created Custom AI Models: OpenAI introduces GPTs, allowing users to build custom versions of ChatGPT for specific purposes with no coding experience required, opening up the AI marketplace. These GPTs can range from simple tasks like recipe assistance to complex ones such as coding or answering specific questions. The GPT Store will soon host these creations, enabling users to publish and potentially monetize them, mirroring the App Store model's success. OpenAI aims to pay creators based on their GPTs' usage, encouraging innovation. However, this move may create challenges in dealing with industry giants like Apple and Microsoft, who have their app models and platforms. 🔹 Elon Musk Drops xAI's Game-Changer: Meet Grok, the LLM with Real-Time Data, Efficiency, and a Dash of Humor! Named after the slang term for "understanding," Grok is intended to compete with AI models like OpenAI's GPT. It's currently available to a limited number of users in the United States through a waitlist on xAI's website. Grok is designed with impressive efficiency, utilizing half the training resources of comparable models. It brings humor and wit to AI interactions, aligning with Musk's goal of creating a "maximum truth-seeking AI." ***************************************************************************************************************************************************************🔮 Expert Insights from Packt Community Machine Learning with PyTorch and Scikit-Learn - By Sebastian Raschka, Yuxi (Hayden) Liu, Vahid Mirjalili Solving interactive problems with reinforcement learning Another type of machine learning is reinforcement learning. In reinforcement learning, the goal is to develop a system (agent) that improves its performance based on interactions with the environment. Since the information about the current state of the environment typically also includes a so-called reward signal, we can think of reinforcement learning as a field related to supervised learning. However, in reinforcement learning, this feedback is not the correct ground truth label or value, but a measure of how well the action was measured by a reward function. Through its interaction with the environment, an agent can then use reinforcement learning to learn a series of actions that maximizes this reward via an exploratory trial-and-error approach or deliberative planning. Discovering hidden structures with unsupervised learning In supervised learning, we know the right answer (the label or target variable) beforehand when we train a model, and in reinforcement learning, we define a measure of reward for particular actions carried out by the agent. In unsupervised learning, however, we are dealing with unlabeled data or data of an unknown structure. Using unsupervised learning techniques, we are able to explore the structure of our data to extract meaningful information without the guidance of a known outcome variable or reward function. Finding subgroups with clustering Clustering is an exploratory data analysis or pattern discovery technique that allows us to organize a pile of information into meaningful subgroups (clusters) without having any prior knowledge of their group memberships. Each cluster that arises during the analysis defines a group of objects that share a certain degree of similarity but are more dissimilar to objects in other clusters, which is why clustering is also sometimes called unsupervised classification. Clustering is a great technique for structuring information and deriving meaningful relationships from data. For example, it allows marketers to discover customer groups based on their interests, in order to develop distinct marketing programs. Dimensionality reduction for data compression Another subfield of unsupervised learning is dimensionality reduction. Often, we are working with data of high dimensionality—each observation comes with a high number of measurements—that can present a challenge for limited storage space and the computational performance of machine learning algorithms. Unsupervised dimensionality reduction is a commonly used approach in feature preprocessing to remove noise from data, which can degrade the predictive performance of certain algorithms. Dimensionality reduction compresses the data onto a smaller dimensional subspace while retaining most of the relevant information. This content is from the book “Machine Learning with PyTorch and Scikit-Learn” writtern by Sebastian Raschka, Yuxi (Hayden) Liu, Vahid Mirjalili (Feb 2022). Start reading a free chapter or access the entire Packt digital library free for 7 days by signing up now. To learn more, click on the button below. Read through the Chapter 1 unlocked here...  🌟 Secret Knowledge: AI/LLM Resources💡 Enhancing User Experiences with AI-PWAs in Web Development: This article explores the integration of AI and Progressive Web Applications (PWAs) to revolutionize website development. Learn how AI chatbots and generative AI, such as OpenAI's GPT-3, can personalize content and streamline coding. Discover the benefits of combining AI technology with PWAs, including improved user engagement, streamlined content generation, and enhanced scalability.  💡 Boosting Your AI App with 9 Open Source Tools: From LLM queries to chatbots and AI app quality, explore projects like LLMonitor for cost analytics and user tracking, Guidance for complex agent flows, LiteLLM for easy integration of various LLM APIs, Zep for chat history management, LangChain for building powerful AI apps, DeepEval for LLM application testing, pgVector for embedding storage and similarity search, promptfoo for testing prompts and models, and Model Fusion, a TypeScript library for AI applications. These tools can help you optimize and streamline your AI projects, improving user experiences and productivity. 💡 Creating Gen AI-Powered Vector Search Applications with Vertex AI Search: Learn how to harness the power of generative AI and vector embeddings to build user experiences and applications. Vector embeddings are a way to represent various types of data in a semantic space, enabling developers to create applications such as finding relevant information in documents, personalized product recommendations, and more. The article introduces vector search, a service within the Vertex AI Search platform, which helps developers find relevant embeddings quickly. It offers scalability, adaptability to changing data, security features, and easy integration with other AI tools.  🔛 Masterclass: AI/LLM Tutorials🔑 Integrating Amazon MSK with CockroachDB for Real-Time Data Streams: This guide offers a comprehensive step-by-step process for integrating Amazon Managed Streaming for Apache Kafka (Amazon MSK) with CockroachDB, creating a robust and scalable pipeline for real-time data processing. The integration enables various use cases, such as real-time analytics, event-driven microservices, and audit logging, enhancing businesses' ability to provide immediate, personalized experiences for customers. 🔑 Understanding GPU Workload Monitoring on Amazon EKS with AWS Managed Services: As the demand for GPU-accelerated ML workloads grows, this post offers valuable insights into monitoring GPU utilization on Amazon Elastic Kubernetes Service (EKS) using AWS managed open-source services. Amazon EC2 instances with NVIDIA GPUs are crucial for efficient ML training. The article explains how GPU metrics can provide essential information for optimizing resource allocation, identifying anomalies, and enhancing system performance.  🔑 Unlocking Zero-Shot Adaptive Prompting for LLMs: This study explores LLMs, emphasizing their prowess in solving problems in both few-shot and zero-shot scenarios. It introduces "Consistency-Based Self-Adaptive Prompting (COSP)" and "Universal Self-Adaptive Prompting (USP)" to generate robust prompts for diverse tasks in natural language understanding and generation. 🔑 Exploring Interactive AI Applications with OpenAI's GPT Assistants and Streamlit: This post unveils a cutting-edge Streamlit app integrating OpenAI's GPT models for interactive Wardley Mapping instruction. It details development, emphasizing GPT-4-1106-preview, covering setup, session management, UI configuration, user input, and real-time content generation, showcasing Streamlit's synergy with OpenAI for dynamic applications.  🔑 Utilizing GPT-4 Code Interpreter API for CSV Analysis: A Step-by-Step Guide: Learn to analyze CSV files with OpenAI's GPT-4 Code Interpreter API. The guide covers step-by-step processes, from uploading files via Postman to creating an Assistant, forming a thread, and executing a run. Gain insights for efficient CSV analysis, unlocking data-driven insights and automation power. 🔑 Creating a Python Chat Web App with OpenAI's API and Reflex: In this tutorial, you'll learn how to develop a chat web application in pure Python, utilizing OpenAI's API for intelligent responses. The guide explains how to use the Reflex open-source framework to build both the backend and frontend entirely in Python. The tutorial also covers styling and handling user input, making it easy for those without JavaScript experience to create a chat application with an AI-driven chatbot. By the end, you'll have a functional AI chatbot web app built in Python.  🚀 HackHub: Trending AI Tools📐tigerlab-ai/tiger: Build customized AI models and language applications, bridging the gap between general LLMs and domain-specific knowledge. 📐 langchain-ai/langchain/tree/master/templates: Reference architectures for various LLM use cases, enabling developers to quickly build production-ready LLM applications. 📐 ggerganov/whisper.cpp/tree/master/examples/talk-llama: Uses the SDL2 library to capture audio from the microphone and combines Whisper and LLaMA models for real-time interactions. 📐 explosion/thinc: Lightweight deep learning library for model composition, offering type-checked, functional-programming APIs, and support for PyTorch, TensorFlow, and MXNet. 
Read more
  • 0
  • 0
  • 15491

article-image-making-the-best-out-of-hugging-face-hub-using-langchain
Ms. Valentina Alto
17 Jun 2023
6 min read
Save for later

Making the best out of Hugging Face Hub using LangChain

Ms. Valentina Alto
17 Jun 2023
6 min read
Since the launch of ChatGPT in November 2022, everyone is talking about GPT models and OpenAI. There is no doubt that the Generative Pre-trained Transformers (GPT) architecture developed by OpenAI has demonstrated incredible results, also given the investments in training (almost 500 billion tokens) and complexity of the model (175 billion parameters for the GPT-3).Nevertheless, there is an incredible number of open-source Large Language Models (LLMs) that have been widespread in the last months. Below are some examples:Dolly: 12 billion parameters LLM developed by Databricks and trained on their ML platform. Source codeàhttps://github.com/databrickslabs/dollyStableML: This is a series of LLM developed by StabilityAI, the company behind the popular image generation model Stable Diffusion. The series encompasses a variety of LLMs, some of which are fine-tuned on specific use cases. Source codeàhttps://github.com/Stability-AI/StableLMFalcon LLM: A 40 billion parameters LLM developed by the Technology Innovation Institute and trained on a particularly high-quality dataset called RefinedWeb. Plus, as for now (June 2023) ranks 1 globally in the latest Hugging Face independent verification of open-source AI models. Source codeà https://huggingface.co/tiiuaeGPT NeoX and GPT-J: An open-source reproduction of the OpenAI’s GPT series developed by Eulether AI, with respectively 40 and 6 billion parameters. Source codeà https://huggingface.co/EleutherAI/gpt-neox-20b and https://huggingface.co/EleutherAI/gpt-j-6bOpenLLaMa: As for the previous class of models, also this one is an open-source reproduction of Meta AI’s LLaMA and has 3.7 billion parameters. Source codeàhttps://github.com/openlm-research/open_llamaIf you are interested in getting deeper into those models and their performance, you can reference the Hugging Face leaderboard here.Image1: Hugging Face Leaderboard Now, LLMs are great, yet to unlock their real power we need them to be positioned within an applicative logic. In other words, we want our LLMs to infuse intelligence within our applications.For this purpose, we will be using LangChain, a powerful lightweight SDK which makes it easier to integrate and orchestrate LLMs within applications. LangChain is one of the most popular LLMs orchestrators, yet if you want to explore further packages I encourage you to read about Semantic Kernel and Jarvis.One of the nice things about LangChain is its integration with external tools: those might be OpenAI (and other LLMs vendors), data sources, search APIs, and so on. In this article, we are going to explore how LangChain makes it easier to leverage open-source LLMs by leveraging its integration with the Hugging Face Hub.Welcome to the realm of open source LLMsThe Hugging Face Hub serves as a comprehensive platform comprising more than 120k models, 20kdatasets, and 50k demo apps (Spaces), all of which are openly accessible and shared as open-source projects. It provides an online environment where developers can effortlessly collaborate and collectively develop machine learning solutions. Thanks to LangChain, it is way easier to start interacting with open-source LLMs. Plus, you can also surround those models with all the libraries provided by LangChain in terms of prompt design, memory retention, chain management, and so on.Let’s see an implementation with Python. To reproduce the code, make sure to have: Python 3.7.1 or higheràyou can check your Python version running python --version in your terminalLangChain installedàyou can install it via pip install langchainThe huggingface_hub Python package installedà you can install it via pip install huggingface_butHugging Face Hub API keyàto get the API key, you can register into the portal here and then generate your secret key.For this example, I’m going to use the lightest version of Dolly, developed by Databricks and available in three sizes: 3, 7, and 12 billion parameters.from langchain import HuggingFaceHub from getpass import getpass HUGGINGFACEHUB_API_TOKEN = "your-api-key" import os os.environ["HUGGINGFACEHUB_API_TOKEN"] = HUGGINGFACEHUB_API_TOKEN repo_id = " databricks/dolly-v2-3b" llm = HuggingFaceHub(repo_id=repo_id, model_kwargs={"temperature":0, "max_length":64})As you can see from the above code, the only information we need are our Hugging Face Hub API key and the model’s repo ID; then, LangChain will take care of initializing our model thanks to the direct integration with Hugging Face Hub.Now that we have initialized our model, it is time to define the structure of the prompt:from langchain import PromptTemplate, LLMChain template = """Question: {question} Answer: Let's think step by step.""" prompt = PromptTemplate(template=template, input_variables=["question"]) llm_chain = LLMChain(prompt=prompt, llm=llm)Finally, we can feed our model with a first question:question = "In the first movie of Harry Potter, what is the name of the three-headed dog? “ print(llm_chain.run(question)) Output: The name of the three-headed dog in Harry Potter and the Philosopher Stone is Fuffy.Even though I tested the light version of Dolly with “only” 3 billion parameters, it came with pretty accurate results. Of course, for more complex tasks or real-world projects, heavier models might be taken into consideration, like the one emerging as top performers in the Hugging Face leaderboard mentioned at the beginning of this article.ConclusionThe realm of open-source LLM is growing exponentially, and this creates a vibrant environment of experimentation and tuning from which anyone can benefit. Plus, some interesting trends are rising, like the reduction of the number of models’ parameters in favour of an increase in quality of the training dataset. In fact, we saw that the current top performer among the open source model is Falcon LLM, with “only” 40 billion parameters, which gained its strength from the high-quality training dataset. Finally, with the development of orchestration frameworks like LangChain and similar, it’s getting easier and easier to leverage open source LLMs and integrate them into our applications. Referenceshttps://huggingface.co/docs/hub/indexOpen LLM Leaderboard — a Hugging Face Space by HuggingFaceH4Hugging Face Hub — 🦜🔗 LangChain 0.0.189Overview (huggingface.co)stabilityai (Stability AI) (huggingface.co)Stability-AI/StableLM: StableLM: Stability AI Language Models (github.com)Author BioValentina Alto graduated in 2021 in data science. Since 2020, she has been working at Microsoft as an Azure solution specialist, and since 2022, she has been focusing on data and AI workloads within the manufacturing and pharmaceutical industries. She has been working closely with system integrators on customer projects to deploy cloud architecture with a focus on modern data platforms, data mesh frameworks, IoT and real-time analytics, Azure Machine Learning, Azure Cognitive Services (including Azure OpenAI Service), and Power BI for dashboarding. Since commencing her academic journey, she has been writing tech articles on statistics, machine learning, deep learning, and AI in various publications and has authored a book on the fundamentals of machine learning with Python.Author of the book: Modern Generative AI with ChatGPT and OpenAI ModelsLink - Medium  LinkedIn  
Read more
  • 0
  • 0
  • 15487

article-image-jaas-based-security-authentication-jsps
Packt
18 Nov 2013
9 min read
Save for later

JAAS-based security authentication on JSPs

Packt
18 Nov 2013
9 min read
(For more resources related to this topic, see here.) The deployment descriptor is the main configuration file of all the web applications. The container first looks out for the deployment descriptor before starting any application. The deployment descriptor is an XML file, web.xml, inside the WEB-INF folder. If you look at the XSD of the web.xml file, you can see the security-related schema. The schema can be accessed using the following URL: http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd. The following is the schema element available in the XSD: <xsd:element name="security-constraint" type="j2ee:securityconstraintType"/> <xsd:element name="login-config" type="j2ee:login-configType"/> <xsd:element name="security-role "type="j2ee:security-roleType"/> Getting ready You will need the following to demonstrate authentication and authorization: JBoss 7 Eclipse Indigo 3.7 Create a dynamic web project and name it Security Demo Create a package, com.servlets Create an XML file in the WebContent folder, jboss-web.xml Create two JSP pages, login.jsp and logoff.jsp How to do it... Perform the following steps to achieve JAAS-based security for JSPs: Edit the login.jsp file with the input fields j_username, j_password, and submit it to SecurityCheckerServlet: <%@ page contentType="text/html; charset=UTF-8" %> <%@ page language="java" %> <html > <HEAD> <TITLE>PACKT Login Form</TITLE> <SCRIPT> function submitForm() { var frm = document. myform; if( frm.j_username.value == "" ) { alert("please enter your username, its empty"); frm.j_username.focus(); return ; } if( frm.j_password.value == "" ) { alert("please enter the password,its empty"); frm.j_password.focus(); return ; } frm.submit(); } </SCRIPT> </HEAD> <BODY> <FORM name="myform" action="SecurityCheckerServlet" METHOD=get> <TABLE width="100%" border="0" cellspacing="0" cellpadding= "1" bgcolor="white"> <TABLE width="100%" border="0" cellspacing= "0" cellpadding="5"> <TR align="center"> <TD align="right" class="Prompt"></TD> <TD align="left"> <INPUT type="text" name="j_username" maxlength=20> </TD> </TR> <TR align="center"> <TD align="right" class="Prompt"> </TD> <TD align="left"> <INPUT type="password" name="j_password" maxlength=20 > <BR> <TR align="center"> <TD align="right" class="Prompt"> </TD> <TD align="left"> <input type="submit" onclick="javascript:submitForm();" value="Login"> </TD> </TR> </TABLE> </FORM> </BODY> </html> The j_username and j_password are the indicators of using form-based authentication. Let's modify the web.xml file to protect all the files that end with .jsp. If you are trying to access any JSP file, you would be given a login form, which in turn calls a SecurityCheckerServlet file to authenticate the user. You can also see role information is displayed. Update the web.xml file as shown in the following code snippet. We have used 2.5 xsd. The following code needs to be placed in between the webapp tag in the web.xml file: <display-name>jaas-jboss</display-name> <welcome-file-list> <welcome-file>index.html</welcome-file> <welcome-file>index.htm</welcome-file> <welcome-file>index.jsp</welcome-file> <welcome-file>default.html</welcome-file> <welcome-file>default.htm</welcome-file> <welcome-file>default.jsp</welcome-file> </welcome-file-list> <security-constraint> <web-resource-collection> <web-resource-name>something</web-resource-name> <description>Declarative security tests</description> <url-pattern>*.jsp</url-pattern> <http-method>HEAD</http-method> <http-method>GET</http-method> <http-method>POST</http-method> <http-method>PUT</http-method> <http-method>DELETE</http-method> </web-resource-collection> <auth-constraint> <role-name>role1</role-name> </auth-constraint> <user-data-constraint> <description>no description</description> <transport-guarantee>NONE</transport-guarantee> </user-data-constraint> </security-constraint> <login-config> <auth-method>FORM</auth-method> <form-login-config> <form-login-page>/login.jsp</form-login-page> <form-error-page>/logoff.jsp</form-error-page> </form-login-config> </login-config> <security-role> <description>some role</description> <role-name>role1</role-name> </security-role> <security-role> <description>packt managers</description> <role-name>manager</role-name> </security-role> <servlet> <description></description> <display-name>SecurityCheckerServlet</display-name> <servlet-name>SecurityCheckerServlet</servlet-name> <servlet-class>com.servlets.SecurityCheckerServlet </servlet-class> </servlet> <servlet-mapping> <servlet-name>SecurityCheckerServlet</servlet-name> <url-pattern>/SecurityCheckerServlet</url-pattern> </servlet-mapping> JAAS Security Checker and Credential Handler: Servlet is a security checker. Since we are using JAAS, the standard framework for authentication, in order to execute the following program you need to import org.jboss.security.SimplePrincipal and org.jboss.security.auth.callback.SecurityAssociationHandle and add all the necessary imports. In the following SecurityCheckerServlet, we are getting the input from the JSP file and passing it to the CallbackHandler. We are then passing the Handler object to the LoginContext class which has the login() method to do the authentication. On successful authentication, it will create Subject and Principal for the user, with user details. We are using iterator interface to iterate the LoginContext object to get the user details retrieved for authentication. In the SecurityCheckerServlet Class: package com.servlets; public class SecurityCheckerServlet extends HttpServlet { private static final long serialVersionUID = 1L; public SecurityCheckerServlet() { super(); } protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { char[] password = null; PrintWriter out=response.getWriter(); try { SecurityAssociationHandler handler = new SecurityAssociationHandler(); SimplePrincipal user = new SimplePrincipal(request.getParameter ("j_username")); password=request.getParameter("j_password"). toCharArray(); handler.setSecurityInfo(user, password); System.out.println("password"+password); CallbackHandler myHandler = new UserCredentialHandler(request.getParameter ("j_username"),request.getParameter ("j_password")); LoginContext lc = new LoginContext("other", handler); lc.login(); Subject subject = lc.getSubject(); Set principals = subject.getPrincipals(); List l=new ArrayList(); Iterator it = lc.getSubject().getPrincipals(). iterator(); while (it.hasNext()) { System.out.println("Authenticated: " + it.next().toString() + "<br>"); out.println("<b><html><body><font color='green'>Authenticated: " + request.getParameter("j_username")+" <br/>"+it.next().toString() + "<br/></font></b></body></html>"); } it = lc.getSubject().getPublicCredentials (Properties.class).iterator(); while (it.hasNext()) System.out.println(it.next().toString()); lc.logout(); } catch (Exception e) { out.println("<b><font color='red'>failed authenticatation.</font>-</b>"+e); } } protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { } } Create the UserCredentialHandler file: package com.servlets; class UserCredentialHandler implements CallbackHandler { private String user, pass; UserCredentialHandler(String user, String pass) { super(); this.user = user; this.pass = pass; } @Override public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException { for (int i = 0; i < callbacks.length; i++) { if (callbacks[i] instanceof NameCallback) { NameCallback nc = (NameCallback) callbacks[i]; nc.setName(user); } else if (callbacks[i] instanceof PasswordCallback) { PasswordCallback pc = (PasswordCallback) callbacks[i]; pc.setPassword(pass.toCharArray()); } else { throw new UnsupportedCallbackException (callbacks[i], "Unrecognized Callback"); } } } } In the jboss-web.xml file: <?xml version="1.0" encoding="UTF-8"?> <jboss-web> <security-domain>java:/jaas/other</security-domain> </jboss-web> Other is the name of the application policy defined in the login-config.xml file. All these will be packed in as a .war file. Configuring the JBoss Application Server. Go to jboss-5.1.0.GAserver defaultconflogin-config.xml in JBoss. If you look at the file, you can see various configurations for database LDAP and a simple one using the properties file, which I have used in the following code snippet: <application-policy name="other"> <!-- A simple server login module, which can be used when the number of users is relatively small. It uses two properties files: users.properties, which holds users (key) and their password (value). roles.properties, which holds users (key) and a commaseparated list of their roles (value). The unauthenticatedIdentity property defines the name of the principal that will be used when a null username and password are presented as is the case for an unauthenticated web client or MDB. If you want to allow such users to be authenticated add the property, e.g., unauthenticatedIdentity="nobody" --> <authentication> <login-module code="org.jboss.security.auth.spi.UsersRoles LoginModule" flag="required"/> <module-option name="usersProperties"> users.properties</module-option> <module-option name="rolesProperties"> roles.properties</module-option> <module-option name="unauthenticatedIdentity"> nobody</module-option> </authentication> </application-policy> Create the users.properties file in the same folder. The following is the Users. properties file with username mapped with role. User.properties anjana=anjana123 roles.properties anjana=role1 Restart the server. How it works... JAAS consists of a set of interfaces to handle the authentication process. They are: The CallbackHandler and Callback interfaces The LoginModule interface LoginContext The CallbackHandler interface gets the user credentials. It processes the credentials and passes them to LoginModule, which authenticates the user. JAAS is container specific. Each container will have its own implementation, here we are using JBoss application server to demonstrate JAAS. In my previous example, I have explicitly called JASS interfaces. UserCredentialHandler implements the CallbackHandler interfaces. So, CallbackHandlers are storage spaces for the user credentials and the LoginModule authenticates the user. LoginContext bridges the CallbackHandler interface with LoginModule. It passes the user credentials to LoginModule interfaces for authentication: CallbackHandler myHandler = new UserCredentialHandler(request. getParameter("j_username"),request.getParameter("j_password")); LoginContext lc = new LoginContext("other", handler); lc.login() The web.xml file defines the security mechanisms and also points us to the protected resources in our application. The following screenshot shows a failed authentication window: The following screenshot shows a successful authentication window: Summary We introduced the JAAs-based mechanism of applying security to authenticate and authorize the users to the application Resources for Article: Further resources on this subject: Spring Security 3: Tips and Tricks [Article] Spring Security: Configuring Secure Passwords [Article] Migration to Spring Security 3 [Article]
Read more
  • 0
  • 0
  • 15486
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-using-openshift
Packt
21 Oct 2013
5 min read
Save for later

Using OpenShift

Packt
21 Oct 2013
5 min read
(For more resources related to this topic, see here.) Each of these utilize the OpenShift REST API at the backend; therefore, as a user, we could potentially orchestrate OpenShift using the API with such common command-line utilities as curl to write scripts for automation. We could also use the API to write our own custom user interface, if we had the desire. In the following sections, we will explore using each of the currently supported user experiences, all of which can be intermixed as they communicate with the backend in a uniform fashion using the REST API previously mentioned. Getting started using OpenShift As discussed previously, we will be using the OpenShift Online free hosted service for example portions. OpenShift Online has the lowest barrier of entry from a user's perspective because we will not have to deploy our own OpenShift PaaS before being able to utilize it. Since we will be using the OpenShift Online service, the very first step is going to be to visit their website and sign up for a free account via https://openshift.redhat.com/app/account/new. New Account Form Once this step is complete, we will find an e-mail in our inbox that was provided during sign up, with a subject line similar to Confirm your Red Hat OpenShift account; inside that e-mail will be a URL that needs to be followed to complete the setup and verification step. Now that we've successfully completed the sign up phase, let's move on to exploring the different ways in which we can use and interact with OpenShift. Command-line utilities Due to the advancements in modern computing and the advent of mobile devices such as tablets, smart phones, and many other devices, we are often accustomed to Graphical User Interface (GUI) over Command-Line Interface (CLI) for most of our computing needs. This trend is heavier in the realm of web applications because of the rich visual experiences that can be delivered using next generation web technologies. However, those of us who are in the development and system administration circles of the world are no strangers to the CLI, and we know that it is often the most powerful way to accomplish an array of tasks pertaining to development and administration. Much of this is a credit to powerful shell environments that have their roots in traditional UNIX environments; popular examples of these are bash and zsh. Also, in more recent years, PowerShell for the Microsoft Windows platform has aimed to provide some relatively similar CLI power. The shell, as it is referenced here, is that of a UNIX shell, which is a command interpreter that supports such features as variables, functions, pipes, I/O redirection, variable substitution, flow control, conditionals, the ability to be scripted, and more. There is also a POSIX standard for a shell that defines a standard set of features and behaviors that must be complied with, allowing for portability of complex scripts. With this inherent power at the fingertips of the person who wields the command line, the development team of the OpenShift PaaS has written a command-line utility, much in the spirit of offering powerful utilities to its users and developers. Before we get too deep into the details, let's quickly look at what a normal application creation and deployment requires in OpenShift using the following command: $ rhc app create myawesomewebapp ruby-1.9 $ cd myawesomewebapp (Write, create, and implement code changes) $ git commit -a -m "wrote awesome code" $ git push It will be discussed at length shortly, but for a quick rundown, the rhc app create myawesomewebapp ruby-1.9 command creates an application, which runs on OpenShift using ruby-1.9 as the programming platform. Behind the scenes, it's provisioning space, resources, and configuring services for us. It also creates a git repository that is then cloned locally—in our example named myawesomewebapp—and in order to access this, we need to change directories into the git repository. That is precisely what the next command cd myawesomewebapp does. And you're live, running your web application in the cloud. While this is an extremely high-level overview and there are some prerequisites necessary, normal use of OpenShift is that easy. In the following section, we will discuss at length all the steps necessary to launch a live application in OpenShift Online using the rhc command-line utility and git. This command-line utility, rhc, is written in the Ruby programming language and is distributed as a RubyGem (https://rubygems.org/). This is the recommended method of installation for Ruby modules, libraries, and utilities due to the platform-independent nature of Ruby and the ease of distribution of gems. The rhc command-line utility is also available using the native package management for both Fedora and Red Hat Enterprise Linux (via the EPEL repository, available at https://fedoraproject.org/wiki/EPEL) by running the yum install rubygem-rhc command. Another noteworthy proponent of RubyGems is that they can be installed to a user's home directory within their local machine's operating system, allowing them to be utilized even in environments where systems are centrally managed by an IT department. RubyGems are also installed using the gem package manager for users of GNU/Linux package managers, such as yum, apt-get, and pacman or Mac OS X's community homebrew (brew) package manager, which will be familiar with the concept. For those unfamiliar with these concepts, a package manager will track a software named "package" and its dependencies, handle installation, updates, as well as removal. We will take a short moment to tangent into the topic of RubyGems before moving on to the command-line utility for OpenShift to ensure that we don't leave out any background information. Summary Hopefully, we can select our preferred method of deploying on OpenShift, and developers of all backgrounds, preferences, and development platforms will feel at home working with OpenShift as a development and deployment platform. Resources for Article: Further resources on this subject: What is Oracle Public Cloud? [Article] Features of CloudFlare [Article] vCloud Networks [Article]
Read more
  • 0
  • 0
  • 15477

article-image-creating-maze-and-animating-cube
Packt
07 Jul 2014
9 min read
Save for later

Creating the maze and animating the cube

Packt
07 Jul 2014
9 min read
(For more resources related to this topic, see here.) A maze is a rather simple shape that consists of a number of walls and a floor. So, what we need is a way to create these shapes. Three.js, not very surprisingly, doesn't have a standard geometry that will allow you to create a maze, so we need to create this maze by hand. To do this, we need to take two different steps: Find a way to generate the layout of the maze so that not all the mazes look the same. Convert that to a set of cubes (THREE.BoxGeometry) that we can use to render the maze in 3D. There are many different algorithms that we can use to generate a maze, and luckily there are also a number of open source JavaScript libraries that implement such an algorithm. So, we don't have to start from scratch. For the example in this book, I've used the following random-maze-generator project that you can find on GitHub at the following link: https://github.com/felipecsl/random-maze-generator Generating a maze layout Without going into too much detail, this library allows you to generate a maze and render it on an HTML5 canvas. The result of this library looks something like the following screenshot: You can generate this by just using the following JavaScript: var maze = new Maze(document, 'maze'); maze.generate(); maze.draw(); Even though this is a nice looking maze, we can't use this directly to create a 3D maze. What we need to do is change the code the library uses to write on the canvas, and change it to create Three.js objects. This library draws the lines on the canvas in a function called drawLine: drawLine: function(x1, y1, x2, y2) { self.ctx.beginPath(); self.ctx.moveTo(x1, y1); self.ctx.lineTo(x2, y2); self.ctx.stroke(); } If you're familiar with the HTML5 canvas, you can see that this function draws lines based on the input arguments. Now that we've got this maze, we need to convert it to a number of 3D shapes so that we can render them in Three.js. Converting the layout to a 3D set of objects To change this library to create Three.js objects, all we have to do is change the drawLine function to the following code snippet: drawLine: function(x1, y1, x2, y2) { var lengthX = Math.abs(x1 - x2); var lengthY = Math.abs(y1 - y2); // since only 90 degrees angles, so one of these is always 0 // to add a certain thickness to the wall, set to 0.5 if (lengthX === 0) lengthX = 0.5; if (lengthY === 0) lengthY = 0.5; // create a cube to represent the wall segment var wallGeom = new THREE.BoxGeometry(lengthX, 3, lengthY); var wallMaterial = new THREE.MeshPhongMaterial({ color: 0xff0000, opacity: 0.8, transparent: true }); // and create the complete wall segment var wallMesh = new THREE.Mesh(wallGeom, wallMaterial); // finally position it correctly wallMesh.position = new THREE.Vector3( x1 - ((x1 - x2) / 2) - (self.height / 2), wallGeom.height / 2, y1 - ((y1 - y2)) / 2 - (self.width / 2)); self.elements.push(wallMesh); scene.add(wallMesh); } In this new drawLine function, instead of drawing on the canvas, we create a THREE.BoxGeometry object whose length and depth are based on the supplied arguments. Using this geometry, we create a THREE.Mesh object and use the position attribute to position the mesh on a specific points with the x, y, and z coordinates. Before we add the mesh to the scene, we add it to the self.elements array. Now we can just use the following code snippet to create a 3D maze: var maze = new Maze(scene,17, 100, 100); maze.generate(); maze.draw(); As you can see, we've also changed the input arguments. These properties now define the scene to which the maze should be added and the size of the maze. The result from these changes can be seen in the following screenshot: Every time you refresh, you'll see a newly generated random maze. Now that we've got our generated maze, the next step is to add the object that we'll move through the maze. Animating the cube Before we dive into the code, let's first look at the result as shown in the following screenshot: Using the controls at the top-right corner, you can move the cube around. What you'll see is that the cube rotates around its edges, not around its center. In this section, we'll show you how to create that effect. Let's first look at the default rotation, which is along an object's central axis, and the translation behavior of Three.js. The standard Three.js rotation behavior Let's first look at all the properties you can set on THREE.Mesh. They are as follows: Function/property Description position This property refers to the position of an object, which is relative to the position of its parent. In all our examples, so far the parent is THREE.Scene. rotation This property defines the rotation of THREE.Mesh around its own x, y, or z axis. scale With this property, you can scale the object along its own x, y, and z axes. translateX(amount) This property moves the object by a specified amount over the x axis. translateY(amount) This property moves the object by a specified amount over the y axis. translateZ(amount) This property moves the object by a specified amount over the z axis. If we want to rotate a mesh around one of its own axes, we can just call the following line of code: plane.rotation.x = -0.5 * Math.PI; We've used this to rotate the ground area from a horizontal position to a vertical one. It is important to know that this rotation is done around its own internal axis, not the x, y, or z axis of the scene. So, if you first do a number of rotations one after another, you have to keep track at the orientation of your mesh to make sure you get the required effect. Another point to note is that rotation is done around the center of the object—in this case the center of the cube. If we look at the effect we want to accomplish, we run into the following two problems: First, we don't want to rotate around the center of the object; we want to rotate around one of its edges to create a walking-like animation Second, if we use the default rotation behavior, we have to continuously keep track of our orientation since we're rotating around our own internal axis In the next section, we'll explain how you can solve these problems by using matrix-based transformations. Creating an edge rotation using matrix-based transformation If we want to perform edge rotations, we have to take the following few steps: If we want to rotate around the edge, we have to change the center point of the object to the edge we want to rotate around. Since we don't want to keep track of all the rotations we've done, we'll need to make sure that after each rotation, the vertices of the cube represent the correct position. Finally, after we've rotated around the edge, we have to do the inverse of the first step. This is to make sure the center point of the object is back in the center of the cube so that it is ready for the next step. So, the first thing we need to do is change the center point of the cube. The approach we use is to offset the position of all individual vertices and then change the position of the cube in the opposite way. The following example will allow us to make a step to the right-hand side: cubeGeometry.applyMatrix(new THREE.Matrix4().makeTranslation (0, width / 2, width / 2)); cube.position.y += -width / 2; cube.position.z += -width / 2; With the cubeGeometry.applyMatrix function, we can change the position of the individual vertices of our geometry. In this example, we will create a translation (using makeTranslation), which offsets all the y and z coordinates by half the width of the cube. The result is that it will look like the cube moved a bit to the right-hand side and then up, but the actual center of the cube now is positioned at one of its lower edges. Next, we use the cube.position property to position the cube back at the ground plane since the individual vertices were offset by the makeTranslation function. Now that the edge of the object is positioned correctly, we can rotate the object. For rotation, we could use the standard rotation property, but then, we will have to constantly keep track of the orientation of our cube. So, for rotations, we once again use a matrix transformation on the vertices of our cube: cube.geometry.applyMatrix(new THREE.Matrix4().makeRotationX(amount); As you can see, we use the makeRotationX function, which changes the position of our vertices. Now we can easily rotate our cube, without having to worry about its orientation. The final step we need to take is reset the cube to its original position; taking into account that we've moved a step to the right, we can take the next step: cube.position.y += width/2; // is the inverse + width cube.position.z += -width/2; cubeGeometry.applyMatrix(new THREE.Matrix4().makeTranslation(0, - width / 2, width / 2)); As you can see, this is the inverse of the first step; we've added the width of the cube to position.y and subtracted the width from the second argument of the translation to compensate for the step to the right-hand side we've taken. If we use the preceding code snippet, we will only see the result of the step to the right. Summary In this article, we have seen how to create a maze and animate a cube. Resources for Article: Further resources on this subject: Working with the Basic Components That Make Up a Three.js Scene [article] 3D Websites [article] Rich Internet Application (RIA) – Canvas [article]
Read more
  • 0
  • 0
  • 15469

article-image-10-useful-google-cloud-ai-services-for-your-next-machine-learning-project-tutorial
Savia Lobo
18 Sep 2018
9 min read
Save for later

10 useful Google Cloud AI services for your next machine learning project [Tutorial]

Savia Lobo
18 Sep 2018
9 min read
Google Cloud seems to be using artificial intelligence as a strategy to unlock more customers in the ever increasing race to hyper-competitive cloud infrastructure landscape.  Cloud AI provides modern machine learning services, including pre-trained models and a service to create your own tailored models.  It has increased accuracy compared to other deep learning systems. Google Cloud AI is fast, scalable, and easy to use. In this tutorial, we will learn the various Google Cloud AI services. This article is an excerpt from a book written by Arvind Ravulavaru titled Google Cloud AI Services Quick Start Guide. Cloud AutoML Alpha As of April 2018, Cloud AutoML is in alpha and is only available on request, subject to GCP terms and conditions. AutoML helps us develop custom machine learning models with minimal ML knowledge and experience, using the power of Google's transfer learning and Neural Architecture Search technology. Under this service, the first custom service that Google is releasing is named AutoML Vision. This service will help users to train custom vision models for their own use cases. There are other services that will follow. Some of the key AutoML features are the following: Integration with human labeling Powered by Google's Transfer Learning and AutoML Fully integrated with other services of Google Cloud You can read more about AutoML here: https://cloud.google.com/automl/. Cloud TPU Beta As of today, this service is in beta, but we need to explicitly request a TPU quota for our processing needs. Using the Cloud TPUs, one can easily request large computation power to run our own machine learning algorithms. This service helps us with not only the required computing, but by using Google's TensorFlow, we can accelerate the complete setup. This service can be used to perform heavy-duty machine learning, both training and prediction. Some of the key Cloud TPU features are the following: High performance Utilizing the power of GCP Referencing data models Fully Integrated with other services of Google Cloud Connecting Cloud TPUs to custom machine types You can read more about Cloud TPU here: https://cloud.google.com/tpu/. Cloud Machine Learning Engine Cloud Machine Learning Engine helps us easily build machine learning models that work on any type of data, of any size. Cloud Machine Learning Engine can take any TensorFlow model and perform large-scale training on a managed cluster. Additionally, it can also manage the trained models for large-scale online and batch predictions. Cloud Machine Learning Engine can seamlessly transition from training to prediction, using online and batch prediction services. Cloud Machine Learning Engine uses the same scalable and distributed infrastructure with GPU acceleration that powers Google ML products. Some of the key Cloud Machine Learning Engine features are the following: Fully integrated with other Google Cloud services Discover and Share Samples HyperTune your models Managed and Scalable Service Notebook Developer Experience Portable Models You can read more about Cloud Machine Learning Engine here: https://cloud.google.com/ml-engine/. Cloud Job Discovery Private Beta Matching qualified people with the right people doesn't have to be so hard; that is the premise of Cloud Job Discovery. Today's job portals and career sites search people for a job role based on keywords. This approach most of the time results in a mismatch of the candidate to the role. That is where Cloud Job Discovery comes into the picture to bridge the gap between employer and employee. Job Discovery provides plug-and-play access to Google's search and machine learning capabilities, enabling the entire recruiting ecosystem—company career sites, job boards, applicant-tracking systems, and staffing agencies—to improve job site engagement and candidate conversion. Before we continue, you can navigate to https://cloud.google.com/job-discovery/ and try out the Job Discovery Demo. You should see results based on your selection, similar to the following screenshot: The key takeaway from the demo is how Discovery relates a profile to a keyword. This diagram explains how Cloud Job Discovery works: Some of the key differences of Cloud Job Discovery over a standard keyword search are the following: Keyword matching Company jargon recognition Abbreviation recognition Commute search Spelling correction Concept recognition Title detection Real-time query broadening Employer recognition Job enrichment Advanced location mapping Location expansion Seniority alignment Dialogflow Enterprise Edition Beta Dialogflow is a development suite which is used for building interfaces for websites, mobile applications, some of the popular machine learning platforms, and IoT devices. It is powered by machine learning to recognize the intent and context of what a user says, allowing your conversational interface to provide highly efficient and accurate responses. Natural language understanding recognizes a user's intent and extracts prebuilt entities such as time, date, and numbers. You can train your agent to identify custom entity types by providing a small dataset of examples. This service offers cross-platform and multi-language support and can work well with the Google Cloud speech service. You can read more about Dialogflow Enterprise Edition here: https://cloud.google.com/dialogflow-enterprise/. Cloud Natural Language Google's Cloud Natural Language service helps us better understand the structure and meaning of a piece of text by providing powerful machine learning models. These models can be queried by REpresentational State Transfer (REST) API. We can use it to understand sentiment about our product on social media, or parse intent from customer conversations happening in a call center or through a messaging app. Before we continue with Cloud Natural Language, I would recommend heading over to https://cloud.google.com/natural-language/ and trying out the API. Here is a quick glimpse of it: As we can see from the previous screenshot, this service offers various insights regarding a piece of text. Some of the key features are: Syntax analysis Entity recognition Sentiment analysis Content classification Multi-language Integrated REST API You can read more about Cloud Natural Language service here: https://cloud.google.com/natural-language/. Cloud Speech API Cloud Speech API uses powerful neural network models to convert audio to text in real time. This service is exposed as a REST API, as we have seen with the Google Cloud Natural Language API. This API can recognize over 110 languages and users can use this service to convert speech to text in real time, recognize audio uploaded in the request, and integrate with our audio storage on Google Cloud Storage, by using the same technology Google uses to power its own products. Before we continue with Cloud Speech API, I would recommend heading over to https://cloud.google.com/speech/ and trying out the API. Here is a quick glimpse of it: I was actually playing a song in the background and tried the speech-to-text. I was very impressed with the results, except for one part, where I said with a song playing and the API represented it as with the song playing; still, pretty good! I think it is only a matter of time and continued use of these services that will increase their accuracy. Some of the key features of Cloud Speech API are: Automatic Speech Recognition (ASR) Global vocabulary Streaming recognition Word hints Real-time or prerecorded audio support Noise robustness Inappropriate content filtering Integrated API You can read more about Cloud Speech API here: https://cloud.google.com/speech/. Cloud Translation API Using the state-of-the-art Neural Machine Translation, the Cloud Translation service converts texts from one language to another. Translation API is highly responsive, so websites and applications can integrate with Translation API for fast, dynamic translation of source text from the source language to a target language. Before we continue with Cloud Translation API, I would recommend heading over to https://cloud.google.com/translate/ and trying out the API. Here is a quick glimpse of it, as shown in the following screenshot: Some of the key features of Cloud Translation API are as follows: Programmatic access – REST API-driven Text translation Language detection Continuous updates You can read more about Cloud Translate API here: https://cloud.google.com/translate/. Cloud Vision API Fred R. Barnard of Printers' Ink stated "A picture is worth ten thousand words". But no one really knows what those words are. Here comes the Google Cloud Vision API to decipher that for us. Cloud Vision API takes an image as input and spits out the contents of the image as text. It can understand the contents of the image. And this service can be accessed over REST API. Before we continue with Cloud Vision API, I would recommend heading over to https://cloud.google.com/vision/ and trying out the API. Here is a quick glimpse of it as shown in the screenshot: That is a photo of me when I was going through a trying-to-grow-long-hair phase, and after having fun at the beach. What is important is how the vision service was able to look at the image and detect my mood. The same service can perform label detection as well as detect web entities related to this image among others. Some of the key features of this service are: Detecting explicit content Detecting logos, labels, landmarks Landmark detection Optical character recognition Face detection Image attributes Integrated REST API To find out more about Cloud Vision API, check this out: https://cloud.google.com/vision/. Cloud Video Intelligence Cloud Video Intelligence is one of the latest cognitive services released by Google. Cloud Video Intelligence API does almost all the things that the Cloud Vision API can do, but on videos. This service extracts the metadata from a video frame by frame, and we can search any moment of the video file. Before we continue with Cloud Video Intelligence, I would recommend heading over to https://cloud.google.com/video-intelligence/ and trying out the API. Here is a quick glimpse of it, as shown in the screenshot: I have selected the dinosaur and the bicycle video, and you can see the analysis. Some of the key features of Cloud Video Intelligence are: Label detection Shot change detection Explicit content detection Video transcription Alpha This concludes the overview of the various services offered as part of the Cloud AI vertical. In this book, we are going to use a few of these to make a simple web application smart. Summary In this tutorial, we have understood what is truly inside Google Cloud AI.  We have seen the different sercives that it offers along with their key features. In order to Leverage the power of various Google Cloud AI Services by building a smart web application using MEAN Stack,  check out this book  Google Cloud AI Services Quick Start Guide Google’s event-driven serverless platform, Cloud Function, is now generally available What’s new in Google Cloud Functions serverless platform Google Cloud Next: Fei-Fei Li reveals new AI tools for developers
Read more
  • 0
  • 0
  • 15466

article-image-tim-berners-lee-is-on-a-mission-to-save-the-web-he-invented
Bhagyashree R
07 Nov 2018
4 min read
Save for later

Tim Berners-Lee is on a mission to save the web he invented

Bhagyashree R
07 Nov 2018
4 min read
On Monday, at Web Summit 2018 in Lisbon, Tim Berners-Lee outlined his plan to ‘save’ the web he invented. His idea is simple: he wants to create a ‘contract’ for the web. The intervention comes at an important time, in a year of ‘techlash’ and increasing scepticism in the belief in technology’s and the web’s ability to deliver progress for everyone. Tim Berners-Lee on Contract for the Web In his talk, Berners-Lee pointed out that one of the properties the web should preserve is universality. He also argued that The web should be independent, with no restrictions on what it can be used for. It should be available in any region, culture, and religion. He sees these values as things that are currently under attack: “Those of us who are online are seeing our rights and freedoms threatened.” He also listed other challenges that have failed to be addressed, including fake news and privacy. His solution to these huge challenges is something called ‘Contract for the Web,’ which will, he claims, outline “clear and tough responsibilities for those who have the power to make it better.” Although Berners-Lee was relatively light on detail, the full contract is due to be published in May 2019. In theory, it should define people’s online rights and lists the key principles and duties government, companies, and citizens should follow. In Berners-Lee’s mind, it will restore some degree of equilibrium and transparency to the digital realm. The core principles of Contract for the Web Tim Berners-Lee did offer some information on what the Contract for the Web will include. Below are some of the key principles for the key stakeholders in the running of the web - government, businesses, and citizens - people like you. Government Anyone should be able to connect to the web irrespective of who they are and where they live and they should be allowed to actively participate online. Internet should be available all the time so that nobody is denied to the right of full internet access. Privacy, people's fundamental right, should be respected so that they can use the internet freely, safely, and without fear. Companies They should present the user an affordable and accessible internet. Consumers’ privacy and personal data should be respected. Make technologies that support the best in humanity and challenge the worst. Citizens Citizens should involve themselves in creating and collaborating to provide rich and relevant content for everyone. They should build strong communities that respect civil discourse and human dignity. To ensure that the open web remains open, they should come together and fight for it. This contract is already seeing a great support with more than 50 organisations signing the contract including some of big shots like Facebook and Google. This contract is being promoted by the campaign, #ForTheWeb. The contract is part of a broader project that Berners-Lee believes is essential if we are to ‘save’ the web from its current problems First, we need to create an open web for the users who are already connected to the web and give them the power of fixing issues that we have with the existing web. Secondly, we need to bring the other half of the world, which is not yet connected to the web. Tim Berners Lee speaks to CNN about Contract for the Web After his talk, Tim Berners-Lee was interviewed by Laurie Segall from CNN. Here are some highlights from the interview: Internally, companies should have the motto of doing the right thing. To make sure the principles are upheld, some measures will be introduced. But also, this contract is not just about creating a set of rules and enforcing them, rather it is about changing the attitude. Privacy is our fundamental right and we should always fight for it. Fighting for privacy is important, not just because of the data breaches we are seeing but to empower individuals to be able to share anything they want without any fear. As a tech company we should realize the implication of each line of code we write and think how it will affect people's lives. To sum it all up Tim Berners-Lee said: We all are going to step back and put aside all the myths we are currently taking as physics of the way things work. People do not have to motivated only by ad-based fund model or by click bates. Contract for the Web is about going back to the values. It is about people coming together to build the web and taking things in their own hand. You can watch the full Tim Berners-Lee's talk and the interview on YouTube. Web Summit 2018: day 2 highlights Tim Berners-Lee’s Solid – Trick or Treat? Sir Tim Berners-Lee on digital ethics and socio-technical systems at ICDPPC 2018
Read more
  • 0
  • 0
  • 15456
article-image-how-create-breakout-game-godot-engine-part-2
George Marques
23 Nov 2016
8 min read
Save for later

How to create a Breakout game with Godot Engine – Part 2

George Marques
23 Nov 2016
8 min read
In part one of this article you learned how to set up a project and create a basic scene with input and scripting. By now you should grasp the basic concepts of the Godot Engine, such as nodes and scenes. Here we're going to complete the game up to a playable demo. Game scene Let's create new scene to hold the game itself. Click on the menu Scene > New Scene and add a Node2D as its root. You may feel tempted to resize this node to occupy the scene, but you shouldn't. If you resize it you'll be changing the scale and position, which will be reflected on child nodes. We want the position and scale both to be (0, 0). Rename the root node to Game and save the scene as game.tscn. Go to the Project Settings, and in the Application section, set this as the main_scene option. This will make the Game scene run when the game starts. Drag the paddle.tscn file from the FileSystem dock and drop it over the root Game node. This will create a new instance of the paddle scene. It's also possible to click on the chain icon on the Scene dock and select a scene to instance. You can then move the instanced paddle to the bottom of the screen where it should stay in the game (use the guides in the editor as reference). Play the project and you can then move the paddle with your keyboard. If you find the movement too slow or too fast, you can select the Paddle node and adjust the Speed value on the Inspector because it's an exported script variable. This is a great way to tweak the gameplay without touching the code. It also allows you to put multiple paddles in the game, each with its own speed if you wish. To make this better, you can click the Debug Options button (the last one on the top center bar) and activate Sync Scene Changes. This will reflect the changes on the editor in the running game, so you can set the speed without having to stop and play again. The ball Let's create a moving object to interact with. Make a new scene and add a RigidBody2D as the root. Rename it to Ball and save the scene as ball.tscn. The rigid body can be moved by the physics engine and interact with other physical objects, like the static body of the paddle. Add a Sprite as a child node and set the following image as its texture: Ball Now add a CollisionShape2D as a child of the Ball. Set its shape to new CircleShape2D and adjust the radius to cover the image. We need to adjust some of the Ball properties to behave in an appropriate way for this game. Set the Mode property to Character to avoid rotation. Set the Gravity Scale to 0 so it doesn't fall. Set the Friction to 0 and the Damp Override > Linear to 0 in to avoid the loss of momentum. Finally, set the Bounce property to 1, as we want the ball to completely bounce when touching the paddle. With that done, add the following script to the ball, so it starts moving when the scene plays: extends RigidBody2D # Export the ball speed to be changed in the editor export var ball_speed = 150.0 func _ready(): # Apply the initial impulse to the ball so it starts moving # It uses a bit of vector math to make the speed consistent apply_impulse(Vector2(), Vector2(1, 1).normalized() * ball_speed) Walls Going back to the Game scene, instance the ball as a child of the root node. We're going to add the walls so the ball doesn't get lost in the world. Add a Node2D as a child of the root and rename it to Walls. This will be the root for the wall nodes, to keep things organized. As a child of that, add four StaticBody2D nodes, each with its own rectangular collision shape to cover the borders of the screen. You'll end up with something like the following: Walls By now you can play the game a little bit and use the paddle to deflect the ball or leave it to bounce on the bottom wall. Bricks The last part of this puzzle left is the bricks. Create a new scene, add a StaticBody2D as the root and rename it to Brick. Save the scene as brick.tscn. Add a Sprite as its child and set the texture to the following image: Brick Add a CollisionShape2D and set its shape to rectangle, making it cover the whole image. Now add the following script to the root to make a little bit of magic: # Tool keyword makes the script run in editor # In this case you can see the color change in the editor itself tool extends StaticBody2D # Export the color variable and a setter function to pass it to the sprite export (Color) var brick_color = Color(1, 1, 1) setget set_color func _ready(): # Set the color when first entering the tree set_color(brick_color) # This is a setter function and will be called whenever the brick_color variable is set func set_color(color): brick_color = color # We make sure the node is inside the tree otherwise it cannot access its children if is_inside_tree(): # Change the modulate property of the sprite to change its color get_node("Sprite").set_modulate(color) This will allow to set the color of the brick using the Inspector, removing the need to make a scene for each brick color. To make it easier to see, you can click the eye icon besides the CollisionShape2D to hide it. Hide CollisionShape2D The last thing to be done is to make the brick disappear when touched by the ball. Using the Node dock, add the group brick to the root Brick node. Then go back to the Ball scene and, again using the Node dock, but this time in the Signals section, double-click the body_enter signal. Click the Connect button with the default values. This will open the script editor with a new function. Replace it with this: func _on_Ball_body_enter( body ): # If the body just touched in member of the "brick" group if body.is_in_group("brick"): # Mark it for deletion in the next idle frame body.queue_free() Using the Inspector, change the Ball node to enable the Contact Monitor property and increase the Contacts Reported to 1. This will make sure the signal is sent when the ball touches something. Level Make a new scene for the level. Add a Node2D as the root, rename it to Level 1 and save the scene as level1.tscn. Now instance a brick in the scene. Position it anywhere, set a color using the Inspector and then duplicate it. You can repeat this process to make the level look the way you want. Using the Edit menu you can set a grid with snapping to make it easier to position the bricks. Then go back to the Game scene, instance the level there as a child of the root. Play the game and you will finally see the ball bouncing around and destroying the bricks it touches. Breakout Game Going further This is just a basic tutorial showing some of the fundamental aspects of the Godot Engine. The Node and Scene system, physics bodies, scripting, signals, and groups are very useful concepts but not all that Godot has to offer. Once you get acquainted with those, it's easy to learn other functions of the engine. The finished game in this tutorial is just bare bones. There are many things you can do, such as adding a start menu, progressing the levels as they are finished and detecting when the player loses. Thankfully, Godot makes all those things very easy and it should not be much effort to make this a complete game. Author: George Marques is a Brazilian software developer who has been playing with programming in a variety of environments since he was a kid. He works as a freelancer programmer for web technologies based on open source solutions such as WordPress and Open Journal Systems. He's also one of the regular contributors of the Godot Engine, helping solving bugs and adding new features to the software, while also giving solutions to the community for the questions they have.
Read more
  • 0
  • 0
  • 15427

article-image-calendars-jquery-13-php-using-jquery-week-calendar-plugin-part-2
Packt
19 Nov 2009
9 min read
Save for later

Calendars in jQuery 1.3 with PHP using jQuery Week Calendar Plugin: Part 2

Packt
19 Nov 2009
9 min read
Deleting events After creating, moving, and editing events, we might come across a case where the edit form is different from the create form. There is no reason why you would want to delete an event that has not yet been created, so there is no reason to add a delete button to the "Create Event" form. We have a choice—add the delete button to the section at the bottom of the modal dialog, next to Save and Cancel, or add it to the body of the form itself. I always try to add delete buttons and links where I think they cannot be hit by accident. Therefore in this case, I chose not to add it to the row of buttons at the bottom. Instead, I placed it in the form itself where there's little chance it will be clicked by accident while saving or closing the form. And even then, if the link is clicked, there is always a secondary "Are you sure?" confirmation box. Client-side code In the calendar_edit_entry function in calender.js, change the beginning of the $.getJSON call to this: $.getJSON('./calendar.php?action=get_event&id='+calEvent.id, function(eventdata){ var controls='<a href="javascript:calendar_delete_entry' +'('+eventdata.id+');">[delete]</a>'; $('<div id="calendar_edit_entry_form" title="Edit Calendar Entry">' +'<div style="float:right;text-align:right">'+controls+'</div>' +'event name<br />' +'<input id="calendar_edit_entry_form_title" value="'+eventdata.title+'" /><br />' +'body text<br />' +'<textarea style="width:400px;height:200px" id="calendar_edit_entry_form_body">' '+eventdata.body+'</textarea></div>' ).appendTo($('body')); $("#calendar_edit_entry_form").dialog({ The only real change is to add the controls variable, which lets us create more buttons if necessary, and add that variable's contained HTML to the form. The only control there at the moment is a delete link, which calls the calendar_delete_entry function when clicked. Add this function now: function calendar_delete_entry(id){ if(confirm('are you sure you want to delete this entry?')){ $('#calendar_edit_entry_form').remove(); $.getJSON('./calendar.php?action=delete_event&id='+id, function(ret){ $('#calendar_wrapper').weekCalendar('refresh'); } ); } } Server-side code On the server side, we add a case to handle deletes: case 'delete_event': // { $id=(int)$_REQUEST['id']; unset($_SESSION['calendar'][$id]); echo 1; exit; // } All it needs to do is to unset the session variable. With that completed, you now have the finished basics of a calendar, where you can create events, move them around, edit them, and delete them. Walk-through of the calendar so far We've built the basics of a weekly calendar, and before we go on to discuss recurring events, let's take the time to walk through the calendar so far with a simple example. Let's say you have an appointment on Tuesday at 2 pm with a business partner. You add that by clicking on that time, as follows: You think that the meeting will go on for about two hours, so you resize it: Now Bob calls up early on Tuesday to say that he's not going to be able to make it, and suggests moving it to Wednesday at 1 pm. You drag the event over: He also says that he won't be able to make it, but Sally would be there. So, you click on the event and edit the form accordingly: Wednesday comes, and of course, something has come up on your end. You call Sally and explain that you won't be able to make it, and delete the event by clicking on the event, and then clicking on the delete link. Simple and quick. What more would you want? Let's do some recurring events. Recurring events Sometimes you will want to have the same event automatically populated in the calendar on a recurrent basis. For example, you go to lunch every day at 1 pm, or there might be a weekly office meeting every Monday morning. In this case, we need to come up with a way of having events recur. This can be simple or very complex. The simplest method is what we'll demonstrate in this article. The simple method involves entering a frequency (daily, monthly, and so on) and a final date, where the events stop recurring. On the server side, when it is asked to create that recurring event, the server actually iterates over the entire requested period and adds each individual event. This is not extremely efficient, but it's simple to write, and it's not likely that anyone would be placing years-long recurrent events on a very regular basis, so it's justifiable. The more complex method is to only create events that are actually visible in the week you are viewing, and whenever you change the week, you check to see if there are any events that are supposed to recur that week but are not visible. This is arguably even less efficient than the simple method, but it would allow us to be a little more flexible—for example, to leave out the final date so that the events just keep recurring. Anyway, given that there are no major drawbacks to either method, we will choose the simpler method. Client-side code On the client side, recurrences are created at the same time as the recurrence of the first event. So, we edit the "Create Event" form. In calendar.js , adapt the calendar_new_entry function by replacing the form-creation line with this: var recurbox='<select id="calendar_new_entry_form_recurring">' +'<option value="0">non-recurring</option>' +'<option value="1">Daily</option>' +'<option value="7">Weekly</option>' +'</select>';$('<div id="calendar_new_entry_form" title="New Calendar Entry">event name<br />' +'<input value="new event" id="calendar_new_entry_form_title" /><br />' +'body text<br />' +'<textarea style="width:400px;height:200px" id="calendar_new_entry_form_body">' event description</textarea>' +recurbox+'</div>') .appendTo($('body')); $('#calendar_new_entry_form_recurring') .change(calendar_new_get_recurring_end); This adds a select box below the body text area, requesting the user to choose a recurring frequency (defaulting to non-recurring). When the select box is changed, the function calendar_new_get_recurring_end is called. This function is used to request the final recurring date. We could use a plain old text field, but jQuery UI includes a really cool date widget, which allows us to request the date and have it stored in our own choice of format. I've chosen yyyy-mm-dd format, as it is easy to manipulate. Add this to calendar.js: function calendar_new_get_recurring_end(){ if(document.getElementById('calendar_new_recurring_end')) return; var year = new Date().getFullYear(); var month = new Date().getMonth(); var day = new Date().getDate(); $('<span> until <input id="calendar_new_recurring_end" value="'+year+'-'+(month+1)+'-'+day+'" style="font-size:14px" class="date" />' +' </span>' ).insertAfter('#calendar_new_entry_form_recurring'); $('.date').datepicker({ 'dateFormat':'yy-mm-dd', 'yearRange':'-10:+50', 'changeYear':true }); } That creates an input field after the dropdown box, and when it is clicked, a calendar pops up: Whoops! What's happened here is that the date pop up's z-index is lower than the modal dialog. That can be corrected by adding this CSS line to the  < head > section of calendar.html: <style type="text/css"> #ui-datepicker-div{ z-index: 2000; }</style> When reloaded, the calendar now looks correct: Great! Now, we just need to send the data to the server. To do that, change the Save button's $.getJSON parameters in the calendar_new_entry function in calendar.js to these (new parameters are highlighted): 'body':$('#calendar_new_entry_form_body').val(), 'title':$('#calendar_new_entry_form_title').val(), 'recurring':$('#calendar_new_entry_form_recurring').val(), 'recurring_end':$('#calendar_new_recurring_end').val() And we're done on the client side. Server-side code On the server side, the save switch case is going to change considerably, so I'll provide the entire section: case 'save': // { $start_date=(int)$_REQUEST['start']; $data=array( 'title'=>$_REQUEST['title'], 'body' =>$_REQUEST['body'], 'start'=>date('c',$start_date), 'end' =>date('c',(int)$_REQUEST['end']) ); $id=(int)$_REQUEST['id']; if($id && isset($_SESSION['calendar'][$id])){ if(isset($_SESSION['calendar'][$id]['recurring'])) $data['recurring']=$_SESSION['calendar'][$id]['recurring']; $_SESSION['calendar'][$id]=$data; } else{ $id=++$_SESSION['calendar']['ids']; $rec=(int)$_REQUEST['recurring']; if($rec) $data['recurring']=$id; $_SESSION['calendar'][$id]=$data; if($rec && $rec==1 || $rec==7){ list($y,$m,$d)=explode('-',$_REQUEST['recurring_end']); $length=(int)$_REQUEST['end']-$start_date; $end_date=mktime(23,59,59,$m,$d,$y); $step=3600*24*$rec; for($j=1,$i=$start_date+$step;$i<$end_date;$j++,$i+=$step){ $data['start']=date('c',$i); $data['end']=date('c',$i+$length); $nextid=++$_SESSION['calendar']['ids']; $_SESSION['calendar'][$nextid]=$data; } } } echo 1; exit; // } OK. From the data point of view, we've added a single field, recurring, which records the first event in the series. This is needed when deleting recurring events that are not needed anymore. When editing an existing event, all that's changed is that the recurring field (if it exists) is copied from the original before the event is overwritten with fresh data (shown highlighted). The real action happens when creating a new event. If a recurring period is required, then the event is copied and pasted at the requested frequency from the event's first creation until the expiry date. This is figured out by counting the seconds, and incrementing as needed. We can immediately see that recurring events work. Here's an example of a week's lunch hours created from the new recurring method: You can shift individual events around, and even delete them, without affecting the rest.
Read more
  • 0
  • 0
  • 15424

article-image-introduction-ansible
Packt
26 Dec 2016
25 min read
Save for later

Introduction to Ansible

Packt
26 Dec 2016
25 min read
In this article by Walter Bentley, the author of the book OpenStack Administration with Ansible 2 - Second Edition. This article will serve as a high-level overview of Ansible 2.0 and components that make up this open source configuration management tool. We will cover the definition of the Ansible components and their typical use. Also, we will discuss how to define variables for the roles and defining/setting facts about the hosts for the playbooks. Next, we will transition into how to set up your Ansible environment and the ways you can define the host inventory used to run your playbooks against. We will then cover some of the new components introduced in Ansible 2.0 named Blocks and Strategies. It will also review the cloud integrations natively part of the Ansible framework. Finally, the article will finish up with a working example of a playbook that will confirm the required host connectivity needed to use Ansible. The following topics are covered: Ansible 2.0 overview What are playbooks, roles, and modules? Setting up the environment Variables and facts Defining the inventory Blocks and Strategies Cloud integrations (For more resources related to this topic, see here.) Ansible 2.0 overview Ansible in its simplest form has been described as a python-based open source IT automation tool that can be used to configuremanage systems, deploy software (or almost anything), and provide orchestration to a process. These are just a few of the many possible use cases for Ansible. In my previous life as a production support infrastructure engineer, I wish such a tool would have existed. Would have surely had much more sleep and a lot less gray hairs. One thing that always stood out to me in regard to Ansible is that the developer's first and foremost goal was to create a tool that offers simplicity and maximum ease of use. In the world filled with complicated and intricate software, keeping it simple goes a long way for most IT professionals. Staying with the goal of keeping things simple, Ansible handles configuration/management of hosts solely through Secure Shell (SSH). Absolutely no daemon or agent is required. The server or workstation where you run the playbooks from only needs python and a few other packages, most likely already present, installed. Honestly, it does not get simpler than that. The automation code used with Ansible is organized into something named playbooks and roles, of which is written in YAML markup format. Ansible follows the YAML formatting and structure within the playbooks/roles. Being familiar with YAML formatting helps in creating your playbooks/roles. If you are not familiar do not worry, as it is very easy to pick up (it is all about the spaces and dashes). The playbooks and roles are in a noncomplied format making the code very simple to read if familiar with standard UnixLinux commands. There is also a suggested directory structure in order to create playbooks. This also is one of my favorite features of Ansible. Enabling the ability to review and/or use playbooks written by anyone else with little to no direction needed. It is strongly suggested that you review the Ansible playbook best practices before getting started: http://docs.ansible.com/playbooks_best_practices.html. I also find the overall Ansible website very intuitive and filled with great examples at http://docs.ansible.com. My favorite excerpt from the Ansible playbook best practices is under the Content Organization section. Having a clear understanding of how to organize your automation code proved very helpful to me. The suggested directory layout for playbooks is as follows: group_vars/ group1 # here we assign variables to particular groups group2 # "" host_vars/ hostname1 # if systems need specific variables, put them here hostname2 # "" library/ # if any custom modules, put them here (optional) filter_plugins/ # if any custom filter plugins, put them here (optional) site.yml # master playbook webservers.yml # playbook for webserver tier dbservers.yml # playbook for dbserver tier roles/ common/ # this hierarchy represents a "role" tasks/ # main.yml # <-- tasks file can include smaller files if warranted handlers/ # main.yml # <-- handlers file templates/ # <-- files for use with the template resource ntp.conf.j2 # <------- templates end in .j2 files/ # bar.txt # <-- files for use with the copy resource foo.sh # <-- script files for use with the script resource vars/ # main.yml # <-- variables associated with this role defaults/ # main.yml # <-- default lower priority variables for this role meta/ # main.yml # <-- role dependencies It is now time to dig deeper into reviewing what playbooks, roles, and modules consist of. This is where we will break down each of these component's distinct purposes. What are playbooks, roles, and modules? The automation code you will create to be run by Ansible is broken down in hierarchical layers. Envision a pyramid with its multiple levels of elevation. We will start at the top and discuss playbooks first. Playbooks Imagine that a playbook is the very topmost triangle of the pyramid. A playbook takes on the role of executing all of the lower level code contained in a role. It can also be seen as a wrapper to the roles created. We will cover the roles in the next section. Playbooks also contain other high-level runtime parameters, such as the host(s) to run the playbook against, the root user to use, and/or if the playbook needs to be run as a sudo user. These are just a few of the many playbook parameters you can add. Below is an example of what the syntax of a playbook looks like: --- # Sample playbooks structure/syntax. - hosts: dbservers remote_user: root become: true roles: - mysql-install In the preceding example, you will note that the playbook begins with ---. This is required as the heading (line 1) for each playbook and role. Also, please note the spacing structure at the beginning of each line. The easiest way to remember it is each main command starts with a dash (-). Then, every subcommand starts with two spaces and repeats the lower in the code hierarchy you go. As we walk through more examples, it will start to make more sense. Let's step through the preceding example and break down the sections. The first step in the playbook was to define what hosts to run the playbook against; in this case, it was dbservers (which can be a single host or list of hosts). The next area sets the user to run the playbook as locally, remotely, and it enables executing the playbook as sudo. The last section of the syntax lists the roles to be executed. The earlier example is similar to the formatting of the other playbooks. This format incorporates defining roles, which allows for scaling out playbooks and reusability (you will find the most advanced playbooks structured this way). With Ansible's high level of flexibility, you can also create playbooks in a simpler consolidated format. An example of such kind is as follows: --- # Sample simple playbooks structure/syntax - name: Install MySQL Playbook hosts: dbservers remote_user: root become: true tasks: - name: Install MySQL apt: name={{item}} state=present with_items: - libselinux-python - mysql - mysql-server - MySQL-python - name: Copying my.cnf configuration file template: src=cust_my.cnf dest=/etc/my.cnf mode=0755 - name: Prep MySQL db command: chdir=/usr/bin mysql_install_db - name: Enable MySQL to be started at boot service: name=mysqld enabled=yes state=restarted - name: Prep MySQL db command: chdir=/usr/bin mysqladmin -u root password 'passwd' Now that we have reviewed what playbooks are, we will move on to reviewing roles and their benefits. Roles Moving down to the next level of the Ansible pyramid, we will discuss roles. The most effective way to describe roles is the breaking up a playbook into multiple smaller files. So, instead of having one long playbook with multiple tasks defined, all handling separately related steps, you can break the playbook into individual specific roles. This format keeps your playbooks simple and leads to the ability to reuse roles between playbooks. The best advice I personally received concerning creating roles is to keep them simple. Try to create a role to do a specific function, such as just installing a software package. You can then create a second role to just do configurations. In this format, you can reuse the initial installation role over and over without needing to make code changes for the next project. The typical syntax of a role can be found here and would be placed into a file named main.yml within the roles/<name of role>/tasks directory: --- - name: Install MySQL apt: name={{item}} state=present with_items: - libselinux-python - mysql - mysql-server - MySQL-python - name: Copying my.cnf configuration file template: src=cust_my.cnf dest=/etc/my.cnf mode=0755 - name: Prep MySQL db command: chdir=/usr/bin mysql_install_db - name: Enable MySQL to be started at boot service: name=mysqld enabled=yes state=restarted - name: Prep MySQL db command: chdir=/usr/bin mysqladmin -u root password 'passwd' The complete structure of a role is identified in the directory layout found in the Ansible Overview section of this article. We will review additional functions of roles as we step through the working examples. With having covered playbooks and roles, we are prepared to cover the last topic in this session, which are modules. Modules Another key feature of Ansible is that it comes with predefined code that can control system functions, named modules. The modules are executed directly against the remote host(s) or via playbooks. The execution of a module generally requires you to pass a set number of arguments. The Ansible website (http://docs.ansible.com/modules_by_category.html) does a great job of documenting every available module and the possible arguments to pass to that module. The documentation for each module can also be accessed via the command line by executing the command ansible-doc <module name>. The use of modules will always be the recommended approach within Ansible as they are written to avoid making the requested change to the host unless the change needs to be made. This is very useful when re-executing a playbook against a host more than once. The modules are smart enough to know not to re-execute any steps that have already completed successfully, unless some argument or command is changed. Another thing worth noting is with every new release of Ansible additional modules are introduced. Personally, there was an exciting addition to Ansible 2.0, and these are the updated and extended set of modules set to ease the management of your OpenStack cloud. Referring back to the previous role example shared earlier, you will note the use of various modules. The modules used are highlighted here again to provide further clarity: --- - name: Install MySQL apt: name={{item}} state=present with_items: - libselinux-python - mysql - mysql-server - MySQL-python - name: Copying my.cnf configuration file template: src=cust_my.cnf dest=/etc/my.cnf mode=0755 - name: Prep MySQL db command: chdir=/usr/bin mysql_install_db - name: Enable MySQL to be started at boot service: name=mysqld enabled=yes state=restarted ... Another feature worth mentioning is that you are able to not only use the current modules, but you can also write your very own modules. Although the core of Ansible is written in python, your modules can be written in almost any language. Underneath it, all the modules technically return JSON format data, thus allowing for the language flexibility. In this section, we were able to cover the top two sections of our Ansible pyramid, playbooks, and roles. We also reviewed the use of modules, that is, the built-in power behind Ansible. Next, we transition into another key features of Ansible—variable substitution and gathering host facts. Setting up the environment Before you can start experimenting with Ansible, you must install it first. There was no need in duplicating all the great documentation to accomplish this already created on http://docs.ansible.com/ . I would encourage you to go to the following URL and choose an install method of your choice: http://docs.ansible.com/ansible/intro_installation.html. If you are installing Ansible on Mac OS, I found using Homebrew was much simpler and consistent. More details on using Homebrew can be found at http://brew.sh. The command to install Ansible with Homebrew is brew install ansible. Upgrading to Ansible 2.0 It is very important to note that in order to use the new features part of Ansible version 2.0, you must update the version running on your OSA deployment node. The version currently running on the deployment node is either 1.9.4 or 1.9.5. The method that seemed to work well every time is outlined here. This part is a bit experimental, so please make a note of any warnings or errors incurred. From the deployment node, execute the following commands: $ pip uninstall -y ansible $ sed -i 's/^export ANSIBLE_GIT_RELEASE.*/export ANSIBLE_GIT_RELEASE=${ANSIBLE_GIT_RELEASE:-"v2.1.1.0-1"}/' /opt/openstack-ansible/scripts/bootstrap-ansible.sh $ cd /opt/openstack-ansible $ ./scripts/bootstrap-ansible.sh New OpenStack client authenticate Alongside of the introduction of the new python-openstackclient, CLI was the unveiling of the os-client-config library. This library offers an additional way to provide/configure authentication credentials for your cloud. The new OpenStack modules part of Ansible 2.0 leverages this new library through a package named shade. Through the use of os-client-config and shade, you can now manage multiple cloud credentials within a single file named clouds.yml. When deploying OSA, I discovered that shade will search for this file in the $HOME/.config/openstack/ directory wherever the playbook/role and CLI command is executed. A working example of the clouds.yml file is shown as follows: # Ansible managed: /etc/ansible/roles/openstack_openrc/templates/clouds.yaml.j2 modified on 2016-06-16 14:00:03 by root on 082108-allinone02 clouds: default: auth: auth_url: http://172.29.238.2:5000/v3 project_name: admin tenant_name: admin username: admin password: passwd user_domain_name: Default project_domain_name: Default region_name: RegionOne interface: internal identity_api_version: "3" Using this new authentication method drastically simplifies creating automation code to work on an OpenStack environment. Instead of passing a series of authentication parameters in line with the command, you can just pass a single parameter, --os-cloud=default. The Ansible OpenStack modules can also use this new authentication method. More details about os-client-config can be found at: http://docs.openstack.org/developer/os-client-config. Installing shade is required to use the Ansible OpenStack modules in version 2.0. Shade will be required to be installed directly on the deployment node and the Utility container (if you decide to use this option). If you encounter problems installing shade, try the command—pip install shade—isolated. Variables and facts Anyone who has ever attempted to create some sort of automation code, whether be via bash or Perl scripts, knows that being able to define variables is an essential component. Although Ansible does not compare with other programming languages mentioned, it does contain some core programming language features such as variable substitution. Variables To start, let's first define the meaning of variables and use in the event this is a new concept. Variable (computer science), a symbolic name associated with a value and whose associated value may be changed Using variable allows you to set a symbolic placeholder in your automation code that you can substitute values for on each execution. Ansible accommodates defining variables within your playbooks and roles in various ways. When dealing with OpenStack and/or cloud technologies in general being able to adjust your execution parameters on the fly is critical. We will step through a few ways how you can set variable placeholders in your playbooks, how to define variable values, and how you can register the result of a task as a variable. Setting variable placeholders In the event you wanted to set a variable placeholder within your playbooks, you would add the following syntax like this: - name: Copying my.cnf configuration file template: src=cust_my.cnf dest={{ CONFIG_LOC }} mode=0755 In the preceding example, the variable CONFIG_LOC was added in the place of the configuration file location (/etc/my.cnf) designated in the earlier example. When setting the placeholder, the variable name must be encased within {{ }} as shown in the example. Defining variable values Now that you have added the variable to your playbook, you must define the variable value. This can be done easily by passing command-line values as follows: $ ansible-playbook base.yml --extra-vars "CONFIG_LOC=/etc/my.cnf" Or you can define the values directly in your playbook, within each role or include them inside of global playbook variable files. Here are the examples of the three options. Define a variable value directly in your playbook by adding the vars section: --- # Sample simple playbooks structure/syntax - name: Install MySQL Playbook hosts: dbservers ... vars: CONFIG_LOC: /etc/my.cnf ... Define a variable value within each role by creating a variable file named main.yml within the vars/ directory of the role with the following contents: --- CONFIG_LOC: /etc/my.cnf To define the variable value inside of the global playbook, you would first create a host-specific variable file within the group_vars/ directory in the root of the playbook directory with the exact same contents as mentioned earlier. In this case, the variable file must be named to match the host or host group name defined within the hosts file. As in the earlier example, the host group name is dbservers; in turn, a file named dbservers would be created within the group_vars/ directory. Registering variables The situation at times arises when you want to capture the output of a task. Within the process of capturing the result you are in essence registering a dynamic variable. This type of variable is slightly different from the standard variables we have covered so far. Here is an example of registering the result of a task to a variable: - name: Check Keystone process shell: ps -ef | grep keystone register: keystone_check The registered variable value data structure can be stored in a few formats. It will always follow a base JSON format, but the value can be stored under different attributes. Personally, I have found it difficult at times to blindly determine the format, The tip given here will save you hours of troubleshooting. To review and have the data structure of a registered variable returned when running a playbook, you can use the debug module, such as adding this to the previous example: - debug: var=keystone_check. Facts When Ansible runs a playbook, one of the first things it does on your behalf is gather facts about the host before executing tasks or roles. The information gathered about the host will range from the base information such as operating system and IP addresses to the detailed information such as the hardware type/resources. The details capture on then stored into a variable named facts. You can find a complete list of available facts on the Ansible website at: http://docs.ansible.com/playbooks_variables.html#information-discovered-from-systems-facts. You have the option to disable the facts gather process by adding the following to your playbook: gather_facts: false. Facts about a host are captured by default unless the feature is disabled. A quick way of viewing all facts associated with a host, you can manually execute the following via a command line: $ ansible dbservers –m setup There is plenty more you can do with facts, and I would encourage you to take some time reviewing them in the Ansible documentation. Next, we will learn more about the base of our pyramid, the host inventory. Without an inventory of hosts to run the playbooks against, you would be creating the automation code for nothing. So to close out this artticle, we will dig deeper into how Ansible handles host inventory whether it be in a static and/or dynamic format. Defining the inventory The process of defining a collection of hosts to Ansible is named the inventory. A host can be defined using its fully qualified domain name (FQDN), local hostname, and/or its IP address. Since Ansible uses SSH to connect to the hosts, you can provide any alias for the host that the machine where Ansible is installed can understand. Ansible expects the inventory file to be in an INI-like format and named hosts. By default, the inventory file is usually located in the /etc/ansible directory and will look as follows: athena.example.com [ocean] aegaeon.example.com ceto.example.com [air] aeolus.example.com zeus.example.com apollo.example.com Personally I have found the default inventory file to be located in different places depending on the operating system Ansible is installed on. With that point, I prefer to use the –i command-line option when executing a playbook. This allows me to designate the specific hosts file location. A working example would look like this: ansible-playbook -i hosts base.yml. In the preceding example, there is a single host and a group of hosts defined. The hosts are grouped together into a group by defining a group name enclosed in [ ] inside the inventory file. Two groups are defined in the earlier-mentioned example—ocean and air. In the event where you do not have any hosts within your inventory file (such as in the case of running a playbook locally only), you can add the following entry to define localhost like this: [localhost] localhost ansible_connection=local The option exists to define variable for hosts and a group inside of your inventory file. More information on how to do this and additional inventory details can be found on the Ansible website at http://docs.ansible.com/intro_inventory.html. Dynamic inventory It seemed appropriate since we are automating functions on a cloud platform to review yet another great feature of Ansible, which is the ability to dynamically capture an inventory of hosts/instances. One of the primary principles of cloud is to be able to create instances on demand directly via an API, GUI, CLI, and/or through automation code, like Ansible. That basic principle will make relying on a static inventory file pretty much a useless choice. This is why, you will need to rely heavily on dynamic inventory. A dynamic inventory script can be created to pull information from your cloud at runtime and then, in turn, use that information for the playbooks execution. Ansible provides the functionality to detect if an inventory file is set as an executable and if so will execute the script to pull current time inventory data. Since creating an Ansible dynamic inventory script is considered more of an advanced activity, I am going to direct you to the Ansible website, (http://docs.ansible.com/intro_dynamic_inventory.html), as they have a few working examples of dynamic inventory scripts there. Fortunately, in our case, we will be reviewing an OpenStack cloud built using openstack-ansible (OSA) repository. OSA comes with a prebuilt dynamic inventory script that will work for your OpenStack cloud. That script is named dynamic_inventory.py and can be found within the playbooks/inventory directory located in the root OSA deployment folder. First, execute the dynamic inventory script manually to become familiar with the data structure and group names defined (this example assumes that you are in the root OSA deployment directory): $ cd playbooks/inventory $ ./dynamic_inventory.py This will print to the screen an output similar to this: ... }, "compute_all": { "hosts": [ "compute1_rsyslog_container-19482f86", "compute1", "compute2_rsyslog_container-dee00ea5", "compute2" ] }, "utility_container": { "hosts": [ "infra1_utility_container-c5589031" ] }, "nova_spice_console": { "hosts": [ "infra1_nova_spice_console_container-dd12200f" ], "children": [] }, ... Next, with this information, you now know that if you wanted to run a playbook against the utility container, all you would have to do is execute the playbook like this: $ ansible-playbook -i inventory/dynamic_inventory.py playbooks/base.yml –l utility_container Blocks & Strategies In this section, we will cover two new features added to version 2.0 of Ansible. Both features add additional functionality to how tasks are grouped or executed within a playbook. So far, they seem to be really nice features when creating more complex automation code. We will now briefly review each of the two new features. Blocks The Block feature can simply be explained as a way of logically grouping tasks together with the option of also applying customized error handling. It gives the option to group a set of tasks together establishing specific conditions and privileges. An example of applying the block functionality to an earlier example can be found here: --- # Sample simple playbooks structure/syntax - name: Install MySQL Playbook hosts: dbservers tasks: - block: - apt: name={{item}} state=present with_items: - libselinux-python - mysql - mysql-server - MySQL-python - template: src=cust_my.cnf dest=/etc/my.cnf mode=0755 - command: chdir=/usr/bin mysql_install_db - service: name=mysqld enabled=yes state=restarted - command: chdir=/usr/bin mysqladmin -u root password 'passwd' when: ansible_distribution == 'Ubuntu' remote_user: root become: true Additional details on how to implement Blocks and any associated error handling can be found at http://docs.ansible.com/ansible/playbooks_blocks.html. Strategies The strategy feature allows you to add control on how a play is executed by the hosts. Currently, the default behavior is described as being the linear strategy, where all hosts will execute each task before any host moves on to the next task. As of today, the two other strategy types that exist are free and debug. Since Strategies are implemented as a new type of plugin to Ansible more can be easily added by contributing code. Additional details on Strategies can be found at http://docs.ansible.com/ansible/playbooks_strategies.html. A simple example of implementing a strategy within a playbook is as follows: --- # Sample simple playbooks structure/syntax - name: Install MySQL Playbook hosts: dbservers strategy: free tasks: ... The new debug strategy is extremely helpful when you need to step through your playbook/role to find something like a missing variable, determine what variable value to supply or figure out why it may be sporadically failing. These are just a few of the possible use cases. Definitely I encourage you to give this feature a try. Here is the URL to more details on the playbook debugger: http://docs.ansible.com/ansible/playbooks_debugger.html. Cloud integrations Since cloud automation is the main and most important theme of this article, it only makes sense that we highlight the many different cloud integrations Ansible 2.0 offers right out of the box. Again, this was one of the reasons why I immediately fell in love with Ansible. Yes, the other automation tools also have hooks into many of the cloud providers, but I found at times they did not work or were not mature enough to leverage. Ansible has gone above and beyond to not fall into that trap. Not saying Ansible has all the bases covered, it does feel like most are and that is what matters most to me. If you have not checked out the cloud modules available for Ansible, take a moment now and take a look at http://docs.ansible.com/ansible/list_of_cloud_modules.html. From time to time check back as I am confident, you will be surprised to find more have been added. I am very proud of my Ansible family for keeping on top of these and making it much easier to write automation code against our clouds. Specific to OpenStack, a bunch of new modules have been added to the Ansible library as of version 2.0. The extensive list can be found at http://docs.ansible.com/ansible/list_of_cloud_modules.html#openstack. You will note that the biggest changes, from the first version of this book to this one, will be focused on using as many of the new OpenStack modules when possible. Summary Let's pause here on exploring the dynamic inventory script capabilities and continue to build upon it as we dissect the working examples. We will create our very first OpenStack administration playbook together. We will start off with a fairly simple task of creating users and tenants. This will also include reviewing a few automation considerations you will need to keep in mind when creating automation code for OpenStack. Ready? Ok, let's get started! Resources for Article:   Further resources on this subject: AIO setup of OpenStack – preparing the infrastructure code environment [article] RDO Installation [article] Creating Multiple Users/Tenants [article]
Read more
  • 0
  • 0
  • 15420
article-image-ble-and-internet-things
Packt
02 Feb 2017
11 min read
Save for later

BLE and the Internet of Things

Packt
02 Feb 2017
11 min read
In this article by Muhammad Usama bin Aftab, the author of the book Building Bluetooth Low Energy (BLE) Systems, this article is a practical guide to the world of Internet of Things (IoT), where readers will not only learn the theoretical concepts of the Internet of Things but also will get a number of practical examples. The purpose of this article is to bridge the gap between the knowledge base and its interpretation. Much literature is available for the understanding of this domain but it is difficult to find something that follows a hands-on approach to the technology. In this article, the readers will get an introduction of Internet of Things with a special focus on Bluetooth Low Energy (BLE). There is no problem justifying that the most important technology for the Internet of Things is Bluetooth Low Energy as it is widely available throughout the world and almost every cell phone user keeps this technology in his pocket. The article will then go beyond Bluetooth Low Energy and will discuss many other technologies available for the Internet of Things. In this article we'll explore the following topics: Introduction to Internet of Things Current statistics about IoT and how we are living in a world which is going towards Machine to Machine (M2M) communication Technologies in IoT (Bluetooth Low Energy, Bluetooth beacons, Bluetooth mesh and wireless gateways and so on) Typical examples of IoT devices (catering wearables, sports gadgets and autonomous vehicles and so on) (For more resources related to this topic, see here.) Internet of Things The Internet is a system of interconnected devices which uses a full stack of protocols over a number of layers. In early 1960, the first packet-switched network ARPANET was introduced by the United States Department of Defense (DOD) which used a variety of protocols. Later, with the invention of TCP/IP protocols the possibilities were infinite. Many standards were evolved over time to facilitate the communication between devices over a network. Application layer protocols, routing layer protocols, access layer protocols, and physical layer protocols were designed to successfully transfer the Internet packets from the source address to the destination address. Security risks were also taken care of during this process and now we live in the world where the Internet is an essential part of our lives. The world had progressed quite afar from ARPANET and the scientific communities had realized that the need of connecting more and more devices was inevitable. Thus came the need of more Internet addresses. The Internet Protocol version 6 (IPv6) was developed to give support to an almost infinite number of devices. It uses 128 bits' address, allowing 2^128 (3.4 e38) devices to successfully transmit packets over the internet. With this powerful addressing mechanism, it was now possible to think beyond the traditional communication over the Internet. The availability of more addresses opened the way to connect more and more devices. Although, there are other limitations in expanding the number of connected devices, addressing scheme opened up significant ways. Modern Day IoT The idea of modern day Internet of Things is not significantly old. In 2013, the perception of the Internet of Things evolved. The reasons being the merger of wireless technologies, increase the range of wireless communication and significant advancement in embedded technology. It was now possible to connect devices, buildings, light bulbs and theoretically any device which has a power source and can be connected wirelessly. The combination of electronics, software, and network connectivity has already shown enough marvels in the computer industry in the last century and Internet of Things is no different. Internet of Things is a network of connected devices that are aware of their surrounding. Those devices are constantly or eventually transferring data to its neighboring devices in order to fulfil certain responsibility. These devices can be automobiles, sensors, lights, solar panels, refrigerators, heart monitoring implants or any day-to-day device. These things have their dedicated software and electronics to support the wireless connectivity. It also implements the protocol stack and the application level programming to achieve the required functionality: An illustration of connected devices in the Internet of Things Real life examples of the Internet of Things Internet of Things is fascinatingly spread in our surroundings and the best way to check it is to go to a shopping mall and turn on your Bluetooth. The devices you will see are merely a drop in the bucket of the Internet of Things. Cars, watches, printers, jackets, cameras, light bulbs, street lights, and other devices that were too simple before are now connected and continuously transferring data. It is to keep in mind that this progress in the Internet of Things is only 3 years old and it is not improbable to expect that the adoption rate of this technology will be something that we have never seen before. Last decade tells us that the increase in the internet users was exponential where it reached the first billion in 2005, second in 2010 and third in 2014. Currently, there are 3.4 billion internet users present in the world. Although this trend looks unrealistic, the adoption rate of the Internet of Things is even more excessive. The reports say that by 2020, there will be 50 billion connected devices in the world and 90 percent of the vehicles will be connected to the Internet. This expansion will bring $19 trillion in profits by the same year. By the end of this year, wearables will become a $6 billion market with 171 million devices sold. As the article suggests, we will discuss different kinds of IoT devices available in the market today. The article will not cover them all, but to an extent where the reader will get an idea about the possibilities in future. The reader will also be able to define and identify the potential candidates for future IoT devices. Wearables The most important and widely recognized form of Internet of Things is wearables. In the traditional definition, wearables can be any item that can be worn. Wearables technology can range from fashion accessories to smart watches. The Apple Watch is a prime example of wearable technology. It contains fitness tracking and health-oriented sensors/apps which work with iOS and other Apple products. A competitor of Apple Watch is Samsung Gear S2 which provides compatibility with Android devices and fitness sensors. Likewise, there are many other manufacturers who are building smart watches including, Motorola, Pebble, Sony, Huawei, Asus, LG and Tag Heuer. These devices are more than just watches as they form a part of the Internet of Things—they can now transfer data, talk to your phone, read your heart rate and connect directly to Wi-Fi. For example, a watch can now keep track of your steps and transfer this information to the cellphone: Fitbit Blaze and Apple Watch The fitness tracker The fitness tracker is another important example of the Internet of Things where the physical activities of an athlete are monitored and maintained. Fitness wearables are not confined to the bands, there are smart shirts that monitor the fitness goals and progress of the athlete. We will discuss two examples of fitness trackers in this article. Fitbit and Athos smart apparel. The Blaze is a new product from Fitbit which resembles a smart watch. Although it resembles a smart watch, it a fitness-first watch targeted at the fitness market. It provides step tracking, sleep monitoring, and 24/7 heart rate monitoring. Some of Fitbit's competitors like Garmin's vívoactive watch provide a built-in GPS capability as well. Athos apparel is another example of a fitness wearable which provides heart rate and EMG sensors. Unlike watch fitness tracker, their sensors are spread across the apparel. The theoretical definition of wearables may include augmented and virtual reality headsets and Bluetooth earphones/headphones in the list. Smart home devices The evolution of the Internet of Things is transforming the way we live our daily lives as people use wearables and other Internet of Things devices in their daily lives. Another growing technology in the field of the Internet of Things is the smart home. Home automation, sometimes referred to as smart homes, results from extending the home by including automated controls to the things like heating, ventilation, lighting, air-conditioning, and security. This concept is fully supported by the Internet of Things which demands the connection of devices in an environment. Although the concept of smart homes has already existed for several decades 1900s, it remained a niche technology that was either too expensive to deploy or with limited capabilities. In the last decade, many smart home devices have been introduced into the market by major technology companies, lowering costs and opening the doors to mass adoption. Amazon Echo A significant development in the world of home automation was the launch of Amazon Echo in late 2014. The Amazon Echo is a voice enabled device that performs tasks just by recognizing voice commands. The device responds to the name Alexa, a key word that can be used to wake up the device and perform an number of tasks. This keyword can be used followed by a command to perform specific tasks. Some basic commands that can be used to fulfil home automation tasks are: Alexa, play some Adele. Alexa, play playlist XYZ. Alexa, turn the bedroom lights on (Bluetooth enabled lights bulbs (for example Philips Hue) should be present in order to fulfil this command). Alexa, turn the heat up to 80 (A connected thermostat should be present to execute this command). Alexa, what is the weather? Alexa, what is my commute? Alexa, play audiobook a Game of Thrones. Alexa, Wikipedia Packt Publishing. Alexa, How many teaspoons are in one cup? Alexa, set a timer for 10 minutes. With these voice commands, Alexa is fully operable: Amazon Echo, Amazon Tap and Amazon Dot (From left to right) Amazon Echo's main connectivity is through Bluetooth and Wi-Fi. Wi-Fi connectivity enables it to connect to the Internet and to other devices present on the network or worldwide. Bluetooth Low Energy, on the other hand, is used to connect to other devices in the home which are Bluetooth Low Energy capable. For example, Philips Hue and Thermostat are controlled through Bluetooth Low Energy. In Google IO 2016, Google announced a competing smart home device that will use Google as a backbone to perform various tasks, similar to Alexa. Google intends to use this device to further increase their presence in the smart home market, challenging Amazon and Alexa. Amazon also launched Amazon Dot and Amazon Tap. Amazon Dot is a smaller version of Echo which does not have speakers. External speakers can be connected to the Dot in order to get full access to Alexa. Amazon Tap is a more affordable, cheaper and wireless version of Amazon Echo. Wireless bulbs The Philips Hue wireless bulb is another example of a smart home device. It is a Bluetooth Low Energy connected light bulb that's give full control to the user through his smartphone. These colored bulbs can display millions of colors and can be also controlled remotely through the away from home feature. The lights are also smart enough to sync with music: Illustration of controlling Philips Hue bulbs with smartphones Smart refrigerators Our discussion of home automation would not be complete incomplete without discussing kitchen and other house electronics, as several major vendors such as Samsung have begun offering smart appliances for a smarter home. The Family Hub refrigerator is a smart fridge that lets you access the Internet and runs applications. It is also categorized in the Internet of Things devices as it is fully connected to the Internet and provides various controls to the users: Samsung Family Hub refrigerator with touch controls Summary In this article we spoke about the Internet of Things technology and how it is rooting in our real lives. The introduction of the Internet of Things discussed wearable devices, autonomous vehicles, smart light bulbs, and portable media streaming devices. Internet of Things technologies like Wireless Local Area Network (WLAN), Mobile Ad-hoc Networks (MANETs) and Zigbee was discussed in order to have a better understanding of the available choices in the IoT. Resources for Article: Further resources on this subject: Get Connected – Bluetooth Basics [article] IoT and Decision Science [article] Building Voice Technology on IoT Projects [article]
Read more
  • 0
  • 0
  • 15404

Packt
21 Nov 2013
10 min read
Save for later

Our First Machine Learning Method – Linear Classification

Packt
21 Nov 2013
10 min read
(For more resources related to this topic, see here.) To get a grip on the problem of machine learning in scikit-learn, we will start with a very simple machine learning problem: we will try to predict the Iris flower species using only two attributes: sepal width and sepal length. This is an instance of a classification problem, where we want to assign a label (a value taken from a discrete set) to an item according to its features. Let's first build our training dataset—a subset of the original sample, represented by the two attributes we selected and their respective target values. After importing the dataset, we will randomly select about 75 percent of the instances, and reserve the remaining ones (the evaluation dataset) for evaluation purposes (we will see later why we should always do that): >>> from sklearn.cross_validation import train_test_split >>> from sklearn import preprocessing >>> # Get dataset with only the first two attributes >>> X, y = X_iris[:, :2], y_iris >>> # Split the dataset into a training and a testing set >>> # Test set will be the 25% taken randomly >>> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=33) >>> print X_train.shape, y_train.shape (112, 2) (112,) >>> # Standardize the features >>> scaler = preprocessing.StandardScaler().fit(X_train) >>> X_train = scaler.transform(X_train) >>> X_test = scaler.transform(X_test) The train_test_split function automatically builds the training and evaluation datasets, randomly selecting the samples. Why not just select the first 112 examples? This is because it could happen that the instance ordering within the sample could matter and that the first instances could be different to the last ones. In fact, if you look at the Iris datasets, the instances are ordered by their target class, and this implies that the proportion of 0 and 1 classes will be higher in the new training set, compared with that of the original dataset. We always want our training data to be a representative sample of the population they represent. The last three lines of the previous code modify the training set in a process usually called feature scaling. For each feature, calculate the average, subtract the mean value from the feature value, and divide the result by their standard deviation. After scaling, each feature will have a zero average, with a standard deviation of one. This standardization of values (which does not change their distribution, as you could verify by plotting the X values before and after scaling) is a common requirement of machine learning methods, to avoid that features with large values may weight too much on the final results. Now, let's take a look at how our training instances are distributed in the two-dimensional space generated by the learning feature. pyplot, from the matplotlib library, will help us with this: >>> import matplotlib.pyplot as plt >>> colors = ['red', 'greenyellow', 'blue'] >>> for i in xrange(len(colors)): >>> xs = X_train[:, 0][y_train == i] >>> ys = X_train[:, 1][y_train == i] >>> plt.scatter(xs, ys, c=colors[i]) >>> plt.legend(iris.target_names) >>> plt.xlabel('Sepal length') >>> plt.ylabel('Sepal width') The scatter function simply plots the first feature value (sepal width) for each instance versus its second feature value (sepal length) and uses the target class values to assign a different color for each class. This way, we can have a pretty good idea of how these attributes contribute to determine the target class. The following screenshot shows the resulting plot: Looking at the preceding screenshot, we can see that the separation between the red dots (corresponding to the Iris setosa) and green and blue dots (corresponding to the two other Iris species) is quite clear, while separating green from blue dots seems a very difficult task, given the two features available. This is a very common scenario: one of the first questions we want to answer in a machine learning task is if the feature set we are using is actually useful for the task we are solving, or if we need to add new attributes or change our method. Given the available data, let's, for a moment, redefine our learning task: suppose we aim, given an Iris flower instance, to predict if it is a setosa or not. We have converted our problem into a binary classification task (that is, we only have two possible target classes). If we look at the picture, it seems that we could draw a straight line that correctly separates both the sets (perhaps with the exception of one or two dots, which could lie in the incorrect side of the line). This is exactly what our first classification method, linear classification models, tries to do: build a line (or, more generally, a hyperplane in the feature space) that best separates both the target classes, and use it as a decision boundary (that is, the class membership depends on what side of the hyperplane the instance is). To implement linear classification, we will use the SGDClassifier from scikit-learn. SGD stands for Stochastic Gradient Descent, a very popular numerical procedure to find the local minimum of a function (in this case, the loss function, which measures how far every instance is from our boundary). The algorithm will learn the coefficients of the hyperplane by minimizing the loss function. To use any method in scikit-learn, we must first create the corresponding classifier object, initialize its parameters, and train the model that better fits the training data. You will see while you advance that this procedure will be pretty much the same for what initially seemed very different tasks. >>> from sklearn.linear_modelsklearn._model import SGDClassifier >>> clf = SGDClassifier() >>> clf.fit(X_train, y_train)</p></pre> The SGDClassifier initialization function allows several parameters. For the moment, we will use the default values, but keep in mind that these parameters could be very important, especially when you face more real-world tasks, where the number of instances (or even the number of attributes) could be very large. The fit function is probably the most important one in scikit-learn. It receives the training data and the training classes, and builds the classifier. Every supervised learning method in scikit-learn implements this function. What does the classifier look like in our linear model method? As we have already said, every future classification decision depends just on a hyperplane. That hyperplane is, then, our model. The coef_ attribute of the clf object (consider, for the moment, only the first row of the matrices), now has the coefficients of the linear boundary and the intercept_ attribute, the point of intersection of the line with the y axis. Let's print them: >>> print clf.coef_ [[-28.53692691 15.05517618] [ -8.93789454 -8.13185613] [ 14.02830747 -12.80739966]] >>> print clf.intercept_ [-17.62477802 -2.35658325 -9.7570213 ] Indeed in the real plane, with these three values, we can draw a line, represented by the following equation: -17.62477802 - 28.53692691 * x1 + 15.05517618 * x2 = 0 Now, given x1 and x2 (our real-valued features), we just have to compute the value of the left-side of the equation: if its value is greater than zero, then the point is above the decision boundary (the red side), otherwise it will be beneath the line (the green or blue side). Our prediction algorithm will simply check this and predict the corresponding class for any new iris flower. But, why does our coefficient matrix have three rows? Because we did not tell the method that we have changed our problem definition (how could we have done this?), and it is facing a three-class problem, not a binary decision problem. What, in this case, the classifier does is the same we did—it converts the problem into three binary classification problems in a one-versus-all setting (it proposes three lines that separate a class from the rest). The following code draws the three decision boundaries and lets us know if they worked as expected: >>> x_min, x_max = X_train[:, 0].min() - .5, X_train[:, 0].max() + .5 >>> y_min, y_max = X_train[:, 1].min() - .5, X_train[:, 1].max() + .5 >>> xs = np.arange(x_min, x_max, 0.5) >>> fig, axes = plt.subplots(1, 3) >>> fig.set_size_inches(10, 6) >>> for i in [0, 1, 2]: >>> axes[i].set_aspect('equal') >>> axes[i].set_title('Class '+ str(i) + ' versus the rest') >>> axes[i].set_xlabel('Sepal length') >>> axes[i].set_ylabel('Sepal width') >>> axes[i].set_xlim(x_min, x_max) >>> axes[i].set_ylim(y_min, y_max) >>> sca(axes[i]) >>> plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=plt.cm.prism) >>> ys = (-clf.intercept_[i] – Xs * clf.coef_[i, 0]) / clf.coef_[i, 1] >>> plt.plot(xs, ys, hold=True) The first plot shows the model built for our original binary problem. It looks like the line separates quite well the Iris setosa from the rest. For the other two tasks, as we expected, there are several points that lie on the wrong side of the hyperplane. Now, the end of the story: suppose that we have a new flower with a sepal width of 4.7 and a sepal length of 3.1, and we want to predict its class. We just have to apply our brand new classifier to it (after normalizing!). The predict method takes an array of instances (in this case, with just one element) and returns a list of predicted classes: >>>print clf.predict(scaler.transform([[4.7, 3.1]])) [0] If our classifier is right, this Iris flower is a setosa. Probably, you have noticed that we are predicting a class from the possible three classes but that linear models are essentially binary: something is missing. You are right. Our prediction procedure combines the result of the three binary classifiers and selects the class in which it is more confident. In this case, we will select the boundary line whose distance to the instance is longer. We can check that using the classifier decision_function method: >>>print clf.decision_function(scaler.transform([[4.7, 3.1]])) [[ 19.73905808 8.13288449 -28.63499119]] Summary In this article we included a very simple example of classification, trying to show the main steps for learning. Resources for Article: Further resources on this subject: Python Testing: Installing the Robot Framework [Article] Inheritance in Python [Article] Python 3: Object-Oriented Design [Article]
Read more
  • 0
  • 0
  • 15384
Modal Close icon
Modal Close icon