Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7019 Articles
article-image-facebook-research-suggests-chatbots-and-conversational-ai-will-empathize-humans
Fatema Patrawala
06 Aug 2019
6 min read
Save for later

Facebook research suggests chatbots and conversational AI are on the verge of empathizing with humans

Fatema Patrawala
06 Aug 2019
6 min read
Last week, the Facebook AI research team published a progress report on dialogue research that is fundamentally building more engageable and personalized AI systems. According to the team, “Dialogue research is a crucial component of building the next generation of intelligent agents. While there’s been progress with chatbots in single-domain dialogue, agents today are far from capable of carrying an open-domain conversation across a multitude of topics. Agents that can chat with humans in the way that people talk to each other will be easier and more enjoyable to use in our day-to-day lives — going beyond simple tasks like playing a song or booking an appointment.” In their blog post, they have described new open source data sets, algorithms, and models that improve five common weaknesses of open-domain chatbots today. The weaknesses identified are maintaining consistency, specificity, empathy, knowledgeability, and multimodal understanding. Let us look at each one in detail: Dataset called Dialogue NLI introduced for maintaining consistency Inconsistencies are a common issue for chatbots partly because most models lack explicit long-term memory and semantic understanding. Facebook team in collaboration with their colleagues at NYU, developed a new way of framing consistency of dialogue agents as natural language inference (NLI) and created a new NLI data set called Dialogue NLI, used to improve and evaluate the consistency of dialogue models. The team showcased an example in the Dialogue NLI model, where in they considered two utterances in a dialogue as the premise and hypothesis, respectively. Each pair was labeled to indicate whether the premise entails, contradicts, or is neutral with respect to the hypothesis. Training an NLI model on this data set and using it to rerank the model’s responses to entail previous dialogues — or maintain consistency with them — improved the overall consistency of the dialogue agent. Across these tests they say they saw 3x lesser contradictions in the sentences. Several conversational attributes were studied to balance specificity As per the team, generative dialogue models frequently default to generic, safe responses, like “I don’t know” to some query which needs specific responses. Hence, the Facebook team in collaboration with Stanford’s AI researcher Abigail See, studied how to fix this by controlling several conversational attributes, like the level of specificity. In one experiment, they conditioned a bot on character information and asked “What do you do for a living?” A typical chatbot responds with the generic statement “I’m a construction worker.” With control methods, the chatbots proposed more specific and engaging responses, like “I build antique homes and refurbish houses." In addition to specificity, the team mentioned, “that balancing question-asking and answering and controlling how repetitive our models are make significant differences. The better the overall conversation flow, the more engaging and personable the chatbots and dialogue agents of the future will be.” Chatbot’s ability to display empathy while responding was measured The team worked with researchers from the University of Washington to introduce the first benchmark task of human-written empathetic dialogues centered on specific emotional labels to measure a chatbot’s ability to display empathy. In addition to improving on automatic metrics, the team showed that using this data for both fine-tuning and as retrieval candidates leads to responses that are evaluated by humans as more empathetic, with an average improvement of 0.95 points (on a 1-to-5 scale) across three different retrieval and generative models. The next challenge for the team is that empathy-focused models should perform well in complex dialogue situations, where agents may require balancing empathy with staying on topic or providing information. Wikipedia dataset used to make dialogue models more knowledgeable The research team has improved dialogue models’ capability of demonstrating knowledge by collecting a data set with conversations from Wikipedia, and creating new model architectures that retrieve knowledge, read it, and condition responses on it. This generative model has yielded the most pronounced improvement and it is rated by humans as 26% more engaging than their knowledgeless counterparts. To engage with images, personality based captions were used To engage with humans, agents should not only comprehend dialogue but also understand images. In this research, the team focused on image captioning that is engaging for humans by incorporating personality. They collected a data set of human comments grounded in images, and trained models capable of discussing images with given personalities, which makes the system interesting for humans to talk to. 64% humans preferred these personality-based captions over traditional captions. To build strong models, the team considered both retrieval and generative variants, and leveraged modules from both the vision and language domains. They defined a powerful retrieval architecture, named TransResNet, that works by projecting the image, personality, and caption in the same space using image, personality, and text encoders. The team showed that their system was able to produce captions that are close to matching human performance in terms of engagement and relevance. And annotators preferred their retrieval model’s captions over captions written by people 49.5% of the time. Apart from this, Facebook team has released a new data collection and model evaluation tool, a Messenger-based Chatbot game called Beat the Bot, that allows people to interact directly with bots and other humans in real time, creating rich examples to help train models. To conclude, the Facebook AI team mentions, “Our research has shown that it is possible to train models to improve on some of the most common weaknesses of chatbots today. Over time, we’ll work toward bringing these subtasks together into one unified intelligent agent by narrowing and eventually closing the gap with human performance. In the future, intelligent chatbots will be capable of open-domain dialogue in a way that’s personable, consistent, empathetic, and engaging.” On Hacker News, this research has gained positive and negative reviews. Some of them discuss that if AI will converse like humans, it will do a lot of bad. While other users say that this is an impressive improvement in the field of conversational AI. A user comment reads, “I gotta say, when AI is able to converse like humans, a lot of bad stuff will happen. People are so used to the other conversation partner having self-interest, empathy, being reasonable. When enough bots all have a “swarm” program to move conversations in a particular direction, they will overwhelm any public conversation. Moreover, in individual conversations, you won’t be able to trust anything anyone says or negotiates. Just like playing chess or poker online now. And with deepfakes, you won’t be able to trust audio or video either. The ultimate shock will come when software can render deepfakes in realtime to carry on a conversation, as your friend but not. As a politician who “said crazy stuff” but really didn’t, but it’s in the realm of believability. I would give it about 20 years until it all goes to shit. If you thought fake news was bad, realtime deepfakes and AI conversations with “friends” will be worse.  Scroll Snapping and other cool CSS features come to Firefox 68 Google Chrome to simplify URLs by hiding special-case subdomains Lyft releases an autonomous driving dataset “Level 5” and sponsors research competition
Read more
  • 0
  • 0
  • 18468

article-image-get-started-with-fabric-create-your-workspace-reports
Arshad Ali, Bradley Schacht
14 Mar 2024
9 min read
Save for later

Get Started with Fabric: Create Your Workspace & Reports

Arshad Ali, Bradley Schacht
14 Mar 2024
9 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!This article is an excerpt from the book, Learn Microsoft Fabric, by Arshad Ali, Bradley Schacht. Harness the power of Microsoft Fabric to develop data analytics solutions for various use cases guided by step-by-step instructionsIntroductionEmbark on a journey to harness the full potential of Microsoft Fabric within Power BI. This article serves as your comprehensive guide, walking you through the essential steps to create your first Fabric workspace seamlessly. From understanding the fundamentals to practical implementation, we'll equip you with the knowledge and tools needed to optimize your data management and reporting processes. Get ready to elevate your Power BI experience and unlock new possibilities with Fabric-enabled workspaces.Creating your first Fabric-enabled workspaceOnce you have confirmed that Fabric is enabled in your tenant and you have access to it, the next step is to create your Fabric workspace. You can think of a Fabric workspace as a logical container that will contain items such as lakehouses, warehouses, notebooks, and pipelines. Follow these steps to create your first Fabric workspace:1. Sign into Power BI (https://app.powerbi.com/).2. Select Workspaces | + New workspace:Figure 2.5 – Creating a new workspace3. Fill out the Create a workspace form, as follows:Name: Enter Learn Microsoft Fabric and some characters for uniqueness.Description: Optionally, enter a description for the workspace:Figure 2.6 – Create a workspace – detailsAdvanced: Select Fabric capacity under License mode and then choose a capacity you have access to. If not, you can start a trial license, as described earlier, and use it here.4. Select Apply. Th e workspace will be created and opened.5. You can click on Workspaces again and then search for your workspace by typing its name in the search box. You can also pin the selected workspace so that it always appears at the top:Figure 2.7 – Searching for a workspace6. Clicking on the name of the workspace will open that workspace. A link to it will become available in the left-hand side navigation bar, allowing you to switch from one item to another quickly. Since we haven’t created anything yet, there is nothing here. You can click on +New to start creating Fabric items:Figure 2.8 – Switching to a workspaceWith a Microsoft  Fabric workspace set up, let’s review the different workloads that are available.Copilot in Power BIPower BI has several key components, including data transformation and data modeling, culminating in a visual report that end users will consume. The Copilot experience is centered around the visual storytelling and reporting aspects of Power BI. This materializes in three ways: report page creation, narrative generation, and improving Q&A.Let’s look at each of these Copilot capabilities.Creating reports with the Power BI CopilotThe most common use for Copilot with Power BI is likely to be for creating reports. There are two features that come together to build reports. The first analyzes the dataset to suggest content for your report by using table relationships and column names, while the second one helps you create intuitive reports quickly. Figure 11.30 shows an example where Copilot has suggested several report pages, each with a short description of what would be displayed:Figure 11.30 – The Power BI Copilot page suggestionsIf you like the page suggestions, simply click on the Create button and the report page will appear.While a suggested set of report content is a good starting point, analysts often have a specific need to meet. You can have Copilot create a report from the criteria you provide using prompts as well. These can be as simple as “create a page that shows customer analysis” or more specific, such as “create a page to show the impact of each sales territory on profit and quantity sold.”Figure 11.31 – Sales impact report created by CopilotOnce the report page is generated, Copilot cannot update the report, but you can interact with and modify the report as necessary. This is a great way to reduce the time to get started building reports.A couple of other important things to note are that in addition to not being able to modify reports, Copilot will not allow you to specify specific visual types, apply filters, or change the report layout. All of these can be changed manually after the initial report generation. It is worth noting that users should not expect Copilot to filter results to a specific time period based on their prompt as an example.Next, let’s look at the smart narrative.Creating a narrative using CopilotVisuals are a wonderful way to tell a story and give users the ability to explore data on their own. However, sometimes a narrative that summarizes what is being displayed in a report can be useful. It can not only tell a story but also provide some additional context and information for users.To get started, open a report and add a narrative visualization to the report as shown in Figure 11.32. You will see two options; click on Copilot. Choose the type of summary you wish to produce and optionally select specific pages or visuals to include in the summary. Then click on Create.Figure 11.32 – The report narrative generated by CopilotAfter the narrative is generated, remember to always review the narrative for accuracy and adjust the prompt, if necessary, to produce more accurate results. In addition to summaries, you can ask it to highlight key information, customize the order in which the data is described to help convey importance, specify specific data points to include in the summary, and even generate impact analysis showing how different factors affect metrics on the report.Report, page, and visual narratives are a great way to guide users through a report, especially if there isn’t a subject matter expert there to explain all the data.Finally, let’s look at using Copilot to improve the Q&A visual.Generating synonyms with CopilotThe Q&A visual has been dazzling users for years at this point. It is impressive to build a model, walk into the room, and tell users that they can use natural language to query their data without needing to build any visuals. This may not be as impressive as the Copilot functionality that we have today, but it is still a very useful tool in your Power BI visualization toolbelt.One piece of important information for the success of Q&A is something called a synonym. These are end-user-specific ways to reference data. For example, a table in the data model may be called Dim Person, but you know that some report consumers always refer to these as “users.” Therefore, you would create a synonym that tells Q&A that when someone asks about users, they are really talking about persons. This can also be done on a column level. A synonym for “postal code” could be “zip code,” while a synonym for an “item” could be “product” or “finished good.”Q&A itself may not use Copilot, but Power BI Desktop can leverage Copilot to generate synonyms. This can be done when creating a new Q&A visual by clicking on Add synonyms from the ribbon with the label Improve Q&A with synonyms from Copilot. They can also be generated from the Q&A settings menu by adding Copilot as a source from the Suggestion settings list.The more synonyms that can be used to describe your data, the more likely you are to produce quality Q&A results. It is important to double-check the synonyms generated by Copilot to ensure they line up with your specific business terminology.With these Copilot experiences for Power BI, you will be able to generate report ideas, report pages and visuals, summaries, and narratives, and improve Q&A.ConclusionIn conclusion, by mastering the creation of Fabric workspaces in Power BI, you've laid a solid foundation for efficient data management and reporting. With Fabric's capabilities at your fingertips, you're equipped to streamline workflows, generate insightful reports, and enhance collaboration within your organization. Keep exploring the diverse functionalities of Fabric to continuously refine your Power BI experience and stay ahead in the realm of data analytics.Author bioArshad Ali is a principal product manager at Microsoft, working on the Microsoft Fabric product team in Redmond, WA. He focuses on Spark Runtime, which empowers both data engineering and data science experiences. In his previous role, he helped strategic customers and partners adopt Azure Synapse and Microsoft Fabric.Arshad has more than 20 years of industry experience and has been with Microsoft for over 16 years. He is the co-author of the book Big Data Analytics with Azure HDInsight and the author of over 200 technical articles and blogs on data and analytics. Arshad holds an MBA from the Foster School of Business at the University of Washington and an MCA from India.Bradley Schacht is a principal program manager on the Microsoft Fabric product team based in Saint Augustine, Florida. Bradley is a former consultant and trainer and has co-authored five books on SQL Server and Power BI. As a member of the Microsoft Fabric product team, Bradley works directly with customers to solve some of their most complex data problems and helps shape the future of Microsoft Fabric. Bradley gives back to the community by speaking at events, such as the PASS Summit, SQL Saturday, Code Camp, and user groups across the country, including locally at the Jacksonville SQL Server User Group (JSSUG). He is a contributor on SQLServerCentral and blogs on his personal site, BradleySchacht.
Read more
  • 0
  • 0
  • 18462

article-image-achieving-high-availability-aws-cloud
Packt
11 Aug 2015
18 min read
Save for later

Achieving High-Availability on AWS Cloud

Packt
11 Aug 2015
18 min read
In this article, by Aurobindo Sarkar and Amit Shah, author of the book Learning AWS, we will introduce some key design principles and approaches to achieving high availability in your applications deployed on the AWS cloud. As a good practice, you want to ensure that your mission-critical applications are always available to serve your customers. The approaches in this article will address availability across the layers of your application architecture including availability aspects of key infrastructural components, ensuring there are no single points of failure. In order to address availability requirements, we will use the AWS infrastructure (Availability Zones and Regions), AWS Foundation Services (EC2 instances, Storage, Security and Access Control, Networking), and the AWS PaaS services (DynamoDB, RDS, CloudFormation, and so on). (For more resources related to this topic, see here.) Defining availability objectives Achieving high availability can be costly. Therefore, it is important to ensure that you align your application availability requirements with your business objectives. There are several options to achieve the level of availability that is right for your application. Hence, it is essential to start with a clearly defined set of availability objectives and then make the most prudent design choices to achieve those objectives at a reasonable cost. Typically, all systems and services do not need to achieve the highest levels of availability possible; at the same time ensure you do not introduce a single point of failure in your architecture through dependencies between your components. For example, a mobile taxi ordering service needs its ordering-related service to be highly available; however, a specific customer's travel history need not be addressed at the same level of availability. The best way to approach high availability design is to assume that anything can fail, at any time, and then consciously design against it. "Everything fails, all the time." - Werner Vogels, CTO, Amazon.com In other words, think in terms of availability for each and every component in your application and its environment because any given component can turn into a single point of failure for your entire application. Availability is something you should consider early on in your application design process, as it can be hard to retrofit it later. Key among these would be your database and application architecture (for example, RESTful architecture). In addition, it is important to understand that availability objectives can influence and/or impact your design, development, test, and running your system on the cloud. Finally, ensure you proactively test all your design assumptions and reduce uncertainty by injecting or forcing failures instead of waiting for random failures to occur. The nature of failures There are many types of failures that can happen at any time. These could be a result of disk failures, power outages, natural disasters, software errors, and human errors. In addition, there are several points of failure in any given cloud application. These would include DNS or domain services, load balancers, web and application servers, database servers, application services-related failures, and data center-related failures. You will need to ensure you have a mitigation strategy for each of these types and points of failure. It is highly recommended that you automate and implement detailed audit trails for your recovery strategy, and thoroughly test as many of these processes as possible. In the next few sections, we will discuss various strategies to achieve high availability for your application. Specifically, we will discuss the use of AWS features and services such as: VPC Amazon Route 53 Elastic Load Balancing, auto-scaling Redundancy Multi-AZ and multi-region deployments Setting up VPC for high availability Before setting up your VPC, you will need to carefully select your primary site and a disaster recovery (DR) site. Leverage AWS's global presence to select the best regions and availability zones to match your business objectives. The choice of a primary site is usually the closest region to the location of a majority of your customers and the DR site could be in the next closest region or in a different country depending on your specific requirements. Next, we need to set up the network topology, which essentially includes setting up the VPC and the appropriate subnets. The public facing servers are configured in a public subnet; whereas the database servers and other application servers hosting services such as the directory services will usually reside in the private subnets. Ensure you chose different sets of IP addresses across the different regions for the multi-region deployment, for example 10.0.0.0/16 for the primary region and 192.168.0.0/16 for the secondary region to avoid any IP addressing conflicts when these regions are connected via a VPN tunnel. Appropriate routing tables and ACLs will also need to be defined to ensure traffic can traverse between them. Cross-VPC connectivity is required so that data transfer can happen between the VPCs (say, from the private subnets in one region over to the other region). The secure VPN tunnels are basically IPSec tunnels powered by VPN appliances—a primary and a secondary tunnel should be defined (in case the primary IPSec tunnel fails). It is imperative you consult with your network specialists through all of these tasks. An ELB is configured in the primary region to route traffic across multiple availability zones; however, you need not necessarily commission the ELB for your secondary site at this time. This will help you avoid costs for the ELB in your DR or secondary site. However, always weigh these costs against the total cost/time required for recovery. It might be worthwhile to just commission the extra ELB and keep it running. Gateway servers and NAT will need to be configured as they act as gatekeepers for all inbound and outbound Internet access. Gateway servers are defined in the public subnet with appropriate licenses and keys to access your servers in the private subnet for server administration purposes. NAT is required for servers located in the private subnet to access the Internet and is typically used for automatic patch updates. Again, consult your network specialists for these tasks. Elastic load balancing and Amazon Route 53 are critical infrastructure components for scalable and highly available applications; we discuss these services in the next section. Using ELB and Route 53 for high availability In this section, we describe different levels of availability and the role ELBs and Route 53 play from an availability perspective. Instance availability The simplest guideline here is to never run a single instance in a production environment. The simplest approach to improving greatly from a single server scenario is to spin up multiple EC2 instances and stick an ELB in front of them. The incoming request load is shared by all the instances behind the load balancer. ELB uses the least outstanding requests routing algorithm to spread HTTP/HTTPS requests across healthy instances. This algorithm favors instances with the fewest outstanding requests. Even though it is not recommended to have different instance sizes between or within the AZs, the ELB will adjust for the number of requests it sends to smaller or larger instances based on response times. In addition, ELBs use cross-zone load balancing to distribute traffic across all healthy instances regardless of AZs. Hence, ELBs help balance the request load even if there are unequal number of instances in different AZs at any given time (perhaps due to a failed instance in one of the AZs). There is no bandwidth charge for cross-zone traffic (if you are using an ELB). Instances that fail can be seamlessly replaced using auto scaling while other instances continue to operate. Though auto-replacement of instances works really well, storing application state or caching locally on your instances can be hard to detect problems. Instance failure is detected and the traffic is shifted to healthy instances, which then carries the additional load. Health checks are used to determine the health of the instances and the application. TCP and/or HTTP-based heartbeats can be created for this purpose. It is worthwhile implementing health checks iteratively to arrive at the right set that meets your goals. In addition, you can customize the frequency and the failure thresholds as well. Finally, if all your instances are down, then AWS will return a 503. Zonal availability or availability zone redundancy Availability zones are distinct geographical locations engineered to be insulated from failures in other zones. It is critically important to run your application stack in more than one zone to achieve high availability. However, be mindful of component level dependencies across zones and cross-zone service calls leading to substantial latencies in your application or application failures during availability zone failures. For sites with very high request loads, a 3-zone configuration might be the preferred configuration to handle zone-level failures. In this situation, if one zone goes down, then other two AZs can ensure continuing high availability and better customer experience. In the event of a zone failure, there are several challenges in a Multi-AZ configuration, resulting from the rapidly shifting traffic to the other AZs. In such situations, the load balancers need to expire connections quickly and lingering connections to caches must be addressed. In addition, careful configuration is required for smooth failover by ensuring all clusters are appropriately auto scaled, avoiding cross-zone calls in your services, and avoiding mismatched timeouts across your architecture. ELBs can be used to balance across multiple availability zones. Each load balancer will contain one or more DNS records. The DNS record will contain multiple IP addresses and DNS round-robin can be used to balance traffic between the availability zones. You can expect the DNS records to change over time. Using multiple AZs can result in traffic imbalances between AZs due to clients caching DNS records. However, ELBs can help reduce the impact of this caching. Regional availability or regional redundancy ELB and Amazon Route 53 have been integrated to support a single application across multiple regions. Route 53 is AWS's highly available and scalable DNS and health checking service. Route 53 supports high availability architectures by health checking load balancer nodes and rerouting traffic to avoid the failed nodes, and by supporting implementation of multi-region architectures. In addition, Route 53 uses Latency Based Routing (LBR) to route your customers to the endpoint that has the least latency. If multiple primary sites are implemented with appropriate health checks configured, then in cases of failure, traffic shifts away from that site to the next closest region. Region failures can present several challenges as a result of rapidly shifting traffic (similar to the case of zone failures). These can include auto scaling, time required for instance startup, and the cache fill time (as we might need to default to our data sources, initially). Another difficulty usually arises from the lack of information or clarity on what constitutes the minimal or critical stack required to keep the site functioning as normally as possible. For example, any or all services will need to be considered as critical in these circumstances. The health checks are essentially automated requests sent over the Internet to your application to verify that your application is reachable, available, and functional. This can include both your EC2 instances and your application. As answers are returned only for the resources that are healthy and reachable from the outside world, the end users can be routed away from a failed application. Amazon Route 53 health checks are conducted from within each AWS region to check whether your application is reachable from that location. The DNS failover is designed to be entirely automatic. After you have set up your DNS records and health checks, no manual intervention is required for failover. Ensure you create appropriate alerts to be notified when this happens. Typically, it takes about 2 to 3 minutes from the time of the failure to the point where traffic is routed to an alternate location. Compare this to the traditional process where an operator receives an alarm, manually configures the DNS update, and waits for the DNS changes to propagate. The failover happens entirely within the Amazon Route 53 data plane. Depending on your availability objectives, there is an additional strategy (using Route 53) that you might want to consider for your application. For example, you can create a backup static site to maintain a presence for your end customers while your primary dynamic site is down. In the normal course, Route 53 will point to your dynamic site and maintain health checks for it. Furthermore, you will need to configure Route 53 to point to the S3 storage, where your static site resides. If your primary site goes down, then traffic can be diverted to the static site (while you work to restore your primary site). You can also combine this static backup site strategy with a multiple region deployment. Setting up high availability for application and data layers In this section, we will discuss approaches for implementing high availability in the application and data layers of your application architecture. The auto healing feature of AWS OpsWorks provides a good recovery mechanism from instance failures. All OpsWorks instances have an agent installed. If an agent does not communicate with the service for a short duration, then OpsWorks considers the instance to have failed. If auto healing is enabled at the layer and an instance becomes unhealthy, then OpsWorks first terminates the instance and starts a new one as per the layer configuration. In the application layer, we can also do cold starts from preconfigured images or a warm start from scaled down instances for your web servers and application servers in a secondary region. By leveraging auto scaling, we can quickly ramp up these servers to handle full production loads. In this configuration, you would deploy the web servers and application servers across multiple AZs in your primary region while the standby servers need not be launched in your secondary region until you actually need them. However, keep the preconfigured AMIs for these servers ready to launch in your secondary region. The data layer can comprise of SQL databases, NoSQL databases, caches, and so on. These can be AWS managed services such as RDS, DynamoDB, and S3, or your own SQL and NoSQL databases such as Oracle, SQL Server, or MongoDB running on EC2 instances. AWS services come with HA built-in, while using database products running on EC2 instances offers a do-it-yourself option. It can be advantageous to use AWS services if you want to avoid taking on database administration responsibilities. For example, with the increasing sizes of your databases, you might choose to share your databases, which is easy to do. However, resharding your databases while taking in live traffic can be a very complex undertaking and present availability risks. Choosing to use the AWS DynamoDB service in such a situation offloads this work to AWS, thereby resulting in higher availability out of the box. AWS provides many different data replication options and we will discuss a few of those in the following several paragraphs. DynamoDB automatically replicates your data across several AZs to provide higher levels of data durability and availability. In addition, you can use data pipelines to copy your data from one region to another. DynamoDB streams functionality that can be leveraged to replicate to another DynamoDB in a different region. For very high volumes, low latency Kinesis services can also be used for this replication across multiple regions distributed all over the world. You can also enable the Multi-AZ setting for the AWS RDS service to ensure AWS replicates your data to a different AZ within the same region. In the case of Amazon S3, the S3 bucket contents can be copied to a different bucket and the failover can be managed on the client side. Depending on the volume of data, always think in terms of multiple machines, multiple threads and multiple parts to significantly reduce the time it takes to upload data to S3 buckets. While using your own database (running on EC2 instances), use your database-specific high availability features for within and cross-region database deployments. For example, if you are using SQL Server, you can leverage the SQL Server Always-on feature for synchronous and asynchronous replication across the nodes. If the volume of data is high, then you can also use the SQL Server log shipping to first upload your data to Amazon S3 and then restore into your SQL Server instance on AWS. A similar approach in case of Oracle databases uses OSB Cloud Module and RMAN. You can also replicate your non-RDS databases (on-premise or on AWS) to AWS RDS databases. You will typically define two nodes in the primary region with synchronous replication and a third node in the secondary region with asynchronous replication. NoSQL databases such as MongoDB and Cassandra have their own asynchronous replication features that can be leveraged for replication to a different region. In addition, you can create Read Replicas for your databases in other AZs and regions. In this case, if your master database fails followed by a failure of your secondary database, then one of the read replicas can be promoted to being the master. In hybrid architectures, where you need to replicate between on-premise and AWS data sources, you can do so through a VPN connection between your data center and AWS. In case of any connectivity issues, you can also temporarily store pending data updates in SQS, and process them when the connectivity is restored. Usually, data is actively replicated to the secondary region while all other servers like the web servers and application servers are maintained in a cold state to control costs. However, in cases of high availability for web scale or mission critical applications, you can also choose to deploy your servers in active configuration across multiple regions. Implementing high availability in the application In this section, we will discuss a few design principles to use in your application from a high availability perspective. We will briefly discuss using highly available AWS services to implement common features in mobile and Internet of Things (IoT) applications. Finally, we also cover running packaged applications on the AWS cloud. Designing your application services to be stateless and following a micro services-oriented architecture approach can help the overall availability of your application. In such architectures, if a service fails then that failure is contained or isolated to that particular service while the rest of your application services continue to serve your customers. This approach can lead to an acceptable degraded experience rather than outright failures or worse. You should also store user or session information in a central location such as the AWS ElastiCache and then spread information across multiple AZs for high availability. Another design principle is to rigorously implement exception handling in your application code, and in each of your services to ensure graceful exit in case of failures. Most mobile applications share common features including user authentication and authorization, data synchronization across devices; user behavior analytics; retention tracking, storing, sharing, and delivering media globally; sending push notifications; store shared data; stream real-time data; and so on. There are a host of highly available AWS services that can be used for implementing such mobile application functionality. For example, you can use Amazon Cognito to authenticate users, Amazon Mobile Analytics for analyzing user behavior and tracking retention, Amazon SNS for push notifications and Amazon Kinesis for streaming real-time data. In addition, other AWS services such as S3, DynamoDB, IAM, and so on can also be effectively used to complete most mobile application scenarios. For mobile applications, you need to be especially sensitive about latency issues; hence, it is important to leverage AWS regions to get as close to your customers as possible. Similar to mobile applications, for IoT applications you can use the same highly available AWS services to implement common functionality such as device analytics and device messaging/notifications. You can also leverage Amazon Kinesis to ingest data from hundreds of thousands of sensors that are continuously generating massive quantities of data. Aside from your own custom applications, you can also run packaged applications such as SAP on AWS. These would typically include replicated standby systems, Multi-AZ and multi-region deployments, hybrid architectures spanning your own data center, and AWS cloud (connected via VPN or AWS Direct Connect service), and so on. For more details, refer to the specific package guides for achieving high availability on the AWS cloud. Summary In this article, we reviewed some of the strategies you can follow for achieving high availability in your cloud application. We emphasized the importance of both designing your application architecture for availability and using the AWS infrastructural services to get the best results. Resources for Article: Further resources on this subject: Securing vCloud Using the vCloud Networking and Security App Firewall [article] Introduction to Microsoft Azure Cloud Services [article] AWS Global Infrastructure [article]
Read more
  • 0
  • 0
  • 18450

article-image-dont-call-us-ninjas-or-rockstars-say-developers
Richard Gall
30 May 2018
5 min read
Save for later

Don't call us ninjas or rockstars, say developers

Richard Gall
30 May 2018
5 min read
Words like 'ninja' and 'rockstar' have been flying around the tech world for some time now. Data revealed by recruitment website Indeed at the end of 2017 showed that the term 'rockstar' has increased in job postings 19% since 2015. We seem to live in a world where 'sexing up' job roles has become the norm. And when top talent is hard to come by it makes sense. The words offer some degree of status to candidates and, and imply the organizations behind them are forward-thinking. But it's starting to get boring. In this year's Skill Up survey, 57% of respondents said they didn't like creative terms like 'rockstar' 'ninja' and 'wizard' - only 26% said they actually liked the term. While words like these might be boring, they can be harmful too. In an age of spin and fake news, using language to dress up and redefine things can have an impact we might not expect. Sign up to the Packt Hub weekly newsletter and receive a free PDF of this year's Skill Up report. Using words like rockstar and ninja pits developers against each other The industry's insistence on using these words in everything from recruitment to conferences cultivates a bizarre class system within tech. When we start calling people rockstars, it suggests something about the role they play within a company or engineering team. It says 'these people are doing something really exciting' while everyone else, presumably, isn't. While it's true that hierarchies are part and parcel of any modern organization, this superficial labeling isn't helpful. 'Collaboration' and 'agile' are buzzwords that are as overused as ninja and rockstar, but at least they offer something practical and positive. And let's be honest - collaborating is important if we're to build better software and have better working lives. The unforeseen impact on developer mental health An unforeseen affect of these words could be a negative impact on mental health in the tech industry. We already know that burnout is becoming a common occurrence, as tech professionals are being overworked and pushed to breaking point. While this is particularly true of startup culture, where engineers are driven to develop products on incredibly tight schedules as owners seek investment and investors look for signs of growth, but this trend is growing. Arguably, the gap between management and engineers is playing into this. The results of innovation look shiny and exciting, but the actual work - which can, as we know, be repetitive, boring, hard as it can be enjoyable - isn't properly understood. Ninjas, rockstars and the commodification of technical skill It has become a truism that communication is important when it comes to tech. Words like ninja and rockstar are making communication hard - they undermine our ability to communicate. They conceal the work that engineers actually do. It's great that they shine a spotlight on technical professionals and skills, but they do so in a way that is actually quite euphemistic. It's hints at the value of the skills, but actually fails to engage with why these skills are important, how you develop them, and how they can be properly leveraged. More specifically, words like 'ninja' and 'rockstar' turn technical expertise into a product. It makes skills marketable; it turns knowledge into a commodity. Good for you if that helps you earn a little more money in your next job; even better if it lands you a book deal or a spot speaking at a conference. But all you're really doing is taking advantage of the technical skills bubble. It won't last forever, and in the long run it probably won't be good news for the industry. Some people will be overvalued, while others will be undervalued. Ninjas, rockstars and open source culture These irritating euphemisms come from a number of different sources. Tech recruitment has played a big part, as companies try and attract top tech talent. So has modern corporate culture, which has been trying to loosen its proverbial tie for the last decade. But it's also worth noting that rockstars and ninjas come out of open source culture. This is a culture that is celebrated for its lack of authority. However, with this decline, it makes opens up a space for 'experts' to take the lead. In the past, we might have quaintly referred to these people as 'community figures'. As open source has moved mainstream, the commodification of technical expertise found form in the tech ninja and rockstar. But while rockstars and ninjas appear to be the shining lights of open source culture, they might also be damaging to it as well. If open source culture is 'led' by a number of people the world begins referring to as rockstars, the very foundations of it begin to move. It's no longer quite as open as it used to be. True, perhaps we need rockstars and ninjas in open source. These are people that evangelize about certain projects. These are people who can offer unique and useful perspectives on important debates and issues in their respective fields, right? Well, sort of. It is important for people to discuss new ideas and pioneer new ways of doing things. But this doesn't mean we need to sex things up. After all, certain people aren't more entitled to an opinion just because they have a book deal. Yes they have experience, but it's important that communities don't get left behind as the tech industry chases after the dream of relentless innovation. Of course it's great to talk about, but we've all got to do the work too.
Read more
  • 0
  • 0
  • 18443

article-image-chatgpt-for-quantum-computing
Anshul Saxena
03 Nov 2023
7 min read
Save for later

ChatGPT for Quantum Computing

Anshul Saxena
03 Nov 2023
7 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!IntroductionHello there, fellow explorer! So, you've been hearing about this thing called 'quantum computing' and how it promises to revolutionize... well, almost everything. And you're curious about how we can harness its power, right? But there's a twist: you want to use ChatGPT to help guide the process. Intriguing! In this tutorial, I'll take you by the hand, and together, we'll craft some amazing use cases for quantum computing, all with the help of ChatGPT prompts.First, we'll lay down our goals. What exactly do we want to achieve with quantum computing? Maybe it's predicting the weather years in advance, or understanding the deep mysteries of our oceans. Once we have our roadmap, it's time to gather our tools and data. Here's where satellites, weather stations, and another cool tech come in.But data can be messy, right? No worries! We'll clean it up and get it ready for our quantum adventure. And then, brace yourself, because we're diving deep into the world of quantum mechanics. But fear not! With ChatGPT by our side, we'll decode the jargon and make it all crystal clear.The next steps? Designing our very own quantum algorithms and giving them a test run. It's like crafting a recipe and then baking the perfect cake. Once our quantum masterpiece is ready, we'll look at the results, decipher what they mean, and integrate them with existing tools. And because we always strive for perfection, we'll continuously refine our approach, ensuring it's the best it can be.Here's a streamlined 10-step process for modeling complex climate systems using quantum computing:Step 1. Objective Definition: Clearly define the specific goals of climate modeling, such as predicting long-term temperature changes, understanding oceanic interactions, or simulating atmospheric phenomena.Step 2. Data Acquisition: Gather comprehensive climate data from satellites, ground stations, and other relevant sources, focusing on parameters crucial for the modeling objectives.Step 3. Data Preprocessing: Clean and transform the climate data into a format suitable for quantum processing, addressing any missing values, inconsistencies, or noise.Step 4. Understanding Quantum Mechanics: Familiarize with the principles and capabilities of quantum computing, especially as they relate to complex system modeling.Step 5. Algorithm Selection/Design: Choose or develop quantum algorithms tailored to model the specific climate phenomena of interest. Consider hybrid algorithms that leverage both classical and quantum computations.Step 6. Quantum Simulation: Before deploying on real quantum hardware, simulate the chosen quantum algorithms on classical systems to gauge their efficacy and refine them as needed.Step 7. Quantum Execution: Implement the algorithms on quantum computers, monitoring performance and ensuring accurate modeling of the climate system.Step 8. Result Interpretation: Analyze the quantum computing outputs, translating them into actionable climate models, predictions, or insights.Step 9. Integration & Application: Merge the quantum-enhanced models with existing climate research tools and methodologies, ensuring the findings are accessible and actionable for researchers, policymakers, and stakeholders.Step 10. Review & Iteration: Regularly evaluate the quantum modeling process, updating algorithms and methodologies based on new data, quantum advancements, or evolving climate modeling objectives.Using quantum computing for modeling complex climate systems holds promise for more accurate and faster simulations, but it's essential to ensure the approach is methodical and scientifically rigorous.So, are you ready to create some quantum magic with ChatGPT? Let's jump right in!1. Objective DefinitionPrompt: "ChatGPT, can you help me outline the primary objectives and goals when modeling complex climate systems? What are the key phenomena and parameters we should focus on?"2. Data AcquisitionPrompt:"ChatGPT, where can I source comprehensive climate data suitable for quantum modeling? Can you list satellite databases, ground station networks, or other data repositories that might be relevant?"3. Data PreprocessingPrompt:"ChatGPT, what are the best practices for preprocessing climate data for quantum computing? How do I handle missing values, inconsistencies, or noise in the dataset?"4. Understanding Quantum MechanicsPrompt:"ChatGPT, can you give me a primer on the principles of quantum computing, especially as they might apply to modeling complex systems like climate?"5. Algorithm Selection/DesignPrompt:"ChatGPT, what quantum algorithms or techniques are best suited for climate modeling? Are there hybrid algorithms that combine classical and quantum methods for this purpose?"6. Quantum SimulationPrompt:"ChatGPT, how can I simulate quantum algorithms on classical systems before deploying them on quantum hardware? What tools or platforms would you recommend?"7. Quantum Execution Prompt:"ChatGPT, what are the steps to implement my chosen quantum algorithms on actual quantum computers? Are there specific quantum platforms or providers you'd recommend for climate modeling tasks?"8. Result InterpretationPrompt:"ChatGPT, once I have the outputs from the quantum computation, how do I interpret and translate them into meaningful climate models or predictions?"9. Integration & ApplicationPrompt:"ChatGPT, how can I integrate quantum-enhanced climate models with existing research tools and methodologies? What steps should I follow to make these models actionable for the broader research community?"10. Review & IterationPrompt:"ChatGPT, how should I periodically evaluate and refine my quantum modeling approach? What metrics or feedback mechanisms can help ensure the process remains optimal and up-to-date?"These prompts are designed to guide a user in leveraging ChatGPT's knowledge and insights for each step of the quantum computing-based climate modeling process.ConclusionAnd there you have it! From setting clear goals to diving into the intricate world of quantum mechanics and finally crafting our very own quantum algorithms, we've journeyed through the fascinating realm of quantum computing together. With ChatGPT as our trusty guide, we've unraveled complex concepts, tackled messy data, and brewed some quantum magic. It's been quite the adventure, hasn't it? Remember, the world of quantum computing is vast and ever-evolving, so there's always more to explore and learn. Whether you're a seasoned quantum enthusiast or just starting out, I hope this guide has ignited a spark of curiosity in you. As we part ways on this tutorial journey, I encourage you to keep exploring, questioning, and innovating. The quantum realm awaits your next adventure. Until next time, happy quantum-ing!Author BioDr. Anshul Saxena is an author, corporate consultant, inventor, and educator who assists clients in finding financial solutions using quantum computing and generative AI. He has filed over three Indian patents and has been granted an Australian Innovation Patent. Anshul is the author of two best-selling books in the realm of HR Analytics and Quantum Computing (Packt Publications). He has been instrumental in setting up new-age specializations like decision sciences and business analytics in multiple business schools across India. Currently, he is working as Assistant Professor and Coordinator – Center for Emerging Business Technologies at CHRIST (Deemed to be University), Pune Lavasa Campus. Dr. Anshul has also worked with reputed companies like IBM as a curriculum designer and trainer and has been instrumental in training 1000+ academicians and working professionals from universities and corporate houses like UPES, CRMIT, and NITTE Mangalore, Vishwakarma University, Pune & Kaziranga University, and KPMG, IBM, Altran, TCS, Metro CASH & Carry, HPCL & IOC. With a work experience of 5 years in the domain of financial risk analytics with TCS and Northern Trust, Dr. Anshul has guided master's students in creating projects on emerging business technologies, which have resulted in 8+ Scopus-indexed papers. Dr. Anshul holds a PhD in Applied AI (Management), an MBA in Finance, and a BSc in Chemistry. He possesses multiple certificates in the field of Generative AI and Quantum Computing from organizations like SAS, IBM, IISC, Harvard, and BIMTECH.Author of the book: Financial Modeling Using Quantum Computing
Read more
  • 0
  • 0
  • 18404

article-image-using-chatgpt-with-text-to-speech
Denis Rothman
04 Jun 2023
7 min read
Save for later

Using ChatGPT with Text to Speech

Denis Rothman
04 Jun 2023
7 min read
This article provides a quick guide to using the OpenAI API to jump-start ChatGPT. The guide includes instructions on how to use a microphone to speak to ChatGPT and how to create a ChatGPT request with variables. Additionally, the article explains how to use Google gTTS, a text-to-speech tool, to listen to ChatGPT's response. By following these steps, you can have a more interactive experience with ChatGPT and make use of its advanced natural language processing capabilities. We’re using the GPT-3.5-Turbo architecture in this example. We are also running the examples within Google Colab, but they should be applicable to other environments. In this article, we’ll cover: Installing OpenAI, your API key, and Google gTTS for Text-to-SpeechGenerating content with ChatGPTSpeech-to-text ChatGPT's responseTranscribing with WhisperTo understand GPT-3 Transformers in detail, read Transformers for NLP, 2nd Edition 1. Installing OpenAI, gTTS, and your API Key There are a few libraries that we’ll need to install into Colab for this project. We’ll install them as required, starting with OpenAI. Installing and Importing OpenAI To start using OpenAI's APIs and tools, we'll need to install the OpenAI Python package and import it into your project. To do this, you can use pip, a package manager for Python. First, make sure you have pip installed on your system. !pip install --upgrade pipNext, run the following script in your notebook to install the OpenAI package. It should come pre-installed in Colab:#Importing openaitry:import openaiexcept:!pip install openaiimport openai Installing gTTS Next, install Google gTTS a Python library that provides an easy-to-use interface for text-to-speech synthesis using the Google Text-to-Speech API:#Importing gTTStry:from gtts import gTTSexcept:!pip install gTTS   from gtts import gTTS API Key Finally, import your API key. Rather than enter your key directly into your notebook, I recommend keeping it in a local file and importing it from your script. You will need to provide the correct path and filename in the code below.from google.colab import drivedrive.mount('/content/drive')f = open("drive/MyDrive/files/api_key.txt", "r")API_KEY=f.readline()f.close()#The OpenAI Keyimport osos.environ['OPENAI_API_KEY'] =API_KEYopenai.api_key = os.getenv("OPENAI_API_KEY") 2. Generating Content Let’s look at how to pass prompts into the OpenAI API to generate responses. Speech to text When it comes to speech recognition, Windows provides built-in speech-to-text functionality. However, third-party speech-to-text modules are also available, offering features such as multiple language support, speaker identification, and audio transcription. For simple speech-to-text, this notebook uses the built-in functionality in Windows. Press Windows key + H to bring up the Windows speech interface. You can read the documentation for more information.Note: For this notebook, press Enter when you have finished asking for a request in Colab. You could also adapt the function in your application with a timed input function that automatically sends a request after a certain amount of time has elapsed. Preparing the Prompt Note: you can create variables for each part of the OpenAI messages object. This object contains all the information needed to generate a response from ChatGPT, including the text prompt, the model ID, and the API key. By creating variables for each part of the object, you can make it easier to generate requests and responses programmatically. For example, you could create a prompt variable that contains the text prompt for generating a response. You could also create variables for the model ID and API key, making it easier to switch between different OpenAI models or accounts as needed.For more on implementing each part of the messages object, take a look at: Prompt_Engineering_as_an_alternative_to_fine_tuning.ipynb.Here’s the code for accepting the prompt and passing the request to OpenAI:#Speech to text. Use OS speech-to-text app. For example,   Windows: press Windows Key + H def prepare_message():#enter the request with a microphone or type it if you wish  # example: "Where is Tahiti located?"  print("Enter a request and press ENTER:")  uinput = input("")  #preparing the prompt for OpenAI   role="user"  #prompt="Where is Tahiti located?" #maintenance or if you do not want to use a microphone  line = {"role": role, "content": uinput}  #creating the message   assert1={"role": "system", "content": "You are a helpful assistant."}  assert2={"role": "assistant", "content": "Geography is an important topic if you are going on a once in a lifetime trip."}  assert3=line  iprompt = []  iprompt.append(assert1)  iprompt.append(assert2)  iprompt.append(assert3)  return iprompt#run the cell to start/continue a dialogiprompt=prepare_message() #preparing the messages for ChatGPTresponse=openai.ChatCompletion.create(model="gpt-3.5-turbo",messages=iprompt) #ChatGPT dialogtext=response["choices"][0]["message"]["content"] #response in JSONprint("ChatGPT response:",text) Here's a sample of the output: Enter a request and press ENTER:Where is Tahiti locatedChatGPT response: Tahiti is located in the South Pacific Ocean, specifically in French Polynesia. It is part of a group of islands called the Society Islands and is located approximately 4,000 kilometers (2,500 miles) south of Hawaii and 7,850 kilometers (4,880 miles) east of Australia. 3. Speech-to-text the response GTTS and IPython Once you've generated a response from ChatGPT using the OpenAI package, the next step is to convert the text into speech using gTTS (Google Text-to-Speech) and play it back using  IPython audio.from gtts import gTTSfrom IPython.display import Audiotts = gTTS(text)tts.save('1.wav')sound_file = '1.wav'Audio(sound_file, autoplay=True) 4. Transcribing with Whisper If your project requires the transcription of audio files, you can use OpenAI’s Whisper.First, we’ll install the ffmpeg audio processing library. ffmpeg is a popular open-source software suite for handling multimedia data, including audio and video files:!pip install ffmpegNext, we’ll install Whisper:!pip install git+https://github.com/openai/whisper.git With that done, we can use a simple command to transcribe the WAV file and store it as a JSON file with the same name:!whisper  1.wavYou’ll see Whisper transcribe the file in chunks:[00:00.000 --> 00:06.360]  Tahiti is located in the South Pacific Ocean, specifically in the archipelago of society[00:06.360 --> 00:09.800]  islands and is part of French Polynesia.[00:09.800 --> 00:22.360]  It is approximately 4,000 miles, 6,400 km, south of Hawaii and 5,700 miles, 9,200 km,[00:22.360 --> 00:24.640]  west of Santiago, Chile.Once that’s done, we can read the JSON file and display the text object:import json with open('1.json') as f:     data = json.load(f) text = data['text'] print(text)This gives the following output:Tahiti is located in the South Pacific Ocean, specifically in the archipelago of society islands and is part of French Polynesia. It is approximately 4,000 miles, 6,400 km, south of Hawaii and 5,700 miles, 9,200 km, west of Santiago, Chile. By using Whisper in combination with ChatGPT and gTTS, you can create a fully featured AI-powered application that enables users to interact with your system using natural language inputs and receive audio responses. This might be useful for applications that involve transcribing meetings, conferences, or other audio files. About the Author Denis Rothman graduated from Sorbonne University and Paris-Diderot University, designing one of the very first word2matrix patented embedding and patented AI conversational agents. He began his career authoring one of the first AI cognitive natural language processing (NLP) chatbots applied as an automated language teacher for Moet et Chandon and other companies. He authored an AI resource optimizer for IBM and apparel producers. He then authored an advanced planning and scheduling (APS) solution used worldwide.You can follow Denis on LinkedIn:  https://www.linkedin.com/in/denis-rothman-0b034043/Copyright 2023 Denis Rothman, MIT License
Read more
  • 0
  • 0
  • 18392
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at ₹800/month. Cancel anytime
article-image-using-openai-python-library-to-interact-with-the-chatgpt-api
Martin Yanev
04 Jun 2023
7 min read
Save for later

Using OpenAI Python Library to Interact with the ChatGPT API

Martin Yanev
04 Jun 2023
7 min read
Using the ChatGPT API with Python is a relatively simple process. You'll first need to make sure you create a new PyCharm project called ChatGPTResponse as shown in the following screenshot: Fig 1: New Project Window in PyCharm Once you have that setup, you can use the OpenAI Python library to interact with the ChatGPT API. Open a new Terminal in PyCharm, make sure that you are in your project folder, and install the openai package: $ pip install openai Next, you need to create a new Python file in your PyCharm project on the left top corner perform a Right-click on the folder ChatGPTResponse | New | Python File. Name the file app.py and click hit Enter. You should now have a new Python file in your project directory: Fig 2: New Python File To get started, you'll need to import the openai library into your Python file. Also, you'll need to provide your OpenAI API Key. You can obtain an API Key from the OpenAI website by following the steps outlined in the previous sections of this book. Then you'll need to set it as a parameter in your Python code. Once your API Key is set up, you can start interacting with the ChatGPT API. import openaiopenai.api_key = "YOUR_API_KEY" Replace YOUR_API_KEY with the API key you obtained from the OpenAI platform page. Now, you can ask the user a question using the input() function: question = input("What would you like to ask ChatGPT? ") The input() function is used to prompt the user to input a question they would like to ask the ChatGPT API. The function takes a string as an argument, which is displayed to the user when the program is run. In this case, the question string is "What would you Like to ask ChatGPT?". When the user types their question and presses enter, the input() function will return the string that the user typed. This string is then assigned to the variable question. To pass the user question from your Python script to ChatGPT, you will need to use the ChatGPT API Completion function: response = openai.Completion.create(    engine="text-davinci-003",    prompt=question,    max_tokens=1024,    n=1,    stop=None,    temperature=0.8,) The openai.Completion.create() function in the code is used to send a request to the ChatGPT API to generate a completion of the user's input prompt. The engine parameter specifies the ChatGPT engine to use for the request, and in this case, it is set to "text-davinci-003". The prompt parameter specifies the text prompt for the API to complete, which is the user's input question in this case. The max_tokens parameter specifies the maximum number of tokens the response should contain.  The n parameter specifies the number of completions to generate for the prompt. The stop parameter specifies the sequence where the API should stop generating the response. The temperature parameter controls the creativity of the generated response. It ranges from 0 to 1. Higher values will result in more creative but potentially less coherent responses, while lower values will result in more predictable but potentially fewer interesting responses. Later in the book, we will delve into how these parameters impact the responses received from ChatGPT.The function returns a JSON object containing the generated response from the ChatGPT API, which then can be accessed and printed to the console in the next line of code.print(response)In the project pane on the left-hand side of the screen, locate the Python file you want to run. Right-click on the app.py file and select Run app.py from the context menu. You should receive a message in the Run window that asks you to write a question to the ChatGPT.  Fig 3: Run WindowOnce you have entered your question, press the Enter key to submit your request to the ChatGPT API. The response generated by the ChatGPT API model will be displayed in the Run window as a complete JSON object: {   "choices": [    {       "finish_reason": "stop",       "index": 0,       "logprobs": null,       "text": "\n\n1. Start by getting in the water. If you're swimming in a pool, you can enter the water from the side, ………….    }  ],   "created": 1681010983,   "id": "cmpl-73G2JJCyBTfwCdIyZ7v5CTjxMiS6W",   "model": "text-davinci-003",   "object": "text_completion",   "usage": {    "completion_tokens": 415,     "prompt_tokens": 4,     "total_tokens": 419  }} This JSON response produced by the OpenAI API contains information about the response generated by the GPT-3 model. This response consists of the following fields:The choices field contains an array of objects with the generated responses, which in this case only contains one response object. The text field within the response object contains the actual response generated by the GPT-3 model.The finish_reason field indicates the reason why the response was generated; in this case it was because the model reached the stop condition specified in the API request.The created field specifies the Unix timestamp of when the response was created. The id field is a unique identifier for the API request that generated this response.The model field specifies the GPT-3 model that was used to generate the response. The object field specifies the type of object that was returned, which in this case is text_completion.The usage field provides information about the resource usage of the API request. It contains information about the number of tokens used for the completion, the number of tokens in the prompt, and the total number of tokens used.The two most important parameter from the response is the text field, that contains the answer to the question asked to ChatGPT API. This is why most API users would like to access only that parameter from the JSON object. You can easily separate the text from the main body as follows: answer = response["choices"][0]["text"]print(answer)By following this approach, you can guarantee that the variable answer will hold the complete ChatGPT API text response, which you can then print to verify. Keep in mind that ChatGPT responses can significantly differ depending on the input, making each response unique.OpenAI:1. Start by getting in the water. If you're swimming in a pool, you can enter the water from the side, ladder, or diving board. If you are swimming in the ocean or lake, you can enter the water from the shore or a dock.2. Take a deep breath in and then exhale slowly. This will help you relax and prepare for swimming.Summary In this tutorial, you implemented a simple ChatGPT API response, by sending a request to generate a completion of a user's input prompt/question. You have also learned how to set up your API Key and how to prompt the user to input a question, and finally, how to access the generated response from ChatGPT in the form of a JSON object containing information about the response.  About the Author Martin Yanev is an experienced Software Engineer who has worked in the aerospace and medical industries for over 8 years. He specializes in developing and integrating software solutions for air traffic control and chromatography systems. Martin is a well-respected instructor with over 280,000 students worldwide, and he is skilled in using frameworks like Flask, Django, Pytest, and TensorFlow. He is an expert in building, training, and fine-tuning AI systems with the full range of OpenAI APIs. Martin has dual Master's degrees in Aerospace Systems and Software Engineering, which demonstrates his commitment to both practical and theoretical aspects of the industry.https://www.linkedin.com/in/martinyanev/https://www.udemy.com/user/martin-yanev-3/
Read more
  • 0
  • 0
  • 18383

article-image-no-new-peps-will-be-approved-for-python-until-a-new-bdfl-is-chosen-next-year
Natasha Mathur
24 Jul 2018
3 min read
Save for later

No new PEPS will be approved for Python in 2018, with BDFL election pending.

Natasha Mathur
24 Jul 2018
3 min read
According to an email exchange, Python will not approve of any proposals or PEPs until January 2019 as the team is planning to choose a new leader, with a deadline set for January 1, 2019. The news comes after Python founder and “Benevolent Dictator for Life (BDFL)” Guido van Rossum, announced earlier this month that he was standing down from the decision-making process.   Here’s a quick look at Why Guido may have quit as the Python chief (BDFL). A PEP - or Python Enhancement Proposal - is a design document consisting of information important to the Python community. If a new feature is to be added to the language, it is first detailed in a PEP. It will include technical specifications as well as the rationale for the feature. A passionate discussion regarding PEP 576, 579 and 580 started last week on the python-dev community with Jeroen Demeyer, a Cython contributor, stating how he finally came up with real-life benchmarks proving why a faster C calling protocol is required in Python. He implemented Cython compilation of SageMath on Python 2.7.15 (SageMath is not yet ported to Python 3). But, he mentioned that the conclusions of his implementation should be valid for newer Python versions as well. Guido’s, in his capacity as a core developer, responded back to this implementation request. In the email thread, he writes that “Jeroen was asked to provide benchmarks but only provided them for Python 2. The reasoning that not much has changed could affect the benchmarks feels a bit optimistic, that's all. The new BDFL may be less demanding though.” Stefan Behnel, a core Cython developer, was disappointed with Rossum’s response.. He writes that because Demeyer has already done “the work to clean up the current implementation… it would be very sad if this chance was wasted, and we would have to remain with the current implementation” But it seems that Guido is not convinced. According to Guido, no PEPS will be approved until the Python core dev team have elected a “new BDFL or come up with some other process for accepting PEPs, no action will be taken on this PEP.” The mail says “right now, there is no-one who can approve this PEP, and you will have to wait until 2019 until there is”. Many Python users are worried about this decision:                    Reddit All that’s left to do now is wait for more updates from the Python-dev community. Behnel states “I just hope that Python development won't stall completely. Even if no formal action can be taken on this PEP (or any other), I hope that there will still be informal discussion”. For more coverage on this news, check out the official mail posts on python-dev. Top 7 Python programming books you need to read Python, Tensorflow, Excel and more – Data professionals reveal their top tools  
Read more
  • 0
  • 0
  • 18382

article-image-learning-r-geospatial-analysis
Packt
08 May 2015
3 min read
Save for later

Learning R for Geospatial Analysis

Packt
08 May 2015
3 min read
The defining feature of spatial data analysis is the reference, within the data being analyzed, to locations on the surface of the earth. This is a very broad subject encompassing distinct areas of expertise such as spatial statistics, geometric computation, and image processing. In practice, spatial data is commonly stored, viewed, and analyzed in Geographic Information System (GIS) software, of which the most well-known example is ArcGIS. However, most often, menu-based interfaces of GIS software are too narrow in scope to meet with specialized demands or too inflexible to feasibly accomplish customized, repetitive tasks. Writing scripts rather than using menus or working in combination with external software are two commonly used paths to solve such problems. However, what if we can use a single environment, combining the advantages of programming and spatial data analysis capabilities with a comprehensive ecosystem of computational tools that are readily implementable in customized procedures? This book will demonstrate that the R programming language is indeed such an environment and teach you how to use it in order to perform various spatial data analysis tasks. (For more resources related to this topic, see here.) What you will learn This book covers the basic concepts related to writing R code. You will also learn how to work with vectors, time series, tables, rasters, points, lines, and polygons. The book also covers several advanced themes associated with raster data analysis in R. Demonstrations on how rasters and vector layers can be combined in a single analysis are shown. Transformation between raster and vector data structures as well as data extraction from a raster based on vector layers are covered in this book. Moreover, we will also learn how spatial interpolation can be carried out in R through examples of interpolating meteorological point measurements to create annual temperature maps of Spain. You will also explore some of the most useful methods for advanced visualization of spatial data in R, using the ggplot2, ggmap, and lattice packages. How the book differs Most currently available books on this subject are focused on advanced applications such as spatial statistics, assuming you have prior knowledge of R and the respective scientific domains. Yet, introductory material on R from the point of view of a spatial data analyst, which is focused on introductory topics such as spatial data handling, computation, and visualization, is scarce. This book aims to fill the gap. Thus, this book is intended for anyone who wants to learn how to efficiently analyze geospatial data with R. No prior experience with R and/or programming is required; only you need to be familiar with basic geographic information concepts (such as spatial coordinates). Required skills To follow through the examples in this book, all you need to do is install R (which is available for free) and download the example datasets from the book's website. Some of the examples also require you to have an Internet connection to download additional datasets and R packages from the R environment. Summary This book is composed of step-by-step tutorials, starting with the language basics before proceeding to cover the main GIS operations and data types. Visualization of spatial data is vital either during the various analysis steps and/or as the final product, and this book shows you how to get the most out of R's visualization capabilities. The book culminates with examples of cutting-edge applications utilizing R's strengths as a statistical and graphical tool. Resources for Article:  Further resources on this subject: Data visualization [article] Machine Learning in Bioinformatics [article] Specialized Machine Learning Topics [article]
Read more
  • 0
  • 0
  • 18379

article-image-understanding-websockets-and-server-sent-events-detail
Packt
29 Oct 2013
10 min read
Save for later

Understanding WebSockets and Server-sent Events in Detail

Packt
29 Oct 2013
10 min read
(For more resources related to this topic, see here.) Encoders and decoders in Java API for WebSockets As seen in the previous chapter, the class-level annotation @ServerEndpoint indicates that a Java class is a WebSocket endpoint at runtime. The value attribute is used to specify a URI mapping for the endpoint. Additionally the user can add encoder and decoder attributes to encode application objects into WebSocket messages and WebSocket messages into application objects. The following table summarizes the @ServerEndpoint annotation and its attributes: Annotation Attribute Description @ServerEndpoint   This class-level annotation signifies that the Java class is a WebSockets server endpoint.   value The value is the URI with a leading '/.'   encoders The encoders contains a list of Java classes that act as encoders for the endpoint. The classes must implement the Encoder interface.   decoders The decoders contains a list of Java classes that act as decoders for the endpoint. The classes must implement the Decoder interface.   configurator The configurator attribute allows the developer to plug in their implementation of ServerEndpoint.Configurator that is used when configuring the server endpoint.   subprotocols The sub protocols attribute contains a list of sub protocols that the endpoint can support. In this section we shall look at providing encoder and decoder implementations for our WebSockets endpoint. The preceding diagram shows how encoders will take an application object and convert it to a WebSockets message. Decoders will take a WebSockets message and convert to an application object. Here is a simple example where a client sends a WebSockets message to a WebSockets java endpoint that is annotated with @ServerEndpoint and decorated with encoder and decoder class. The decoder will decode the WebSockets message and send back the same message to the client. The encoder will convert the message to a WebSockets message. This sample is also included in the code bundle for the book. Here is the code to define ServerEndpoint with value for encoders and decoders: @ServerEndpoint(value="/book", encoders={MyEncoder.class}, decoders = {MyDecoder.class} ) public class BookCollection { @OnMessage public void onMessage(Book book,Session session) { try { session.getBasicRemote().sendObject(book); } catch (Exception ex) { ex.printStackTrace(); } } @OnOpen public void onOpen(Session session) { System.out.println("Opening socket" +session.getBasicRemote() ); } @OnClose public void onClose(Session session) { System.out.println("Closing socket" + session.getBasicRemote()); } } In the preceding code snippet, you can see the class BookCollection is annotated with @ServerEndpoint. The value=/book attribute provides URI mapping for the endpoint. The @ServerEndpoint also takes the encoders and decoders to be used during the WebSocket transmission. Once a WebSocket connection has been established, a session is created and the method annotated with @OnOpen will be called. When the WebSocket endpoint receives a message, the method annotated with @OnMessage will be called. In our sample the method simply sends the book object using the Session.getBasicRemote() which will get a reference to the RemoteEndpoint and send the message synchronously. Encoders can be used to convert a custom user-defined object in a text message, TextStream, BinaryStream, or BinaryMessage format. An implementation of an encoder class for text messages is as follows: public class MyEncoder implements Encoder.Text<Book> { @Override public String encode(Book book) throws EncodeException { return book.getJson().toString(); } } As shown in the preceding code, the encoder class implements Encoder.Text<Book>. There is an encode method that is overridden and which converts a book and sends it as a JSON string. (More on JSON APIs is covered in detail in the next chapter) Decoders can be used to decode WebSockets messages in custom user-defined objects. They can decode in text, TextStream, and binary or BinaryStream format. Here is a code for a decoder class: public class MyDecoder implements Decoder.Text<Book> { @Override public Book decode(String string) throws DecodeException { javax.json.JsonObject jsonObject = javax.json.Json.createReader(new StringReader(string)).readObject(); return new Book(jsonObject); } @Override public boolean willDecode(String string) { try { javax.json.Json.createReader(new StringReader(string)).readObject(); return true; } catch (Exception ex) { } return false; } In the preceding code snippet, the Decoder.Text needs two methods to be overridden. The willDecode() method checks if it can handle this object and decode it. The decode() method decodes the string into an object of type Book by using the JSON-P API javax.json.Json.createReader(). The following code snippet shows the user-defined class Book: public class Book { public Book() {} JsonObject jsonObject; public Book(JsonObject json) { this.jsonObject = json; } public JsonObject getJson() { return jsonObject; } public void setJson(JsonObject json) { this.jsonObject = json; } public Book(String message) { jsonObject = Json.createReader(new StringReader(message)).readObject(); } public String toString () { StringWriter writer = new StringWriter(); Json.createWriter(writer).write(jsonObject); return writer.toString(); } } The Book class is a user-defined class that takes the JSON object sent by the client. Here is an example of how the JSON details are sent to the WebSockets endpoints from JavaScript. var json = JSON.stringify({ "name": "Java 7 JAX-WS Web Services", "author":"Deepak Vohra", "isbn": "123456789" }); function addBook() { websocket.send(json); } The client sends the message using websocket.send() which will cause the onMessage() of the BookCollection.java to be invoked. The BookCollection.java will return the same book to the client. In the process, the decoder will decode the WebSockets message when it is received. To send back the same Book object, first the encoder will encode the Book object to a WebSockets message and send it to the client. The Java WebSocket Client API WebSockets and Server-sent Events , covered the Java WebSockets client API. Any POJO can be transformed into a WebSockets client by annotating it with @ClientEndpoint. Additionally the user can add encoders and decoders attributes to the @ClientEndpoint annotation to encode application objects into WebSockets messages and WebSockets messages into application objects. The following table shows the @ClientEndpoint annotation and its attributes: Annotation Attribute Description @ClientEndpoint   This class-level annotation signifies that the Java class is a WebSockets client that will connect to a WebSockets server endpoint.   value The value is the URI with a leading /.   encoders The encoders contain a list of Java classes that act as encoders for the endpoint. The classes must implement the encoder interface.   decoders The decoders contain a list of Java classes that act as decoders for the endpoint. The classes must implement the decoder interface.   configurator The configurator attribute allows the developer to plug in their implementation of ClientEndpoint.Configurator, which is used when configuring the client endpoint.   subprotocols The sub protocols attribute contains a list of sub protocols that the endpoint can support. Sending different kinds of message data: blob/binary Using JavaScript we can traditionally send JSON or XML as strings. However, HTML5 allows applications to work with binary data to improve performance. WebSockets supports two kinds of binary data Binary Large Objects (blob) arraybuffer A WebSocket can work with only one of the formats at any given time. Using the binaryType property of a WebSocket, you can switch between using blob or arraybuffer: websocket.binaryType = "blob"; // receive some blob data websocket.binaryType = "arraybuffer"; // now receive ArrayBuffer data The following code snippet shows how to display images sent by a server using WebSockets. Here is a code snippet for how to send binary data with WebSockets: websocket.binaryType = 'arraybuffer'; The preceding code snippet sets the binaryType property of websocket to arraybuffer. websocket.onmessage = function(msg) { var arrayBuffer = msg.data; var bytes = new Uint8Array(arrayBuffer); var image = document.getElementById('image'); image.src = 'data:image/png;base64,'+encode(bytes); } When the onmessage is called the arrayBuffer is initialized to the message.data. The Uint8Array type represents an array of 8-bit unsigned integers. The image.src value is in line using the data URI scheme. Security and WebSockets WebSockets are secured using the web container security model. A WebSockets developer can declare whether the access to the WebSocket server endpoint needs to be authenticated, who can access it, or if it needs an encrypted connection. A WebSockets endpoint which is mapped to a ws:// URI is protected under the deployment descriptor with http:// URI with the same hostname,port path since the initial handshake is from the HTTP connection. So, WebSockets developers can assign an authentication scheme, user roles, and a transport guarantee to any WebSockets endpoints. We will take the same sample as we saw in , WebSockets and Server-sent Events , and make it a secure WebSockets application. Here is the web.xml for a secure WebSocket endpoint: <web-app version="3.0" xsi_schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"> <security-constraint> <web-resource-collection> <web-resource-name>BookCollection</web-resource-name> <url-pattern>/index.jsp</url-pattern> <http-method>PUT</http-method> <http-method>POST</http-method> <http-method>DELETE</http-method> <http-method>GET</http-method> </web-resource-collection> <user-data-constraint> <description>SSL</description> <transport-guarantee>CONFIDENTIAL</transport-guarantee> </user-data-constraint> </security-constraint> </web-app> As you can see in the preceding snippet, we used <transport-guarantee>CONFIDENTIAL</transport-guarantee>. The Java EE specification followed by application servers provides different levels of transport guarantee on the communication between clients and application server. The three levels are: Data Confidentiality (CONFIDENTIAL) : We use this level to guarantee that all communication between client and server goes through the SSL layer and connections won't be accepted over a non-secure channel. Data Integrity (INTEGRAL) : We can use this level when a full encryption is not required but we want our data to be transmitted to and from a client in such a way that, if anyone changed the data, we could detect the change. Any type of connection (NONE) : We can use this level to force the container to accept connections on HTTP and HTTPs. The following steps should be followed for setting up SSL and running our sample to show a secure WebSockets application deployed in Glassfish. Generate the server certificate: keytool -genkey -alias server-alias -keyalg RSA -keypass changeit --storepass changeit -keystore keystore.jks Export the generated server certificate in keystore.jks into the file server.cer: keytool -export -alias server-alias -storepass changeit -file server.cer -keystore keystore.jks Create the trust-store file cacerts.jks and add the server certificate to the trust store: keytool -import -v -trustcacerts -alias server-alias -file server.cer -keystore cacerts.jks -keypass changeit -storepass changeit Change the following JVM options so that they point to the location and name of the new keystore. Add this in domain.xml under java-config: <jvm-options>-Djavax.net.ssl.keyStore=${com.sun.aas.instanceRoot}/config/keystore.jks</jvm-options> <jvm-options>-Djavax.net.ssl.trustStore=${com.sun.aas.instanceRoot}/config/cacerts.jks</jvm-options> Restart GlassFish. If you go to https://localhost:8181/helloworld-ws/, you can see the secure WebSocket application. Here is how the the headers look under Chrome Developer Tools: Open Chrome Browser and click on View and then on Developer Tools . Click on Network . Select book under element name and click on Frames . As you can see in the preceding screenshot, since the application is secured using SSL the WebSockets URI, it also contains wss://, which means WebSockets over SSL. So far we have seen the encoders and decoders for WebSockets messages. We also covered how to send binary data using WebSockets. Additionally we have demonstrated a sample on how to secure WebSockets based application. We shall now cover the best practices for WebSocket based-applications.
Read more
  • 0
  • 0
  • 18374
article-image-introduction-clustering-and-unsupervised-learning
Packt
23 Feb 2016
16 min read
Save for later

Introduction to Clustering and Unsupervised Learning

Packt
23 Feb 2016
16 min read
The act of clustering, or spotting patterns in data, is not much different from spotting patterns in groups of people. In this article, you will learn: The ways clustering tasks differ from the classification tasks How clustering defines a group, and how such groups are identified by k-means, a classic and easy-to-understand clustering algorithm The steps needed to apply clustering to a real-world task of identifying marketing segments among teenage social media users Before jumping into action, we'll begin by taking an in-depth look at exactly what clustering entails. (For more resources related to this topic, see here.) Understanding clustering Clustering is an unsupervised machine learning task that automatically divides the data into clusters, or groups of similar items. It does this without having been told how the groups should look ahead of time. As we may not even know what we're looking for, clustering is used for knowledge discovery rather than prediction. It provides an insight into the natural groupings found within data. Without advance knowledge of what comprises a cluster, how can a computer possibly know where one group ends and another begins? The answer is simple. Clustering is guided by the principle that items inside a cluster should be very similar to each other, but very different from those outside. The definition of similarity might vary across applications, but the basic idea is always the same—group the data so that the related elements are placed together. The resulting clusters can then be used for action. For instance, you might find clustering methods employed in the following applications: Segmenting customers into groups with similar demographics or buying patterns for targeted marketing campaigns Detecting anomalous behavior, such as unauthorized network intrusions, by identifying patterns of use falling outside the known clusters Simplifying extremely large datasets by grouping features with similar values into a smaller number of homogeneous categories Overall, clustering is useful whenever diverse and varied data can be exemplified by a much smaller number of groups. It results in meaningful and actionable data structures that reduce complexity and provide insight into patterns of relationships. Clustering as a machine learning task Clustering is somewhat different from the classification, numeric prediction, and pattern detection tasks we examined so far. In each of these cases, the result is a model that relates features to an outcome or features to other features; conceptually, the model describes the existing patterns within data. In contrast, clustering creates new data. Unlabeled examples are given a cluster label that has been inferred entirely from the relationships within the data. For this reason, you will, sometimes, see the clustering task referred to as unsupervised classification because, in a sense, it classifies unlabeled examples. The catch is that the class labels obtained from an unsupervised classifier are without intrinsic meaning. Clustering will tell you which groups of examples are closely related—for instance, it might return the groups A, B, and C—but it's up to you to apply an actionable and meaningful label. To see how this impacts the clustering task, let's consider a hypothetical example. Suppose you were organizing a conference on the topic of data science. To facilitate professional networking and collaboration, you planned to seat people in groups according to one of three research specialties: computer and/or database science, math and statistics, and machine learning. Unfortunately, after sending out the conference invitations, you realize that you had forgotten to include a survey asking which discipline the attendee would prefer to be seated with. In a stroke of brilliance, you realize that you might be able to infer each scholar's research specialty by examining his or her publication history. To this end, you begin collecting data on the number of articles each attendee published in computer science-related journals and the number of articles published in math or statistics-related journals. Using the data collected for several scholars, you create a scatterplot: As expected, there seems to be a pattern. We might guess that the upper-left corner, which represents people with many computer science publications but few articles on math, could be a cluster of computer scientists. Following this logic, the lower-right corner might be a group of mathematicians. Similarly, the upper-right corner, those with both math and computer science experience, may be machine learning experts. Our groupings were formed visually; we simply identified clusters as closely grouped data points. Yet in spite of the seemingly obvious groupings, we unfortunately have no way to know whether they are truly homogeneous without personally asking each scholar about his/her academic specialty. The labels we applied required us to make qualitative, presumptive judgments about the types of people that would fall into the group. For this reason, you might imagine the cluster labels in uncertain terms, as follows: Rather than defining the group boundaries subjectively, it would be nice to use machine learning to define them objectively. This might provide us with a rule in the form if a scholar has few math publications, then he/she is a computer science expert. Unfortunately, there's a problem with this plan. As we do not have data on the true class value for each point, a supervised learning algorithm would have no ability to learn such a pattern, as it would have no way of knowing what splits would result in homogenous groups. On the other hand, clustering algorithms use a process very similar to what we did by visually inspecting the scatterplot. Using a measure of how closely the examples are related, homogeneous groups can be identified. In the next section, we'll start looking at how clustering algorithms are implemented. This example highlights an interesting application of clustering. If you begin with unlabeled data, you can use clustering to create class labels. From there, you could apply a supervised learner such as decision trees to find the most important predictors of these classes. This is called semi-supervised learning. The k-means clustering algorithm The k-means algorithm is perhaps the most commonly used clustering method. Having been studied for several decades, it serves as the foundation for many more sophisticated clustering techniques. If you understand the simple principles it uses, you will have the knowledge needed to understand nearly any clustering algorithm in use today. Many such methods are listed on the following site, the CRAN Task View for clustering at http://cran.r-project.org/web/views/Cluster.html. As k-means has evolved over time, there are many implementations of the algorithm. One popular approach is described in : Hartigan JA, Wong MA. A k-means clustering algorithm. Applied Statistics. 1979; 28:100-108. Even though clustering methods have advanced since the inception of k-means, this is not to imply that k-means is obsolete. In fact, the method may be more popular now than ever. The following table lists some reasons why k-means is still used widely: Strengths Weaknesses Uses simple principles that can be explained in non-statistical terms Highly flexible, and can be adapted with simple adjustments to address nearly all of its shortcomings Performs well enough under many real-world use cases Not as sophisticated as more modern clustering algorithms Because it uses an element of random chance, it is not guaranteed to find the optimal set of clusters Requires a reasonable guess as to how many clusters naturally exist in the data Not ideal for non-spherical clusters or clusters of widely varying density The k-means algorithm assigns each of the n examples to one of the k clusters, where k is a number that has been determined ahead of time. The goal is to minimize the differences within each cluster and maximize the differences between the clusters. Unless k and n are extremely small, it is not feasible to compute the optimal clusters across all the possible combinations of examples. Instead, the algorithm uses a heuristic process that finds locally optimal solutions. Put simply, this means that it starts with an initial guess for the cluster assignments, and then modifies the assignments slightly to see whether the changes improve the homogeneity within the clusters. We will cover the process in depth shortly, but the algorithm essentially involves two phases. First, it assigns examples to an initial set of k clusters. Then, it updates the assignments by adjusting the cluster boundaries according to the examples that currently fall into the cluster. The process of updating and assigning occurs several times until changes no longer improve the cluster fit. At this point, the process stops and the clusters are finalized. Due to the heuristic nature of k-means, you may end up with somewhat different final results by making only slight changes to the starting conditions. If the results vary dramatically, this could indicate a problem. For instance, the data may not have natural groupings or the value of k has been poorly chosen. With this in mind, it's a good idea to try a cluster analysis more than once to test the robustness of your findings. To see how the process of assigning and updating works in practice, let's revisit the case of the hypothetical data science conference. Though this is a simple example, it will illustrate the basics of how k-means operates under the hood. Using distance to assign and update clusters As with k-NN, k-means treats feature values as coordinates in a multidimensional feature space. For the conference data, there are only two features, so we can represent the feature space as a two-dimensional scatterplot as depicted previously. The k-means algorithm begins by choosing k points in the feature space to serve as the cluster centers. These centers are the catalyst that spurs the remaining examples to fall into place. Often, the points are chosen by selecting k random examples from the training dataset. As we hope to identify three clusters, according to this method, k = 3 points will be selected at random. These points are indicated by the star, triangle, and diamond in the following diagram: It's worth noting that although the three cluster centers in the preceding diagram happen to be widely spaced apart, this is not always necessarily the case. Since they are selected at random, the three centers could have just as easily been three adjacent points. As the k-means algorithm is highly sensitive to the starting position of the cluster centers, this means that random chance may have a substantial impact on the final set of clusters. To address this problem, k-means can be modified to use different methods for choosing the initial centers. For example, one variant chooses random values occurring anywhere in the feature space (rather than only selecting among the values observed in the data). Another option is to skip this step altogether; by randomly assigning each example to a cluster, the algorithm can jump ahead immediately to the update phase. Each of these approaches adds a particular bias to the final set of clusters, which you may be able to use to improve your results. In 2007, an algorithm called k-means++ was introduced, which proposes an alternative method for selecting the initial cluster centers. It purports to be an efficient way to get much closer to the optimal clustering solution while reducing the impact of random chance. For more information, refer to Arthur D, Vassilvitskii S. k-means++: The advantages of careful seeding. Proceedings of the eighteenth annual ACM-SIAM symposium on discrete algorithms. 2007:1027–1035. After choosing the initial cluster centers, the other examples are assigned to the cluster center that is nearest according to the distance function. You will remember that we studied distance functions while learning about k-Nearest Neighbors. Traditionally, k-means uses Euclidean distance, but Manhattan distance or Minkowski distance are also sometimes used. Recall that if n indicates the number of features, the formula for Euclidean distance between example x and example y is: For instance, if we are comparing a guest with five computer science publications and one math publication to a guest with zero computer science papers and two math papers, we could compute this in R as follows: > sqrt((5 - 0)^2 + (1 - 2)^2) [1] 5.09902 Using this distance function, we find the distance between each example and each cluster center. The example is then assigned to the nearest cluster center. Keep in mind that as we are using distance calculations, all the features need to be numeric, and the values should be normalized to a standard range ahead of time. As shown in the following diagram, the three cluster centers partition the examples into three segments labeled Cluster A, Cluster B, and Cluster C. The dashed lines indicate the boundaries for the Voronoi diagram created by the cluster centers. The Voronoi diagram indicates the areas that are closer to one cluster center than any other; the vertex where all the three boundaries meet is the maximal distance from all three cluster centers. Using these boundaries, we can easily see the regions claimed by each of the initial k-means seeds: Now that the initial assignment phase has been completed, the k-means algorithm proceeds to the update phase. The first step of updating the clusters involves shifting the initial centers to a new location, known as the centroid, which is calculated as the average position of the points currently assigned to that cluster. The following diagram illustrates how as the cluster centers shift to the new centroids, the boundaries in the Voronoi diagram also shift and a point that was once in Cluster B (indicated by an arrow) is added to Cluster A: As a result of this reassignment, the k-means algorithm will continue through another update phase. After shifting the cluster centroids, updating the cluster boundaries, and reassigning points into new clusters (as indicated by arrows), the figure looks like this: Because two more points were reassigned, another update must occur, which moves the centroids and updates the cluster boundaries. However, because these changes result in no reassignments, the k-means algorithm stops. The cluster assignments are now final: The final clusters can be reported in one of the two ways. First, you might simply report the cluster assignments such as A, B, or C for each example. Alternatively, you could report the coordinates of the cluster centroids after the final update. Given either reporting method, you are able to define the cluster boundaries by calculating the centroids or assigning each example to its nearest cluster. Choosing the appropriate number of clusters In the introduction to k-means, we learned that the algorithm is sensitive to the randomly-chosen cluster centers. Indeed, if we had selected a different combination of three starting points in the previous example, we may have found clusters that split the data differently from what we had expected. Similarly, k-means is sensitive to the number of clusters; the choice requires a delicate balance. Setting k to be very large will improve the homogeneity of the clusters, and at the same time, it risks overfitting the data. Ideally, you will have a priori knowledge (a prior belief) about the true groupings and you can apply this information to choosing the number of clusters. For instance, if you were clustering movies, you might begin by setting k equal to the number of genres considered for the Academy Awards. In the data science conference seating problem that we worked through previously, k might reflect the number of academic fields of study that were invited. Sometimes the number of clusters is dictated by business requirements or the motivation for the analysis. For example, the number of tables in the meeting hall could dictate how many groups of people should be created from the data science attendee list. Extending this idea to another business case, if the marketing department only has resources to create three distinct advertising campaigns, it might make sense to set k = 3 to assign all the potential customers to one of the three appeals. Without any prior knowledge, one rule of thumb suggests setting k equal to the square root of (n / 2), where n is the number of examples in the dataset. However, this rule of thumb is likely to result in an unwieldy number of clusters for large datasets. Luckily, there are other statistical methods that can assist in finding a suitable k-means cluster set. A technique known as the elbow method attempts to gauge how the homogeneity or heterogeneity within the clusters changes for various values of k. As illustrated in the following diagrams, the homogeneity within clusters is expected to increase as additional clusters are added; similarly, heterogeneity will also continue to decrease with more clusters. As you could continue to see improvements until each example is in its own cluster, the goal is not to maximize homogeneity or minimize heterogeneity, but rather to find k so that there are diminishing returns beyond that point. This value of k is known as the elbow point because it looks like an elbow. There are numerous statistics to measure homogeneity and heterogeneity within the clusters that can be used with the elbow method (the following information box provides a citation for more detail). Still, in practice, it is not always feasible to iteratively test a large number of k values. This is in part because clustering large datasets can be fairly time consuming; clustering the data repeatedly is even worse. Regardless, applications requiring the exact optimal set of clusters are fairly rare. In most clustering applications, it suffices to choose a k value based on convenience rather than strict performance requirements. For a very thorough review of the vast assortment of cluster performance measures, refer to: Halkidi M, Batistakis Y, Vazirgiannis M. On clustering validation techniques. Journal of Intelligent Information Systems. 2001; 17:107-145. The process of setting k itself can sometimes lead to interesting insights. By observing how the characteristics of the clusters change as k is varied, one might infer where the data have naturally defined boundaries. Groups that are more tightly clustered will change a little, while less homogeneous groups will form and disband over time. In general, it may be wise to spend little time worrying about getting k exactly right. The next example will demonstrate how even a tiny bit of subject-matter knowledge borrowed from a Hollywood film can be used to set k such that actionable and interesting clusters are found. As clustering is unsupervised, the task is really about what you make of it; the value is in the insights you take away from the algorithm's findings. Summary This article covered only the fundamentals of clustering. As a very mature machine learning method, there are many variants of the k-means algorithm as well as many other clustering algorithms that bring unique biases and heuristics to the task. Based on the foundation in this article, you will be able to understand and apply other clustering methods to new problems. To learn more about different machine learning techniques, the following books published by Packt Publishing (https://www.packtpub.com/) are recommended: Learning Data Mining with R (https://www.packtpub.com/big-data-and-business-intelligence/learning-data-mining-r) Mastering Scientific Computing with R (https://www.packtpub.com/application-development/mastering-scientific-computing-r) R for Data Science (https://www.packtpub.com/big-data-and-business-intelligence/r-data-science) Resources for Article:   Further resources on this subject: Displaying SQL Server Data using a Linq Data Source [article] Probability of R? [article] Working with Commands and Plugins [article]
Read more
  • 0
  • 0
  • 18351

article-image-installing-panda3d
Packt
11 Feb 2011
4 min read
Save for later

Installing Panda3D

Packt
11 Feb 2011
4 min read
Getting started with Panda3D installation packages The kind folks who produce Panda3D have made it very easy to get Panda3D up and working. You don't need to worry about any compiling, library linking, or other difficult, multi-step processes. The Panda3D website provides executable files that take care of all the work for you. These files even install the version of Python they need to operate correctly, so you don't need to go elsewhere for it. Time for action - downloading and installing Panda3D I know what you're thinking: "Less talk, more action!" Here are the step-by-step instructions for installing Panda3D: Navigate your web browser to www.Panda3D.org. Under the Downloads option, you'll see a link labeled SDK. Click it. If you are using Windows, scroll down this page you'll find a section titled Download other versions. Find the link to Panda3D SDK 1.6.2 and click it. If you aren't using Windows, click on the platform you are using (Mac, Linux, or any other OS.). That will take you to a page that has the downloads for that platform. Scroll down to the Download other versions section and find the link to Panda3D SDK 1.6.2, as before. When the download is complete, run the file and this screen will pop up: Click Next to continue and then accept the terms. After that, you'll be prompted about where you want to install Panda3D. The default location is just fine. Click the Install button to continue. Wait for the progress bar to fill up. When it's done, you'll see another prompt. This step really isn't necessary. Just click No and move on. When you have finished the installation, you can verify that it's working by going to Start Menu | All Programs | Panda3D 1.6.2 | Sample Programs | Ball in Maze | Run Ball in Maze. A window will open, showing the Ball in Maze sample game, where you tilt a maze to make a ball roll around while trying to avoid the holes. What just happened? You may be wondering why we skipped a part of the installation during step 7. That step of the process caches some of the assets, like 3D models and such that come with Panda3D. Essentially, by spending a few minutes caching these files now, the sample programs that come with Panda3d will load a few seconds faster the first time we run them, that's all. Now that we've got Panda3D up and running let's get ourselves an advanced text editor to do our coding in. Switching to an advanced text editor The next thing we need is Notepad++. Why, you ask? Well, to code with Python all you really need is a text editor, like the notepad that comes with Windows XP. After typing your code you just have to save the file with .py extension. Notepad itself is kind of dull, though, and it doesn't have many features to make coding easier. Notepad++ is a text editor very similar to Notepad. It can open pretty much any text file and it comes with a pile of features to make coding easier. To highlight some fan favorites, it provides language mark-up, a Find and Replace feature, and file tabs to organize multiple open files. The language mark-up will change the color and fonts of specific parts of your code to help you visually understand and organize it. With Find and Replace you can easily change a large number of variable names and also quickly and easily update code. File tabbing keeps all of your open code files in one window and makes it easy to switch back and forth between them.
Read more
  • 0
  • 0
  • 18340

article-image-exploring-new-reality-oculus-rift
Packt
21 Dec 2016
7 min read
Save for later

Exploring a New Reality with the Oculus Rift

Packt
21 Dec 2016
7 min read
In this article by Jack Donovan the author of the book Mastering Oculus Rift Development explains about virtual reality. What made you feel like you were truly immersed in a game world for the first time? Was it graphics that looked impressively realistic, ambient noise that perfectly captured the environment and mood, or the way the game's mechanics just started to feel like a natural reflex? Game developers constantly strive to replicate scenarios that are as real and as emotionally impactful as possible, and they've never been as close as they are now with the advent of virtual reality. Virtual reality has been a niche market since the early 1950s, often failing to evoke a meaningful sense of presence that the concept hinges on—that is, until the first Oculus Rift prototype was designed in 2010 by Oculus founder Palmer Luckey. The Oculus Rift proved that modern rendering and display technology was reaching a point that an immersive virtual reality could be achieved, and that's when the new era of VR development began. Today, virtual reality development is as accessible as ever, comprehensively supported in the most popular off-the-shelf game development engines such as Unreal Engine and Unity 3D. In this article, you'll learn all of the essentials that go into a complete virtual reality experience, and master the techniques that will enable you to bring any idea you have into VR. This article will cover everything you need to know to get started with virtual reality, including the following points: The concept of virtual reality The importance of intent in VR design Common limitations of VR games (For more resources related to this topic, see here.) The concept of virtual reality Virtual reality has taken many forms and formats since its inception, but this article will be focused on modern virtual reality experienced with a Head-Mounted Display (HMD). HMDs like the Oculus Rift are typically treated like an extra screen attached to your computer (more on that later) but with some extra components that enable it to capture its own orientation (and position, in some cases). This essentially amounts to a screen that sits on your head and knows how it moves, so it can mirror your head movements in the VR experience and enable you to look around. In the following example from the Oculus developer documentation, you can see how the HMD translates this rotational data into the game world: Depth perception Depth perception is another big principle of VR. Because the display of the HMD is always positioned right in front of the user's eyes, the rendered image is typically split into two images: one per eye, with each individual image rendered from the position of that eye. You can observe the difference between normal rendering and VR rendering in the following two images. This first image is how normal 3D video games are rendered to a computer screen, created based on the position and direction of a virtual camera in the game world: This next image shows how VR scenes are rendered, using a different virtual camera for each eye to create a stereoscopic depth effect: Common limitations of VR games While virtual reality provides the ability to immerse a player's senses like never before, it also creates some new, unique problems that must be addressed by responsible VR developers. Locomotion sickness Virtual reality headsets are meant to make you feel like you're somewhere else, and it only makes sense that you'd want to be able to explore that somewhere. Unfortunately, common game mechanics like traditional joystick locomotion are problematic for VR. Our inner ears and muscular system are accustomed to sensing inertia while we move from place to place, so if you were to push a joystick forward to walk in virtual reality, your body would get confused when it sensed that you're still in a chair. Typically when there's a mismatch between what we're seeing and what we're feeling, our bodies assume that a nefarious poison or illness is at work, and they prepare to rid the body of the culprit; that's the motion sickness you feel when reading in a car, standing on a boat, and yes, moving in virtual reality. This doesn't mean that we have to prevent users from moving in VR, we just might want to be more clever about it—more on that later. The primary cause of nausea with traditional joystick movement in VR is acceleration; your brain gets confused when picking up speed or slowing down, but not so much when it's moving at a constant rate (think of being stationary in a car that's moving at a constant speed). Rotation can get even more complicated, because rotating smoothly even at a constant speed causes nausea. Some developers get around this by using hard increments instead of gradual acceleration, such as rotating in 30 degree "snaps" once per second instead of rotating smoothly. Lack of real-world vision One of the potentially clumsiest aspects of virtual reality is getting your hands where they need to be without being able to see them. Whether you're using a gamepad, keyboard, or motion controller, you'll likely need to use your hands to interact with VR—which you can't see with an HMD sitting over your eyes. It's good practice to centralize input around resting positions (i.e. the buttons naturally closest to your thumbs on a gamepad or the home row of a computer keyboard), but shy away from anything that requires complex precise input, like writing sentences on a keyboard or hitting button combos on a controller. Some VR headsets, such as the HTC Vive, have a forward-facing camera (sometimes called a passthrough camera) that users can choose to view in VR, enabling basic interaction with the real world without taking the headset off. The Oculus Rift doesn't feature a built-in camera, but you could still display the feed from an external camera on any surface in virtual reality. Even before modern VR, developers were creating applications that overlay smart information over what a camera is seeing; that's called augmented reality (AR). Experiences that ride the line between camera output and virtual environments are called mixed reality (MR). Unnatural head movements You may not have thought about it before, but looking around in a traditional first-person shooter (FPS) is quite different than looking around using your head. The right analog stick is typically used to direct the camera and make adjustments as necessary, but in VR, players actually move their head instead of using their thumb to move their virtual head. Don't expect players in VR to be able to make the same snappy pivots and 180-degree turns on a dime that are trivial in a regular console game. Summary In this article, we approached the topic of virtual reality from a fundamental level. The HMD is the crux of modern VR simulation, and it uses motion tracking components as well as peripherals like the constellation system to create immersive experiences that transport the player into a virtual world. Now that we've scratched the surface of the hardware, development techniques and use cases of virtual reality—particularly the Oculus Rift—you're probably beginning to think about what you'd like to create in virtual reality yourself Resources for Article: Further resources on this subject: Cardboard is Virtual Reality for Everyone [article] Virtually Everything for Everyone [article] Customizing the Player Character [article]
Read more
  • 0
  • 0
  • 18314
article-image-understanding-python-regex-engine
Packt
18 Feb 2014
8 min read
Save for later

Understanding the Python regex engine

Packt
18 Feb 2014
8 min read
(For more resources related to this topic, see here.) These are the most common characteristics of the algorithm: It supports "lazy quantifiers" such as *?, +?, and ??. It matches the first coincidence, even though there are longer ones in the string. >>>re.search("engineer | engineering", "engineering").group()'engineer' This also means that order is important. The algorithm tracks only one transition at one step, which means that the engine checks one character at a time. Backreferences and capturing parentheses are supported. Backtracking is the ability to remember the last successful position so that it can go back and retry if needed In the worst case, complexity is exponential O(Cn). We'll see this later in Backtracking. Backtracking Backtracking allows going back and repeating the different paths of the regular expression. It does so by remembering the last successful position, this applies to alternation and quantifiers, let’s see an example: Backtracking As we see in the image the regex engine tries to match one character at a time until it fails and starts again with the following path it can retry. The regex used in the image is the perfect example of the importance of how the regex is built, in this case the expression can be rebuild as spa (in | niard), so that the regex engine doesn’t have to go back up to the start of the string in order to retry the second alternative. This leads us to what is called catastrophic backtracking a well-known problem with backtracking that can give you several problems, ranging from slow regex to a crash with a stack overflow. In the previous example, you can see that the behavior grows not only with the input but also with the different paths in the regex, that’s why the algorithm is exponential O(Cn), with this in mind it’s easy to understand why we can end up with a stack overflow. The problem arises when the regex fails to match the String. Let’s benchmark a regex with technique we’ve seen previously, so we can understand the problem better. First let’s try a simple regex: >>> def catastrophic(n): print "Testing with %d characters" %n pat = re.compile('(a+)+c') text = "%s" %('a' * n) pat.search(text) As you can see the text we’re trying to match it’s always going to fail due to there is no c at the end. Let’s test it with different inputs. >>> for n in range(20, 30): test(catastrophic, n) Testing with 20 characters The function catastrophic lasted: 0.130457 Testing with 21 characters The function catastrophic lasted: 0.245125 …… The function catastrophic lasted: 14.828221 Testing with 28 characters The function catastrophic lasted: 29.830929 Testing with 29 characters The function catastrophic lasted: 61.110949 The behavior of this regex looks like quadratic. But why? what’s happening here? The problem is that (a+) starts greedy, so it tries to get as many a’s as possible and after that it fails to match the (a+)+, that is, it backtracks to the second a, and continue consuming a’s until it fails to match c, when it tries again (backtrack) the whole process starting with the second a. Let’s see another example, in this case with an exponential behavior: >>> def catastrophic(n): print "Testing with %d characters" %n pat = re.compile('(x+)+(b+)+c') text = 'x' * n text += 'b' * n pat.search(text) >>> for n in range(12, 18): test(catastrophic, n) Testing with 12 characters The function catastrophic lasted: 1.035162 Testing with 13 characters The function catastrophic lasted: 4.084714 Testing with 14 characters The function catastrophic lasted: 16.319145 Testing with 15 characters The function catastrophic lasted: 65.855182 Testing with 16 characters The function catastrophic lasted: 276.941307 As you can see the behavior is exponential, which can lead to a catastrophic scenarios. And finally let’s see what happen when regex has a match. >>> def non_catastrophic(n): print "Testing with %d characters" %n pat = re.compile('(x+)+(b+)+c') text = 'x' * n text += 'b' * n text += 'c' pat.search(text) >>> for n in range(12, 18): test(non_catastrophic, n) Testing with 10 characters The function catastrophic lasted: 0.000029 …… Testing with 19 characters The function catastrophic lasted: 0.000012 Optimization recommendations In the following sections we will find a number of recommendations that could be used to apply to improve regular expressions. The best tool will always be the common sense, and even following these recommendations common sense will need to be used. It has to be understood when the recommendation is applicable and when not. For instance the recommendation don’t be greedy cannot be used in the 100% of the cases. Reuse compiled patterns To use a regular expression we have to convert it from the string representation to a compiled form as RegexObject. This compilation takes some time. If instead of using the compile function we are using the rest of the methods to avoid the creation of the RegexObject, we should understand that the compilation is executed anyway and a number of compiled RegexObject are cached automatically. However, when we are compiling that cache won’t back us. Every single compile execution will consume an amount of time that perhaps could be negligible for a single execution, but it’s definitely relevant if many executions are performed. Extract common parts in alternation Alternation is always a performance risk point in regular expressions. When using them in Python, and therefore in a sort of NFA implementation, we should extract any common part outside of the alternation. For instance if we have /(Hello⇢World|Hello⇢Continent|Hello⇢Country,)/, we could easily extract Hello⇢ having the following expression /Hello⇢(World|Continent|Country)/. This will make our engine to just check Hello⇢ once, and not going back and recheck for each possibility. Shortcut the alternation Ordering in alternation is relevant; each of the different options present in the alternation will be checked one by one, from the left to the right. This can be used in favor of performance. If we place the more likely options at the beginning of the alternation, more checks will mark the alternation as matched sooner. For instance, we know that the more common colors of cars are white and black. If we are writing a regular expression accepting some colors, we should put white and black first, as those are the more likely to appear. This is: /(white|black|red|blue|green)/. For the rest of the elements, if they have the very same odds of appearing, if could be favorable to put the shortest ones before the longer ones. Use non capturing groups when appropriate Capturing groups will consume some time per each group defined in an expression. This time is not very important but is still relevant if we are executing a regular expression many times. Sometimes we are using groups but we might not be interested in the result. For instance when using alternation. If that is the case we can save some execution time to the engine by marking that group as non-capturing. This is: (?:person|company).. Be specific When the patterns we define are very specific, the engine can help us performing quick integrity checks before the actual pattern matching is executed. For instance, if we pass to the engine the expression /w{15}/ to be matched against the text hello, the engine could decide to check if the input string is actually at least 15 characters long instead of matching the expression. Don’t be greedy The quantifiers and we learnt the difference between greedy and reluctant quantifiers. We also found that the quantifiers are greedy by default. What does this mean to performance? It means that the engine will always try to catch as many characters as possible and then reducing the scope step by step until it's done. This could potentially make the regular expression slow if the match is typically short. Keep in mind, however, this is only applicable if the match is usually short. Summary In this article, we understood how to see the engine working behind the scenes. We learned some theory of the engine design and how it's easy to fall in a common pitfall—the catastrophic backtracking. Finally, we reviewed different general recommendations to improve the performance of our regular expressions. Resources for Article: Further resources on this subject: Python LDAP Applications: Part 1 - Installing and Configuring the Python-LDAP Library and Binding to an LDAP Directory [Article] Python Data Persistence using MySQL Part III: Building Python Data Structures Upon the Underlying Database Data [Article] Python Testing: Installing the Robot Framework [Article]
Read more
  • 0
  • 0
  • 18289

article-image-using-protocols-and-protocol-extensions
Packt
13 Oct 2015
22 min read
Save for later

Using Protocols and Protocol Extensions

Packt
13 Oct 2015
22 min read
In this article by Jon Hoffman, the author of Mastering Swift 2, we'll see how protocols are used as a type, how we can implement polymorphism in Swift using protocols, how to use protocol extensions, and why we would want to use protocol extensions. (For more resources related to this topic, see here.) While watching the presentations from WWDC 2015 about protocol extensions and protocol-oriented programming, I will admit that I was very skeptical. I have worked with object-oriented programming for so long that I was unsure if this new programming paradigm would solve all of the problems that Apple was claiming it would. Since I am not one to let my skepticism get in the way of trying something new, I set up a new project that mirrored the one I was currently working on, but wrote the code using Apple's recommendations for protocol-oriented programming and used protocol extensions extensively in the code. I can honestly say that I was amazed with how much cleaner the new project was compared to the original one. I believe that protocol extensions is going to be one of those defining features that set one programming language apart from the rest. I also believe that many major languages will soon have similar features. Protocol extensions are the backbone for Apple's new protocol-oriented programming paradigm and is arguably one of the most important additions to the Swift programming language. With protocol extensions, we are able to provide method and property implementations to any type that conforms to a protocol. To really understand how useful protocols and protocol extensions are, let's get a better understanding of protocols. While classes, structs, and enums can all conform to protocols in Swift, for this article, we will be focusing on classes and structs. Enums are used when we need to represent a finite number of cases and while there are valid use cases where we would have an enum conform to a protocol, they are very rare in my experience. Just remember that anywhere that we refer to a class or struct, we can also use an enum. Let's begin exploring protocols by seeing how they are full-fledged types in Swift. Protocol as a type Even though no functionality is implemented in a protocol, they are still considered a full-fledged type in the Swift programming language and can be used like any other type. What this means is we can use protocols as a parameter type or a return type in a function. We can also use them as the type for variables, constants, and collections. Let's take a look at some examples. For these few examples, we will use the PersonProtocol protocol: protocol PersonProtocol { var firstName: String {get set} var lastName: String {get set} var birthDate: NSDate {get set} var profession: String {get} init (firstName: String, lastName: String, birthDate: NSDate) } In this first example, we will see how we would use protocols as a parameter type or return type in functions, methods, or initializers: func updatePerson(person: PersonProtocol) -> PersonProtocol { // Code to update person goes here return person } In this example, the updatePerson() function accepts one parameter of the PersonProtocol protocol type and then returns a value of the PersonProtocol protocol type. Now let's see how we can use protocols as a type for constants, variables, or properties: var myPerson: PersonProtocol In this example, we create a variable of the PersonProtocol protocol type that is named myPerson. We can also use protocols as the item type to store in collection such as arrays, dictionaries, or sets: var people: [PersonProtocol] = [] In this final example, we create an array of PersonProtocol protocol types. As we can see from these three examples, even though the PersonProtocol protocol does not implement any functionality, we can still use protocols when we need to specify a type. We cannot, however, create an instance of a protocol. This is because no functionality is implemented in a protocol. As an example, if we tried to create an instance of the PersonProtocol protocol, we would be receiving the error: protocol type 'PersonProtocol' cannot be instantiated error, as shown in the following example: var test = PersonProtocol(firstName: "Jon", lastName: "Hoffman", birthDate: bDateProgrammer) We can use the instance of any class or struct that conforms to our protocol anywhere that the protocol type is required. As an example, if we defined a variable to be of the PersonProtocol protocol type, we could then populate that variable with any class or struct that conforms to the PersonProtocol protocol. For this example, let's assume that we have two types named SwiftProgrammer and FootballPlayer, which conform to the PersonProtocol protocol: var myPerson: PersonProtocol myPerson = SwiftProgrammer(firstName: "Jon", lastName: "Hoffman", birthDate: bDateProgrammer) print("(myPerson.firstName) (myPerson.lastName)") myPerson = FootballPlayer(firstName: "Dan", lastName: "Marino", birthDate: bDatePlayer) print("(myPerson.firstName) (myPerson.lastName)") In this example, we start off by creating the myPerson variable of the PersonProtocol protocol type. We then set the variable with an instance of the SwiftProgrammer type and print out the first and last names. Next, we set the myPerson variable to an instance of the FootballPlayer type and print out the first and last names again. One thing to note is that Swift does not care if the instance is a class or struct. It only matters that the type conforms to the PersonProtocol protocol type. Therefore, if our SwiftProgrammer type was a struct and the FootballPlayer type was a class, our previous example would be perfectly valid. As we saw earlier, we can use our PersonProtocol protocol as the type for an array. This means that we can populate the array with instances of any type that conforms to the PersonProtocol protocol. Once again, it does not matter if the type is a class or a struct as long as it conforms to the PersonProtocol protocol. Here is an example of this: var programmer = SwiftProgrammer(firstName: "Jon", lastName: "Hoffman", birthDate: bDateProgrammer) var player = FootballPlayer(firstName: "Dan", lastName: "Marino", birthDate: bDatePlayer) var people: [PersonProtocol] = [] people.append(programmer) people.append(player) In this example, we create an instance of the SwiftProgrammer type and an instance of the FootballPlayer type. We then add both instances to the people array. Polymorphism with protocols What we were seeing in the previous examples is a form of Polymorphism. The word polymorphism comes from the Greek roots Poly, meaning many and morphe, meaning form. In programming languages, polymorphism is a single interface to multiple types (many forms). In the previous example, the single interface was the PersonProtocol protocol and the multiple types were any type that conforms to that protocol. Polymorphism gives us the ability to interact with multiple types in a uniform manner. To illustrate this, we can extend our previous example where we created an array of the PersonProtocol types and loop through the array. We can then access each item in the array using the properties and methods define in the PersonProtocol protocol, regardless of the actual type. Let's see an example of this: for person in people { print("(person.firstName) (person.lastName): (person.profession)") } If we ran this example, the output would look similar to this: Jon Hoffman: Swift Programmer Dan Marino: Football Player We have mentioned a few times in this article that when we define the type of a variable, constant, collection type, and so on to be a protocol type, we can then use the instance of any type that conforms to that protocol. This is a very important concept to understand and it is what makes protocols and protocol extensions so powerful. When we use a protocol to access instances, as shown in the previous example, we are limited to using only properties and methods that are defined in the protocol. If we want to use properties or methods that are specific to the individual types, we would need to cast the instance to that type. Type casting with protocols Type casting is a way to check the type of the instance and/or to treat the instance as a specified type. In Swift, we use the is keyword to check if an instance is a specific type and the as keyword to treat the instance as a specific type. To start with, let's see how we would check the instance type using the is keyword. The following example shows how would we do this: for person in people { if person is SwiftProgrammer { print("(person.firstName) is a Swift Programmer") } } In this example, we use the if conditional statement to check whether each element in the people array is an instance of the SwiftProgrammer type and if so, we print that the person is a Swift programmer to the console. While this is a good method to check whether we have an instance of a specific class or struct, it is not very efficient if we wanted to check for multiple types. It is a lot more efficient to use the switch statement, as shown in the next example, if we want to check for multiple types: for person in people { switch (person) { case is SwiftProgrammer: print("(person.firstName) is a Swift Programmer") case is FootballPlayer: print("(person.firstName) is a Football Player") default: print("(person.firstName) is an unknown type") } } In the previous example, we showed how to use the switch statement to check the instance type for each element of the array. To do this check, we use the is keyword in each of the case statements in an attempt to match the instance type. We can also use the where statement with the is keyword to filter the array, as shown in the following example: for person in people where person is SwiftProgrammer { print("(person.firstName) is a Swift Programmer") } Now let's look at how we can cast an instance of a class or struct to a specific type. To do this, we can use the as keyword. Since the cast can fail if the instance is not of the specified type, the as keyword comes in two forms: as? and as!. With the as? form, if the casting fails, it returns a nil, and with the as! form, if the casting fails, we get a runtime error; therefore, it is recommended to use the as? form unless we are absolutely sure of the instance type or we perform a check of the instance type prior to doing the cast. Let's look at how we would use the as? keyword to cast an instance of a class or struct to a specified type: for person in people { if let p = person as? SwiftProgrammer { print("(person.firstName) is a Swift Programmer") } } Since the as? keyword returns an optional, we can use optional binding to perform the cast, as shown in this example. If we are sure of the instance type, we can use the as! keyword. The following example shows how to use the as! keyword when we filter the results of the array to only return instances of the SwiftProgrammer type: for person in people where person is SwiftProgrammer { let p = person as! SwiftProgrammer } Now that we have covered the basics of protocols, that is, how polymorphism works and type casting, let's dive into one of the most exciting new features of Swift protocol extensions. Protocol extensions Protocol extensions allow us to extend a protocol to provide method and property implementations to conforming types. It also allows us to provide common implementations to all the confirming types eliminating the need to provide an implementation in each individual type or the need to create a class hierarchy. While protocol extensions may not seem too exciting, once you see how powerful they really are, they will transform the way you think about and write code. Let's begin by looking at how we would use protocol extension with a very simplistic example. We will start off by defining a protocol called DogProtocol as follows: protocol DogProtocol { var name: String {get set} var color: String {get set} } With this protocol, we are saying that any type that conforms to the DogProtocol protocol, must have the two properties of the String type, namely, name and color. Now let's define the three types that conform to this protocol. We will name these types JackRussel, WhiteLab, and Mutt as follows: struct JackRussel: DogProtocol { var name: String var color: String } class WhiteLab: DogProtocol { var name: String var color: String init(name: String, color: String) { self.name = name self.color = color } } struct Mutt: DogProtocol { var name: String var color: String } We purposely created the JackRussel and Mutt types as structs and the WhiteLab type as a class to show the differences between how the two types are set up and to illustrate how they are treated in the same way when it comes to protocols and protocol extensions. The biggest difference that we can see in this example is the struct types provide a default initiator, but in the class, we must provide the initiator to populate the properties. Now let's say that we want to provide a method named speak to each type that conforms to the DogProtocol protocol. Prior to protocol extensions, we would start off by adding the method definition to the protocol, as shown in the following code: protocol DogProtocol { var name: String {get set} var color: String {get set} func speak() -> String } Once the method is defined in the protocol, we would then need to provide an implementation of the method in every type that conform to the protocol. Depending on the number of types that conformed to this protocol, this could take a bit of time to implement. The following code sample shows how we might implement this method: struct JackRussel: DogProtocol { var name: String var color: String func speak() -> String { return "Woof Woof" } } class WhiteLab: DogProtocol { var name: String var color: String init(name: String, color: String) { self.name = name self.color = color } func speak() -> String { return "Woof Woof" } } struct Mutt: DogProtocol { var name: String var color: String func speak() -> String { return "Woof Woof" } } While this method works, it is not very efficient because anytime we update the protocol, we would need to update all the types that conform to it and we may be duplicating a lot of code, as shown in this example. Another concern is, if we need to change the default behavior of the speak() method, we would have to go in each implementation and change the speak() method. This is where protocol extensions come in. With protocol extensions, we could take the speak() method definition out of the protocol itself and define it with the default behavior, in protocol extension. The following code shows how we would define the protocol and the protocol extension: protocol DogProtocol { var name: String {get set} var color: String {get set} } extension DogProtocol { func speak() -> String { return "Woof Woof" } } We begin by defining DogProtocol with the original two properties. We then create a protocol extension that extends DogProtocol and contains the default implementation of the speak() method. With this code, there is no need to provide an implementation of the speak() method in each of the types that conform to DogProtocol because they automatically receive the implementation as part of the protocol. Let's see how this works by setting our three types that conform to DogProtocol back to their original implementations and they should receive the speak() method from the protocol extension: struct JackRussel: DogProtocol { var name: String var color: String } class WhiteLab: DogProtocol { var name: String var color: String init(name: String, color: String) { self.name = name self.color = color } } struct Mutt: DogProtocol { var name: String var color: String } We can now use each of the types as shown in the following code: let dash = JackRussel(name: "Dash", color: "Brown and White") let lily = WhiteLab(name: "Lily", color: "White") let buddy = Mutt(name: "Buddy", color: "Brown") let dSpeak = dash.speak() // returns "woof woof" let lSpeak = lily.speak() // returns "woof woof" let bSpeak = buddy.speak() // returns "woof woof" As we can see in this example, by adding the speak() method to the DogProtocol protocol extension, we are automatically adding that method to all the types that conform to DogProtocol. The speak() method in the DogProtocol protocol extension can be considered a default implementation of the speak() method because we are able to override it in the type implementations. As an example, we could override the speak() method in the Mutt struct, as shown in the following code: struct Mutt: DogProtocol { var name: String var color: String func speak() -> String { return "I am hungry" } } When we call the speak() method for an instance of the Mutt type, it will return the string, "I am hungry". Now that we have seen how we would use protocols and protocol extensions, let's look at a more real-world example. In numerous apps, across multiple platforms (iOS, Android, and Windows), I have had the requirement to validate user input as it is entered. This validation can be done very easily with regular expressions; however, we do not want various regular expressions littered through out our code. It is very easy to solve this problem by creating different classes or structs that contains the validation code; however, we would have to organize these classes to make them easy to use and maintain. Prior to protocol extensions in Swift, I would use protocols to define the validation requirements and then create a struct that would conform to the protocol for each validation that I needed. Let's take a look at this preprotocol extension method. A regular expression is a sequence of characters that define a particular pattern. This pattern can then be used to search a string to see whether the string matches the pattern or contains a match of the pattern. Most major programming languages contain a regular expression parser, and if you are not familiar with regular expressions, it may be worth to learn more about them. The following code shows the TextValidationProtocol protocol that defines the requirements for any type that we want to use for text validation: protocol TextValidationProtocol { var regExMatchingString: String {get} var regExFindMatchString: String {get} var validationMessage: String {get} func validateString(str: String) -> Bool func getMatchingString(str: String) -> String? } In this protocol, we define three properties and two methods that any type that conforms to TextValidationProtocol must implement. The three properties are: regExMatchingString: This is a regular expression string used to verify that the input string contains only valid characters. regExFindMatchString: This is a regular expression string used to retrieve a new string from the input string that contains only valid characters. This regular expression is generally used when we need to validate the input real time, as the user enters information, because it will find the longest matching prefix of the input string. validationMessage: This is the error message to display if the input string contains non-valid characters. The two methods for this protocol are as follows: validateString: This method will return true if the input string contains only valid characters. The regExMatchingString property will be used in this method to perform the match. getMatchingString: This method will return a new string that contains only valid characters. This method is generally used when we need to validate the input real time as the user enters information because it will find the longest matching prefix of the input string. We will use the regExFindMatchString property in this method to retrieve the new string. Now let's see how we would create a struct that conforms to this protocol. The following struct would be used to verify that the input string contains only alpha characters: struct AlphaValidation1: TextValidationProtocol { static let sharedInstance = AlphaValidation1() private init(){} let regExFindMatchString = "^[a-zA-Z]{0,10}" let validationMessage = "Can only contain Alpha characters" var regExMatchingString: String { get { return regExFindMatchString + "$" } } func validateString(str: String) -> Bool { if let _ = str.rangeOfString(regExMatchingString, options: .RegularExpressionSearch) { return true } else { return false } } func getMatchingString(str: String) -> String? { if let newMatch = str.rangeOfString(regExFindMatchString, options: .RegularExpressionSearch) { return str.substringWithRange(newMatch) } else { return nil } } } In this implementation, the regExFindMatchString and validationMessage properties are stored properties, and the regExMatchingString property is a computed property. We also implement the validateString() and getMatchingString() methods within the struct. Normally, we would have several different types that conform to TextValidationProtocol where each one would validate a different type of input. As we can see from the AlphaValidation1 struct, there is a bit of code involved with each validation type. A lot of the code would also be duplicated in each type. The code for both methods (validateString() and getMatchingString()) and the regExMatchingString property would be duplicated in every validation class. This is not ideal, but if we wanted to avoid creating a class hierarchy with a super class that contains the duplicate code (I personally prefer using value types over classes), we would have no other choice. Now let's see how we would implement this using protocol extensions. With protocol extensions we need to think about the code a little differently. The big difference is, we do not need, nor want to define everything in the protocol. With standard protocols or when we use class hierarchy, all the methods and properties that you would want to access using the generic superclass or protocol would have to be defined within the superclass or protocol. With protocol extensions, it is preferred for us not to define a property or method in the protocol if we are going to be defining it within the protocol extension. Therefore, when we rewrite our text validation types with protocol extensions, TextValidationProtocol would be greatly simplified to look similar to this: protocol TextValidationProtocol { var regExFindMatchString: String {get} var validationMessage: String {get} } In original TextValidationProtocol, we defined three properties and two methods. As we can see in this new protocol, we are only defining two properties. Now that we have our TextValidationProtocol defined, let's create the protocol extension for it: extension TextValidationProtocol { var regExMatchingString: String { get { return regExFindMatchString + "$" } } func validateString(str: String) -> Bool { if let _ = str.rangeOfString(regExMatchingString, options: .RegularExpressionSearch) { return true } else { return false } } func getMatchingString(str: String) -> String? { if let newMatch = str.rangeOfString(regExFindMatchString, options: .RegularExpressionSearch) { return str.substringWithRange(newMatch) } else { return nil } } } In the TextValidationProtocol protocol extension, we define the two methods and the third property that were defined in original TextValidationProtocol, but were not defined in the new one. Now that we have created our protocol and protocol extension, we are able to define our text validation types. In the following code, we define three structs that we will use to validate text when a users types it in: struct AlphaValidation: TextValidationProtocol { static let sharedInstance = AlphaValidation() private init(){} let regExFindMatchString = "^[a-zA-Z]{0,10}" let validationMessage = "Can only contain Alpha characters" } struct AlphaNumericValidation: TextValidationProtocol { static let sharedInstance = AlphaNumericValidation() private init(){} let regExFindMatchString = "^[a-zA-Z0-9]{0,15}" let validationMessage = "Can only contain Alpha Numeric characters" } struct DisplayNameValidation: TextValidationProtocol { static let sharedInstance = DisplayNameValidation() private init(){} let regExFindMatchString = "^[\s?[a-zA-Z0-9\-_\s]]{0,15}" let validationMessage = "Display Name can contain only contain Alphanumeric Characters" } In each one of the text validation structs, we create a static constant and a private initiator so that we can use the struct as a singleton. After we define the singleton pattern, all we do in each type is set the values for the regExFindMatchString and validationMessage properties. Now we have not duplicated the code virtually because even if we could, we would not want to define the singleton code in the protocol extension because we would not want to force that pattern on all the conforming types. To use the text validation classes, we would want to create a dictionary object that would map the UITextField objects to the validation class to use it like this: var validators = [UITextField: TextValidationProtocol]() We could then populate the validators dictionary as shown here: validators[alphaTextField] = AlphaValidation.sharedInstance validators[alphaNumericTextField] = AlphaNumericValidation.sharedInstance validators[displayNameTextField] = DisplayNameValidation.sharedInstance We can now set the EditingChanged event of the text fields to a single method named keyPressed(). To set the edition changed event for each field, we would add the following code to the viewDidLoad() method of our view controller: alphaTextField.addTarget(self, action:Selector("keyPressed:"), forControlEvents: UIControlEvents.EditingChanged) alphaNumericTextField.addTarget(self, action: Selector("keyPressed:"), forControlEvents: UIControlEvents.EditingChanged) displayNameTextField.addTarget(self, action: Selector("keyPressed:"), forControlEvents: UIControlEvents.EditingChanged) Now lets create the keyPressed() method that each text field calls when a user types a character into the field: @IBAction func keyPressed(textField: UITextField) { if let validator = validators[textField] where !validator.validateString(textField.text!) { textField.text = validator.getMatchingString(textField.text!) messageLabel?.text = validator.validationMessage } } In this method, we use the if let validator = validators[textField] statement to retrieve the validator for the particular text field and then we use the where !validator.validateString(textField.text!) statement to validate the string that the user has entered. If the string fails validation, we use the getMatchingString() method to update the text in the text field by removing all the characters from the input string, starting with the first invalid character and then displaying the error message from the text validation class. If the string passes validation, the text in the text field is left unchanged. Summary In this article, we saw that protocols are treated as full-fledged types by Swift. We also saw how polymorphism can be implemented in Swift with protocols. We concluded this article with an in-depth look at protocol extensions and saw how we would use them in Swift. Protocols and protocol extensions are the backbone of Apple's new protocol-oriented programming paradigm. This new model for programming has the potential to change the way we write and think about code. While we did not specifically cover protocol-oriented programming in this article, understanding the topics in this article gives us the solid understanding of protocols and protocol extensions needed to learn about this new programming model. Resources for Article: Further resources on this subject: Using OpenStack Swift [Article] Dragging a CCNode in Cocos2D-Swift [Article] Installing OpenStack Swift [Article]
Read more
  • 0
  • 0
  • 18285
Modal Close icon
Modal Close icon