You're reading from Robotic Process Automation with Blue Prism Quick Start Guide

Product typeBook

Published inNov 2018

Reading LevelIntermediate

PublisherPackt

ISBN-139781789610444

Edition1st Edition

Languages

Tools

UiPath Blue Prism

Concepts

Robotic Process Automation

Author (1)

Lim Mei Ying

Spying Elements

In the previous chapter, we got Amazon up and running by opening the site in Internet Explorer. However, we want to do a lot more than opening websites. In this chapter, we will do the following:

Teach the robot to spy elements using Application Modeller
Learn how to tweak the match criteria so that the robot is able to find the elements again
Add more elements to the Modeller and categorize them
Learn about the various spy modes available in Blue Prism, including HTML, UI Automation, and region modes

What is spying? Spying is the way to teach the robot to look for elements such as buttons, links, and textboxes on the screen. It acts as the robot's eyes, telling it where to find the items that it needs to click, press, or key text into.

Spying is crucial to the building of business objects. Without spying, the robot will not be able to see and interact with...

Spying elements on a web page

One of the tasks performed by the purchasing process is to search the Amazon site for the items to purchase. Before the robot can perform a search, it needs to interact with the following elements, as shown in the following screenshot:

The search textbox for entering the keywords
The search button

Let's train the robot to identify these elements on the page by using Application Modeller to spy on it:

Open the Amazon - Search business object for editing. From the toolbar, click Application Modeller.
Let's familiarize ourselves with the Application Modeller dialog. On the left is the Application Explorer. It's a directory that lists all the items that you have spied. At this point, we have not spied any elements. The first element, Element1, was automatically created when we ran the wizard.
Click on the empty Element1. On the left...

How does spying work?

A robot does not have real eyes to see what is on the screen. How then is it able to look for elements that it needs to interact with? Here's how it does it for HTML pages:

It scans through the page by looking at the HTML source code. A web page is made up of many HTML components. Here is an example of a page that contains a textbox and a button:

     <HTML>
      <BODY>
      Search: <input type= "text" id= "twotabsearchtextbox" />
      <input type="submit" value="Go" />
      </BODY>
      </HTML>

The robot breaks the HTML down into elements, such as textboxes, labels, buttons, and links. Each element is defined by a markup tag. For example, a textbox is an <INPUT/> tag, while a hyperlink is an <A/>.
Let's take the preceding code, for example. The robot sees...

Tweaking the match criteria

Earlier, when we spied the Search textbox, Blue Prism came up with a list of suggested match criteria to correctly identify the element, as shown in the following screenshot:

All these conditions must be satisfied in order for the robot to locate the Search textbox. What happens if Amazon decides to change the appearance of the textbox and give it a different class name?

Well, we can't hack into Amazon's site and effect the change. For the purposes of the demonstration, let's simulate that by changing the Class Name attribute from nav-input to nav-input2.

Now, click the Highlight button again. Oops, are you getting the same error message as the one here?

What just happened? We only changed the value of one of the many match attributes. Why isn't Blue Prism able to find the Search textbox now? The answer lies in the way matching...

Tightening the match criteria

Let's tighten up the match criteria for the Search textbox to make it more robust to changes on Amazon's site. Since the developers at Amazon have used the ID and Input Identifier attributes to uniquely label the element, we shall get Blue Prism to use these two attributes to look for the search box. Let's tighten the match criteria further:

Back in Application Modeller, select Textbox – Search. Sort the Attributes according to the Match column such that the matching attributes are listed at the top.
Uncheck all the match criteria, except for the following:
- Input Identifier
- ID
Click Highlight. Check that the Search textbox can still be found on the screen.
Click Apply to save the changes without closing the window.

Now, we have made the robot a lot more resilient to changes to the target website. Unless Amazon changes the...

Adding elements

We have successfully spied the search textbox. In order to execute the search, we will need to press the search button as well. Let's proceed to spy the Search button by adding an element to Application Modeller:

With the Textbox - Search element selected, click Add Element. A new empty element named Element1 is added to Application Explorer. Rename Element1 to Button - Search.
Click Identify and highlight the Search button . Press Ctrl + left mouse click when the button is highlighted in green.
Once the button has been identified, tighten the match criteria by unchecking all the matches and leaving only the following:
- Tag name
- Path
- Input type

Hey, wait a minute. Didn't we just mention that the position of the Search button may change, so therefore we don't use the path attribute in the match criteria? It really isn't a hard and fast rule...

Categorizing elements

As we build up the Application Modeller and add more elements to the list, it is going to grow and grow. It is not uncommon to have lists that number in the hundreds of elements. Looking for the element that you want to work with becomes a huge exercise akin to looking for a needle in a haystack. One nice bit about elements is that they can be ordered and categorized by nesting them. Observe the following steps to see how it's done:

In Application Explorer, select Amazon - Search. Click Add Child. This creates a new element directly beneath the root.

Alternatively, you could have chosen any of the elements that we created earlier and clicked Add Element. This creates the new element at the same level as the other elements.

Name the new element Search.
Drag the two elements that we created previously and drop them on top of the newly created Search...

More spy modes

When we spied the Search textbox and button, we used HTML mode. We saw it indicated in a green box when we tried to highlight the elements on the Amazon home page. HTML mode has worked well so far. We were able to identify the elements we needed.

What if the application that we are working with is not a web page but, say, a Windows application such as Microsoft Office? Or perhaps the web page uses modern web technologies that are too dynamic for Blue Prism to find elements using HTML mode.

To solve these common spying issues, Blue Prism supports the following alternative spy modes:

Win32 mode uses the base Windows 32 API to look for elements in an application. It is typically used to spy Windows-based applications.
Accessibility mode uses the Microsoft Active Accessibility (AA) framework. Its origins started when there was a demand for providing alternative means...

UI Automation mode

Let's try out the other spy modes. Observe the following steps to spy the first item that appears in the search results listing using the UI Automation mode:

In the search box, enter the keywords to look for any item that you may want to purchase. Later on, we will get the robot to enter the keywords for us. For now, as we are still spying the elements, we will enter it manually. When you are ready, click the Search button and wait for the results to be displayed.
Let's create a new element for storing all the elements related to the search results page. Back in Application Explorer, add an element beneath Amazon – Search. Name the element Search Results. This will be our container for storing all the elements related to the Search Results page.

Add another element. This time, place it beneath the Search Results element. Name the element...

UI Automation navigator

In cases where it's difficult to locate the element on the screen using the mouse, we have the option of using the navigators. Navigators scan an entire page for elements that can be identified and present it as a list. All we need to do is to select the element that we want from the list. These elements are located in the sub-menu (refer to the following screenshot) that appears when you click on the arrow next to the Identify button.

The menu has three options:

Spy Element: This is the default option and is the one that we have used to identify elements in HTML and UIA mode by clicking on them in spy mode.
Open Application Navigator: This option uses the older AA framework to scan the entire page for all the elements that can be identified.
Open UI Automation Navigator: This option also scans the page for elements, but it uses the newer and better...

Surface automation with region mode

Methods of identification through point and click and using the navigator works most of the time to find the elements that we need to interact with. They work especially well if the application is installed on the same computer as the robot. However, we are seeing more and more technologies that work off a remote machine. The application is installed on a server somewhere in the network and what is sent to your computer is just an image that you can interact with. Examples of such applications (also known as thin-clients) include Citrix, Microsoft Terminal Services, and Mainframes.

In such special cases, we need to fall back on the good old method of looking at the position of the element on the screen to figure out where they are. This method of spying is also known as surface automation. The spy mode that supports surface automation is region...

Summary

In this chapter, we learned about spying. Blue Prism uses Application Modeller to define the elements that the robot has to interact with. To be able to work with the various types of applications out there, Blue Prism offers several spy modes.

We tried out HTML mode and captured the search box and button. Later, we used UI Automation to spy the search results list. We also sampled region mode to have an idea of how to capture elements that can't be found using other means.

In the next chapter, we will look at how the elements that we have spied can be used in the business object diagram.

The rest of the chapter is locked

You have been reading a chapter from

Robotic Process Automation with Blue Prism Quick Start Guide

Published in: Nov 2018Publisher: PacktISBN-13: 9781789610444

A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.

undefined

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Start free trial

Renews at $15.99/month. Cancel anytime

Author (1)

Lim Mei Ying

Lim Mei Ying has extensive experience in designing, implementing and supporting Blue Prism processes as well as setting up a Robotic Operating Model for the enterprise. She has spent many hours figuring out the dos and don'ts of Blue Prism technologies and thrives on the challenge of finding new ways to solve automation problems. Mei Ying lives on the sunny island of Singapore.
Read more about Lim Mei Ying

Personalised recommendations for you

Based on your interests and search pattern

Engineering Manager's Handbook

Engineering Manager's Handbook is a comprehensive guide for managers to excel in their role, foster customer-centric digital products, learn leadership, team building, and balancing technical work with management. You’ll also explore how to develop trust, authority, and collaboration to drive success and make a lasting impact.

BookSep 2023278 pages

C++ Game Animation Programming

Video game characters have a fascinating history, evolving from simple 2D sprites to high-polygon 3D models. Take a look behind the curtain and learn how to build a 3D renderer, load character models, play animations and blend between them, and create large crowds of animated people with this comprehensive C++ game animation programming guide.

BookDec 2023480 pages

Gamification for Product Excellence

This book helps you to take your product management strategy to the next level by standing out in crowded markets. Along with boosting user adoption rates by creating engaging products that incorporate playful elements, learn gamification theory and how to integrate it into your design, product development, and product management processes.

BookSep 2023350 pages

Supercharging Productivity with Trello

Supercharging Productivity with Trello is the ultimate guide for anyone looking to boost their productivity with digital tools. Whether you're new to Trello or a seasoned professional, this book covers everything from core features to advanced automation, and Power-Ups.

BookAug 2023342 pages

Automate It with Zapier and Generative AI

This comprehensive guide takes you through the concepts of business process automation, showing you how Zapier can facilitate it without having to write code and helping you to boost productivity. You’ll learn how to save time, reduce costs, and make your business recession-proof by using Zapier to automate tasks in your cloud-based business apps.

BookAug 2023706 pages

Scoring to Picture in Logic Pro

In this book, you’ll explore a variety of techniques to synchronize music to picture using Logic Pro. Though this is not a technical manual, it will teach you how to make the best use of Logic Pro and how to wield this technology to maximize your potential when scoring to picture.

BookSep 2023412 pages

Mastering Information Security Compliance Management

This concise book equips you with the knowledge and practices needed to establish and maintain an effective information security management system. The chapters provide insights into ISO/IEC 27001/27002:2022, risk management, ISMS development, incident management, audit processes, and strategies for continuous improvement.

BookAug 2023236 pages1

Implementing Atlassian Confluence

Implementing Atlassian Confluence provides both a high-level overview and an insightful path for remote collaboration with Atlassian Confluence. With this multi-layered yet practical guide, you’ll be able to set up Confluence-based collaboration with minimum external consultancy services to ensure smooth and close coordination between teams.

BookSep 2023406 pages

R Bioinformatics Cookbook

This book takes a unique problem–solution approach to handling complex tasks in the bioinformatics domain using different datasets present in the book. With the help of real-world examples, you’ll learn to put each independent recipe to use to tackle problems in the field of bioinformatics.

BookOct 2023396 pages

Build Your Own Metaverse with Unity

Build Your own Metaverse with Unity is a practical guide for developers to create their own metaverse - a virtual world with infinite possibilities. It empowers you to identify gaps in existing metaverses and improve upon them, enabling you to shape your virtual world.

BookSep 2023586 pages5