Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-introduction-kohana-php-framework
Packt
28 Dec 2009
6 min read
Save for later

Introduction to Kohana PHP Framework

Packt
28 Dec 2009
6 min read
Overview Kohana PHP Framework is an open source PHP software development framework that helps php developers to build web applications faster, and also, more effectively by providing them with a set of built-in objects/classes. It also enforces highly organized coding standards. The Kohana PHP Framework is just like Ruby on Rails; it implements the well known software engineering design pattern—Model View Controller(MVC). The Model View Controller software design pattern guides engineers to design their software codes into three separate parts which includes: Models: The objects that manipulate data sources and data stores. Views: The html and css files with inline php codes that present the user interface and controls to the application users. Controllers: Objects in charge of the business logic, displaying the page(views), and routing the click actions from the views to the model and back to the views. Kohana was originally based upon the well documented codeigniter php framework, but it stands out due to its strict use of OOP best practices and standards. Kohana PHP is officially defined by its creators to be a PHP 5 framework that uses the model view controller architectural pattern. It aims to be secure, lightweight, and easy to use. Why use Kohana PHP framework? PHP is a very easy to use programming language, that's the reason why its loved by the web development community. It's easy to use nature and learning curve has attracted a lot of users. But PHP has one downside. Most experienced developers don’t love coding php due to its little Object orientation, and its often nasty codes. With Kohana PHP, any php developer from beginner to expert will all get to write standard codes which we see in the Java World, or the .NET world. Kohana bridges the gap between amateur web developers and designers who love php because its easy, and the experienced web developers who go fully into Object Oriented codes and nothing less. Kohana enforces true software engineering and brings it closer to the php world which has little or no standards. So the major reason why you should use Kohana PHP Framework for your php work is to ensure that your codes are of standards, and you follow the best practices which go a long way to help teams of developers from beginners to experts to easily work together with the PHP programming language. Kohana PHP Framework solves the problem of beginners, and intermediate developers who have nasty codes and are killing the overall work of the team. With the well documented and standard practices of Kohana, a developer in India can write the code and a developer in Africa can optimise the code without asking a lot of questions, because everything will clearly be a model, view or a controller, and it will be easy to understand what it was meant for exactly. Key features of Kohana PHP framework Kohana PHP has some very good features that makes it stand out from all other PHP frameworks. The features include:  Fully Object Oriented: Object oriented programming is an industry standard since the introduction of C++. Many developers working with Java programming languages are comfortable with Object and classes, and they prefer the order it brings to programming. Kohana PHP being fully Object oriented, it automatically brings this order and industry standard programming to PHP; with this, you wont get developers saying you are just a php developer because Kohana PHP is a standard software programming platform for enterprise and web applications. In built templating: Kohana PHP like most other standard web frameworks has a very easy to use template engine. You get a simple html file, covert it into a kohana template by renaming it to template.php, fit in two php variables— $title and $content, to signify where the title of the page and where the content will be displayed. That’s all you need to have a kohana php template. In built Internationalization: Kohana PHP strongly supports internationalization and good documentation. With Kohana, all you need to do is create the various language files and use an internationalization object to reference the files depending on the language from page to page. Switching the language, automatically switches the language file referenced. Robust ORM Engine: ORM stands for Object Relational Mapping. It’s a programming concept in which your codes directly manipulate the database tables as objects, while the SQL gets generated for you behind the scenes. The ORM in Kohana is easy due to its use of convention over configuration. Its the best ORM I have seen in the php world and its 100% stable. To make use of the ORM, you simply extend the ORM class in your model and that model will be able to manipulate your database without any SQL code. MVC: the framework is based on MVC, and this helps to bring about a lot of order, eases code maintenance and team work. I can get any Kohana PHP application, and easily understand what every code does because it must flow from a controller to a view or to a model then to a view. And that’s all about it. Clean and Search Engine Friendly URLs: Kohana PHP framework has a good implementation of controllers where every url call or reference in a kohana based site refers to a controller and its function, as in site/controller/function/parammeters. For example, calling a blog controller to show a post with ID 1 will be as simple as your-site.com/index.php/blog/showPost/1 where index.php is the front controller, blog is a controller object and show post a blog->showPost function that takes a parameter which is the post ID. Libraries and helpers and third party classes: With kohana php, you have a set of libraries and helpers for almost everything you need as a web developer. For instance, sending out an email is as easy as calling email::send() function. There are more helpers and libraries, getting to read about them on the Kohana php website will make you feel like your life as a developer has been taken away and you are just a secretary. Also with Kohana PHP, you can get any available php class from the internet and use it in the kohana framework by simple dropping the file in a folder in your kohana site known as vendor, and then, instantiating the class in your codes. That’s all! It sounds too good to be true but thats all you need to use that PEAR object or that zend class or that php class you have been using for years with kohana framework on your next project.
Read more
  • 0
  • 0
  • 15104

article-image-vm-it-not-what-you-think
Packt
10 Mar 2016
10 min read
Save for later

VM, It Is Not What You Think!

Packt
10 Mar 2016
10 min read
In this article by Iwan 'e1' Rahabok, the author of the book VMware Performance and Capacity Management, Second Edition, we will look at why a seemingly simple technology, a virtualized x86 machine, has huge ramifications for the IT industry. In fact, it is turning a lot of things upside down and breaking down silos that have existed for decades in large IT organizations. We will cover the following topics: Why virtualization is not what we think it is Virtualization versus partitioning A comparison between a physical server and a virtual machine (For more resources related to this topic, see here.) Our journey into the virtual world A virtual machine, or simply, VM - who doesn't know what it is? Even a business user who has never seen one knows what it is. It is just a physical server, virtualized. Nothing more. Wise men say that small leaks sink the ship. This is a good way to explain why IT departments that manage physical servers well struggle when the same servers are virtualized. We can also use the Pareto principle (80/20 rule). 80 percent of a VM is identical to a physical server. But it's the 20 percent of difference that hits you. We will highlight some of this 20 percent portion, focusing on areas that impact data center management. The change caused by virtualization is much larger than the changes brought about by previous technologies. In the past two or more decades, we transitioned from mainframes to the client/server-based model and then to the web-based model. These are commonly agreed upon as the main evolutions in IT architecture. However, all of these are just technological changes. They changed the architecture, yes, but they did not change the operation in a fundamental way. Both the client-server and web shifts did not talk about the journey. There was no journey to the client-server based model. However, with virtualization, we talk about the journey. It is a journey because the changes are massive and involve a lot of people. In 2007, Gartner correctly predicted the impact of virtualization (http://www.gartner.com/newsroom/id/505040). More than 8 years later, we are still in the midst of the journey. Proving how pervasive the change is, here is the summary on the article from Gartner: Notice how Gartner talks about a change in culture. Virtualization has a cultural impact too. In fact, if your virtualization journey is not fast enough, look at your organization's structure and culture. Have you broken the silos? Do you empower your people to take risks and do things that have never been done before? Are you willing to flatten the organizational chart? The silos that have served you well are likely your number one barrier to a hybrid cloud. So why exactly is virtualization causing such a fundamental shift? To understand this, we need to go back to the basics, which is exactly what virtualization is. It's pretty common that chief information officers (CIOs) have a misconception about what it is. Take a look at the following comments. Have you seen them in your organization? VM is just a virtualized physical machine. Even VMware says that the guest OS is not aware it's virtualized and that it does not run differently. It is still about monitoring CPU, RAM, disk, network, and other resources. No difference. It is a technological change. Our management process does not have to change. All of these VMs must still feed into our main enterprise IT management system. This is how we have run our business for decades, and it works. If only life were that simple, we would all be 100-percent virtualized and have no headaches! Virtualization has been around for years, and yet, most organizations have not mastered it. The proof of mastering it is when you have completed the journey and have reached the highest level of the virtualization maturity model. Not all virtualizations are equal There are plenty of misconceptions about the topic of virtualization, especially among IT folks who are not familiar with virtualization. CIOs who have not felt the strategic impact of virtualization (be it a good or bad experience) tend to carry these misconceptions. Although virtualization looks similar to a physical system from the outside, it is completely re-architected under the hood. So, let's take a look at the first misconception: what exactly is virtualization? Because it is an industry trend, virtualization is often generalized to include other technologies that are not virtualized. This is a typical strategy by IT vendors who have similar technologies. A popular technology often branded under virtualization is hardware partitioning; since it is parked under the umbrella of virtualization, both should be managed in the same way. Since both are actually different, customers who try to manage both with a single piece of management software struggle to do well. Partitioning and virtualization are two different architectures in computer engineering, resulting in there being major differences between their functionalities. They are shown in the following screenshot: Virtualization versus partitioning With partitioning, there is no hypervisor that virtualizes the underlying hardware. There is no software layer separating the VM and the physical motherboard. There is, in fact, no VM. This is why some technical manuals for partitioning technology do not even use the term VM. The manuals use the term domain, partition, or container instead. There are two variants of partitioning technology, hardware-level and OS-level partitioning, which are covered in the following bullet points: In hardware-level partitioning, each partition runs directly on the hardware. It is not virtualized. This is why it is more scalable and has less of a performance hit. Because it is not virtualized, it has to have an awareness of the underlying hardware. As a result, it is not fully portable. You cannot move the partition from one hardware model to another. The hardware has to be built for a purpose to support that specific version of the partition. The partitioned OS still needs all the hardware drivers and will not work on other hardware if the compatibility matrix does not match. As a result, even the version of the OS matters, as it is just like a physical server. In OS-level partitioning, there is a parent OS that runs directly on the server motherboard. This OS then creates an "OS partition", where other OSes can run. We use double quotes as it is not exactly the full OS that runs inside that partition. The OS has to be modified and qualified to be able to run as a zone or container. Because of this, application compatibility is affected. This is different in a VM, where there is no application compatibility issue as the hypervisor is transparent to the guest OS. Hardware partitioning We covered the difference between virtualization and partitioning from an engineering point of view. However, does it translate into different data center architectures and operations? We will focus on hardware partitioning since there are fundamental differences between hardware partitioning and software partitioning. The use case for both is also different. Software partitioning is typically used in native cloud applications. With that, let's do a comparison between hardware partitioning and virtualization. We will start with availability. With virtualization, all VMs are protected by vSphere High Availability (vSphere HA), which provides 100 percent protection and that too without VM awareness. Nothing needs to be done at the VM layer. No shared or quorum disk and no heartbeat-network VM is required to protect a VM with basic HA. With hardware partitioning, the protection has to be configured manually, one by one for each logical partition (LPAR) or logical domain (LDOM). The underlying platform does not provide that. With virtualization, you can even go beyond five nines (99.999 percent) and move to 100 percent with vSphere Fault Tolerance. This is not possible in the partitioning approach as there is no hypervisor that replays CPU instructions. Also, because it is virtualized and transparent to the VM, you can turn the Fault Tolerance capability on and off on demand. Fault Tolerance is completely defined in the software. Another area of difference between partitioning and virtualization is disaster recovery (DR). With partitioning technology, the DR site requires another instance to protect the production instance. It is a different instance, with its own OS image, hostname, and IP address. Yes, we can perform a Storage Area Network (SAN) boot, but that means another Logical Unit Number (LUN) is required to manage, zone, replicate, and so on. Disaster recovery is not scalable to thousands of servers. To make it scalable, it has to be simpler. Compared to partitioning, virtualization takes a different approach. The entire VM fits inside a folder; it becomes like a document and we migrate the entire folder as if the folder is one object. This is what vSphere Replication or Site Recovery Manager do. They perform a replication per VM; there is no need to configure a SAN boot. The entire DR exercise, which can cover thousands of virtual servers, is completely automated and has audit logs automatically generated. Many large enterprises have automated their DR with virtualization. There is probably no company that has automated DR for their entire LPAR, LDOM, or container. In the previous paragraph, we're not implying LUN-based or hardware-based replication as inferior solutions. We're merely driving the point that virtualization enables you to do things differently. We're also not saying that hardware partitioning is an inferior technology. Every technology has its advantages and disadvantages and addresses different use cases. Before joining VMware, the author was a Sun Microsystems sales engineer for five years, so he is aware of the benefits of UNIX partitioning. This article is merely trying to dispel the misunderstanding that hardware partitioning equals virtualization. OS partitioning We've covered the differences between hardware partitioning and virtualization. Let's switch gear to software partitioning. In 2016, the adoption of Linux containers will continue its rapid rise. You can actually use both containers and virtualization, and they complement each other in some use cases. There are two main approaches to deploying containers: Run them directly on bare metal Run them inside a virtual machine As both technologies evolve, the gap gets wider. As a result, managing a software partition is different from managing a VM. Securing a container is different to securing a VM. Be careful when opting for a management solution that claims to manage both. You will probably end up with the most common denominator. This is one reason why VMware is working on vSphere Integrated Containers and the Photon platform. Now that's a separate topic by itself! Summary We hope you enjoyed the comparison and found it useful. We covered, to a great extent, the impact caused by virtualization and the changes it introduces. We started by clarifying that virtualization is a different technology compared to partitioning. We then explained that once a physical server is converted to a virtual machine, it takes on a different form and has radically different properties. Resources for Article: Further resources on this subject: Deploying New Hosts with vCenter [article] VMware vCenter Operations Manager Essentials - Introduction to vCenter Operations Manager [article] VMware vRealize Operations Performance and Capacity Management [article]
Read more
  • 0
  • 0
  • 15101

article-image-facebooks-outgoing-head-of-communications-and-policy-takes-blame-for-hiring-pr-firm-definers-and-reveals-more
Melisha Dsouza
22 Nov 2018
4 min read
Save for later

Facebook's outgoing Head of communications and policy takes blame for hiring PR firm ‘Definers’ and reveals more

Melisha Dsouza
22 Nov 2018
4 min read
On 4th November, the New York Times published a scathing report on Facebook that threw the tech giant under scrutiny for its leadership morales. The report pointed out how Facebook has been following the strategy of 'delaying, denying and deflecting’ the blame for all the controversies surrounding it. One of the recent scandals it was involved in was hiring a PR firm- called Definers- who did opposition research and shared content that criticized Facebook’s rivals Google and Apple, diverting focus from the impact of Russian interference on Facebook. They also pushed the idea that liberal financier George Soros was behind a growing anti-Facebook movement. Now, in a memo sent by Elliot Schrage (Facebook’s outgoing Head of Communications and Policy) to Facebook employees and obtained by TechCrunch, he takes the blame for hiring The Definers. Elliot Schrage, who after the Cambridge Analytica scandal, announced in June that he was leaving, admitted that his team asked Definers to push negative narratives about Facebook's competitors. He also stated that Facebook asked Definers to conduct research on liberal financier George Soros. His argument was that after George Soros attacked Facebook in a speech at Davos, calling them a “menace to society”, they wanted to determine if he had any financial motivation. According to the TechCrunch report, Elliot denied that the company asked the PR firm to distribute or create fake news. "I knew and approved of the decision to hire Definers and similar firms. I should have known of the decision to expand their mandate," Schrage said in the memo. He further stresses on being disappointed that a lot of the company’s internal discussion has become public. According to the memo, “This is a serious threat to our culture and ability to work together in difficult times.” Saving Mark and Sheryl from additional finger pointing, Schrage further added "Over the past decade, I built a management system that relies on the teams to escalate issues if they are uncomfortable about any project, the value it will provide or the risks that it creates. That system failed here and I'm sorry I let you all down. I regret my own failure here." As a follow-up note to the memo, Sheryl Sandberg (COO, Facebook) also shares accountability of hiring Deniers. She says “I want to be clear that I oversee our Comms team and take full responsibility for their work and the PR firms who work with us” Conveniently enough, this memo comes after the announcement that Elliot is stepping down from his post at Facebook. Elliot’s replacement, Facebook’s new head of global policy and former U.K. Deputy Prime Minister, Nick Clegg will now be reviewing its work with all political consultants. The entire scandal has led to harsh criticism from the media circle like Kara Swisher and from academics like Scott Galloway. On an episode of Pivot with Kara Swisher and Scott Galloway,  Kara comments that “Sheryl Sandberg ... really comes off the worst in this story, although I still cannot stand the ability of people to pretend that this is not all Mark Zuckerberg’s responsibility,” She further followed up with a jarring comment stating “He is the CEO. He has 60 percent. He’s an adult, and they’re treating him like this sort of adult boy king who doesn’t know what’s going on. It’s ridiculous. He knows exactly what’s going on.” Galloway added that since Sheryl had “written eloquently on personal loss and the important discussion around gender equality”, these accomplishments gave her “unfair” protection, and that it might also be true that she will be “unfairly punished.” He raises questions on both, Mark and Sheryl’s leadership saying “Can you think of any individuals who have made so much money doing so much damage? I mean, they make tobacco executives look like Mister Rogers.” On 19th November, he tweeted a detailed theory on why Sandberg is yet a part of Facebook; because “The Zuck can't be (fired)” and nobody wants to be the board who "fires the woman". https://twitter.com/profgalloway/status/1064559077819326464 Here’s another recent tweet thread from Scott which is a sarcastic take on what a “Big Tech” company actually is: https://twitter.com/profgalloway/status/1065315074259202048 Head over to CNBC to know more about this news. What is Facebook hiding? New York Times reveals Facebook’s insidious crisis management strategy NYT Facebook exposé fallout: Board defends Zuckerberg and Sandberg; Media call and transparency report Highlights BuzzFeed Report: Google’s sexual misconduct policy “does not apply retroactively to claims already compelled to arbitration”  
Read more
  • 0
  • 0
  • 15093

article-image-tour-around-gimp
Packt
21 May 2013
5 min read
Save for later

A Tour Around GIMP

Packt
21 May 2013
5 min read
(For more resources related to this topic, see here.)   We will go to produce the following image: Step 1 – Creating a new file Working with GIMP requires you to work on canvas, just like an artist would do. In GIMP, the canvas is a file on your system. You may have a clean canvas (a new file), or you may also work on an image obtained from various sources. The File menu presents you with a number of options. From here, you can create a clean canvas by selecting New. From the next screenshot, we can see that there is an extensive menu for Create. You may start to import images copied from somewhere else, scan them, or take a screenshot to begin with. To create a new canvas, follow these steps: Click on File. From the menu, click on New. Now the Create a New Image dialog box will appear. Adjust the width and height.Before clicking on OK, set the background to be filled with nothing by selecting Transparency. Now you have the canvas to work on your first art, as shown in the following screenshot: If you have hand drawn the outline of the picture and scanned it previously like I do, locate the scanned picture on your machine, and all you got to do is drag the file into the canvas. If you happen to have a stylus pen, you could easily draw on the canvas. Step 2 – Setting the picture for painting It is the perfect time to introduce the toolbox. The Toolbox option provides you with a set of tools to complete your work. The collections of tools here involve tools to select areas, painting, transform, and touch-up tools. If the Toolbox window does not show up, press Ctrl + B. Following is a screenshot of the Toolbox window. The previous screenshot consists of two parts, the tools and also tool options that further define the properties of the tool. The default installation of GIMP puts the Toolbox window and Tool option together, but they can be separated. It is one of the GIMP feature to be able to dock windows together or separate them according to user preference. Click on Windows and choose Dockable Dialogs to expose various dialog boxes that you might want to bring up front while working. If you're not sure how to go with each item of the menu, place the cursor on the icon, and a tooltip will appear together with a shortcut key in bold. At the bottom of the Toolbox window is the color chooser, where user can select an active color and alternative color, making painting easier when these colors are switchable. Click on the magic wand icon .Then, by clicking on the hair of the boy, the hair will be selected.   Now, duplicate the layer, so we can have a layer exclusively for his hair. This step involves working with layers. Step 3 – Create layers It is a good idea to have a layer for each part of the image. Hair, shirt, skin – all should have their own layer. This way you can paint each part without accidently filling other parts. The concept of layers is important when you are working with GIMP. Think of it as if you are drawing on different set of papers or transparencies that you may cut and rearrange before you paste them on the drawing board. The Layers dockable window can be accessed from Windows | Dockable Dialogs | Layers. The Layers window is useful to navigate between layers. Refer to the following screenshot: To navigate between layers, follow these steps: Drag a layer to the   icon to quickly create a duplicate of the layer. Press Ctrl + I (or click on Select | Invert) to select the opposite part and delete it. Now you have a layer specific for a part. Create a layer for each of the parts. The layers should be aligned and overlapped on top of the original outline. To rename the layers, double-click on the layer name or click on the layer and press F2. The following is a screenshot of how the layers should look: There are various controls that can be used from the Layers window alone. You may set certain layers' visibility by clicking on the eye icon and control how it blends with other layers by selecting the mode. Layers are best used when composing images. Each of the images can be on a layer, so it is easier to adjust them around images on other layers. These layers can be organized to let certain layer sit on top of others. To do so, drag the layer to the top to make it sit on top of the other image. Step 4 – Coloring Click the Paintbrush Tool , and pick a color of your choice and paint away.   To select another color, click on , and you'll be given a menu, which has a set of colors to choose from:   It is up to you to pick any color from the spectrum. Once done, click on OK, and you're back to the canvas. Repeat the select-and-paint process for each part. To illustrate the separate layers being colored, let's look at the following screenshot: Every fill on the drawing is separated by layers to enable each layer to be modified without affecting the others. You may also specify the opacity of individual layers. Summery This concludes a brief tour of GIMP. These few basic steps are just the tip of the iceberg. It is amazing what GIMP has in store for you. Resources for Article : Further resources on this subject: Creating a Quick Logo for a Company with GIMP 2.6 [Article] Blender 2.5: Creating a UV Texture [Article] How To Create Amazing Text and Font Effects in Gimp 2.6 [Article]
Read more
  • 0
  • 0
  • 15090

article-image-microsoft-airbnb-genentech-toyota-pytorch-to-build-deploy-production-ready-ai
Sugandha Lahoti
10 Dec 2019
6 min read
Save for later

How Microsoft, Airbnb, Genentech, and Toyota are using PyTorch to build and deploy production-ready AI

Sugandha Lahoti
10 Dec 2019
6 min read
Built by Facebook engineers and researchers, Pytorch is an open-source Python-based deep learning framework for developing new machine learning models, explore neural network architecture and deploy them at scale in production.  PyTorch is known for its advanced indexing and functions, imperative style, integration support, and API simplicity. This is one of the key reasons why developers prefer this framework for research and hackability. PyTorch is also the second-fastest-growing open source project on the GitHub community which includes anybody from developers starting to get acquainted with AI to some of the best known AI researchers and some of the best-known companies doing AI.  At its F8 annual developer conference, Facebook shared how production-ready PyTorch 1.0 is being adopted by the community and the industry. If you want to learn how you can use this framework to build projects in machine intelligence and deep learning, you may go through our book PyTorch Deep Learning Hands-On by authors Sherin Thomas and Sudhanshu Passi. This book demonstrates numerous examples and dynamic AI applications and demonstrates the simplicity and efficiency of PyTorch.  A number of companies are using PyTorch for research and for production. At F8 developer conference this year, Jerome Pesenti, Vice President of AI at Facebook introduced representatives from Microsoft, Airbnb, Genentech, and Toyota Research Institute who talked about how the framework is helping them build, train, and deploy production-ready AI. Below are some excerpts from their talks. Read also: How PyTorch is bridging the gap between research and production at Facebook: PyTorch team at F8 conference How Microsoft uses PyTorch for its language modeling service David Aronchick, Head of Open Source Machine Learning Strategy at Microsoft Azure  At Microsoft, PyTorch is being used in their language modeling service. Language modeling service uses state-of-the-art language models for both 1 P (first-party) and 3 P (third party). Microsoft explored a number of deep learning frameworks but was running into several issues. These included a slow transition from research to production, inconsistent and frequently changing APIs, and a trade-off between high-level ease-of-use and low-level flexibility.  To overcome these issues, in partnership with Facebook Microsoft built an internal language modeling toolkit on top of PyTorch. Using the native extensibility that PyTorch provided, Microsoft was able to build advanced/custom tasks and architecture. It also improved the onboarding of new users and was an active and inviting community. As a result of this work, Microsoft was able to scale the language modeling features to billions of words. It also led to intuitive, static, and consistent APIs which resulted in seamless migration from Language modeling toolkit v0.4 to 1.0. They also saw improvements in model sizes. Microsoft have partnered with ics.ai to deliver conversational AI bots across the public sector in the UK. ICS.ai, based in Basingstoke, have trained their Microsoft AI driven chat bots to scale to the demands of large county councils, healthcare trusts and universities. How Airbnb is using conversational AI tools in PyTorch to enhance customer experience Cindy Chen, Senior machine learning Data Scientist at Airbnb Airbnb has built a dialog assistant to integrate smart replies and enhance their customer experience. The core of their Dialog assistant for customer service at Airbnb is powered by PyTorch. They have built the smart replies recommendation model by treating it as a machine translation problem.  Airbnb is translating the customer's input message into agent responses by building a sequence to sequence model. They leverage PyTorch’s Open neural machine translation library to build the sequence to sequence model.  Using Pytorch has significantly sped up the Airbnb’s model development cycle as PyTorch provides them with state-of-the-art technologies such as various attention mechanisms and beam search.  How Genentech uses Pytorch in drug discovery and cancer therapy Daniel Bozinov, Head of AI - Early clinical development informatics, Genentech At Genentech, PyTorch is being used to develop personalized cancer medicine as well as for drug discovery and in cancer therapy.  For drug development, Genentech has built deep learning models for specific domains to make some predictions about the properties of molecules such as toxicity. They're also applying AI to come up with new cancer therapies. They identify unique molecules specific to cancer cells that are only produced by those cancer cells, potentially sensitizing the immune system to attack those cancer cells and basically treat them like an infection.   PyTorch has been their deep learning framework of choice because of features such as easier debugging, more flexible control structures, being natively pythonic, and it’s Dynamic graphs which yield in faster execution. Their model architecture is inspired by textual entailment in natural language processing. They use a partially recurrent neural network as well as a straightforward feed-forward network, combine the outputs of these two networks and predict the peptide binding. Toyota Research Institute adds new driver support features in cars Adrien Gaidon, Machine Learning Lead, Toyota Research Institute Toyota developed a cutting-edge cloud platform for distributed deep learning on high-resolution sensory inputs, especially video. This was designed to add new driver support features to the cars. PyTorch was instrumental in scaling up Toyota’s deep learning system because of features like simple API, integration with the global Python ecosystem, and overall a great user experience for fast exploration. It’s also fast for training on a very large scale. In addition to amping up TRI’s creativity and expertise, Pytorch has also amplified Toyota’s capabilities to iterate quickly from idea to real-world use cases. The team at TRI is excited for new Pytorch production features that will help them accelerate Toyota even further.  In this post, we have only summarized the talks. At F8, these researchers spoke in length about each of their company’s projects and how PyTorch has been instrumental in their growth. You can watch the full video on YouTube.  If you are inspired to build your PyTorch-based deep learning and machine learning models, we recommend you to go through our book PyTorch Deep Learning Hands-On. Facebook releases PyTorch 1.3 with named tensors, PyTorch Mobile, 8-bit model quantization, and more François Chollet, creator of Keras on TensorFlow 2.0 and Keras integration, tricky design decisions in Deep Learning, and more PyTorch announces the availability of PyTorch Hub for improving machine learning research reproducibility
Read more
  • 0
  • 0
  • 15088

article-image-learning-ngui-unity
Packt
08 May 2015
2 min read
Save for later

Learning NGUI for Unity

Packt
08 May 2015
2 min read
NGUI is a fantastic GUI toolkit for Unity 3D, allowing fast and simple GUI creation and powerful functionality, with practically zero code required. NGUI is a robust UI system both powerful and optimized. It is an effective plugin for Unity, which gives you the power to create beautiful and complex user interfaces while reducing performance costs. Compared to Unity's GUI features, NGUI is much more powerful and optimized. GUI development in Unity requires users to createUI features by scripting lines that display labels, textures and other UI element on the screen. These lines have to be written inside a special function, OnGUI(), that is called for every frame. However, this is no longer necessary with NGUI, as they makethe GUI process much simpler. This book by Charles Pearson, the author of Learning NGUI for Unity, will help you leverage the power of NGUI for Unity to create stunning mobile and PC games and user interfaces. Based on this, this book covers the following topics: Getting started with NGUI Creating NGUI widgets Enhancing your UI C# with NGUI Atlas and font customization The in-game user interface 3D user interface Going mobile Screen sizes and aspect ratios User experience and best practices This book is a practical tutorial that will guide you through creating a fully functional and localized main menu along with 2D and 3D in-game user interfaces. The book starts by teaching you about NGUI's workflow and creating a basic UI, before gradually moving on to building widgets and enhancing your UI. You will then switch to the Android platform to take care of different issues mobile devices may encounter. By the end of this book, you will have the knowledge to create ergonomic user interfaces for your existing and future PC or mobile games and applications developed with Unity 3D and NGUI. The best part of this book is that it covers the user experience and also talks about the best practices to follow when using NGUI for Unity. If you are a Unity 3D developer who wants to create an effective and user-friendly GUI using NGUI for Unity, then this book is for you. Prior knowledge of C# scripting is expected; however, no knowledge of NGUI is required. Resources for Article: Further resources on this subject: Unity Networking – The Pong Game [article] Unit and Functional Tests [article] Components in Unity [article]
Read more
  • 0
  • 0
  • 15083
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-how-to-query-sharded-data-in-mongodb
Amey Varangaonkar
26 Feb 2018
6 min read
Save for later

How to query sharded data in MongoDB

Amey Varangaonkar
26 Feb 2018
6 min read
[box type="note" align="" class="" width=""]The following excerpt is taken from the book Mastering MongoDB 3.x written by Alex Giamas. This book covers the essential as well as advanced administration concepts in MongoDB.[/box] Querying data using a MongoDB shard is different than a single server deployment or a replica set. Instead of connecting to the single server or the primary of the replica set, we connect to the mongos router which decides which shard to ask for our data. In this article, we will explore how the MongoDB query router operates and showcase how the process is similar to working with a replica set, using Ruby. The query router The query router, also known as mongos process, acts as the interface and entry point to our MongoDB cluster. Applications connect to it instead of connecting to the underlying shards and replica sets; mongos executes queries, gathers results, and passes them to our application. mongos doesn't hold any persistent state and is typically low on system resources, and is typically hosted in the same instance as the application server. It is acting as a proxy for requests. When a query comes in, mongos will examine and decide which shards need to execute the query and establish a cursor in each one of them. The Find operation If our query includes the shard key or a prefix of the shard key, mongos will perform a targeted operation, only querying the shards that hold the keys that we are looking for. For example, with a composite shard key of {_id, email, address} on our collection User, we can have a targeted operation with any of the following queries: > db.User.find({_id: 1}) > db.User.find({_id: 1, email: 'alex@packt.com'}) > db.User.find({_id: 1, email: 'janluc@packt.com', address: 'Linwood Dunn'}) All three of them are either a prefix (the first two) or the complete shard key. On the other hand, a query on {email, address} or {address} will not be able to target the right shards, resulting in a broadcast operation. A broadcast operation is any operation that doesn't include the shard key or a prefix of the shard key and results in mongos querying every shard and gathering results from them. It's also known as a scatter-and-gather operation or a fanout query. Sort/limit/skip operations If we want to sort our results, there are two options: If we are using the shard key in our sort criteria, then mongos can determine the order in which it has to query the shard or shards. This results in an efficient and, again, targeted operation. If we are not using the shard key in our sort criteria, then as with a query without sort, it's going to be a fanout query. To sort the results when we are not using the shard key, the primary shard executes a distributed merge sort locally before passing on the sorted result set to mongos. Limit on queries is enforced on each individual shard and then again at the mongos level as there may be results from multiple shards. Skip, on the other hand, cannot be passed on to individual shards and will be applied by mongos after retrieving all the results locally. Update/remove operations In document modifier operations like update and remove, we have a similar situation to find. If we have the shard key in the find section of the modifier, then mongos can direct the query to the relevant shard. If we don't have the shard key in the find section, then it will again be a fanout operation. In essence, we have the following cases for operations with sharding: Type of operation Query topology insert Must have the shard key update Can have the shard key Query with shard key Targeted operation Query without shard key Scatter gather/fanout query Indexed/sorted query with shard key Targeted operation Indexed/sorted query without shard key Distributed sort merge Querying using Ruby Connecting to a sharded cluster using Ruby is no different than connecting to a replica set. Using the Ruby official driver we have to configure the client object to define the set of mongos servers: client = Mongo::Client.new('mongodb://key:password@mongos-server1- Host:mongos-server1-port,mongos-server2-host:mongos-server2- port/admin?ssl=true&authSource=admin') The mongo-ruby-driver will then return a client object that is no different than connecting to a replica set from the Mongo Ruby client. We can then use the client object like we did in previous chapters, with all the caveats around how sharding behaves differently than a standalone server or a replica set with regards to querying and performance. Performance comparison with replica sets Developers and architects are always looking out for ways to compare performance between replica sets and sharded configurations. The way MongoDB implements sharding, it is based on top of replica sets. Every shard in production should be a replica set. The main difference in performance comes from fan out queries. When we are querying without the shard key, MongoDB's execution time is limited by the worst-performing replica set. In addition, when using sorting without the shard key, the primary server has to implement the distributed merge sort on the entire dataset. This means that it has to collect all data from different shards, merge-sort them, and pass them as sorted to mongos. In both cases, network latency and limitations in bandwidth can slow down operations as opposed to a replica set. On the flip side, by having three shards, we can distribute our working set requirements across different nodes, thus serving results from RAM instead of reaching out to the underlying storage, HDD or SSD. On the other hand, writes can be sped up significantly since we are no longer bound by a single node's I/O capacity but we can have writes in as many nodes as there are shards. To sum up, in most cases and especially for the cases that we are using the shard key, both queries and modification operations will be significantly sped up by sharding. If you found this post useful, check out our book Mastering MongoDB 3.x for more tips and techniques on sharding, replication and other database administration tasks related to MongoDB.  
Read more
  • 0
  • 0
  • 15083

article-image-getting-started-json
Packt
28 Oct 2013
6 min read
Save for later

Getting Started with JSON

Packt
28 Oct 2013
6 min read
(For more resources related to this topic, see here.) JSON was developed by Douglas Crockford. It is text-based, lightweight, and a human-readable format for data exchange between clients and servers. JSON is derived from JavaScript and bears a close resemblance to JavaScript objects, but it is not dependent on JavaScript. JSON is language-independent, and support for the JSON data format is available in all the popular languages, some of which are C#, PHP, Java, C++, Python, and Ruby JSON is a format and not a language. Prior to JSON, XML was considered to be the chosen data interchange format. XML parsing required an XML DOM implementation on the client side that would ingest the XML response, and then XPath was used to query the response in order to access and retrieve the data. That made life tedious, as querying for data had to be performed at two levels: first on the server side where the data was being queried from a database, and the second time was on the client side using XPath. JSON does not need any specific implementations; the JavaScript engine in the browser handles JSON parsing. XML messages often tend to be heavy and verbose, and take up a lot of bandwidth while sending the data over a network connection. Once the XML message is retrieved, it has to be loaded into memory to parse it; let us take a look at a students data feed in XML and JSON. The following is an example in XML: Let us take a look at the example in JSON: As we notice, the size of the XML message is bigger when compared to its JSON counterpart, and this is just for two records. A real-time feed will begin with a few thousands and go upwards. Another point to note is the amount of data that has to be generated by the server and then transmitted over the Internet is already big, and XML, as it is verbose, makes it bigger. Given that we are in the age of mobile devices where smart phones and tablets are getting more and more popular by the day, transmitting such large volumes of data on a slower network causes slow page loads, hang ups, and poor user experience, thus driving the users away from the site. JSON has come about to be the preferred Internet data interchange format, to avoid the issues mentioned earlier. Since JSON is used to transmit serialized data over the Internet, we will need to make a note of its MIME type. A MIME (Multipurpose Internet Mail Extensions) type is an Internet media type, which is a two-part identifier for content that is being transferred over the Internet. The MIME types are passed through the HTTP headers of an HTTP Request and an HTTP Response. The MIME type is the communication of content type between the server and the browser. In general, a MIME type will have two or more parts that give the browser information about the type of data that is being sent either in the HTTP Request or in the HTTP Response. The MIME type for JSON data is application/json. If the MIME type headers are not sent across the browser, it treats the incoming JSON as plain text. The Hello World program with JSON Now that we have a basic understanding of JSON, let us work on our Hello World program. This is shown in the screenshot that follows: The preceding program will alert World onto the screen when it is invoked from a browser. Let us pay close attention to the script between the <script> tags. In the first step, we are creating a JavaScript variable and initializing the variable with a JavaScript object. Similar to how we retrieve data from a JavaScript object, we use the key-value pair to retrieve the value. Simply put, JSON is a collection of key and value pairs, where every key is a reference to the memory location where the value is stored on the computer. Now let us take a step back and analyze why we need JSON, if all we are doing is assigning JavaScript objects that are readily available. The answer is, JSON is a different format altogether, unlike JavaScript, which is a language. JSON keys and values have to be enclosed in double quotes, if either are enclosed in single quotes, we will receive an error. Now, let us take a quick look at the similarities and differences between JSON and a normal JavaScript object. If we were to create a similar JavaScript object like our hello_world JSON variable from the earlier example, it would look like the JavaScript object that follows: The big difference here is that the key is not wrapped in double quotes. Since a JSON key is a string, we can use any valid string for a key. We can use spaces, special characters, and hyphens in our keys, which is not valid in a normal JavaScript object. When we use special characters, hyphens, or spaces in our keys, we have to be careful while accessing them. The reason the preceding JavaScript statement doesn't work is that JavaScript doesn't accept keys with special characters, hyphens, or strings. So we have to retrieve the data using a method where we will handle the JSON object as an associative array with a string key. This is shown in the screenshot that follows: Another difference between the two is that a JavaScript object can carry functions within, while a JSON object cannot carry any functions. The example that follows has the property getName, which has a function that alerts the name John Doe when it is invoked: Finally, the biggest difference is that a JavaScript object was never intended to be a data interchange format, while the sole purpose of JSON was to use it as a data interchange format. Summary This article introduced us to JSON, took us through its history, and its advantages over XML. It focussed on how JSON can be used in web applications for data transfer Resources for Article: Further resources on this subject: Syntax Validation in JavaScript Testing [Article] Enhancing Page Elements with Moodle and JavaScript [Article] Making a Better Form using JavaScript [Article]
Read more
  • 0
  • 0
  • 15081

article-image-measures-and-measure-groups-microsoft-analysis-services-part-1
Packt
15 Oct 2009
12 min read
Save for later

Measures and Measure Groups in Microsoft Analysis Services: Part 1

Packt
15 Oct 2009
12 min read
In this two-part article by Chris Webb, we will look at measures and measure groups, ways to control how measures aggregate up, and how dimensions can be related to measure groups. In this part, will cover useful properties of measures, along with built-in measure aggregation types and dimension calculations. Measures and aggregation Measures are the numeric values that our users want to aggregate, slice, dice and otherwise analyze, and as a result, it's important to make sure they behave the way we want them to. One of the fundamental reasons for using Analysis Services is that, unlike a relational database it allows us to build into our cube design business rules about measures: how they should be formatted, how they should aggregate up, how they interact with specific dimensions and so on. It's therefore no surprise that we'll spend a lot of our cube development time thinking about measures. Useful properties of Measures Apart from the AggregateFunction property of a measure, which we'll come to next, there are two other important properties we'll want to set on a measure, once we've created it. Format string The Format String property of a measure specifies how the raw value of the measure gets formatted when it's displayed in query results. Almost all client tools will display the formatted value of a measure, and this allows us to ensure consistent formatting of a measure across all applications that display data from our cube. A notable exception is Excel 2003 and earlier versions, which can only display raw measure values and not formatted values. Excel 2007 will display properly formatted measure values in most cases, but not all. For instance, it ignores the fourth section of the Format String which controls formatting for nulls. Reporting Services can display formatted values in reports, but doesn't by default; this blog entry describes how you can make it do so:  http://tinyurl.com/gregformatstring. There are a number of built-in formats that you can choose from, and you can also build your own by using syntax very similar to the one used by Visual BASIC for Applications (VBA) for number formatting. The Books Online topic FORMAT_STRING Contents gives a complete description of the syntax used. Here are some points to bear in mind when setting the Format String property: If you're working with percentage values, using the % symbol will display your values multiplied by one hundred and add a percentage sign to the end. Note that only the display value gets multiplied by hundred—the real value of the measure will not be, so although your user might see a value of 98% the actual value of the cell would be 0.98. If you have a measure that returns null values in some circumstances and you want your users to see something other than null, don't try to use a MDX calculation to replace the nulls—this will cause severe query performance problems. You can use the fourth section of the Format String property to do this instead—for example, the following: #,#.00;#,#.00;0;NA will display the string NA for null values, while keeping the actual cell value as null without affecting performance. Be careful while using the Currency built-in format: it will format values with the currency symbol for the locale specified in the Language property of the cube. This combination of the Currency format and the Language property is frequently recommended for formatting measures that contain monetary values, but setting this property will also affect the way number formats are displayed: for example, in the UK and the USA, the comma is used as a thousands separator, but in continental Europe it is used as a decimal separator. As a result, if you wanted to display a currency value to a user in a locale that didn't use that currency, then you could end up with confusing results. The value €100,101 would be interpreted as a value just over one hundred Euros to a user in France, but in the UK, it would be interpreted as a value of just over one hundred thousand Euros. You can use the desired currency symbol in a Format String instead, for example '$#,#.00', but this will not have an effect on the thousands and decimal separators used, which will always correspond to the Language setting. You can find an example of how to change the language property using a scoped assignment in the MDX Script here: http://tinyurl.com/gregformatstring. Similarly, while Analysis Services 2008 supports the translation of captions and member names for users in different locales, unlike in previous versions, it will not translate the number formats used. As a result, if your cube might be used by users in different locales you need to ensure they understand whichever number format the cube is using. Display folders Many cubes have a lot of measures on them, and as with dimension hierarchies, it's possible to group measures together into folders to make it easier for your users to find the one they want. Most, but not all, client tools support display folders, so it may be worth checking whether the one you intend to use does. By default each measure group in a cube will have its own folder containing all of the measures on the measure group; these top level measure group folders cannot be removed and it's not possible to make a measure from one measure group appear in a folder under another measure group. By entering a folder name in a measure's Display Folder property, you'll make the measure appear in a folder underneath its measure group with that name; if there isn't already a folder with that name, then one will be created, and folder names are case-sensitive. You can make a measure appear under multiple folders by entering a semi-colon delimited list of names as follows: Folder One; Folder Two. You can also create a folder hierarchy by entering either a forward-slash / or back-slash delimited list (the documentation contradicts itself on which is meant to be used—most client tools that support display folders support both) of folder names as follows: Folder One; Folder TwoFolder Three. Calculated measures defined in the MDX Script can also be associated with a measure group, through the Associated_Measure_Group property, and with a display folder through the Display_Folder property. These properties can be set either in code or in Form View in the Calculations tab in the Cube Editor: If you don't associate a calculated measure with a measure group, but do put it in a folder, the folder will appear at the same level as the folders created for each measure group. Built-in measure aggregation types The most important property of a measure is AggregateFunction; it controls how the measure aggregates up through each hierarchy in the cube. When you run an MDX query, you can think of it as being similar to a SQL SELECT statement with a GROUP BY clause—but whereas in SQL you have to specify an aggregate function to control how each column's values get aggregated, in MDX you specify this for each measure when the cube is designed. Basic aggregation types Anyone with a passing knowledge of SQL will understand the four basic aggregation types available when setting the AggregateFunction property: Sum is the commonest aggregation type to use, probably the one you'll use for 90% of all the measures. It means that the values for this measure will be summed up. Count is another commonly used property value and aggregates either by counting the overall number of rows from the fact table that the measure group is built from (when the Binding Type property, found on the Measure Source dialog that appears when you click on the ellipses button next to the Source property of a measure, is set to Row Binding), or by counting non-null values from a specific measure column (when Binding Type property is set to Column Binding). Min and Max return the minimum and maximum measure values. There isn't a built-in Average aggregation type—as we'll soon see, AverageOfChildren does not do a simple average—but it's very easy to create a calculated measure that returns an average by dividing a measure with AggregateFunction Sum by one with AggregateFunction Count, for example: CREATE MEMBER CURRENTCUBE.[Measures].[Average Measure Example] ASIIF([Measures].[Count Measure]=0, NULL,[Measures].[Sum Measure]/[Measures].[Count Measure]); Distinct Count The DistinctCount aggregation type counts the number of distinct values in a column in your fact table, similar to a Count(Distinct) in SQL. It's generally used in scenarios where you're counting some kind of key, for example, finding the number of unique Customers who bought a particular product in a given time period. This is, by its very nature, an expensive operation for Analysis Services and queries that use DistinctCount measures can perform worse than those which use additive measures. It is possible to get distinct count values using MDX calculations but this almost always performs worse; it is also possible to use many-to-many dimensions to get the same results and this may perform better in some circumstances; see the section on "Distinct Count" in the "Many to Many Revolution" white paper, available at http://tinyurl.com/m2mrev. When you create a new distinct count measure, BIDS will create a new measure group to hold it automatically. Each distinct count measure needs to be put into its own measure group for query performance reasons, and although it is possible to override BIDS and create a distinct count measure in an existing measure group with measures that have other aggregation types, we strongly recommend that you do not do this. None The None aggregation type simply means that no aggregation takes place on the measure at all. Although it might seem that a measure with this aggregation type displays no values at all, that's not true: it only contains values at the lowest possible granularity in the cube, at the intersection of the key attributes of all the dimensions. It's very rarely used, and only makes sense for values such as prices that should never be aggregated. If you ever find that your cube seems to contain no data even though it has processed successfully, check to see if you have accidentally deleted the Calculate statement from the beginning of your MDX Script. Without this statement, no aggregation will take place within the cube and you'll only see data at the intersection of the leaves of every dimension, as if every measure had AggregateFunction None. Semi-additive aggregation types The semi-additive aggregation types are: AverageOfChildren FirstChild LastChild FirstNonEmpty LastNonEmpty They behave the same as measures with aggregation type Sum on all dimensions except Time dimensions. In order to get Analysis Services to recognize a Time dimension, you'll need to have set the dimension's Type property to Time in the Dimension Editor. Sometimes you'll have multiple, role-playing Time dimensions in a cube, and if you have semi-additive measures, they'll be semi-additive for just one of these Time dimensions. In this situation, Analysis Services 2008 RTM uses the first Time dimension in the cube that has a relationship with the measure group containing the semi-additive measure. You can control the order of dimensions in the cube by dragging and dropping them in the Dimensions pane in the bottom left-hand corner of the Cube Structure tab of the Cube Editor; the following blog entry describes how to do this in more detail: http://tinyurl.com/gregsemiadd. However, this behavior has changed between versions in the past and may change again in the future. Semi-additive aggregation is extremely useful when you have a fact table that contains snapshot data. For example, if you had a fact table containing information on the number of items in stock in a warehouse, then it would never make sense to aggregate these measures over time: if you had ten widgets in stock on January 1, eleven in stock on January 2, eight on January 3 and so on, the value you would want to display for the whole of January would never be the sum of the number of items in stock on each day in January. The value you do display depends on your organization's business rules. Let's take a look at what each of the semi-additive measure values actually do: AverageOfChildren displays the average of all the values at the lowest level of granularity on the Time dimension. So, for example, if Date was the lowest level of granularity, when looking at a Year value, then Analysis Services would display the average value for all days in the year. FirstChild displays the value of the first time period at the lowest level of granularity, for example, the first day of the year. LastChild displays the value of the last time period at the lowest level of granularity, for example, the last day of the year. FirstNonEmpty displays the value of the first time period at the lowest level of granularity that is not empty, for example the first day of the year that has a value. LastNonEmpty displays the value of the last time period at the lowest level of granularity that is not empty, for example the last day of the year that has a value. This is the most commonly used semi-additive type; a good example of its use would be where the measure group contains data about stock levels in a warehouse, so when you aggregated along the Time dimension what you'd want to see is the amount of stock you had at the end of the current time period. The following screenshot of an Excel pivot table illustrates how each of these semi-additive aggregation types works: Note that the semi-additive measures only have an effect above the lowest level of granularity on a Time dimension. For dates like July 17th in the screenshot above, where there is no data for the Sum measure, the LastNonEmpty measure still returns null and not the value of the last non-empty date.
Read more
  • 0
  • 0
  • 15076

article-image-xamarin-test-cloud-for-api-monitoring-tutorial
Gebin George
16 Jul 2018
7 min read
Save for later

Xamarin Test Cloud for API Monitoring [Tutorial]

Gebin George
16 Jul 2018
7 min read
Xamarin Test Cloud can help us identify applications' functionality-related issues on real devices. It is a great source of application monitoring in terms of testing on different mobile devices and with different versions of operating systems. Getting a detailed analysis of various applications' functions is very important to make sure our application is running as expected on our target devices. With that being said, it is also critical to the application to be able to run on different operating system versions, and to analyze how it performs and how much memory usage it has. In this mobile DevOps tutorial, we will discuss how to use Xamarin Test Cloud and the analytics after running an application on different sets of devices. This article is an excerpt from the book, Mobile DevOps,  written by Rohin Tak and Jhalak Modi. We will be using two different applications here to see the monitoring analytics and compare them, to get a better understanding of how this helps us identify various performance and functionality-related issues in our application. Below are the applications we will be using: PhoneCallApp Xamarin Store PhoneCallApp Let's go through some steps to see how to monitor our PhoneCallApp: Go to https://testcloud.xamarin.com/. Click on the PhoneCallApp icon to get to the details of the test runs: On the next page, you'll see a list of tests run for the application: Now, because we have only run one test so far, Test Cloud does not provide us with the graphical metrics shown in the preceding screenshot. In other examples we'll see next, you'll be able to see a more detailed comparison of different test runs. Click on the test run from the list to see its results: The test run listed is the one we ran earlier in previous chapters and uploaded from our machine to Xamarin Test Cloud using the command line. To get an idea of this interface, let's have a look at different parts of Xamarin Test Cloud's interface. Now, this is an overview screen that shows a summary of all the tests run for this application: This screen shows summary details, such as how many tests failed from the total number of tests run, how many times the app ran on a device, how many devices these tests were run on, and much more. This screen is very useful to get a brief idea when you want to get a report on how your application is doing on different devices and OS versions. The next thing you'll see in the left pane is the list of UITests included in the test run: This screen basically has a list of all the Xamarin.UITests that you included in your project. You can click on these different tests to see their respective results on the right side of the screen. Let's click on the test from the list in the preceding screen. This will take us to the next screen, which has detailed reports for the test run: Have a close look at the left pane on this screen. It gives us some steps of the test run on the device. These steps are only what we had written previously in the code to take a screenshot of every activity the test does. The steps are as mentioned (we are using the screens of the test code written in previous chapters here): App started: Take a screenshot when the app starts; this was written in the BeforeEachTest() method in the Tests.cs file: Call button pressed: This step is when the Xamarin.UITest presses the call button to make a call: Failed step (the assert): This is the last step and is shown to provide proof of the failed step, so you can see the outcome that we received and compare it with what was expected. This was the final assert that decides whether the test passes or not, based on the outcome in the Assert.IsTrue() condition. You can click on each of these steps in the left pane and analyze the screenshots taken to see exactly what went on during the test. This is a great way to see exactly what went wrong when the test failed. Now, sometimes the screenshots are not enough to identify the issue. For a more detailed analysis, Test Cloud also provides us with Device Log, as shown in the following screenshot: Device logs are a great way to see what's going on under the hood and get more detailed information about the application's behavior and how the device itself behaves when the application is run on it. This can help pinpoint the issues when a test fails on the device; logs are always a savior in that sort of scenario. Click on the Device Log and you can see step-by-step logs for each screenshot on the same screen: When a test fails, Test Cloud provides us with one more option, to see the Test Failures: It's very useful for automated test developers to see the exception information when a test fails. Last but not least, there is also a Test Log option, which can be used to get a consolidated log of the entire test run: Xamarin Store app Now that we have seen different options provided by Test Cloud to monitor our application and its functionality using test runs, let's see how the dashboard and tests look when we have multiple test runs on various physical devices with different OS versions. This will give us a better idea of how comparative monitoring can be done on Test Cloud to analyze an application's behavior on different devices, and compare them with one another. The Xamarin Store application is a sample application provided by Test Cloud on its platform to help understand the platform and get an idea of the dashboard. Let's go through the steps to understand how to monitor your application running on multiple devices, and how to compare different test runs: Go to the Test Cloud home page, just like in the previous example, and click on the Xamarin Store icon: On the next screen, you'll see a graphical representation of different test runs and brief information about how many tests failed of the total tests run, what the application size is, and its peak memory usage information during different test runs: This gives us a nice comparative look at how our application is performing on different test runs. It is possible that the application was performing fine during the first run, and then some code changes made some functionality fail. So, this graph is very useful to monitor a timeline of changes that affected application functionality. You can further click on the graph or the test run to see an overview of it. Now, this screen gives us a great view of how an application running on different devices can be monitored. It's a very nice way to keep track of the application on different devices and OS versions: Let's click on one of the steps to see the results of the step on multiple devices: The red icon shows failed tests. This page shows all the devices you chose to run the test on; it shows all the devices the test passed on, and shows a red flag on failed devices. You can further click on each device to get device-specific screens and logs. To summarize, we performed API monitoring efficiently using Xamarin Test Cloud. If you found this post useful, do check out the book Mobile DevOps, to deliver continuous integration and delivery for Mobile applications. API Gateway and its Need API and Intent-Driven Networking What is Azure API Management?
Read more
  • 0
  • 0
  • 15074
article-image-aws-global-infrastructure
Packt
06 Jul 2015
5 min read
Save for later

AWS Global Infrastructure

Packt
06 Jul 2015
5 min read
In this article by Uchit Vyas, who is the author of the Mastering AWS Development book, we will see how to use AWS services in detail. It is important to have a choice of placing applications as close as possible to your users or customers, in order to ensure the lowest possible latency and best user experience while deploying them. AWS offers a choice of nine regions located all over the world (for example, East Coast of the United States, West Coast of the United States, Europe, Tokyo, Singapore, Sydney, and Brazil), 26 redundant Availability Zones, and 53 Amazon CloudFront points of presence. (For more resources related to this topic, see here.) It is very crucial and important to have the option to put applications as close as possible to your customers and end users by ensuring the best possible lowest latency and user-expected features and experience, when you are creating and deploying apps for performance. For this, AWS provides worldwide means to the regions located all over the world. To be specific via name and location, they are as follows: US East (Northern Virginia) region US West (Oregon) region US West (Northern California) region EU (Ireland) Region Asia Pacific (Singapore) region Asia Pacific (Sydney) region Asia Pacific (Tokyo) region South America (Sao Paulo) region US GovCloud In addition to regions, AWS has 25 redundant Availability Zones and 51 Amazon CloudFront points of presence. Apart from these infrastructure-level highlights, they have plenty of managed services that can be the cream of AWS candy bar! The managed services bucket has the following listed services: Security: For every organization, security in each and every aspect is the vital element. For that, AWS has several remarkable security features that distinguishes AWS from other Cloud providers as follows : Certifications and accreditations Identity and Access Management Right now, I am just underlining the very important security features. Global infrastructure: AWS provides a fully-functional, flexible technology infrastructure platform worldwide, with managed services over the globe with certain characteristics, for example: Multiple global locations for deployment Low-latency CDN service Reliable, low-latency DNS service Compute: AWS offers a huge range of various cloud-based core computing services (including variety of compute instances that can be auto scaled to justify the needs of your users and application), a managed elastic load balancing service, and more of fully managed desktop resources on the pathway of cloud. Some of the common characteristics of computer services include the following: Broad choice of resizable compute instances Flexible pricing opportunities Great discounts for always on compute resources Lower hourly rates for elastic workloads Wide-ranging networking configuration selections A widespread choice of operating systems Virtual desktops Save further as you grow with tiered pricing model Storage: AWS offers low cost with high durability and availability with their storage services. With pay-as-you-go pricing model with no commitment, provides more flexibility and agility in services and processes for storage with a highly secured environment. AWS provides storage solutions and services for backup, archive, disaster recovery, and so on. They also support block, file, and object kind of storages with a highly available and flexible infrastructure. A few major characteristics for storage are as follows: Cost-effective, high-scale storage varieties Data protection and data management Storage gateway Choice of instance storage options Content delivery and networking: AWS offers a wide set of networking services that enables you to create a logical isolated network that the architect defines and to create a private network connection to the AWS infrastructure, with fault tolerant, scalable, and highly available DNS service. It also provides delivery services for content to your end users, by very low latency and high data transfer speed with AWS CDN service. A few major characteristics for content delivery and networking are as follows: Application and media files delivery Software and large file distribution Private content Databases: AWS offers fully managed, distributed relational and NoSQL type of database services. Moreover, database services are capable of in-memory caching, sharding, and scaling with/without data warehouse solutions. A few major characteristics for databases are as follows: RDS DynamoDB Redshift ElastiCache Application services: AWS provides a variety of managed application services with lower cost such as application streaming and queuing, transcoding, push notification, searching, and so on. A few major characteristics for databases are as follows: AppStream CloudSearch Elastic transcoder SWF, SES, SNS, SQS Deployment and management: AWS offers management of credentials to explore AWS services such as monitor services, application services, and updating stacks of AWS resources. They also have deployment and security services alongside with AWS API activity. A few major characteristics for deployment and management services are as follows: IAM CloudWatch Elastic Beanstalk CloudFormation Data Pipeline OpsWorks CloudHSM Cloud Trail Summary There are a few more additional important services from AWS, such as support, integration with existing infrastructure, Big Data, and ecosystem, which puts it on the top of other infrastructure providers. As a cloud architect, it is necessary to learn about cloud service offerings and their all-important functionalities. Resources for Article: Further resources on this subject: Amazon DynamoDB - Modelling relationships, Error handling [article] Managing Microsoft Cloud [article] Amazon Web Services [article]
Read more
  • 0
  • 0
  • 15051

article-image-tapping-into-financial-data-via-gpt-4
Anshul Saxena
20 Sep 2023
15 min read
Save for later

Tapping into Financial Data via GPT-4

Anshul Saxena
20 Sep 2023
15 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights and books. Don't miss out – sign up today!IntroductionFor anyone new to the world of stocks, looking at the charts and numbers can feel a bit like trying to read a foreign language. But with the right tools and a step-by-step approach, understanding these numbers becomes a whole lot easier. This guide aims to walk beginners through a basic analysis of APPLE's stock data. Here's what will be covered:Step 1. Descriptive Statistics: The first step is to look at APPLE's stock prices. This means seeing the high points, the low points, and understanding how often they change.Step 2. Time Series Analysis: This is all about checking the price of the stock over time. It helps to see if the price is going up, going down, or staying about the same.Step 3. Volatility Analysis: This part helps to see how quickly and by how much the price changes. It's a way to see if the stock is calm or if it has a lot of ups and downs. Step 4. Technical Indicators: These are simple tools that help guess what might happen next with the stock price. It's like trying to guess the end of a story by looking at what happened in the middle. Step 5. Volume Analysis: This is about seeing how many times the stock was bought or sold. It gives a clue about how many people are interested in this stock. With these five easy steps aided by prompts and Python codes, anyone can start to understand the basics of APPLE's stock data. It's a straightforward way to see the story behind the numbers. You can access the data here (Link)Step 1: Descriptive StatisticsUsing descriptive statistics, we assess the stock's central tendencies and dispersion by calculating the mean, median, and standard deviation of prices and volume, and pinpointing the highest and lowest values over the specified period.Prompt 1: As a stock analyst, you are tasked with understanding the nuances of a given stock dataset. Start by loading the dataset that encompasses stock prices, specifically the Open, High, Low, and Close values, alongside trading volume. Compute fundamental descriptive statistics like the mean, median, standard deviation, as well as the maximum and minimum values for each mentioned category. Once done, provide a succinct summary detailing the key findings, highlighting any discernible trends or potential outliers that could be of significance to investors.Response: We'll start by generating basic descriptive statistics for the stock prices (Open, High, Low, Close) and the trading volume.The descriptive statistics for the stock prices and volume are as follows:  The next step is to visualize the dataset as a time seriesStep 2: Time Series AnalysisThrough time series analysis, we visualize stock price trends by plotting Open, High, Low, and Close values, while also examining seasonality, identifying prevailing trends, and detecting any anomalies or outliers in prices or volume.Prompt 2: As a stock analyst, begin by loading the dataset and ensure it includes Date, Open, High, Low, and Close prices. Standardize the "Date" column format and set it as the index. Plot these stock prices against dates, making each distinct. Analyze the graph for evident trends, significant peaks, or troughs, and ponder potential influencing factors.Response: It appears that the dataset contains the columns "Date", "Close/Last", "Volume", "Open", "High", and "Low". The "Date" column has inconsistent formats, so we'll need to standardize it. Additionally, the prices are represented as strings with dollar signs; we'll need to convert them to float values for analysis.I'll begin by:1.      Standardizing the "Date" column format.2.      Converting the price columns to float values.3.      Setting the "Date" column as the index. import pandas as pd # Load the dataset data = pd.read_excel("/mnt/data/HistoricalData_1Yr.xlsx") # Display the first few rows of the dataset data.head() The dataset has been appropriately formatted. The next step is to plot the stock prices (Open, High, Low, and Close/Last) against their respective dates. Each price will be represented with a distinct line for clarity.import matplotlib.pyplot as plt # Plotting the stock prices plt.figure(figsize=(15, 7)) for col in price_columns:    plt.plot(data.index, data[col], label=col) plt.title("Stock Prices Over Time") plt.xlabel("Date") plt.ylabel("Price ($)") plt.legend() plt.grid(True) plt.tight_layout() plt.show()Interpretation: When looking at a stock chart, I first noticed its overall upward movement. Yet, near the end, it seemed to stabilize or maybe even decline a bit. Two clear points caught my attention: a high point in the middle and a low point at the start. Stock prices can change due to company news or broader events, and sometimes, high trading volumes can make prices shift dramatically. It's also important to remember that the general mood of investors can set the direction for a stock's movement. Now let’s look at the volatility to check if the share is tradable or notStep 3: Volatility AnalysisBy calculating daily returns and computing the rolling standard deviation, we can effectively analyze and track the volatility of a financial instrument over time.Prompt 3: As a stock analyst, it's imperative to grasp the volatility of a stock, a crucial aspect for investors. To achieve this, compute the stock's daily returns by measuring the percentage change in the closing price from one day to the subsequent one. Once computed, proceed to plot these daily returns, which will visually represent the stock's volatility throughout the chosen time frame. Your plot, reflecting the daily returns over time, will offer invaluable insights into the stock's stability or fluctuations, helping stakeholders make informed decisions.Response: To grasp the volatility of the stock:We'll compute the stock's daily returns. This is achieved by measuring the percentage change in the closing price from one day to the subsequent one.We'll plot these daily returns over time, which will offer a visual representation of the stock's volatility throughout the chosen time frame.# Compute the daily returns data['Daily Returns'] = data['Close/Last'].pct_change() # Display the first few rows of the daily returns data['Daily Returns'].head()The daily returns have been successfully computed. As expected, the first entry is NaN since there's no previous day to calculate the percentage change from.Now, let's plot these daily returns over time to visually assess the stock's volatility.# Plotting the daily returns plt.figure(figsize=(15, 7)) plt.plot(data.index, data['Daily Returns'], label='Daily Returns', color='blue') plt.axhline(0, color='red', linestyle='--')  # Adding a horizontal line at y=0 for reference plt.title("Daily Returns Over Time") plt.xlabel("Date") plt.ylabel("Daily Return") plt.legend() plt.grid(True) plt.tight_layout() plt.show()Interpretation: When I look at the stock's daily returns, I see its ups and downs around the red zero line, showing its riskiness. Close-to-zero movements mean the stock is steady, while bigger jumps indicate more unpredictability. Different events, like company news or global happenings, can cause these shifts. So, when choosing stocks, it's crucial to know whether you prefer a calm ride or a roller coaster of returns.Since we know every information is inbuilt price data itself let’s analyze it to decode that informationStep 4: Forming Technical IndicatorsThrough the calculation of moving averages, such as the 50-day and 200-day, and the identification of golden or death crosses, coupled with the computation of the RSI, we can gauge market momentum and discern overbought or oversold conditions.Prompt 4: As a seasoned stock analyst, you recognize the importance of technical indicators like moving averages to gauge the potential direction of a stock's momentum. For this analysis, focus on two primary moving averages: the 50-day and the 200-day. By calculating these, you can ascertain pivotal trend indications. Notably, watch for the "Golden Cross," a bullish signal that arises when the 50-day moving average surpasses the 200-day moving average. Conversely, be wary of the "Death Cross," which is a bearish sign indicating potential downtrends when the 50-day moving average drops below the 200-day one. These cross points can serve as critical markers for investors, helping them strategize their market positions.Response: The 50-day and 200-day moving averages are among the most observed technical indicators in stock analysis. These moving averages help to smooth out price data to create a single flowing line, making it easier to identify the direction of the trend.Let's proceed with the following steps:Calculate the 50-day moving average for the closing price.Calculate the 200-day moving average for the closing price.Plot both moving averages alongside the stock's closing price.Identify and annotate potential "Golden Cross" and "Death Cross" points on the plot.Let's begin by calculating the two moving averages.# Calculate the 50-day and 200-day moving averages data['50-day MA'] = data['Close/Last'].rolling(window=50).mean() data['200-day MA'] = data['Close/Last'].rolling(window=200).mean() # Display the first few rows of the moving averages data[['Close/Last', '50-day MA', '200-day MA']].tail() The 50-day and 200-day moving averages have been calculated successfully. To better understand their implications:We'll plot the stock's closing price alongside the two moving averages.We'll highlight and annotate any "Golden Cross" and "Death Cross" occurrences on the chart. Let's visualize the data. # Plotting the closing price and moving averages plt.figure(figsize=(15, 7)) plt.plot(data.index, data['Close/Last'], label='Close Price', color='black', alpha=0.5) plt.plot(data.index, data['50-day MA'], label='50-day MA', color='blue') plt.plot(data.index, data['200-day MA'], label='200-day MA', color='red') # Identifying the Golden Cross and Death Cross golden_cross = data[(data['50-day MA'] > data['200-day MA']) & (data['50-day MA'].shift(1) <= data['200-day MA'].shift(1))] death_cross = data[(data['50-day MA'] < data['200-day MA']) & (data['50-day MA'].shift(1) >= data['200-day MA'].shift(1))] # Annotating the Golden Cross and Death Cross on the plot for date in golden_cross.index:    plt.annotate('Golden Cross', xy=(date, data['200-day MA'].loc[date]),                 xytext=(date, data['200-day MA'].loc[date] + 10),                 arrowprops=dict(facecolor='green', arrowstyle='->'), color='green')   for date in death_cross.index:    plt.annotate('Death Cross', xy=(date, data['200-day MA'].loc[date]),                 xytext=(date, data['200-day MA'].loc[date] - 10),                 arrowprops=dict(facecolor='red', arrowstyle='->'), color='red') # Other plot settings plt.title("Stock Price with 50-day and 200-day Moving Averages") plt.xlabel("Date") plt.ylabel("Price ($)") plt.legend() plt.grid(True) plt.tight_layout() plt.show() Interpretation: The chart shows the stock's closing price with two different line averages: a 50-day (blue) and a 200-day (red). When the 50-day line goes above the 200-day line, marked by green arrows, it usually suggests the stock might go up. On the other hand, red arrows where the 50-day line dips below indicate the stock might drop. These crossing points, known as the "Golden" and "Death" Cross, can help investors gauge the stock's potential direction. Simply put, these averages provide hints on whether the stock is on an upward or downward path. Let’s have a look at the volume of the APPLE stock.Step 5: Volume AnalysisBy analyzing volume spikes in relation to stock price movements and plotting volume over time, we can discern potential patterns and their impact on market dynamics.Prompt 5: In your role as a stock analyst, understanding trading volume's nuances is equally as vital as tracking price movements. Now, turn your attention to plotting the trading volume against time. This visualization can reveal patterns or substantial spikes in trading volume, which might correlate with notable price shifts. Such correlations can often indicate key market events or investor sentiment shifts. The graph you produce, illustrating the trading volume over the period, will provide a clearer picture of the stock's liquidity and the market's interest in it. This, in turn, will guide investment decisions and strategies.Response: Trading volume is a critical metric for stock analysts. It provides insights into the stock's liquidity and the market's interest in it. High trading volumes can indicate strong interest in a stock, and when paired with price movements, can reveal significant market events or shifts in investor sentiment.Let's proceed with the following steps:1. Plot the trading volume against time.2. Highlight and annotate any substantial spikes in trading volume.3. Overlay the stock's closing price to see if there's any correlation between trading volume spikes and significant price shifts.# Plotting the trading volume against time fig, ax1 = plt.subplots(figsize=(15, 7)) # Twin the axes for two different y-axes ax2 = ax1.twinx() # Plotting trading volume on ax1 ax1.bar(data.index, data['Volume'], color='gray', alpha=0.5, label='Volume') ax1.set_xlabel('Date') ax1.set_ylabel('Trading Volume', color='gray') ax1.tick_params(axis='y', labelcolor='gray') ax1.set_title("Trading Volume and Stock Price Over Time") # Plotting closing price on ax2 ax2.plot(data.index, data['Close/Last'], color='blue', label='Close Price') ax2.set_ylabel('Close Price ($)', color='blue') ax2.tick_params(axis='y', labelcolor='blue') # Highlighting and annotating substantial volume spikes average_volume = data['Volume'].mean() high_volume_dates = data[data['Volume'] > 1.5 * average_volume] for date in high_volume_dates.index:    ax1.annotate('High Volume', xy=(date, high_volume_dates['Volume'].loc[date]),                 xytext=(date, high_volume_dates['Volume'].loc[date] + 5e6),                 arrowprops=dict(facecolor='red', arrowstyle='->'), color='red') plt.grid(True) plt.tight_layout() plt.show() Let's begin by visualizing the trading volume against time.Interpretation: The chart displays both the stock's closing price (blue line) and how often it was traded (gray bars). Larger gray bars show days when the stock was traded more frequently. Some days, highlighted with red arrows, even saw an unusually high trading activity. Looking at these spikes alongside the blue line, we can guess if positive or negative news affected the stock. Generally, when a stock is traded often, it's easier to buy or sell without greatly changing its price. This chart helps investors gauge interest in the stock and its stability, supporting smarter investment choices.ConclusionIn conclusion, while the world of stocks might initially seem like an intricate puzzle, it truly isn't as daunting as it first appears. By systematically breaking down APPLE's stock data through the aforementioned steps, even a beginner can gain valuable insights into the dynamics of the stock market. Think of it as assembling the pieces of a story – from setting the scene with descriptive statistics to reaching the climax with volume analysis. And with the added advantage of specific prompts and Python code to guide the way, understanding the ebb and flow of stocks becomes a clear and attainable goal. So, here's to turning those charts and numbers from intimidating to intriguing, and uncovering the fascinating story they hold within. Author BioDr. Anshul Saxena is an author, corporate consultant, inventor, and educator who assists clients in finding financial solutions using quantum computing and generative AI. He has filed over three Indian patents and has been granted an Australian Innovation Patent. Anshul is the author of two best-selling books in the realm of HR Analytics and Quantum Computing (Packt Publications). He has been instrumental in setting up new-age specializations like decision sciences and business analytics in multiple business schools across India. Currently, he is working as Assistant Professor and Coordinator – Center for Emerging Business Technologies at CHRIST (Deemed to be University), Pune Lavasa Campus. Dr. Anshul has also worked with reputed companies like IBM as a curriculum designer and trainer and has been instrumental in training 1000+ academicians and working professionals from universities and corporate houses like UPES, CRMIT, and NITTE Mangalore, Vishwakarma University, Pune & Kaziranga University, and KPMG, IBM, Altran, TCS, Metro CASH & Carry, HPCL & IOC. With a work experience of 5 years in the domain of financial risk analytics with TCS and Northern Trust, Dr. Anshul has guided master's students in creating projects on emerging business technologies, which have resulted in 8+ Scopus-indexed papers. Dr. Anshul holds a PhD in Applied AI (Management), an MBA in Finance, and a BSc in Chemistry. He possesses multiple certificates in the field of Generative AI and Quantum Computing from organizations like SAS, IBM, IISC, Harvard, and BIMTECH.Author of the book: Financial Modeling Using Quantum Computing
Read more
  • 0
  • 0
  • 15036

article-image-push-your-data-web
Packt
22 Feb 2016
27 min read
Save for later

Push your data to the Web

Packt
22 Feb 2016
27 min read
This article covers the following topics: An introduction to the Shiny app framework Creating your first Shiny app The connection between the server file and the user interface The concept of reactive programming Different types of interface layouts, widgets, and Shiny tags How to create a dynamic user interface Ways to share your Shiny applications with others How to deploy Shiny apps to the web (For more resources related to this topic, see here.) Introducing Shiny – the app framework The Shiny package delivers a powerful framework to build fully featured interactive Web applications just with R and RStudio. Basic Shiny applications typically consist of two components: ~/shinyapp |-- ui.R |-- server.R While the ui.R function represents the appearance of the user interface, the server.R function contains all the code for the execution of the app. The look of the user interface is based on the famous Twitter bootstrap framework, which makes the look and layout highly customizable and fully responsive. In fact, you only need to know R and how to use the shiny package to build a pretty web application. Also, a little knowledge of HTML, CSS, and JavaScript may help. If you want to check the general possibilities and what is possible with the Shiny package, it is advisable to take a look at the inbuilt examples. Just load the library and enter the example name: library(shiny) runExample("01_hello") As you can see, running the first example opens the Shiny app in a new window. This app creates a simple histogram plot where you can interactively change the number of bins. Further, this example allows you to inspect the corresponding ui.R and server.R code files. There are currently eleven inbuilt example apps: 01_hello 02_text 03_reactivity 04_mpg 05_sliders 06_tabsets 07_widgets 08_html 09_upload 10_download 11_timer These examples focus mainly on the user interface possibilities and elements that you can create with Shiny. Creating a new Shiny web app with RStudio RStudio offers a fast and easy way to create the basis of every new Shiny app. Just click on New Project and select the New Directory option in the newly opened window: After that, click on the Shiny Web Application field: Give your new app a name in the next step, and click on Create Project: RStudio will then open a ready-to-use Shiny app by opening a prefilled ui.R and server.R file: You can click on the now visible Run App button in the right corner of the file pane to display the prefilled example application. Creating your first Shiny application In your effort to create your first Shiny application, you should first create or consider rough sketches for your app. Questions that you might ask in this context are, What do I want to show? How do I want it to show?, and so on. Let's say we want to create an application that allows users to explore some of the variables of the mtcars dataset. The data was extracted from the 1974 Motor Trend US magazine, and comprises fuel consumption and 10 aspects of automobile design and performance for 32 automobiles (1973–74 models). Sketching the final app We want the user of the app to be able to select one out of the three variables of the dataset that gets displayed in a histogram. Furthermore, we want users to get a summary of the dataset under the main plot. So, the following figure could be a rough project sketch: Constructing the user interface for your app We will reuse the already opened ui.R file from the RStudio example, and adapt it to our needs. The layout of the ui.R file for your first app is controlled by nested Shiny functions and looks like the following lines: library(shiny) shinyUI(pageWithSidebar( headerPanel("My First Shiny App"), sidebarPanel( selectInput(inputId = "variable", label = "Variable:", choices = c ("Horsepower" = "hp", "Miles per Gallon" = "mpg", "Number of Carburetors" = "carb"), selected = "hp") ), mainPanel( plotOutput("carsPlot"), verbatimTextOutput ("carsSummary") ) )) Creating the server file The server file holds all the code for the execution of the application: library(shiny) library(datasets) shinyServer(function(input, output) { output$carsPlot <- renderPlot({ hist(mtcars[,input$variable], main = "Histogram of mtcars variables", xlab = input$variable) }) output$carsSummary <- renderPrint({ summary(mtcars[,input$variable]) }) }) The final application After changing the ui.R and the server.R files according to our needs, just hit the Run App button and the final app opens in a new window: As planned in the app sketch, the app offers the user a drop-down menu to choose the desired variable on the left side, and shows a histogram and data summary of the selected variable on the right side. Deconstructing the final app into its components For a better understanding of the Shiny application logic and the interplay of the two main files, ui.R and server.R, we will disassemble your first app again into its individual parts. The components of the user interface We have divided the user interface into three parts: After loading the Shiny library, the complete look of the app gets defined by the shinyUI() function. In our app sketch, we chose a sidebar look; therefore, the shinyUI function holds the argument, pageWithSidebar(): library(shiny) shinyUI(pageWithSidebar( ... The headerPanel() argument is certainly the simplest component, since usually only the title of the app will be stored in it. In our ui.R file, it is just a single line of code: library(shiny) shinyUI(pageWithSidebar( titlePanel("My First Shiny App"), ... The sidebarPanel() function defines the look of the sidebar, and most importantly, handles the input of the variables of the chosen mtcars dataset: library(shiny) shinyUI(pageWithSidebar( titlePanel("My First Shiny App"), sidebarPanel( selectInput(inputId = "variable", label = "Variable:", choices = c ("Horsepower" = "hp", "Miles per Gallon" = "mpg", "Number of Carburetors" = "carb"), selected = "hp") ), ... Finally, the mainPanel() function ensures that the output is displayed. In our case, this is the histogram and the data summary for the selected variables: library(shiny) shinyUI(pageWithSidebar( titlePanel("My First Shiny App"), sidebarPanel( selectInput(inputId = "variable", label = "Variable:", choices = c ("Horsepower" = "hp", "Miles per Gallon" = "mpg", "Number of Carburetors" = "carb"), selected = "hp") ), mainPanel( plotOutput("carsPlot"), verbatimTextOutput ("carsSummary") ) )) The server file in detail While the ui.R file defines the look of the app, the server.R file holds instructions for the execution of the R code. Again, we use our first app to deconstruct the related server.R file into its main important parts. After loading the needed libraries, datasets, and further scripts, the function, shinyServer(function(input, output) {} ), defines the server logic: library(shiny) library(datasets) shinyServer(function(input, output) { The marked lines of code that follow translate the inputs of the ui.R file into matching outputs. In our case, the server side output$ object is assigned to carsPlot, which in turn was called in the mainPanel() function of the ui.R file as plotOutput(). Moreover, the render* function, in our example it is renderPlot(), reflects the type of output. Of course, here it is the histogram plot. Within the renderPlot() function, you can recognize the input$ object assigned to the variables that were defined in the user interface file: library(shiny) library(datasets) shinyServer(function(input, output) { output$carsPlot <- renderPlot({ hist(mtcars[,input$variable], main = "Histogram of mtcars variables", xlab = input$variable) }) ... In the following lines, you will see another type of the render function, renderPrint() , and within the curly braces, the actual R function, summary(), with the defined input variable: library(shiny) library(datasets) shinyServer(function(input, output) { output$carsPlot <- renderPlot({ hist(mtcars[,input$variable], main = "Histogram of mtcars variables", xlab = input$variable) }) output$carsSummary <- renderPrint({ summary(mtcars[,input$variable]) }) }) There are plenty of different render functions. The most used are as follows: renderPlot: This creates normal plots renderPrin: This gives printed output types renderUI: This gives HTML or Shiny tag objects renderTable: This gives tables, data frames, and matrices renderText: This creates character strings Every code outside the shinyServer() function runs only once on the first launch of the app, while all the code in between the brackets and before the output functions runs as often as a user visits or refreshes the application. The code within the output functions runs every time a user changes the widget that belongs to the corresponding output. The connection between the server and the ui file As already inspected in our decomposed Shiny app, the input functions of the ui.R file are linked with the output functions of the server file. The following figure illustrates this again: The concept of reactivity Shiny uses a reactive programming model, and this is a big deal. By applying reactive programming, the framework is able to be fast, efficient, and robust. Briefly, changing the input in the user interface, Shiny rebuilds the related output. Shiny uses three reactive objects: Reactive source Reactive conductor Reactive endpoint For simplicity, we use the formal signs of the RStudio documentation: The implementation of a reactive source is the reactive value; that of a reactive conductor is a reactive expression; and the reactive endpoint is also called the observer. The source and endpoint structure As taught in the previous section, the defined input of the ui.R links is the output of the server.R file. For simplicity, we use the code from our first Shiny app again, along with the introduced formal signs: ... output$carsPlot <- renderPlot({ hist(mtcars[,input$variable], main = "Histogram of mtcars variables", xlab = input$variable) }) ... The input variable, in our app these are the Horsepower; Miles per Gallon, and Number of Carburetors choices, represents the reactive source. The histogram called carsPlot stands for the reactive endpoint. In fact, it is possible to link the reactive source to numerous reactive endpoints, and also conversely. In our Shiny app, we also connected the input variable to our first and second output—carsSummary: ... output$carsPlot <- renderPlot({ hist(mtcars[,input$variable], main = "Histogram of mtcars variables", xlab = input$variable) }) output$carsSummary <- renderPrint({ summary(mtcars[,input$variable]) }) ... To sum it up, this structure ensures that every time a user changes the input, the output refreshes automatically and accordingly. The purpose of the reactive conductor The reactive conductor differs from the reactive source and the endpoint is so far that this reactive type can be dependent and can have dependents. Therefore, it can be placed between the source, which can only have dependents and the endpoint, which in turn can only be dependent. The primary function of a reactive conductor is the encapsulation of heavy and difficult computations. In fact, reactive expressions are caching the results of these computations. The following graph displays a possible connection of the three reactive types: In general, reactivity raises the impression of a logically working directional system; after input, the output occurs. You get the feeling that an input pushes information to an output. But this isn't the case. In reality, it works vice versa. The output pulls the information from the input. And this all works due to sophisticated server logic. The input sends a callback to the server, which in turn informs the output that pulls the needed value from the input and shows the result to the user. But of course, for a user, this all feels like an instant updating of any input changes, and overall, like a responsive app's behavior. Of course, we have just touched upon the main aspects of reactivity, but now you know what's really going on under the hood of Shiny. Discovering the scope of the Shiny user interface After you know how to build a simple Shiny application, as well as how reactivity works, let us take a look at the next step: the various resources to create a custom user interface. Furthermore, there are nearly endless possibilities to shape the look and feel of the layout. As already mentioned, the entire HTML, CSS, and JavaScript logic and functions of the layout options are based on the highly flexible bootstrap framework. And, of course, everything is responsive by default, which makes it possible for the final application layout to adapt to the screen of any device. Exploring the Shiny interface layouts Currently, there are four common shinyUI () page layouts: pageWithSidebar() fluidPage() navbarPage() fixedPage() These page layouts can be, in turn, structured with different functions for a custom inner arrangement structure of the page layout. In the following sections, we are introducing the most useful inner layout functions. As an example, we will use our first Shiny application again. The sidebar layout The sidebar layout, where the sidebarPanel() function is used as the input area, and the mainPanel() function as the output, just like in our first Shiny app. The sidebar layout uses the pageWithSidebar() function: library(shiny) shinyUI(pageWithSidebar( headerPanel("The Sidebar Layout"), sidebarPanel( selectInput(inputId = "variable", label = "This is the sidebarPanel", choices = c("Horsepower" = "hp", "Miles per Gallon" = "mpg", "Number of Carburetors" = "carb"), selected = "hp") ), mainPanel( tags$h2("This is the mainPanel"), plotOutput("carsPlot"), verbatimTextOutput("carsSummary") ) )) When you only change the first three functions, you can create exactly the same look as the application with the fluidPage() layout. This is the sidebar layout with the fluidPage() function: library(shiny) shinyUI(fluidPage( titlePanel("The Sidebar Layout"), sidebarLayout( sidebarPanel( selectInput(inputId = "variable", label = "This is the sidebarPanel", choices = c("Horsepower" = "hp", "Miles per Gallon" = "mpg", "Number of Carburetors" = "carb"), selected = "hp") ), mainPanel( tags$h2("This is the mainPanel"), plotOutput("carsPlot"), verbatimTextOutput("carsSummary") ) ) ))   The grid layout The grid layout is where rows are created with the fluidRow() function. The input and output are made within free customizable columns. Naturally, a maximum of 12 columns from the bootstrap grid system must be respected. This is the grid layout with the fluidPage () function and a 4-8 grid: library(shiny) shinyUI(fluidPage( titlePanel("The Grid Layout"), fluidRow( column(4, selectInput(inputId = "variable", label = "Four-column input area", choices = c("Horsepower" = "hp", "Miles per Gallon" = "mpg", "Number of Carburetors" = "carb"), selected = "hp") ), column(8, tags$h3("Eight-column output area"), plotOutput("carsPlot"), verbatimTextOutput("carsSummary") ) ) )) As you can see from inspecting the previous ui.R file, the width of the columns is defined within the fluidRow() function, and the sum of these two columns adds up to 12. Since the allocation of the columns is completely flexible, you can also create something like the grid layout with the fluidPage() function and a 4-4-4 grid: library(shiny) shinyUI(fluidPage( titlePanel("The Grid Layout"), fluidRow( column(4, selectInput(inputId = "variable", label = "Four-column input area", choices = c("Horsepower" = "hp", "Miles per Gallon" = "mpg", "Number of Carburetors" = "carb"), selected = "hp") ), column(4, tags$h5("Four-column output area"), plotOutput("carsPlot") ), column(4, tags$h5("Another four-column output area"), verbatimTextOutput("carsSummary") ) ) )) The tabset panel layout The tabsetPanel() function can be built into the mainPanel() function of the aforementioned sidebar layout page. By applying this function, you can integrate several tabbed outputs into one view. This is the tabset layout with the fluidPage() function and three tab panels: library(shiny) shinyUI(fluidPage( titlePanel("The Tabset Layout"), sidebarLayout( sidebarPanel( selectInput(inputId = "variable", label = "Select a variable", choices = c("Horsepower" = "hp", "Miles per Gallon" = "mpg", "Number of Carburetors" = "carb"), selected = "hp") ), mainPanel( tabsetPanel( tabPanel("Plot", plotOutput("carsPlot")), tabPanel("Summary", verbatimTextOutput("carsSummary")), tabPanel("Raw Data", dataTableOutput("tableData")) ) ) ) )) After changing the code to include the tabsetPanel() function, the three tabs with the tabPanel() function display the respective output. With the help of this layout, you are no longer dependent on representing several outputs among themselves. Instead, you can display each output in its own tab, while the sidebar does not change. The position of the tabs is flexible and can be assigned to be above, below, right, and left. For example, in the following code file detail, the position of the tabsetPanel() function was assigned as follows: ... mainPanel( tabsetPanel(position = "below", tabPanel("Plot", plotOutput("carsPlot")), tabPanel("Summary", verbatimTextOutput("carsSummary")), tabPanel("Raw Data", tableOutput("tableData")) ) ) ... The navlist panel layout The navlistPanel() function is similar to the tabsetPanel() function, and can be seen as an alternative if you need to integrate a large number of tabs. The navlistPanel() function also uses the tabPanel() function to include outputs: library(shiny) shinyUI(fluidPage( titlePanel("The Navlist Layout"), navlistPanel( "Discovering The Dataset", tabPanel("Plot", plotOutput("carsPlot")), tabPanel("Summary", verbatimTextOutput("carsSummary")), tabPanel("Another Plot", plotOutput("barPlot")), tabPanel("Even A Third Plot", plotOutput("thirdPlot"), "More Information", tabPanel("Raw Data", tableOutput("tableData")), tabPanel("More Datatables", tableOutput("moreData")) ) ))   The navbar page as the page layout In the previous examples, we have used the page layouts, fluidPage() and pageWithSidebar(), in the first line. But, especially when you want to create an application with a variety of tabs, sidebars, and various input and output areas, it is recommended that you use the navbarPage() layout. This function makes use of the standard top navigation of the bootstrap framework: library(shiny) shinyUI(navbarPage("The Navbar Page Layout", tabPanel("Data Analysis", sidebarPanel( selectInput(inputId = "variable", label = "Select a variable", choices = c("Horsepower" = "hp", "Miles per Gallon" = "mpg", "Number of Carburetors" = "carb"), selected = "hp") ), mainPanel( plotOutput("carsPlot"), verbatimTextOutput("carsSummary") ) ), tabPanel("Calculations" … ), tabPanel("Some Notes" … ) )) Adding widgets to your application After inspecting the most important page layouts in detail, we now look at the different interface input and output elements. By adding these widgets, panels, and other interface elements to an application, we can further customize each page layout. Shiny input elements Already, in our first Shiny application, we got to know a typical Shiny input element: the selection box widget. But, of course, there are a lot more widgets with different types of uses. All widgets can have several arguments; the minimum setup is to assign an inputId, which instructs the input slot to communicate with the server file, and a label to communicate with a widget. Each widget can also have its own specific arguments. As an example, we are looking at the code of a slider widget. In the previous screenshot are two versions of a slider; we took the slider range for inspection: sliderInput(inputId = "sliderExample", label = "Slider range", min = 0, max = 100, value = c(25, 75)) Besides the mandatory arguments, inputId and label, three more values have been added to the slider widget. The min and max arguments specify the minimum and maximum values that can be selected. In our example, these are 0 and 100. A numeric vector was assigned to the value argument, and this creates a double-ended range slider. This vector must logically be within the set minimum and maximum values. Currently, there are more than twenty different input widgets, which in turn are all individually configurable by assigning to them their own set of arguments. A brief overview of the output elements As we have seen, the output elements in the ui.R file are connected to the rendering functions in the server file. The mainly used output elements are: htmlOutput imageOutput plotOutput tableOutput textOutput verbatimTextOutput downloadButton Due to their unambiguous naming, the purpose of these elements should be clear. Individualizing your app even further with Shiny tags Although you don't need to know HTML to create stunning Shiny applications, you have the option to create highly customized apps with the usage of raw HTML or so-called Shiny tags. To add raw HTML, you can use the HTML() function. We will focus on Shiny tags in the following list. Currently, there are over a 100 different Shiny tag objects, which can be used to add text styling, colors, different headers, visual and audio, lists, and many more things. You can use these tags by writing tags $tagname. Following is a brief list of useful tags: tags$h1: This is first level header; of course you can also use the known h1 -h6 tags$hr: This makes a horizontal line, also known as a thematic break tags$br: This makes a line break, a popular way to add some space tags$strong = This makes the text bold tags$div: This makes a division of text with a uniform style tags$a: This links to a webpage tags$iframe: This makes an inline frame for embedding possibilities The following ui.R file and corresponding screenshot show the usage of Shiny tags by an example: shinyUI(fluidPage( fluidRow( column(6, tags$h3("Customize your app with Shiny tags!"), tags$hr(), tags$a(href = "http://www.rstudio.com", "Click me"), tags$hr() ), column(6, tags$br(), tags$em("Look - the R project logo"), tags$br(), tags$img(src = "http://www.r-project.org/Rlogo.png") ) ), fluidRow( column(6, tags$strong("We can even add a video"), tags$video(src = "video.mp4", type = "video/mp4", autoplay = NA, controls = NA) ), column(6, tags$br(), tags$ol( tags$li("One"), tags$li("Two"), tags$li("Three")) ) ) ))   Creating dynamic user interface elements We know how to build completely custom user interfaces with all the bells and whistles. But all the introduced types of interface elements are fixed and static. However, if you need to create dynamic interface elements, Shiny offers three ways to achieve it: The conditinalPanel() function: The renderUI() function The use of directly injected JavaScript code In the following section, we only show how to use the first two ways, because firstly, they are built into the Shiny package, and secondly, the JavaScript method is indicated as experimental. Using conditionalPanel The condtionalPanel() functions allow you to show or hide interface elements dynamically, and is set in the ui.R file. The dynamic of this function is achieved by JavaScript expressions, but as usual in the Shiny package, all you need to know is R programming. The following example application shows how this function works for the ui.R file: library(shiny) shinyUI(fluidPage( titlePanel("Dynamic Interface With Conditional Panels"), column(4, wellPanel( sliderInput(inputId = "n", label= "Number of points:", min = 10, max = 200, value = 50, step = 10) )), column(5, "The plot below will be not displayed when the slider value", "is less than 50.", conditionalPanel("input.n >= 50", plotOutput("scatterPlot", height = 300) ) ) )) The following example application shows how this function works for the Related server.R file: library(shiny) shinyServer(function(input, output) { output$scatterPlot <- renderPlot({ x <- rnorm(input$n) y <- rnorm(input$n) plot(x, y) }) }) The code for this example application was taken from the Shiny gallery of RStudio (http://shiny.rstudio.com/gallery/conditionalpanel-demo.html). As readable in both code files, the defined function, input.n, is the linchpin for the dynamic behavior of the example app. In the conditionalPanel() function, it is defined that inputId="n" must have a value of 50 or higher, while the input and output of the plot will work as already defined. Taking advantage of the renderUI function The renderUI() function is hooked, contrary to the previous model, to the server file to create a dynamic user interface. We have already introduced different render output functions in this article. The following example code shows the basic functionality using the ui.R file: # Partial example taken from the Shiny documentation numericInput("lat", "Latitude"), numericInput("long", "Longitude"), uiOutput("cityControls") The following example code shows the basic functionality of the Related sever.R file: # Partial example output$cityControls <- renderUI({ cities <- getNearestCities(input$lat, input$long) checkboxGroupInput("cities", "Choose Cities", cities) }) As described, the dynamic of this method gets defined in the renderUI() process as an output, which then gets displayed through the uiOutput() function in the ui.R file. Sharing your Shiny application with others Typically, you create a Shiny application not only for yourself, but also for other users. There are a two main ways to distribute your app; either you let users download your application, or you deploy it on the web. Offering a download of your Shiny app By offering the option to download your final Shiny application, other users can run your app locally. Actually, there are four ways to deliver your app this way. No matter which way you choose, it is important that the user has installed R and the Shiny package on his/her computer. Gist Gist is a public code sharing pasteboard from GitHub. To share your app this way, it is important that both the ui.R file and the server.R file are in the same Gist and have been named correctly. Take a look at the following screenshot: There are two options to run apps via Gist. First, just enter runGist("Gist_URL") in the console of RStudio; or second, just use the Gist ID and place it in the shiny::runGist("Gist_ID") function. Gist is a very easy way to share your application, but you need to keep in mind that your code is published on a third-party server. GitHub The next way to enable users to download your app is through a GitHub repository: To run an application from GitHub, you need to enter the command, shiny::runGitHub ("Repository_Name", "GitHub_Account_Name"), in the console: Zip file There are two ways to share a Shiny application by zip file. You can either let the user download the zip file over the web, or you can share it via email, USB stick, memory card, or any other such device. To download a zip file via the Web, you need to type runUrl ("Zip_File_URL") in the console: Package Certainly, a much more labor-intensive but also publically effective way is to create a complete R package for your Shiny application. This especially makes sense if you have built an extensive application that may help many other users. Another advantage is the fact that you can also publish your application on CRAN. Later in the book, we will show you how to create an R package. Deploying your app to the web After showing you the ways users can download your app and run it on their local machines, we will now check the options to deploy Shiny apps to the web. Shinyapps.io http://www.shinyapps.io/ is a Shiny app- hosting service by RStudio. There is a free-to- use account package, but it is limited to a maximum of five applications, 25 so-called active hours, and the apps are branded with the RStudio logo. Nevertheless, this service is a great way to publish one's own applications quickly and easily to the web. To use http://www.shinyapps.io/ with RStudio, a few R packages and some additional operating system software needs to be installed: RTools (If you use Windows) GCC (If you use Linux) XCode Command Line Tools (If you use Mac OS X) The devtools R package The shinyapps package Since the shinyapps package is not on CRAN, you need to install it via GitHub by using the devtools package: if (!require("devtools")) install.packages("devtools") devtools::install_github("rstudio/shinyapps") library(shinyapps) When everything that is needed is installed ,you are ready to publish your Shiny apps directly from the RStudio IDE. Just click on the Publish icon, and in the new window you will need to log in to your http://www.shinyapps.io/ account once, if you are using it for the first time. All other times, you can directly create a new Shiny app or update an existing app: After clicking on Publish, a new tab called Deploy opens in the console pane, showing you the progress of the deployment process. If there is something set incorrectly, you can use the deployment log to find the error: When the deployment is successful, your app will be publically reachable with its own web address on http://www.shinyapps.io/.   Setting up a self-hosted Shiny server There are two editions of the Shiny Server software: an open source edition and the professional edition. The open source edition can be downloaded for free and you can use it on your own server. The Professional edition offers a lot more features and support by RStudio, but is also priced accordingly. Diving into the Shiny ecosystem Since the Shiny framework is such an awesome and powerful tool, a lot of people, and of course, the creators of RStudio and Shiny have built several packages around it that are enormously extending the existing functionalities of Shiny. These almost infinite possibilities of technical and visual individualization, which are possible by deeply checking the Shiny ecosystem, would certainly go beyond the scope of this article. Therefore, we are presenting only a few important directions to give a first impression. Creating apps with more files In this article, you have learned how to build a Shiny app consisting of two files: the server.R and the ui.R. To include every aspect, we first want to point out that it is also possible to create a single file Shiny app. To do so, create a file called app.R. In this file, you can include both the server.R and the ui.R file. Furthermore, you can include global variables, data, and more. If you build larger Shiny apps with multiple functions, datasets, options, and more, it could be very confusing if you do all of it in just one file. Therefore, single-file Shiny apps are a good idea for simple and small exhibition apps with a minimal setup. Especially for large Shiny apps, it is recommended that you outsource extensive custom functions, datasets, images, and more into your own files, but put them into the same directory as the app. An example file setup could look like this: ~/shinyapp |-- ui.R |-- server.R |-- helper.R |-- data |-- www |-- js |-- etc   To access the helper file, you just need to add source("helpers.R") into the code of your server.R file. The same logic applies to any other R files. If you want to read in some data from your data folder, you store it in a variable that is also in the head of your server.R file, like this: myData &lt;- readRDS("data/myDataset.rds") Expanding the Shiny package As said earlier, you can expand the functionalities of Shiny with several add-on packages. There are currently ten packages available on CRAN with different inbuilt functions to add some extra magic to your Shiny app. shinyAce: This package makes available Ace editor bindings to enable a rich text-editing environment within Shiny. shinybootstrap2: The latest Shiny package uses bootstrap 3; so, if you built your app with bootstrap 2 features, you need to use this package. shinyBS: This package adds the additional features of the original Twitter Bootstraptheme, such as tooltips, modals, and others, to Shiny. shinydashboard: This packages comes from the folks at RStudio and enables the user to create stunning and multifunctional dashboards on top of Shiny. shinyFiles: This provides functionality for client-side navigation of the server side file system in Shiny apps. shinyjs: By using this package, you can perform common JavaScript operations in Shiny applications without having to know any JavaScript. shinyRGL: This package provides Shiny wrappers for the RGL package. This package exposes RGL's ability to export WebGL visualization in a shiny-friendly format. shinystan: This package is, in fact, not a real add-on. Shinystan is a fantastic full-blown Shiny application to give users a graphical interface for Markov chain Monte Carlo simulations. shinythemes: This packages gives you the option of changing the whole look and feel of your application by using different inbuilt bootstrap themes. shinyTree: This exposes bindings to jsTree—a JavaScript library that supports interactive trees—to enable rich, editable trees in Shiny. Of course, you can find a bunch of other packages with similar or even more functionalities, extensions, and also comprehensive Shiny apps on GitHub. Summary To learn more about Shiny, the following books published by Packt Publishing (https://www.packtpub.com/) are recommended: Learning Shiny (https://www.packtpub.com/application-development/learning-shiny) Mastering Machine Learning with R (https://www.packtpub.com/big-data-and-business-intelligence/mastering-machine-learning-r) Mastering Data Analysis with R (https://www.packtpub.com/big-data-and-business-intelligence/mastering-data-analysis-r)
Read more
  • 0
  • 0
  • 15036
article-image-internet-connected-smart-water-meter-0
Packt
23 Oct 2015
15 min read
Save for later

Internet-Connected Smart Water Meter

Packt
23 Oct 2015
15 min read
In this article by Pradeeka Seneviratne author of the book, Internet of Things with Arduino Blueprints, explains that for many years and even now, water meter readings have been collected manually. To do this, a person has to visit the location where the water meter is installed. In this article, you will learn how to make a smart water meter with an LCD screen that has the ability to connect to the internet and serve meter readings to the consumer through the Internet. In this article, you shall do the following: Learn about water flow sensors and its basic operation Learn how to mount and plumb a water flow meter on and into the pipeline Read and count the water flow sensor pulses Calculate the water flow rate and volume Learn about LCD displays and connecting with Arduino Convert a water flow meter to a simple web server and serve meter readings through the Internet (For more resources related to this topic, see here.) Prerequisites An Arduino UNO R3 board (http://store.arduino.cc/product/A000066) Arduino Ethernet Shield R3 (https://www.adafruit.com/products/201) A liquid flow sensor (http://www.futurlec.com/FLOW25L0.shtml) A Hitachi HD44780 DRIVER compatible LCD Screen (16 x 2) (https://www.sparkfun.com/products/709) A 10K ohm resistor A 10K ohm potentiometer (https://www.sparkfun.com/products/9806) Few Jumper wires with male and female headers (https://www.sparkfun.com/products/9140) A breadboard (https://www.sparkfun.com/products/12002) Water flow sensors The heart of a water flow sensor consists of a Hall effect sensor (https://en.wikipedia.org/wiki/Hall_effect_sensor) that outputs pulses for magnetic field changes. Inside the housing, there is a small pinwheel with a permanent magnet attached to it. When the water flows through the housing, the pinwheel begins to spin, and the magnet attached to it passes very close to the Hall effect sensor in every cycle. The Hall effect sensor is covered with a separate plastic housing to protect it from the water. The result generates an electric pulse that transitions from low voltage to high voltage, or high voltage to low voltage, depending on the attached permanent magnet's polarity. The resulting pulse can be read and counted using the Arduino. For this project, we will use a Liquid Flow sensor from Futurlec (http://www.futurlec.com/FLOW25L0.shtml). The following image shows the external view of a Liquid Flow Sensor: Liquid flow sensor – the flow direction is marked with an arrow The following image shows the inside view of the liquid flow sensor. You can see a pinwheel that is located inside the housing: Pinwheel attached inside the water flow sensor Wiring the water flow sensor with Arduino The water flow sensor that we are using with this project has three wires, which are the following: Red (or it may be a different color) wire, which indicates the Positive terminal Black (or it may be a different color) wire, which indicates the Negative terminal Brown (or it may be a different color) wire, which indicates the DATA terminal All three wire ends are connected to a JST connector. Always refer to the datasheet of the product for wiring specifications before connecting them with the microcontroller and the power source. When you use jumper wires with male and female headers, do the following: Connect positive terminal of the water flow sensor to Arduino 5V. Connect negative terminal of the water flow sensor to Arduino GND. Connect DATA terminal of the water flow sensor to Arduino digital pin 2. Water flow sensor connected with Arduino Ethernet Shield using three wires You can directly power the water flow sensor using Arduino since most residential type water flow sensors operate under 5V and consume a very low amount of current. Read the product manual for more information about the supply voltage and supply current range to save your Arduino from high current consumption by the water flow sensor. If your water flow sensor requires a supply current of more than 200mA or a supply voltage of more than 5v to function correctly, then use a separate power source with it. The following image illustrates jumper wires with male and female headers: Jumper wires with male and female headers Reading pulses The water flow sensor produces and outputs digital pulses that denote the amount of water flowing through it. These pulses can be detected and counted using the Arduino board. Let's assume the water flow sensor that we are using for this project will generate approximately 450 pulses per liter (most probably, this value can be found in the product datasheet). So 1 pulse approximately equals to [1000 ml/450 pulses] 2.22 ml. These values can be different depending on the speed of the water flow and the mounting polarity of the water flow sensor. Arduino can read digital pulses generating by the water flow sensor through the DATA line. Rising edge and falling edge There are two type of pulses, as listed here:. Positive-going pulse: In an idle state, the logic level is normally LOW. It goes HIGH state, stays there for some time, and comes back to the LOW state. Negative-going pulse: In an idle state, the logic level is normally HIGH. It goes LOW state, stays LOW state for time, and comes back to the HIGH state. The rising and falling edges of a pulse are vertical. The transition from LOW state to HIGH state is called rising edge and the transition from HIGH state to LOW state is called falling edge. Representation of Rising edge and Falling edge in digital signal You can capture digital pulses using either the rising edge or the falling edge. In this project, we will use the rising edge. Reading and counting pulses with Arduino In the previous step, you attached the water flow sensor to Arduino UNO. The generated pulse can be read by Arduino digital pin 2 and the interrupt 0 is attached to it. The following Arduino sketch will count the number of pulses per second and display it on the Arduino Serial Monitor: Open a new Arduino IDE and copy the sketch named B04844_03_01.ino. Change the following pin number assignment if you have attached your water flow sensor to a different Arduino pin: int pin = 2; Verify and upload the sketch on the Arduino board: int pin = 2; //Water flow sensor attached to digital pin 2 volatile unsigned int pulse; const int pulses_per_litre = 450; void setup() { Serial.begin(9600); pinMode(pin, INPUT); attachInterrupt(0, count_pulse, RISING); } void loop() { pulse = 0; interrupts(); delay(1000); noInterrupts(); Serial.print("Pulses per second: "); Serial.println(pulse); } void count_pulse() { pulse++; } Open the Arduino Serial Monitor and blow air through the water flow sensor using your mouth. The number of pulses per second will print on the Arduino Serial Monitor for each loop, as shown in the following screenshot: Pulses per second in each loop The attachInterrupt() function is responsible for handling the count_pulse() function. When the interrupts() function is called, the count_pulse() function will start to collect the pulses generated by the liquid flow sensor. This will continue for 1000 milliseconds, and then the noInterrupts() function is called to stop the operation of count_pulse() function. Then, the pulse count is assigned to the pulse variable and prints it on the serial monitor. This will repeat again and again inside the loop() function until you press the reset button or disconnect the Arduino from the power. Calculating the water flow rate The water flow rate is the amount of water flowing in at a given point of time and can be expressed in gallons per second or liters per second. The number of pulses generated per liter of water flowing through the sensor can be found in the water flow sensor's specification sheet. Let's say there are m pulses per liter of water. You can also count the number of pulses generated by the sensor per second: Let's say there are n pulses per second. The water flow rate R can be expressed as: In litres per second Also, you can calculate the water flow rate in liters per minute using the following formula: For example, if your water flow sensor generates 450 pulses for one liter of water flowing through it, and you get 10 pulses for the first second, then the elapsed water flow rate is: 10/450 = 0.022 liters per second or 0.022 * 1000 = 22 milliliters per second. The following steps will explain you how to calculate the water flow rate using a simple Arduino sketch: Open a new Arduino IDE and copy the sketch named B04844_03_02.ino. Verify and upload the sketch on the Arduino board. The following code block will calculate the water flow rate in milliliters per second: Serial.print("Water flow rate: "); Serial.print(pulse * 1000/pulses_per_litre); Serial.println("milliliters per second"); Open the Arduino Serial Monitor and blow air through the water flow sensor using your mouth. The number of pulses per second and the water flow rate in milliliters per second will print on the Arduino Serial Monitor for each loop, as shown in the following screenshot: Pulses per second and water flow rate in each loop Calculating the water flow volume The water flow volume can be calculated by summing up the product of flow rate and the time interval: Volume = ∑ Flow Rate * Time_Interval The following Arduino sketch will calculate and output the total water volume since the device startup: Open a new Arduino IDE and copy the sketch named B04844_03_03.ino. The water flow volume can be calculated using following code block: volume = volume + flow_rate * 0.1; //Time Interval is 0.1 second Serial.print("Volume: "); Serial.print(volume); Serial.println(" milliliters"); Verify and upload the sketch on the Arduino board. Open the Arduino Serial Monitor and blow air through the water flow sensor using your mouth. The number of pulses per second, water flow rate in milliliters per second, and total volume of water in milliliters will be printed on the Arduino Serial Monitor for each loop, as shown in the following screenshot: Pulses per second, water flow rate and in each loop and sum of volume  To accurately measure water flow rate and volume, the water flow sensor needs to be carefully calibrated. The hall effect sensor inside the housing is not a precision sensor, and the pulse rate does vary a bit depending on the flow rate, fluid pressure, and sensor orientation. Adding an LCD screen to the water meter You can add an LCD screen to your newly built water meter to display readings, rather than displaying them on the Arduino serial monitor. You can then disconnect your water meter from the computer after uploading the sketch on to your Arduino. Using a Hitachi HD44780 driver compatible LCD screen and Arduino Liquid Crystal library, you can easily integrate it with your water meter. Typically, this type of LCD screen has 16 interface connectors. The display has two rows and 16 columns, so each row can display up to 16 characters. The following image represents the top view of a Hitachi HD44760 driver compatible LCD screen. Note that the 16-pin header is soldered to the PCB to easily connect it with a breadboard. Hitachi HD44780 driver compatible LCD screen (16 x 2)—Top View The following image represents the bottom view of the LCD screen. Again, you can see the soldered 16-pin header. Hitachi HD44780 driver compatible LCD screen (16x2)—Bottom View Wire your LCD screen with Arduino as shown in the next diagram. Use the 10k potentiometer to control the contrast of the LCD screen. Now, perform the following steps to connect your LCD screen with your Arduino: LCD RS pin (pin number 4 from left) to Arduino digital pin 8. LCD ENABLE pin (pin number 6 from left) to Arduino digital pin 7. LCD READ/WRITE pin (pin number 5 from left) to Arduino GND. LCD DB4 pin (pin number 11 from left) to Arduino digital pin 6. LCD DB5 pin (pin number 12 from left) to Arduino digital pin 5. LCD DB6 pin (pin number 13 from left) to Arduino digital pin 4. LCD DB7 pin (pin number 14 from left) to Arduino digital pin 3. Wire a 10K pot between Arduino +5V and GND, and wire its wiper (center pin) to LCD screen V0 pin (pin number 3 from left). LCD GND pin (pin number 1 from left) to Arduino GND. LCD +5V pin (pin number 2 from left) to Arduino 5V pin. LCD Backlight Power pin (pin number 15 from left) to Arduino 5V pin. LCD Backlight GND pin (pin number 16 from left) to Arduino GND. Fritzing representation of the circuit Open a new Arduino IDE and copy the sketch named B04844_03_04.ino. First initialize the Liquid Crystal library using following line: #include <LiquidCrystal.h> To create a new LCD object with following parameters, the syntax is LiquidCrystal lcd (RS, ENABLE, DB4, DB5, DB6, DB7): LiquidCrystal lcd(8, 7, 6, 5, 4, 3); Then initialize number of rows and columns in the LCD. Syntax is lcd.begin(number_of_columns, number_of_rows): lcd.begin(16, 2); You can set the starting location to print a text on the LCD screen using following function, syntax is lcd.setCursor(column, row): lcd.setCursor(7, 1); The column and row numbers are 0 index based and the following line will start to print a text in the intersection of the 8th column and 2nd row. Then, use the lcd.print() function to print some text on the LCD screen: lcd.print(" ml/s"); Verify and upload the sketch on the Arduino board. Blow some air through the water flow sensor using your mouth. You can see some information on the LCD screen such as pulses per second, water flow rate, and total water volume from the beginning of the time:  LCD screen output Converting your water meter to a web server In the previous steps, you learned how to display your water flow sensor's readings and calculate water flow rate and total volume on the Arduino serial monitor. In this step, you will learn how to integrate a simple web server to your water flow sensor and remotely read your water flow sensor's readings. You can make an Arduino web server with Arduino WiFi Shield or Arduino Ethernet shield. The following steps will explain how to convert the Arduino water flow meter to a web server with Arduino Wi-Fi shield: Remove all the wires you have connected to your Arduino in the previous sections in this article. Stack the Arduino WiFi shield on the Arduino board using wire wrap headers. Make sure the Arduino WiFi shield is properly seated on the Arduino board. Now, reconnect the wires from water flow sensor to the Wi-Fi shield. Use the same pin numbers as used in the previous steps. Connect the 9VDC power supply to the Arduino board. Connect your Arduino to your PC using the USB cable and upload the next sketch. Once the upload is completed, remove your USB cable from the Arduino. Open a new Arduino IDE and copy the sketch named B04844_03_05.ino. Change the following two lines according to your WiFi network settings, as shown here: char ssid[] = "MyHomeWiFi"; char pass[] = "secretPassword"; Verify and upload the sketch on the Arduino board. Blow the air through the water flow sensor using your mouth, or it would be better if you can connect the water flow sensor to a water pipeline to see the actual operation with the water. Open your web browser, type the WiFi shield's IP address assigned by your network, and hit the Enter key: http://192.168.1.177 You can see your water flow sensor's pulses per second, flow rate, and total volume on the Web page. The page refreshes every 5 seconds to display updated information. You can add an LCD screen to the Arduino WiFi shield as discussed in the previous step. However, remember that you can't use some of the pins in the Wi-Fi shield because they are reserved for SD (pin 4), SS (pin 10), and SPI (pin 11, 12, 13). We have not included the circuit and source code here in order to make the Arduino sketch simple. A little bit about plumbing Typically, the direction of the water flow is indicated by an arrow mark on top of the water flow meter's enclosure. Also, you can mount the water flow meter either horizontally or vertically according to its specifications. Some water flow meters can mount both horizontally and vertically. You can install your water flow meter to a half-inch pipeline using normal BSP pipe connectors. The outer diameter of the connector is 0.78" and the inner thread size is half-inch. The water flow meter has threaded ends on both sides. Connect the threaded side of the PVC connectors to both ends of the water flow meter. Use a thread seal tape to seal the connection, and then connect the other ends to an existing half-inch pipeline using PVC pipe glue or solvent cement. Make sure that you connect the water flow meter with the pipe line in the correct direction. See the arrow mark on top of the water flow meter for flow direction. BNC pipe line connector made by PVC Securing the connection between the water flow meter and BNC pipe connector using thread seal PVC solvent cement. Image taken from https://www.flickr.com/photos/ttrimm/7355734996/ Summary In this article, you gained hands-on experience and knowledge about water flow sensors and counting pulses while calculating and displaying them. Finally, you made a simple web server to allow users to read the water meter through the Internet. You can apply this to any type of liquid, but make sure to select the correct flow sensor because some liquids react chemically with the material that the sensor is made of. You can Google and find which flow sensors support your preferred liquid type. Resources for Article: Further resources on this subject: The Arduino Mobile Robot [article] Arduino Development [article] Getting Started with Arduino [article]
Read more
  • 0
  • 0
  • 15034

article-image-our-first-program
Packt
03 Jan 2017
5 min read
Save for later

Our First Program!

Packt
03 Jan 2017
5 min read
In this article by Syed Omar Faruk Towaha, author of Learning C for Arduino, we will be learning how we can connect our Arduino to our system and we will learn how to write program on the Arduino IDE. (For more resources related to this topic, see here.) First connect your A-B cable to your PC and then connect the cable to the PC. Your PC will make a sound which will confirm that the Arduino has connected to the PC. Now open the Arduino IDE. From menu go to Tools | Board:"Arduino/Genuino Uno". You can select any board you have bought from the list. See the following screenshot for the list: You have to select the port on which the Arduino is connected. There are a lot of things you can do to see on which port your Arduino is connected. Hello Arduino! Let's write our first program on the Arduino IDE. Go to File and click on New. A text editor will open with few lines of code. Delete those lines first, and then type the following code: void setup() { Serial.begin(9600); } void loop() { Serial.print("Hello Arduino!n"); } From menu go to Sketch and click Upload. It is a good practice to verify or compile the code before uploading to the Arduino board. To verify or compile the code you need to go to Sketch and click Verify/Compile. You will see a message Done compiling. on the bottom of the IDE if the code is error free. See the following figure for the explanation: After the successful upload, you need to open the serial monitor of the Arduino IDE. To open the serial monitor, you need to go to Tools and click on Serial Monitor. You will see the following screen: setup() function The setup() function helps to initialize variables, set pin modes, and we can also use libraries here. This function is called first when we compile or upload the whole code. The setup() runs only for the first time it is uploaded to the board and later it runs every time we press the reset button on the Arduino. On our code we used Serial.begin(9600); which sets the data rate per second for the serial communication. We already know that serial communication is the process by which we send data one bit at a time over a communication channel. For Arduino to communicate with the computer we used the data rate 9600 which is a standard data rate. This is also known as baud rate. We can set the baud rate any of the following depending on the connection speed and type. I will recommend using 9600 for general data communication purposes. If you use higher baud rate, the characters we print might be broken: 300 1200 2400 4800 9600 19200 38400 57600 74880 115200 230400 250000 If we do not declare the baud rate inside the setup() function there will be no output on the serial monitor. Baud rate 9600 means 960 characters per second. This is because in serial communication, you need to transmit one start bit, eight data bits and one stop bit, for a total of 10 bits at a speed of 9600 bits per second. loop() function loop() function will let the code run again and again until we disconnect the Arduino. We have written a print statement on the loop() function which will execute for the infinity time. To print something on the serial monitor we need to write the following line: Serial.print("Anything We Want To Print"); Between the quotations we can write anything we want to print. On the last code we have written Serial.print("Hello Arduino!n");. That's why on the serial monitor we have seen Hello Arduino! had been printing for an infinity time. We have used n after Hello Arduino!.This is called escape sequence. For now, just remember we need to put this after each of our line inside the print statement to break a line and print the next command on the net line. We can use Serial.println("Hello Arduino!"); instead of Serial.print("Hello Arduino!n");. Both will give the same result. We need to put Serial.println("Hello Arduino!"); inside the setup() function. Now let's see what happens if we put a print statement inside the setup() function. Have a look at the following figure. Hello Arduino! is printed for only one time: Since C is a case sensitive language we need to be careful about the casing. We cannot use serial.print("Hello Arduino!"); instead of Serial.print("Hello Arduino!n");. Summary In this article we have learned how to connect the Arduino to our system and uploaded the code to our board. We have learned how the Arduino code work. If you are new to Arduino this article is most important. Resources for Article: Further resources on this subject: Zabbix Configuration [article] A Configuration Guide [article] Thorium and Salt API [article]
Read more
  • 0
  • 0
  • 15024
Modal Close icon
Modal Close icon