Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7018 Articles
article-image-bi-pro49-microsoft-fabric-lifecycle-management-data-factory-adds-cicd-to-fabric-data-pipelines-database-mirroring-aws-well-architected-data-analytics-lens
Merlyn Shelley
04 Apr 2024
11 min read
Save for later

BI-Pro#49: Microsoft Fabric Lifecycle Management, Data Factory Adds CI/CD to Fabric Data Pipelines, Database Mirroring, AWS Well-Architected Data Analytics Lens

Merlyn Shelley
04 Apr 2024
11 min read
Subscribe to our BI Pro newsletter for the latest insights. Don't miss out – sign up today!👋 Hello,Welcome to BI-Pro #49, your ultimate guide to data and BI insights! 🚀 ⏩ What's Inside? Python Simplified: Master data validation with Pydantic. Visualize Like a Pro: 30+ tools for stunning data visuals. R for Bioinformatics: Custom visuals for bio data. Interactive Data: JavaScript meets Handsontable. Seaborn Stories: Craft data tales with line plots. MetaGPT Insights: Next-gen data solutions unveiled. 🏭 Industry Scoop: Power BI’s Latest: March's must-know features. Fabric Innovations: Updates and new tools from Microsoft Fabric. AWS Well-Architected Data Analytics Lens: Analytics strategies for the real world. Google Cloud Savings: Cut costs on ETL workflows. Tableau Journeys: From student to BI analyst. 💎 Expert Takes: Deep Dive into Python Deep Learning: The latest from Packt. 👉 Community Buzz: Twitch Chat Analysis, Graph Networks, LLM Data Quality, and Ethical AI: Key conversations this week! Dive into the trends shaping data and BI today! 📥 Feedback on the Weekly EditionTake our weekly survey and get a free PDF copy of our best-selling book, "Interactive Data Visualization with Python - Second Edition."📣 And here's the twist – we're tuning into YOUR frequency! Inspired by a reader's request, we're launching a column just for you. Got a burning question or a topic you're itching to dive into? Drop your suggestions in our content box – because your journey of discovery is our blueprint.We appreciate your input and hope you enjoy the book!Share your thoughts and opinions here! Cheers,Merlyn ShelleyEditor-in-Chief, PacktThanks for reading Packt BI-Pro! Subscribe for free to receive new posts and support our work.Pledge your supportSign Up | Advertise | Archives🚀 GitHub's Most Sought-After Repos🌀 man-group/ArcticDB: ArcticDB is a high-performance DataFrame database designed for Python Data Science, with a Python-centric API for Pandas DataFrames. 🌀 gradio-app/gradio: Gradio is an open-source Python package for building demos or web apps for ML models or Python functions, with easy sharing via built-in features. 🌀 Sinaptik-AI/pandas-ai: PandasAI is a Python library using generative AI to explore, clean, and analyze data with natural language queries.🌀 OpenRefine/OpenRefine: OpenRefine is a powerful Java-based tool for loading, understanding, cleaning, reconciling, and augmenting data, accessible from a web browser. 🌀 Kanaries/pygwalker: PyGWalker simplifies Jupyter Notebook workflows by converting pandas dataframes into interactive user interfaces for data analysis and visualization. 🌀 cleanlab/cleanlab: cleanlab aids in data and label cleaning by identifying issues in ML datasets automatically, enabling better model training with real-world data.Email Forwarded? Join BI-Pro Here!🔮 Data Viz with Python Libraries  🌀 Pydantic Tutorial: Data Validation in Python Made Simple. This blog tutorial explains how to use Pydantic, a data validation and serialization library in Python, to validate and serialize data classes, offering support for custom validators and Python's type hints for field validation. 🌀 30+ Data Visualization Libraries, Frameworks and Apps, Mastering Data Presentation: Explore over 30 data visualization tools like Metabase, Gephi, and Grafana, offering a range of features to transform raw data into meaningful visualizations for better decision-making in industries like tech, healthcare, finance, and marketing. 🌀 Mastering Data Visualization in R for Bioinformatics:  The article delves into data visualization in R for bioinformatics, stressing its role in understanding complex biological data, communicating findings, hypothesis generation, and decision-making. It also discusses Anscombe's Quartet, highlighting the importance of visualizing data before analysis and the limitations of summary statistics. 🌀 Integrating JavaScript charting libraries with Handsontable: The article guides developers on integrating Highcharts, Recharts, and Chart.js with Handsontable for data visualization. It explains the features of each library and provides demos for creating a stock portfolio with interactive charts. 🌀 Data Visualization with Seaborn Line Plot: The article introduces Seaborn, a Python library for data visualization, built on top of Matplotlib. It covers installation and demonstrates creating single line plots and customizing styles for better presentation of data. 🌀 MetaGPT’s Data Interpreter: SOTA Open Source LLM-based Data Solutions. MetaGPT introduces its Data Interpreter, a new agent for streamlined data interpretation and analysis. The Data Interpreter employs advanced techniques for real-time data adaptability, tool integration, and logical inconsistency identification, showcasing superior performance in machine learning tasks. ⚡Stay Informed with Industry HighlightsPower BI 🌀 Power BI March 2024 Feature Summary: The Power BI update introduces visual calculation editing, data model editing in the Power BI Service, and report subscription delivery to OneDrive SharePoint. A new Microsoft Fabric certification exam, DP-600, is also available, with free certification opportunities through the Fabric AI Skills Challenge. 🌀 Announcing the Public Preview of Database Mirroring in Microsoft Fabric: Mirroring, now in Public Preview, allows seamless integration of databases into Microsoft Fabric's OneLake, providing real-time insights without ETL. It simplifies data replication and warehousing, enabling easy data access and analysis across different sources, including data lakes and warehouses. 🌀 Get data with Power Query available in Power BI Report Builder (Preview): Power BI Report Builder now allows connecting to 100+ data sources like Snowflake, Databricks, and AWS Redshift. You can transform data using M-Query for paginated reports. Install the latest version and connect from the "Data" tab. Microsoft Fabric🌀 Microsoft Fabric March 2024 Update: This update brings new features like OneLake File Explorer, Autotune Query Tuning, and Test Framework for Power Query SDK in VS Code to Power BI, enhancing reporting, modeling, service, mobile, and developer experiences. 🌀 Data Factory Adds CI/CD to Fabric Data Pipelines: Fabric engineers with Azure Synapse Analytics and Azure Data Factory experience can now utilize Git integration and built-in Deployment Pipelines in Data Factory data pipelines in Fabric. This public preview offers source control, CI/CD features, and collaborative development environments, enhancing data analytics projects. 🌀 Microsoft Fabric Lifecycle Management – Getting started with Git Integration and Deployment Pipelines: Microsoft Fabric makes Lifecycle Management easy, enabling continuous releases through Git and Deployment Pipelines. Git allows reliable updates for supported items like Lakehouse, Notebooks, and Reports, while Deployment Pipelines clone content between stages like DEV, TEST, UAT, and PROD. AWS BI  🌀 Announcing the AWS Well-Architected Data Analytics Lens: The Data Analytics Lens helps assess and improve analytics platforms on AWS. It offers best practices, such as building ACID-compliant data lakes and leveraging Serverless for data pipelines, aligned with the AWS Well-Architected Framework's pillars for secure, efficient, and cost-effective solutions. 🌀 Improve healthcare services through patient 360: A zero-ETL approach to enable near real-time data analytics. The post discusses how healthcare providers can improve patient care by leveraging AWS services for real-time analytics and personalized healthcare, focusing on a zero-ETL approach to data integration.Google Cloud Dat🌀 Enrich streaming data in Bigtable with Dataflow: The post discusses the importance of event stream processing in data engineering and introduces Apache Beam's Enrichment transform, which simplifies the process of enriching streaming data with Bigtable, improving data context and enabling more meaningful analysis.🌀 Dataflow at-least-once vs. exactly-once streaming modes: The post compares exactly-once and at-least-once processing modes in Dataflow Streaming Engine for streaming jobs. It explains the trade-offs between the two modes and provides guidance on choosing the right mode based on use case requirements. Tableau🌀 Data is both art and science - My Tableau Story: Andy Cotgreave. The post highlights Andy Cotgreave's journey from a data analyst at Oxford to becoming a Senior Technical Evangelist at Tableau. It emphasizes the importance of community engagement, innovation, building a portfolio, and having fun in data visualization. 🌀 Student to BI Analyst, How Tableau Can Lead to a Successful Data Career: This blog discusses Karolina Grodzinska's data visualization journey, from discovering Tableau to winning Iron Viz: Student Edition and becoming a Business Intelligence Analyst at Schneider Electric. Karolina emphasizes the importance of an active Tableau Public profile in career development and shares tips for building a strong portfolio and networking with the Tableau Community. ✨ Expert Insights from Packt CommunityPython Deep Learning - Third Edition - By Ivan VasilevDeveloping NN models for edge devices with TF Lite TF Lite is a TF-derived set of tools that allows us to run models on mobile, embedded, and edge devices. Its versatility is part of TF’s appeal for industrial applications (as opposed to research applications, where PyTorch dominates).The key paradigm of TF Lite is that the models run on-device, contrary to client-server architecture, where the model is deployed on remote, more powerful, hardware. This organization has the following implications (both good and bad): Low-latency execution: The lack of server-round trip significantly reduces the model inference time and allows us to run real-time applications. Privacy: The user data never leaves the device. Internet connectivity: Internet connectivity is not required. Small model size: The devices have limited computational ability, hence the need for small and computationally efficient models. More specifically, TF Lite models are stored in the FlatBuffers (https://flatbuffers.dev/) special efficient portable format, identified by the .tflite file extension. Besides its small size, it allows us to access data directly without parsing/unpacking it first. TF Lite models support a subset of the TF Core operations and allow us to define custom ones: Low power consumption: The devices often run on battery. Divergent training and inference: NN training is a lot more computationally intensive compared to inference. Because of this, the model training runs on a different, more powerful, piece of hardware than the actual devices, where the models will run inference. In addition, TF Lite has the following key features: Multi-platform and multi-language support, including Android (Java), iOS (Objective-C and Swift) devices, web (JavaScript), and Python for all other environments. Google provides a TF Lite wrapper API called MediaPipe Solutions (https://developers.google.com/mediapipe, https://github.com/google/mediapipe/), which supersedes the previous TF Lite API. Optimized for performance. It has end-to-end solution pipelines. TF Lite is oriented toward practical applications, rather than research. Because of this, it includes different pipelines for common ML tasks such as image classification, object detection, text classification, and question answering among others. The computer vision pipelines use modified versions of EfficientNet or MobileNet, and the natural language processing pipelines use BERT-based models. So, how does TF Lite model development work? First, we’ll select a model in one of the following ways:  An existing pre-trained .tflite model (https://tfhub.dev/s?deployment-format=lite). Use MediaPipe Model Maker (https://developers.google.com/mediapipe/solutions/model_maker) to apply feature engineering transfer learning on an existing .tflite model with a custom training dataset. Model Maker only works with Python. Convert a full-fledged TF model into .tflite format. Discover more insights from 'Python Deep Learning - Third Edition' by Ivan Vasilev. Unlock access to the full book and a wealth of other titles with a 7-day free trial in the Packt Library. Start exploring today! Read Here💡 What's the Latest Scoop from the BI Community? 🌀 Real-Time Twitch Chat Sentiment Analysis with Apache Flink: This blog explores building a real-time sentiment analysis application for Twitch chat using Apache Flink. It covers setting up the project, reading Twitch chat messages, performing sentiment analysis, and concludes with a demo. 🌀 Entity Type Prediction with Relational Graph Convolutional Network (PyTorch): This post discusses a Python setup for predicting entity types on heterogeneous graphs using the Relational Graph Convolutional Network (R-GCN) and the RGCNConv module from PyTorch. It explains knowledge graphs, entity type prediction, and the R-GCN model. 🌀 Data Quality Error Detection powered by LLMs: This article explores automating the identification of data errors in tabular datasets using Large Language Models (LLMs). It discusses the Data Dirtiness Score, challenges in data cleaning, and the potential of LLMs in detecting data quality issues. 🌀 Building Ethical AI Starts with the Data Team — Here’s Why: This article discusses the ethical considerations of AI, focusing on model bias, AI usage, and data responsibility. It emphasizes the role of data teams in ensuring ethical AI and suggests steps for data teams to take towards a more ethical future. See you next time!
Read more
  • 0
  • 0
  • 23293

article-image-what-do-we-really-mean-when-we-say-that-software-is-dead-or-dying
Richard Gall
22 Oct 2019
12 min read
Save for later

What do we really mean when we say that software is ‘dead’ or ‘dying’?

Richard Gall
22 Oct 2019
12 min read
“The report of my death was an exaggeration,” Mark Twain once wrote in a letter to journalist Frank Marshall White. Twain's quip is a fitting refrain for much of the software industry. Year after year there is a new wave of opinion from experts declaring this or that software or trend to be dead or, if it’s lucky, merely dying. I was inclined to think this was a relatively new phenomenon, but the topic comes up on Jeff Atwood’s blog Coding Horror as far back as 2009. Atwood quotes from an article from influential software engineer Tom DeMarco in which DeMarco writes that “software engineering is an idea whose time has come and gone.” (The hyperlink that points to DeMarco’s piece is, ironically, now dead). So, it’s clearly not something new to the software industry. In fact, this rhetorical trope can tell us a lot about identity, change, and power in tech. Declaring something to be dead is an expression of insecurity, curiosity and sometimes plain obnoxiousness. Consider these questions from Quora - all of them appear to reflect an odd combination of status anxiety and technical interest: Is DevOps dead? Is React.js dead? Is Linux dead? Why? (yes, really) Is web development a dying career? (It’s also something I wrote about last year.) These questions don’t come out of a vacuum. They’re responses to existing opinions and discussions that are ongoing in different areas. To a certain extent they’re valuable: asking questions like those above and using these sorts of metaphors are ways of assessing the value and relevance of different technologies. That being said, they probably should be taken with a pinch of salt. Although they can be an indicator of community feeling towards a particular approach or tool (the plural of anecdote isn’t exactly data, but it’s not as far off as many people pretend it is), they often tell you as much about the person doing the talking as the thing they’re talking about. To be explicit: saying something is dead or dying is very often a mark of perspective or even identity. This might be frustrating, but it nevertheless provides an insight on how different technologies are valued or being used at any given time. What’s important, then, is to be mindful about why someone would describe this or that technology as dead. What might they be really trying to say? “X software is dead because companies just aren’t hiring for it” One of the reasons you might hear people proclaim a certain technology to be dead is because companies are, apparently, no longer hiring for those skills. It stops appearing on job boards; it’s no longer ‘in-demand’. While this undoubtedly makes sense, and it’s true that what’s in demand will shift and change over time, all too often assertions that ‘no one is hiring x developers any more’ is anecdotal. For example, although there have been many attempts to start rumors that JavaScript is ‘dead’ or that its time has come - like this article from a few years back in which the writer claims that JavaScript developers have been “mind-fucked into thinking that JavaScript is a good programming language” - research done earlier this year shows that 70% of companies are searching for JavaScript developers. So, far from dead. In the same survey Java came out as another programming language that is eagerly sought out by companies: 48% were on the lookout for Java developers. And while this percentage has almost certainly decreased over the last decade (admittedly I couldn’t find any research to back this up), that’s still a significant chunk to debunk the notion that Java is ‘dead’ or ‘dying.’ Software ecosystems don't develop chronologically This write up of the research by Free Code Camp argues that it shows that variation can be found “not between tech stacks but within them.” This suggests that the tech world isn’t quite as Darwinian as it’s often made out to be. It’s not a case of life and death, but instead different ecosystems all evolving in different ways and at different times. So, yes, maybe there is some death (after all, there aren’t as many people using Ember or Backbone as there were in 2014), but, as with many things, it’s actually a little more complicated… “X software is dead because no one’s learning it” If no one’s learning something, it’s presumably a good sign that X technology is dead or dying, right? Well, to a certain extent. It might give you an indication of how wider trends are evolving and even the types of problems that engineers and their employers are trying to solve, but again, it’s important to be cautious. It sounds obvious but just because no one seems to be learning something, it doesn’t mean that people aren’t. So yes, in certain developer communities it might be weird to consider people are learning Java. But with such a significant employer demand there are undoubtedly thousands of people trying to get to grips with it. Indeed, they might well be starting out in their career - but you can’t overlook the importance of established languages as a stepping stone into more niche and specialised roles and into more ‘exclusive’ communities. That said, what people are learning can be instructive in the context of the argument made in the Free Code Camp article mentioned above. If variation inside tech stacks is where change and fluctuation is actually happening, then the fact that people are learning a given library or framework will give a clear indication as to how that specific ecosystem is evolving. Software only really dies when their use cases do But even then, it’s still a leap to say that something’s dead. It’s also somewhat misleading and unhelpful. So, although it might be the case that more people are learning React or Kotlin than other related technologies, that doesn’t cancel out the fact that those other technologies may still have a part to play in a particular use case. Another important aspect to consider when thinking about what people are learning is that it’s often part of the whole economics of the hype cycle. The more something gets talked about, the more individual developers might be intrigued about how it actually works. This doesn’t, of course, mean they would necessarily start using it for professional projects or that we’d start to see adoption across large enterprises. There is a mesh of different forces at play when thinking about tech life cycles - sometimes our metaphors don’t really capture what’s going on. “It’s dead because there are better options out there” There’s one thing I haven’t really touched on that’s important when thinking about death and decline in the context of software: the fact that there are always options. You use one tool, language, framework, library, whatever, because it’s the best suited to your needs. If one option comes to supersede another for whatever reason it’s only natural to regard what came before as obsolete in some way. You can see this way of thinking on a large scale in the context of infrastructure - from virtual machines, to containers, to serverless, the technologies that enable each of those various phases might be considered ‘dead’ as we move from one to the other. Except that just isn’t the case. While containerized solutions might be more popular than virtual machines, and while serverless hints at an alternative to containers, each of these different approaches are still very much in play. Indeed, you might even see these various approaches inside the same software architecture - virtual machines might make sense here, but for this part of an application over there serverless functions are the best option for the job. With this in mind, throwing around the D word is - as mentioned above - misleading. In truth it’s really just a way for the speaker to signal that X just doesn’t work for them anymore and that Y is a much better option for what they’re trying to do. The vanity of performed expertise And that’s fine - learning from other people’s experiences is arguably the best way to learn when it comes to technology (far better than, say, a one dimensional manual or bone dry documentation). But when we use the word ‘dead’ we hide what actually might still be interesting or valuable about a given technology. In our bid to signal our own expertise and knowledge we close down avenues of exploration. And vanity only feeds the potentially damaging circus of hype cycles and burn out even more. So, if Kotlin really is a better option for you then that’s great. But it doesn’t mean Java is dead or dying. Indeed, it’s more likely the case that what we’re seeing are use cases growing and proliferating, with engineering teams and organizations requiring a more diverse set of options for an increasingly diverse and multi faceted range of options. If software does indeed die, then it’s not really a linear process. Various use cases will all evolve and over time they will start to impact one another. Maybe eventually we’ll see Kotlin replace Java as the language evolves to tackle a wider range of use cases. “X software pays more money, so Y software must be dying” The idea that certain technologies are worth more than others feeds into the narrative that certain technologies and tools are dying. But it’s just a myth - and a dangerous one at that. Although there is some research on which technologies are the most high paying, much of it lacks context. So, although this piece on Forbes might look insightful (wow, HBase engineers earn more than $120K!) it doesn’t really give you the wider picture of why these technologies command certain salaries. And, more to the point, it ignores the fact that these technologies are just tools used by people in certain job roles. Indeed, it’s more accurate to say that big data engineers and architects are commanding high salaries than to think anything as trite as Kafka developers are really well-respected by their employers! Talent gaps and industry needs It’s probably more useful to look at variation within specific job roles. By this I mean look at what tools the highest earning full-stack developers or architects are using. At least that would be a little more interesting and instructive. But even then it wouldn’t necessarily tell you whether something has ‘died’. It would simply hint at two things: where the talent gaps are, and what organizations are trying to do. That might give you a flavor of how something is evolving - indeed, it might be useful if you’re a developer or engineer looking for a new job. However, it doesn’t mean that something is dead. Java developers might not be paid a great deal but that doesn’t mean the language is dead. If anything, the opposite is true. It’s alive and well with a massive pool of programmers from which employers can choose. The hype cycle might give us an indication of new opportunities and new solutions, but it doesn’t necessarily offer the solutions we need right now. But what about versioning? And end of life software? Okay, these are important issues. In reality, yes, these are examples of when software really is dead. Well, not quite - there are still complications. For example, even as a new version of a particular technology is released it still takes time for individual projects and wider communities to make the move. Even end of life software that's no longer supported by vendors or maintainers can still have an afterlife in poorly managed projects (the existence of this articles like this suggest that this is more common than you’d hope). In a sense this is zombie software that keeps on living years after debates about whether its alive or dead have ceased. In theory, versioning should be the formal way through which we manage death in the software industry. But the fact that even then our nicely ordered systems still fail to properly bury and retire editions of software packages, languages, and tools, highlights that in reality it's actually really hard to properly kill off software. For all the communities that want to kill software, there are always other groups, whether through force of will or plain old laziness, that want to keep it alive. Conclusion: Software is hard to kill Perhaps that’s why we like to say that certain technologies are dead: not only do such proclamations help to signify how we identify (ie. the type of developer we are), it’s also a rhetorical trick that banishes nuance and complexity. If we are managing complexity and solving tricky and nuanced problems every day, the idea that we can simplify our own area of expertise into something that is digestible - quotable, even - is a way of establishing some sort of control in a field where it feels we have anything but. So, if you’re tempted to ever say that one piece of software product is ‘dead,’ ask yourself what you really mean. And if you overhear someone obnoxiously proclaiming a framework, library or language to be dying, consider what they’re trying to say. Are they just trying to make a point about themselves? What’s the other side of the story? And even if they’re almost correct, to what extent aren’t they correct?
Read more
  • 0
  • 0
  • 23285

article-image-chef-language-and-style
Packt
17 Jan 2017
14 min read
Save for later

Chef Language and Style

Packt
17 Jan 2017
14 min read
In this article by Matthias Marschall, author of the book, Chef Cookbook, Third Edition, we will cover the following section: Using community Chef style Using attributes to dynamically configure recipes Using templates Mixing plain Ruby with Chef DSL (For more resources related to this topic, see here.) Introduction If you want to automate your infrastructure, you will end up using most of Chef's language features. In this article, we will look at how to use the Chef Domain Specific Language (DSL) from basic to advanced level. Using community Chef style It's easier to read code that adheres to a coding style guide. It is important to deliver consistently styled code, especially when sharing cookbooks with the Chef community. In this article, you'll find some of the most important rules (out of many more—enough to fill a short book on their own) to apply to your own cookbooks. Getting ready As you're writing cookbooks in Ruby, it's a good idea to follow general Ruby principles for readable (and therefore maintainable) code. Chef Software, Inc. proposes Ian Macdonald's Ruby Style Guide (http://www.caliban.org/ruby/rubyguide.shtml#style), but to be honest, I prefer Bozhidar Batsov's Ruby Style Guide (https://github.com/bbatsov/ruby-style-guide) due to its clarity. Let's look at the most important rules for Ruby in general and for cookbooks specifically. How to do it… Let's walk through a few Chef style guide examples: Use two spaces per indentation level: remote_directory node['nagios']['plugin_dir'] do source 'plugins' end Use Unix-style line endings. Avoid Windows line endings by configuring Git accordingly: mma@laptop:~/chef-repo $ git config --global core.autocrlf true For more options on how to deal with line endings in Git, go to https://help.github.com/articles/dealing-with-line-endings. Align parameters spanning more than one line: variables( mon_host: 'monitoring.example.com', nrpe_directory: "#{node['nagios']['nrpe']['conf_dir']}/nrpe.d" ) Describe your cookbook in metadata.rb (you should always use the Ruby DSL: Version your cookbook according to Semantic Versioning standards (http://semver.org): version "1.1.0" List the supported operating systems by looping through an array using the each method: %w(redhat centos ubuntu debian).each do |os| supports os end Declare dependencies and pin their versions in metadata.rb: depends "apache2", ">= 1.0.4" depends "build-essential" Construct strings from variable values and static parts by using string expansion: my_string = "This resource changed #{counter} files" Download temporary files to Chef::Config['file_cache_path'] instead of /tmp or some local directory. Use strings to access node attributes instead of Ruby symbols: node['nagios']['users_databag_group'] Set attributes in my_cookbook/attributes/default.rb by using default: default['my_cookbook']['version'] = "3.0.11" Create an attribute namespace by using your cookbook name as the first level in my_cookbook/attributes/default.rb: default['my_cookbook']['version'] = "3.0.11" default['my_cookbook']['name'] = "Mine" How it works... Using community Chef style helps to increase the readability of your cookbooks. Your cookbooks will be read much more often than changed. Because of this, it usually pays off to put a little extra effort into following a strict style guide when writing cookbooks. There's more... Using Semantic Versioning (see http://semver.org) for your cookbooks helps to manage dependencies. If you change anything that might break cookbooks, depending on your cookbook, you need to consider this as a backwards incompatible API change. In such cases, Semantic Versioning demands that you increase the major number of your cookbook, for example from 1.1.3 to 2.0.0, resetting minor level and patch levels. Using attributes to dynamically configure recipes Imagine some cookbook author has hardcoded the path where the cookbook puts a configuration file, but in a place that does not comply with your rules. Now, you're in trouble! You can either patch the cookbook or rewrite it from scratch. Both options leave you with a headache and lots of work. Attributes are there to avoid such headaches. Instead of hardcoding values inside cookbooks, attributes enable authors to make their cookbooks configurable. By overriding default values set in cookbooks, users can inject their own values. Suddenly, it's next to trivial to obey your own rules. In this section, we'll see how to use attributes in your cookbooks. Getting ready Make sure you have a cookbook called my_cookbook and the run_list of your node includes my_cookbook. How to do it... Let's see how to define and use a simple attribute: Create a default file for your cookbook attributes: mma@laptop:~/chef-repo $ subl cookbooks/my_cookbook/attributes/default.rb Add a default attribute: default['my_cookbook']['message'] = 'hello world!' Use the attribute inside a recipe: mma@laptop:~/chef-repo $ subl cookbooks/my_cookbook/recipes/default.rb message = node['my_cookbook']['message'] Chef::Log.info("** Saying what I was told to say: #{message}") Upload the modified cookbook to the Chef server: mma@laptop:~/chef-repo $ knife cookbook upload my_cookbook Uploading my_cookbook [0.1.0] Run chef-client on your node: user@server:~$ sudo chef-client ...TRUNCATED OUTPUT... [2016-11-23T19:29:03+00:00] INFO: ** Saying what I was told to say: hello world! ...TRUNCATED OUTPUT... How it works… Chef loads all attributes from the attribute files before it executes the recipes. The attributes are stored with the node object. You can access all attributes stored with the node object from within your recipes and retrieve their current values. Chef has a strict order of precedence for attributes: Default is the lowest, then normal (which is aliased with set), and then override. Additionally, attribute levels set in recipes have precedence over the same level set in an attribute file. Also, attributes defined in roles and environments have the highest precedence. You will find an overview chart at https://docs.chef.io/attributes.html#attribute-precedence. There's more… You can set and override attributes within roles and environments. Attributes defined in roles or environments have the highest precedence (on their respective levels: default and override): Create a role: mma@laptop:~/chef-repo $ subl roles/german_hosts.rb name "german_hosts" description "This Role contains hosts, which should print out their messages in German" run_list "recipe[my_cookbook]" default_attributes "my_cookbook" => { "message" => "Hallo Welt!" } Upload the role to the Chef server: mma@laptop:~/chef-repo $ knife role from file german_hosts.rb Updated Role german_hosts! Assign the role to a node called server: mma@laptop:~/chef-repo $ knife node run_list add server 'role[german_hosts]' server: run_list: role[german_hosts] Run the Chef client on your node: user@server:~$ sudo chef-client ...TRUNCATED OUTPUT... [2016-11-23T19:40:56+00:00] INFO: ** Saying what I was told to say: Hallo Welt! ...TRUNCATED OUTPUT... Calculating values in the attribute files Attributes set in roles and environments (as shown earlier) have the highest precedence and they're already available when the attribute files are loaded. This enables you to calculate attribute values based on role or environment-specific values: Set an attribute within a role: mma@laptop:~/chef-repo $ subl roles/german_hosts.rb name "german_hosts" description "This Role contains hosts, which should print out their messages in German" run_list "recipe[my_cookbook]" default_attributes "my_cookbook" => { "hi" => "Hallo", "world" => "Welt" } Calculate the message attribute, based on the two attributes hi and world: mma@laptop:~/chef-repo $ subl cookbooks/my_cookbook/attributes/default.rb default['my_cookbook']['message'] = "#{node['my_cookbook']['hi']} #{node['my_cookbook']['world']}!" Upload the modified cookbook to your Chef server and run the Chef client on your node to see that it works, as shown in the preceding example. See also Read more about attributes in Chef at https://docs.chef.io/attributes.html Using templates Configuration Management is all about configuring your hosts well. Usually, configuration is carried out by using configuration files. Chef's template resource allows you to recreate these configuration files with dynamic values that are driven by the attributes we've discussed so far in this article. You can retrieve dynamic values from data bags, attributes, or even calculate them on the fly before passing them into a template. Getting ready Make sure you have a cookbook called my_cookbook and that the run_list of your node includes my_cookbook. How to do it… Let's see how to create and use a template to dynamically generate a file on your node: Add a template to your recipe: mma@laptop:~/chef-repo $ subl cookbooks/my_cookbook/recipes/default.rb template '/tmp/message' do source 'message.erb' variables( hi: 'Hallo', world: 'Welt', from: node['fqdn'] ) end Add the ERB template file: mma@laptop:~/chef-repo $ mkdir -p cookbooks/my_cookbook/templates mma@laptop:~/chef-repo $ subl cookbooks/my_cookbook/templates/default/message.erb <%- 4.times do %> <%= @hi %>, <%= @world %> from <%= @from %>! <%- end %> Upload the modified cookbook to the Chef server: mma@laptop:~/chef-repo $ knife cookbook upload my_cookbook Uploading my_cookbook [0.1.0] Run the Chef client on your node: user@server:~$ sudo chef-client ...TRUNCATED OUTPUT... [2016-11-23T19:36:30+00:00] INFO: Processing template[/tmp/message] action create (my_cookbook::default line 9) [2016-11-23T19:36:31+00:00] INFO: template[/tmp/message] updated content ...TRUNCATED OUTPUT... Validate the content of the generated file: user@server:~$ sudo cat /tmp/message Hallo, Welt from vagrant.vm! Hallo, Welt from vagrant.vm! Hallo, Welt from vagrant.vm! Hallo, Welt from vagrant.vm! How it works… Chef uses Erubis as its template language. It allows you to use pure Ruby code by using special symbols inside your templates. These are commonly called the 'angry squid' You use <%= %> if you want to print the value of a variable or Ruby expression into the generated file. You use <%- %> if you want to embed Ruby logic into your template file. We used it to loop our expression four times. When you use the template resource, Chef makes all the variables you pass available as instance variables when rendering the template. We used @hi, @world, and @from in our earlier example. There's more… The node object is available in a template as well. Technically, you could access node attributes directly from within your template: <%= node['fqdn'] %> However, this is not a good idea because it will introduce hidden dependencies to your template. It is better to make dependencies explicit, for example, by declaring the fully qualified domain name (FQDN) of your node as a variable for the template resource inside your cookbook: template '/tmp/fqdn' do source 'fqdn.erb' variables( fqdn: node['fqdn'] ) end Avoid using the node object directly inside your templates because this introduces hidden dependencies to node variables in your templates. If you need a different template for a specific host or platform, you can put those specific templates into various subdirectories of the templates directory. Chef will try to locate the correct template by searching these directories from the most specific (host) to the least specific (default). You can place message.erb in the cookbooks/my_cookbook/templates/host-server.vm ("host-#{node[:fqdn]}") directory if it is specific to that host. If it is platform-specific, you can place it in cookbooks/my_cookbook/templates/ubuntu ("#{node[:platform]}"); and if it is specific to a certain platform version, you can place it in cookbooks/my_cookbook/templates/ubuntu-16.04 ("#{node[:platform]}-#{node[:platorm_version]}"). Only place it in the default directory if your template is the same for any host or platform. Know the templates/default directory means that a template file is the same for all hosts and platforms—it does not correspond to a recipe name. See also Read more about templates at https://docs.chef.io/templates.html Mixing plain Ruby with Chef DSL To create simple recipes, you only need to use resources provided by Chef such as template, remote_file, or service. However, as your recipes become more elaborate, you'll discover the need to do more advanced things such as conditionally executing parts of your recipe, looping, or even making complex calculations. Instead of declaring the gem_package resource ten times, simply use different name attributes; it is so much easier to loop through an array of gem names creating the gem_package resources on the fly. This is the power of mixing plain Ruby with Chef Domain Specific Language (DSL). We'll see a few tricks in the following sections. Getting ready Start a chef-shell on any of your nodes in Client mode to be able to access your Chef server, as shown in the following code: user@server:~$ sudo chef-shell --client loading configuration: /etc/chef/client.rb Session type: client ...TRUNCATED OUTPUT... run `help' for help, `exit' or ^D to quit. Ohai2u user@server! chef > How to do it… Let's play around with some Ruby constructs in chef-shell to get a feel for what's possible: Get all nodes from the Chef server by using search from the Chef DSL: chef > nodes = search(:node, "hostname:[* TO *]") => [#<Chef::Node:0x00000005010d38 @chef_server_rest=nil, @name="server", ...TRUNCATED OUTPUT... Sort your nodes by name by using plain Ruby: chef > nodes.sort! { |a, b| a.hostname <=> b.hostname }.collect { |n| n.hostname } => ["alice", "server"] Loop through the nodes, printing their operating systems: chef > nodes.each do |n| chef > puts n['os'] chef ?> end linux windows => [node[server], node[alice]] Log only if there are no nodes: chef > Chef::Log.warn("No nodes found") if nodes.empty? => nil Install multiple Ruby gems by using an array, a loop, and string expansion to construct the gem names: chef > recipe_mode chef:recipe > %w{ec2 essentials}.each do |gem| chef:recipe > gem_package "knife-#{gem}" chef:recipe ?> end => ["ec2", "essentials"] How it works... Chef recipes are Ruby files, which get evaluated in the context of a Chef run. They can contain plain Ruby code, such as if statements and loops, as well as Chef DSL elements such as resources (remote_file, service, template, and so on). Inside your recipes, you can declare Ruby variables and assign them any values. We used the Chef DSL method search to retrieve an array of Chef::Node instances and stored that array in the variable nodes. Because nodes is a plain Ruby array, we can use all methods the array class provides such as sort! or empty? Also, we can iterate through the array by using the plain Ruby each iterator, as we did in the third example. Another common thing is to use if, else, or case for conditional execution. In the fourth example, we used if to only write a warning to the log file if the nodes array are empty. In the last example, we entered recipe mode and combined an array of strings (holding parts of gem names) and the each iterator with the Chef DSL gem_package resource to install two Ruby gems. To take things one step further, we used plain Ruby string expansion to construct the full gem names (knife-ec2 and knife-essentials) on the fly. There's more… You can use the full power of Ruby in combination with the Chef DSL in your recipes. Here is an excerpt from the default recipe from the nagios cookbook, which shows what's possible: # Sort by name to provide stable ordering nodes.sort! { |a, b| a.name <=> b.name } # maps nodes into nagios hostgroups service_hosts = {} search(:role, ‚*:*') do |r| hostgroups << r.name nodes.select { |n| n[‚roles'].include?(r.name) if n[‚roles'] }.each do |n| service_hosts[r.name] = n[node[‚nagios'][‚host_name_attribute']] end end First, they use Ruby to sort an array of nodes by their name attributes. Then, they define a Ruby variable called service_hosts as an empty Hash. After this, you will see some more array methods in action such as select, include?, and each. See also Find out more about how to use Ruby in recipes here: https://docs.chef.io/chef/dsl_recipe.html There's more… If you don't want to modify existing cookbooks, this is currently the only way to modify parts of recipes which are not meant to be configured via attributes. This approach is exactly the same thing as monkey-patching any Ruby class by reopening it in your own source files. This usually leads to brittle code, as your code now depends on implementation details of another piece of code instead of depending on its public interface (in Chef recipes, the public interface is its attributes). Keep such cookbook modifications in a separate place so that you can easily find out what you did later. If you bury your modifications deep inside your complicated cookbooks, you might experience issues later that are very hard to debug. Resources for Article: Further resources on this subject: Getting started with using Chef [article] Going Beyond the Basics [article] An Overview of Automation and Advent of Chef [article]
Read more
  • 0
  • 0
  • 23261

article-image-web-assembly-beyond-browser-bytecode-alliance-mozilla-fastly-intel-red-hat
Sugandha Lahoti
13 Nov 2019
4 min read
Save for later

Expanding Web Assembly beyond the browser with Bytecode Alliance, a Mozilla, Fastly, Intel and Red Hat partnership

Sugandha Lahoti
13 Nov 2019
4 min read
Mozilla has partnered with Fastly, Intel and Red Hat to form the Bytecode Alliance to build a secure-by-default future for WebAssembly and to take it beyond the browser. “WebAssembly is changing the web, but we believe WebAssembly can play an even bigger role in the software ecosystem as it continues to expand beyond browsers,” explained Luke Wagner, Distinguished Engineer at Mozilla and co-creator of WebAssembly. “This is a unique moment in time at the dawn of a new technology, where we have the opportunity to fix what’s broken and build new, secure-by-default foundations for native development that are portable and scalable. But we need to take deliberate, cross-industry action to ensure this happens in the right way.” What is the Bytecode Alliance This year we saw many initiatives by Mozilla pushing WebAssembly beyond the web. The Bytecode Alliance is also a step forward in that direction with an aim to make it safe for developers to use untrusted code, no matter where they’re running it—whether on the cloud, desktop or IoT devices. This common, reusable set of foundations can be used on their own or embedded in other libraries and applications. As developers run untrusted code in many new places, from the cloud to IoT devices, they open up security and portability challenges. With WebAssembly and emerging related standards such as WASI and WebAssembly Interface Types, the Alliance plans to address challenges that occur when you try to run the same code across these different systems (server, edge, browser, mobile, and more platforms). Bytecode will combine Wasmtime, a runtime for WebAssembley and WASI, Fastly’s Lucet, WebAssembly compiler, and runtime, Intel’s WebAssembly Micro Runtime and code generator Cranelift. It will also include cargo-wasi, a lightweight Cargo subcommand that compiles Rust code to target WebAssembly as well as wat and wasmparser.  These set of projects are likely to expand as the Alliance grows. Once the Alliance is formalized, an open governance model consistent with open-source best practices and strong community norms will be established. Individual developers can also participate in an open-source project in the Bytecode Alliance. Each project is governed by its own committer group. Developers who are very active in shaping a project are eligible for nomination to the project’s committer group. Introducing WebAssembly nanoprocesses Apart from these existing projects, the Alliance is also focusing on a new architectural pattern emerging in WebAssembly. This pattern called the "WebAssembly nanoprocess", gives you most of the benefits of a process (the tool that operating systems use to protect programs from each other), but with much less overhead and much faster communication between (nano)processes. This process, Mozilla thinks, can make massively-modular code (like the ones you see in npm, crates, and PyPI) secure by default. This means developers won't need to worry as much about vulnerabilities in dependencies, or attackers sneaking malicious code into their codebases. Nanoprocess can also provide memory isolation that fits anywhere. It can handle requests for tens of thousands of customers on the same machine. It can also provide software fault isolation for individual libraries in native applications. The blog mentions, “With wasm, we can replace microservices with nanoprocesses and get the same security and language independence benefits. It gives us the composability of microservices without the weight." This alliance was well appreciated by people and developers posting their reactions on Twitter. https://twitter.com/tschneidereit/status/1194382954858000384 https://twitter.com/curioman2/status/1194475559012556800 https://twitter.com/fabianfranz/status/1194408360608698370 Learn more about this alliance on Mozilla’s blog. Introducing Woz, a Progressive WebAssembly Application (PWA + Web Assembly) generator written entirely in Rust Wasmer’s first Postgres extension to run WebAssembly is here! Mozilla proposes WebAssembly Interface Types to enable language interoperability Wasmer introduces WebAssembly Interfaces for validating the imports and exports of a Wasm module
Read more
  • 0
  • 0
  • 23241

article-image-cloud-native-applications
Packt
09 Feb 2017
5 min read
Save for later

Cloud Native Applications

Packt
09 Feb 2017
5 min read
In this article by Ranga Rao Karanam, the author of the book Mastering Spring, we will see what are Cloud Native applications and Twelve Factor App. (For more resources related to this topic, see here.) Cloud Native applications Cloud is disrupting the world. A number of possibilities emerge that were never possible before. Organizations are able to provision computing, network and storage devices on demand. This has high potential to reduce costs in a number of industries. Consider the retail industry where there is high demand in pockets (Black Friday, Holiday Season and so on). Why should they pay for hardware round the year when they could provision it on demand? While we would like to be benefit from the possibilities of the cloud, these possibilities are limited by architecture and the nature of applications. How do we build applications that can be easily deployed on the cloud? That's where Cloud Native applications come into picture. Cloud Native applications are those that can easily be deployed on the cloud. These applications share a few common characteristics. We will begin with looking at Twelve Factor App - A combination of common patterns among Cloud Native applications. Twelve Factor App Twelve Factor App evolved from experiences of engineers at Heroku. This is a list of patterns that are typically used in Cloud Native application architectures. It is important to note, that an App here refers to a single deployable unit. Essentially every microservice is an App (because each microservice is independently deployable). One codebase Each App has one codebase in revision control. There can be multiple environments where the App can be deployed. However, all these environments use code from a single codebase. An example for anti-pattern is building a deployable from multiple codebases. Dependencies Explicitly declare and isolate dependencies. Typical Java applications use build management tools like Maven and Gradle to isolate and track dependencies. The following screenshot shows the typical Java applications managing dependencies using Maven: The following screenshot shows the content of the file: Config All applications have configuration that varies from one environment to another environment. Configuration is typically littered at multiple locations - Application code, property files, databases, environment variables, Java Naming and Directory Interface (JNDI) and system variables are a few examples. A Twelve Factor App should store config in the environment. While environment variables are recommended to manage configuration in a Twelve Factor App, other alternatives like having a centralized repository for application configuration should be considered for more complex systems. Irrespective of mechanism used, we recommended to manage configuration outside application code (independent of the application deployable unit). Use one standardized way of configuration Backing services Typically applications depend on other services being available - data-stores, external services among others. Twelve Factor App treats backing services as attached resources. A backing service is typically declared via an external configuration. Loose coupling to a backing service has many advantages including ability to gracefully handle an outage of a backing service. Build, release, run Strictly separate build and run stages. Build: Creates an executable bundle (ear, war or jar) from code and dependencies that can be deployed to multiple environments. Release: Combine the executable bundle with specific environment configuration to deploy in an environment. Run: Run the application in an execution environment using a specific release An anti-pattern is to build separate executable bundles specific for each environment. Stateless A Twelve Factor App does not have state. All data that it needs is stored in a persistent store. An anti-pattern is a sticky session. Port binding A Twelve Factor App exposes all services using port binding. While it is possible to have other mechanisms to expose services, these mechanisms are implementation dependent. Port binding gives full control of receiving and handling messages irrespective of where an application is deployed. Concurrency A Twelve Factor App is able to achieve more concurrency by scaling out horizontally. Scaling vertically has its limits. Scaling out horizontally provides opportunities to expand without limits. Disposability A Twelve Factor App should promote elastic scaling. Hence, they should be disposable. They can be started and stopped when needed. A Twelve Factor App should: Have minimum start up time. Long start up times means long delay before an application can take requests. Shutdown gracefully. Handle hardware failures gracefully. Environment parity All the environments - development, test, staging, and production - should be similar. They should use same processes and tools. With continuous deployment, they should have similar code very frequently. This makes finding and fixing problems easier. Logs as event streams Visibility is critical to a Twelve Factor App. Since applications are deployed on the cloud and are automatically scaled, it is important to have a centralized visibility into what's happening across different instances of the applications. Treating all logs as stream enables routing of the log stream to different destinations for viewing and archival. This stream can be used to debug issues, perform analytics and create alerting systems based on error patterns. No distinction of admin processes Twelve Factor Apps treat administrative tasks (migrations, scripts) similar to normal application processes. Summary This article thus explains about Cloud Native applications and what are Twelve Factor Apps. Resources for Article: Further resources on this subject: Cloud and Async Communication [article] Setting up of Software Infrastructure on the Cloud [article] Integrating Accumulo into Various Cloud Platforms [article]
Read more
  • 0
  • 0
  • 23211

article-image-web-design-principles-inkscape
Packt
04 Nov 2010
8 min read
Save for later

Web Design Principles in Inkscape

Packt
04 Nov 2010
8 min read
Blogs contain entries or posts that can be anything from commentary of life events, videos and pictures to anything else that can be 'posted' online. Blog posts are in reverse-chronological order and allow readers to provide comments and sometime ratings. Sometimes a blog is the entire website, or a blog can be a subset of the entire site. Another critical element of a blog is comments. Readers expect to be able to leave comments about the posts and sometimes rate the content. Thus, the posts themselves and the way they are displayed on a site are important, as it is the main content portion of the site. Clever designs allow for the common elements around the posts to stay the same and consistent, while allowing posts to be updated (and aggregated) frequently. Small business storefronts, or any site that sells a commodity, often has an area that allows you to search through its products and then purchase them through a "shopping cart". These storefronts then need a well designed area to be able to view products, get the specifications, and then add the products to a cart to checkout. It is a process that would be repeated often and used whenever a product is purchased. Web design principles for blogs and storefronts As with general web design there are some guidelines that can help you create effective, clean, and inspired blog and storefront designs. The web design basics should also be taken into account: proximity, alignment, repetition, and contrast—but also here are a few other pointers. Keeping it simple For blogs, there are more widgets and plugins you can add into your blog than you have space for—and more than is ever really needed. Be particular and choosy. Use only those that make sense for the type of reader you want to attract. Otherwise it becomes visual clutter. Choose sidebar features wisely and keep it to a bare minimum. Some basics are: a picture or more information about yourself, how to contact you (if you like), links to other similar blogs links to some of your past posts (favorites, archives, most mentioned), and links to any other websites you contribute to and/or own. When designing storefronts or any websites where your end goal is to have someone purchase an item—don't distract a user from their main goal of purchasing an item. If you add in sidebars with too much distraction, you'll lose the ever-dwindling window of opportunity for a purchase. Keep the important items front and center: a shopping cart, account information, check out. Identifiable You have about a five second window to attract and keep a reader on your website. So, use those seconds wisely and give your viewers everything they need to know about you and your blog or store within that first five seconds. How can you do this? Use your header wisely. A graphical title, tagline that is prominent and expressive can do it all, and quickly. This should be placed in the "header" location of your web design. When blogging this is critical, since most blogs are text-based, most viewers won't take the time to read but a few sentences unless you entice them to stay on your site and read along for a while. For any storefront, if you keep your company name, and a graphical representation of what they might find in your online store, then you too will encourage a casual browser to "click around" and see what else you have for them to browse. This also increases your chances at making a purchase. Making your site navigationally easy Everyone hates a website where they can't figure out "where to go next". So make your links and navigation very easy to find. Top or sidebar locations are the most obvious and natural locations for these, and probably the best locations for any of the key places you want any visitor to your site to see. You can also incorporate other forms of navigation like breadcrumb trails, tree menus, and submenus. Breadcrumbs are visual navigational elements. They show the "trail" of where you are in the site, and how you arrived at your current location. Most often breadcrumb trails are near the top of the web page. An example of breadcrumb trails is seen in the following screenshot from the online Inkscape Manual http://tavmjong.free.fr/INKSCAPE/MANUAL/html/index.php: An example of a tree menu is as follows: Tree menus are a bit more complicated. They show the main level web pages and then, when clicked, expand to show the next "tier" of pages. Each level can continue to expand if needed. As seen in the tree menu, main level pages can expand to submenus. This could also be done in a drop down in any horizontal-based navigation as well, like the example shown below: No matter what navigational elements used, the key is to keep them simple and identifiable throughout your web page design and obvious that a user can use these links to get where they need to be on your website. Feeds and social networking links More and more people today are taking advantage of RSS feeds. What are RSS feeds? RSS (Really Simple Syndication) feeds are a way to continually broadcast (or publish) your blog entries. These feeds are in a standardized XML format and pull the metadata tags you assigned when publishing your blog post and display them in "readers" for others to automatically receive. Feed readers allow subscribers (those who want to favor your site), to receive updates as soon as you post any new information, in this case, every time you would release a new blog post. When using an RSS reader, the user needs to enter you blog's feed URL to complete a subscription. Then, the RSS reader checks these subscribed feeds' URLs regularly for new updates and then alerts them when new podcasts are available. So, how do you create an RSS feed for your blog? For a website with hosting where you have your blog as part of an entire site, you can create RSS feeds by using a simple RSS feed provider, such as FeedBurner (www.feedburner.com). These services are free, and once you provide a link to your blog, they do the rest automatically for you. You'll have to set up an account, and then make a note of the blog RSS feed URL (it typically ends in XML). Then on your website, you can offer a link to subscribe to your blog feed. When using a blogging service, there are usually ways to automatically set up RSS feeds to your blog entries. In fact, most blog software has two types of RSS feeds links built in. These can be found in the <head> element of the HTML. There are also many orange 'RSS' buttons that can be easily set up and used within the design itself. Explore your blogging service provider site and set up to determine how best to set up RSS feeds. Either way you set up a feed, make sure you display the RSS feed link prominently. Put it in the sidebar, under each blog post, up near the titles. And give it a distinct link title like: Subscribe to my blog or RSS feed or something similar. RSS feeds are most commonly used for blogs, but as a storefront or online merchandiser, you might set up you own blog announcement area where you would want to announce new products or features of your online store. This would be an easy way for your shoppers to stay tuned to your latest offerings. An alternative to RSS feeds are Atom feeds. Atom stands for Atom Syndication Format which is an XML language used for web feeds. This differs from RSS feeds that focus on post updates; web feeds focus on allowing software programs to check for updates on an entire website. The site owner can use a content management system or other special software that publishes a feed of all the new content that then can be used by a feed reader, another program, or fed into other content management systems. Making checkout easy Design the checkout process of your online store to be as easy as entering brief information and clicking a few buttons. Ideally you need to have the shopping cart, checkout, continue shopping, and my account access available from any page within the site. Then, any potential purchase could also be made from any page on the site as well. Work with the programmers of the shopping cart to create the most effi cient checkout process possible. Give them design options that can allow for the fewest number of screens, but that captures the critical information from the purchaser. Summary In this article we learnt about designing for blogs and online merchant stores. We walked through simple designs for each, defining common elements for each website type and even some sub-level pages to help keep the designs consistent. In the next article, Creating Web templates in Inkscape, we will look at how to create templates so that you can create the design and each part of the web page faster. Further resources on this subject: Logos in Inkscape [Article] Creating Web templates in Inkscape [Article]
Read more
  • 0
  • 0
  • 23191
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-googles-event-driven-serverless-platform-cloud-function-is-now-generally-available
Vijin Boricha
26 Jul 2018
2 min read
Save for later

Google's event-driven serverless platform, Cloud Function, is now generally available

Vijin Boricha
26 Jul 2018
2 min read
Early this week, Google announced the general availability of its most awaited service Cloud Function at its Google Cloud Next ‘18 conference, San Francisco. Google finally managed to board the serverless bus, that it missed two years ago allowing AWS and Azure to reach their current milestones. This move takes the current cloud platform war to a new level. Google’s Cloud Function now directly competes with Amazon’s Lambda and Microsoft’s Azure Functions. Of late, application development has changed massively, with developers now focusing on application logic instead of infrastructure management, thanks to serverless computing. Developers can now prioritize agility, application quality, and faster deployment with zero server management, auto-scaling traffic management, and integrated security.   Source: Google Cloud website Google’s event driven serverless platform showcases the ability to scale automatically, run codes in response to events, pay while your code runs, and zero server management. Cloud Functions can be used to build: Serverless application backends Developers can quickly build highly available, secure and cost-effective applications as a connective layer of logic is present in Cloud Functions that helps integrate and extend GCP and third-party services. In other words, you can directly call your code from any web, mobile, or backend application or trigger it from GCP services. Real-time data processing Cloud Functions can provide a variety of real-time data processing systems as it responds to events from GCP services such as Stackdriver logging, Cloud Storage, and more. This helps developers to execute their code in response to any change in data. Intelligent applications Developers can leverage Cloud Functions to build intelligent applications with Google Cloud AI integration. One can easily introduce pre-trained machine learning models into the application that can later analyze videos, classify images, convert speech to text, perform NLP (natural language processing) and more. Developers can start making the most of Google Cloud Functions unless they are deploying functions written in Node.js 8 or Python, as these still remain in Beta. In addition to Cloud Functions, Google also announced a preview of serverless containers, a refreshed way of running containers in a fully managed environment. You can read more about this release from Google Cloud release notes. Related Links A serverless online store on AWS could save you money. Build one Learn Azure serverless computing for free – Download a free eBook from Microsoft AWS SAM (AWS Serverless Application Model) is now open source!
Read more
  • 0
  • 0
  • 23187

article-image-tensorflow-models-mobile-embedded-devices
Savia Lobo
15 May 2018
12 min read
Save for later

How to Build TensorFlow Models for Mobile and Embedded devices

Savia Lobo
15 May 2018
12 min read
TensorFlow models can be used in applications running on mobile and embedded platforms. TensorFlow Lite and TensorFlow Mobile are two flavors of TensorFlow for resource-constrained mobile devices. TensorFlow Lite supports a subset of the functionality compared to TensorFlow Mobile. It results in better performance due to smaller binary size with fewer dependencies. The article covers topics for training a model to integrate TensorFlow into an application. The model can then be saved and used for inference and prediction in the mobile application. [box type="note" align="" class="" width=""]This article is an excerpt from the book Mastering TensorFlow 1.x written by Armando Fandango. This book will help you leverage the power of TensorFlow and Keras to build deep learning models, using concepts such as transfer learning, generative adversarial networks, and deep reinforcement learning.[/box] To learn how to use TensorFlow models on mobile devices, following topics are covered: TensorFlow on mobile platforms TF Mobile in Android apps TF Mobile demo on Android TF Mobile demo on iOS TensorFlow Lite TF Lite demo on Android TF Lite demo on iOS TensorFlow on mobile platforms TensorFlow can be integrated into mobile apps for many use cases that involve one or more of the following machine learning tasks: Speech recognition Image recognition Gesture recognition Optical character recognition Image or text classification Image, text, or speech synthesis Object identification To run TensorFlow on mobile apps, we need two major ingredients: A trained and saved model that can be used for predictions A TensorFlow binary that can receive the inputs, apply the model, produce the predictions, and send the predictions as output The high-level architecture looks like the following figure: The mobile application code sends the inputs to the TensorFlow binary, which uses the trained model to compute predictions and send the predictions back. TF Mobile in Android apps The TensorFlow ecosystem enables it to be used in Android apps through the interface class  TensorFlowInferenceInterface, and the TensorFlow Java API in the jar file libandroid_tensorflow_inference_java.jar. You can either use the jar file from the JCenter, download a precompiled jar from ci.tensorflow.org, or build it yourself. The inference interface has been made available as a JCenter package and can be included in the Android project by adding the following code to the build.gradle file: allprojects  { repositories  { jcenter() } } dependencies  { compile  'org.tensorflow:tensorflow-android:+' } Note : Instead of using the pre-built binaries from the JCenter, you can also build them yourself using Bazel or Cmake by following the instructions at this link: https://github.com/tensorflow/tensorflow/blob/r1.4/ tensorflow/contrib/android/README.md Once the TF library is configured in your Android project, you can call the TF model with the following four steps:  Load the model: TensorFlowInferenceInterface  inferenceInterface  = new  TensorFlowInferenceInterface(assetManager,  modelFilename);  Send the input data to the TensorFlow binary: inferenceInterface.feed(inputName, floatValues,  1,  inputSize,  inputSize,  3);  Run the prediction or inference: inferenceInterface.run(outputNames,  logStats);  Receive the output from the TensorFlow binary: inferenceInterface.fetch(outputName,  outputs); TF Mobile demo on Android In this section, we shall learn about recreating the Android demo app provided by the TensorFlow team in their official repo. The Android demo will install the following four apps on your Android device: TF  Classify: This is an object identification app that identifies the images in the input from the device camera and classifies them in one of the pre-defined classes. It does not learn new types of pictures but tries to classify them into one of the categories that it has already learned. The app is built using the inception model pre-trained by Google. TF  Detect: This is an object detection app that detects multiple objects in the input from the device camera. It continues to identify the objects as you move the camera around in continuous picture feed mode. TF  Stylize: This is a style transfer app that transfers one of the selected predefined styles to the input from the device camera. TF  Speech: This is a speech recognition app that identifies your speech and if it matches one of the predefined commands in the app, then it highlights that specific command on the device screen. Note: The sample demo only works for Android devices with an API level greater than 21 and the device must have a modern camera that supports FOCUS_MODE_CONTINUOUS_PICTURE. If your device camera does not have this feature supported, then you have to add the path submitted to TensorFlow by the author: https://github.com/ tensorflow/tensorflow/pull/15489/files. The easiest way to build and deploy the demo app on your device is using Android Studio. To build it this way, follow these steps:  Install Android Studio. We installed Android Studio on Ubuntu 16.04 from the instructions at the following link: https://developer.android.com/studio/ install.html  Check out the TensorFlow repository, and apply the patch mentioned in the previous tip. Let's assume you checked out the code in the tensorflow folder in your home directory.  Using Android Studio, open the Android project in the path ~/tensorflow/tensorflow/examples/Android.     Your screen will look similar to this:  Expand the Gradle Scripts option from the left bar and then open the  build.gradle file.  In the build.gradle file, locate the def  nativeBuildSystem definition and set it to 'none'. In the version of  the code we checked out, this definition is at line 43: def  nativeBuildSystem  =  'none'  Build the demo and run it on either a real or simulated device. We tested the app on these devices: 7.  You can also build the apk and install the apk file on the virtual or actual connected device. Once the app installs on the device, you will see the four apps we discussed earlier: You can also build the whole demo app from the source using Bazel or Cmake by following the instructions at this link: https://github.com/tensorflow/tensorflow/tree/r1.4/tensorflow/examples/android TF Mobile in iOS apps TensorFlow enables support for iOS apps by following these steps:  Include TF Mobile in your app by adding a file named Profile in the root directory of your project. Add the following content to the Profile: target  'Name-Of-Your-Project' pod  'TensorFlow-experimental'  Run the pod  install command to download and install the TensorFlow Experimental pod.  Run the myproject.xcworkspace command to open the workspace so you can add the      prediction code to your application logic. Note: To create your own TensorFlow binaries for iOS projects, follow the instructions at this link: https://github.com/tensorflow/tensorflow/ tree/master/tensorflow/examples/ios Once the TF library is configured in your iOS project, you can call the TF model with the following four steps:  Load the model: PortableReadFileToProto(file_path,  &tensorflow_graph);  Create a session: tensorflow::Status  s  =  session->Create(tensorflow_graph);  Run the prediction or inference and get the outputs: std::string  input_layer  =  "input"; std::string  output_layer  =  "output"; std::vector<tensorflow::Tensor>  outputs; tensorflow::Status  run_status  =  session->Run( {{input_layer,  image_tensor}}, {output_layer},  {},  &outputs);  Fetch the output data: tensorflow::Tensor*  output  =  &outputs[0]; TF Mobile demo on iOS In order to build the demo on iOS, you need Xcode 7.3 or later. Follow these steps to build the iOS demo apps:  Check out the TensorFlow code in a tensorflow folder in your home directory.  Open a terminal window and execute the following commands from your home folder to download the Inception V1 model, extract the label and graph files, and move these files into the data folders inside the sample app code: $ mkdir -p ~/Downloads $ curl -o ~/Downloads/inception5h.zip https://storage.googleapis.com/download.tensorflow.org/models/incep tion5h.zip && unzip ~/Downloads/inception5h.zip -d ~/Downloads/inception5h $ cp ~/Downloads/inception5h/* ~/tensorflow/tensorflow/examples/ios/benchmark/data/ $ cp ~/Downloads/inception5h/* ~/tensorflow/tensorflow/examples/ios/camera/data/ $ cp ~/Downloads/inception5h/* ~/tensorflow/tensorflow/examples/ios/simple/data/  Navigate to one of the sample folders and download the experimental pod: $ cd ~/tensorflow/tensorflow/examples/ios/camera $ pod install  Open the Xcode workspace: $ open tf_simple_example.xcworkspace  Run the sample app in the device simulator. The sample app will appear with a Run Model button. The camera app requires an Apple device to be connected, while the other two can run in a simulator too. TensorFlow Lite TF Lite is the new kid on the block and still in the developer view at the time of writing this book. TF Lite is a very small subset of TensorFlow Mobile and TensorFlow, so the binaries compiled with TF Lite are very small in size and deliver superior performance. Apart from reducing the size of binaries, TensorFlow employs various other techniques, such as: The kernels are optimized for various device and mobile architectures The values used in the computations are quantized The activation functions are pre-fused It leverages specialized machine learning software or hardware available on the device, such as the Android NN API The workflow for using the models in TF Lite is as follows:  Get the model: You can train your own model or pick a pre-trained model available from different sources, and use the pre-trained as is or retrain it with your own data, or retrain after modifying some parts of the model. As long as you have a trained model in the file with an extension .pb or .pbtxt, you are good to proceed to the next step. We learned how to save the models in the previous chapters.  Checkpoint the model: The model file only contains the structure of the graph, so you need to save the checkpoint file. The checkpoint file contains the serialized variables of the model, such as weights and biases. We learned how to save a checkpoint in the previous chapters.  Freeze the model: The checkpoint and the model files are merged, also known as freezing the graph. TensorFlow provides the freeze_graph tool for this step, which can be executed as follows: $ freeze_graph --input_graph=mymodel.pb --input_checkpoint=mycheckpoint.ckpt --input_binary=true --output_graph=frozen_model.pb --output_node_name=mymodel_nodes  Convert the model: The frozen model from step 3 needs to be converted to TF Lite format with the toco tool provided by TensorFlow: $ toco --input_file=frozen_model.pb --input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE --input_type=FLOAT --input_arrays=input_nodes --output_arrays=mymodel_nodes --input_shapes=n,h,w,c  The .tflite model saved in step 4 can now be used inside an Android or iOS app that employs the TFLite binary for inference. The process of including the TFLite binary in your app is continuously evolving, so we recommend the reader follows the information at this link to include the TFLite binary in your Android or iOS app: https://github.com/tensorflow/tensorflow/tree/master/ tensorflow/contrib/lite/g3doc Generally, you would use the graph_transforms:summarize_graph tool to prune the model obtained in step 1. The pruned model will only have the paths that lead from input to output at the time of inference or prediction. Any other nodes and paths that are required only for training or for debugging purposes, such as saving checkpoints, are removed, thus making the size of the final model very small. The official TensorFlow repository comes with a TF Lite demo that uses a pre-trained mobilenet to classify the input from the device camera in the 1001 categories. The demo app displays the probabilities of the top three categories. TF Lite Demo on Android To build a TF Lite demo on Android, follow these steps: Install Android Studio. We installed Android Studio on Ubuntu 16.04 from the instructions at the following link: https://developer.android.com/studio/ install.html Check out the TensorFlow repository, and apply the patch mentioned in the previous tip. Let's assume you checked out the code in the tensorflow folder in your home directory. Using Android Studio, open the Android project from the path ~/tensorflow/tensorflow/contrib/lite/java/demo. If it complains about a missing SDK or Gradle components, please install those components and sync Gradle. Build the project and run it on a virtual device with API > 21. We received the following warnings, but the build succeeded. You may want to resolve the warnings if the build fails: Warning:The  Jack  toolchain  is  deprecated  and  will  not run.  To  enable  support  for  Java  8 language  features  built into  the  plugin,  remove  'jackOptions  {  ...  }'  from  your build.gradle  file, and  add android.compileOptions.sourceCompatibility  1.8 android.compileOptions.targetCompatibility  1.8 Note:  Future  versions  of  the  plugin  will  not  support  usage 'jackOptions'  in  build.gradle. To learn  more,  go  to https://d.android.com/r/tools/java-8-support-message.html Warning:The  specified  Android  SDK  Build  Tools  version (26.0.1)  is  ignored,  as  it  is  below  the minimum  supported version  (26.0.2)  for  Android  Gradle  Plugin  3.0.1. Android  SDK  Build  Tools 26.0.2  will  be  used. To  suppress  this  warning,  remove  "buildToolsVersion '26.0.1'"  from  your  build.gradle  file,  as  each  version  of the  Android  Gradle  Plugin  now  has  a  default  version  of the  build  tools. TF Lite demo on iOS In order to build the demo on iOS, you need Xcode 7.3 or later. Follow these steps to build the iOS demo apps:  Check out the TensorFlow code in a  tensorflow folder in your home directory.  Build the TF Lite binary for iOS from the instructions at this link: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite  Navigate to the sample folder and download the pod: $ cd ~/tensorflow/tensorflow/contrib/lite/examples/ios/camera $ pod install  Open the Xcode workspace: $ open tflite_camera_example.xcworkspace  Run the sample app in the device simulator. We learned about using TensorFlow models on mobile applications and devices. TensorFlow provides two ways to run on mobile devices: TF Mobile and TF Lite. We learned how to build TF Mobile and TF Lite apps for iOs and Android. We used TensorFlow demo apps as an example.   If you found this post useful, do check out the book Mastering TensorFlow 1.x  to skill up for building smarter, faster, and efficient machine learning and deep learning systems. The 5 biggest announcements from TensorFlow Developer Summit 2018 Getting started with Q-learning using TensorFlow Implement Long-short Term Memory (LSTM) with TensorFlow  
Read more
  • 0
  • 0
  • 23157

article-image-looking-back-looking-forward
Packt
09 Feb 2015
24 min read
Save for later

Looking Back, Looking Forward

Packt
09 Feb 2015
24 min read
In this article by Simon Jackson, author of the book Unity 3D UI Essentials we will see how the new Unity UI has long been sought by developers; it has been announced and re-announced over several years, and now it is finally here. The new UI system is truly awesome and (more importantly for a lot of developers on a shoestring budget) it's free. (For more resources related to this topic, see here.) As we start to look forward to the new UI system, it is very important to understand the legacy GUI system (which still exists for backwards compatibility) and all it has to offer, so you can fully understand just how powerful and useful the new system is. It's crucial to have this understanding, especially since most tutorials will still speak of the legacy GUI system (so you know what on earth they are talking about). With an understanding of the legacy system, you will then peer over the diving board and walk through a 10,000-foot view of the new system, so you get a feel of what to expect from the rest of this book. The following is the list of topics that will be covered in this article: A look back into what legacy Unity GUI is Tips, tricks, and an understanding of legacy GUI and what it has done for us A high level overview of the new UI features State of play You may not expect it, but the legacy Unity GUI has evolved over time, "adding new features and improving performance. However, because it has "kept evolving based on the its original implementation, it has been hampered "with many constraints and the ever pressing need to remain backwards compatible (just look at Windows, which even today has to cater for" programs written in BASIC (http://en.wikipedia.org/wiki/BASIC)). Not to say the old system is bad, it's just not as evolved as some of the newer features being added to the Unity 4.x and Unity 5.x series, which are based on newer and more enhanced designs, and more importantly, a new core. The main drawback of the legacy GUI system is that it is only drawn in screen space (drawn on" the screen instead of within it) on top of any 3D elements or drawing in your scenes. This is fine if you want menus or overlays in your title but if you want to integrate it further within your 3D scene, then it is a lot more difficult. For more information about world space and screen space, see this Unity Answers article (http://answers.unity3d.com/questions/256817/about-world-space-and-local-space.html). So before we can understand how good the new system is, we first need to get to grips with where we are coming from. (If you are already familiar with the legacy GUI system, feel free to skip over this section.) A point of reference Throughout this book, we will refer to the legacy GUI simply as GUI. When we talk about the new system, it will be referred to as UI or Unity UI, just so you don't get mixed-up when reading. When looking around the Web (or even in the Unity support forums), you may hear about or see references to uGUI, which was the development codename for the new Unity UI system. GUI controls The legacy" GUI controls provide basic and stylized controls for use in your titles. All legacy GUI controls are drawn during the GUI rendering phase from the built-in "OnGUI method. In the sample that accompanies this title, there are examples of all the controls in the AssetsBasicGUI.cs script. For GUI controls to function, a camera in the scene must have the GUILayer component attached to it. It is there by default on any Camera in a scene, so for most of the time you won't notice it. However, if you have removed it, then you will have to add it back for GUI to work. The component is just the hook for the OnGUI delegate handler, to ensure it has called each frame. Like "the Update method in scripts, the OnGUI method can be called several times per frame if rendering is slowing things down. Keep this in mind when building your own legacy GUI scripts. The controls that are available in the legacy GUI are: Label Texture Button Text fields (single/multiline and password variant) Box Toolbars Sliders ScrollView Window So let's go through them in more detail: All the following code is implemented in the sample project in the basic GUI script located in the AssetsScripts folder of the downloadable code. To experiment yourself, create a new project, scene, and script, placing the code for each control in the script and attach the script to the camera (by dragging it from the project view on to the Main Camera GameObject in the scene hierarchy). You can then either run the project or adorn the class in the script with the [ExecuteInEditMode] attribute to see it in the game view. The Label control Most "GUI systems start with a Label control; this simply "provides a stylized control to display read-only text on the screen, it is initiated by including the following OnGUI method in your script: void OnGUI() {GUI.Label(new Rect(25, 15, 100, 30), "Label");} This results in the following on-screen display: The Label control "supports altering its font settings through the" use of the guiText GameObject property (guiText.font) or GUIStyles. To set guiText.font in your script, you would simply apply the following in your script, either in the Awake/Start functions or before drawing the next section of text you want drawn in another font: public Font myFont = new Font("arial");guiText.font = myFont; You can also set the myFont property in Inspector using an "imported font. The Label control forms the basis for all controls to display text, and as such, "all other controls inherit from it and have the same behaviors for styling the displayed text. The Label control also supports using a Texture for its contents, but not both text "and a texture at the same time. However, you can layer Labels and other controls on top of each other to achieve the same effect (controls are drawn implicitly in the order they are called), for example: public Texture2D myTexture;void Start() {myTexture = new Texture2D(125, 15);}void OnGUI() {//Draw a textureGUI.Label(new Rect(125, 15, 100, 30), myTexture);//Draw some text on top of the texture using a labelGUI.Label(new Rect(125, 15, 100, 30), "Text overlay");} You can override the order in which controls are drawn by setting GUI.depth = /*<depth number>*/; in between calls; however, I would advise against this unless you have a desperate need. The texture" will then be drawn to fit the dimensions of the Label field, By" default it scales on the shortest dimension appropriately. This too can be altered using GUIStyle to alter the fixed width and height or even its stretch characteristics. Texture drawing Not "specifically a control in itself, the GUI framework" also gives you the ability to simply draw a Texture to the screen Granted there is little difference to using DrawTexture function instead of a Label with a texture or any other control. (Just another facet of the evolution of the legacy GUI). This is, in effect, the same as the previous Label control but instead of text it only draws a texture, for example: public Texture2D myTexture;void Start() {myTexture = new Texture2D(125, 15);}void OnGUI() {GUI.DrawTexture(new Rect(325, 15, 100, 15), myTexture);} Note that in all the examples providing a Texture, I have provided a basic template to initialize an empty texture. In reality, you would be assigning a proper texture to be drawn. You can also provide scaling and alpha blending values when drawing the texture to make it better fit in the scene, including the ability to control the aspect ratio that the texture is drawn in. A warning though, when you scale the image, it affects the rendering properties for that image under the legacy GUI system. Scaling the image can also affect its drawn position. You may have to offset the position it is drawn at sometimes. For" example: public Texture2D myTexture;void Start() {myTexture = new Texture2D(125, 15);}void OnGUI() {GUI.DrawTexture(new Rect(325, 15, 100, 15), myTexture, "   ScaleMode.ScaleToFit,true,0.5f);} This will do its best to draw the source texture with in the drawn area with alpha "blending (true) and an aspect ratio of 0.5. Feel free to play with these settings "to get your desired effect. This is useful in certain scenarios in your game when you want a simple way to "draw a full screen image on the screen on top of all the 3D/2D elements, such as pause or splash screen. However, like the Label control, it does not receive any "input events (see the Button control for that). There is also a variant of the DrawTexture function called DrawTextureWithTexCoords. This allows you to not only pick where you want the texture drawn on to the screen, but also from which part of the source texture you want to draw, for example: public Texture2D myTexture;void Start() {myTexture = new Texture2D(125, 15);}void OnGUI() {GUI.DrawTextureWithTexCoords (new Rect(325, 15, 100, 15), "   myTexture ,new Rect(10, 10, 50, 5));} This will pick a region from the source texture (myTexture) 50 pixels wide by "5 pixels high starting from position 10, 10 on the texture. It will then draw this to "the Rect region specified. However, the DrawTextureWithTexCoords function cannot perform scaling, it "can only perform alpha blending! It will simply draw to fit the selected texture region to the size specified in the initial Rect. DrawTextureWithTexCoords has also been used to draw individual sprites using the legacy GUI, which has a notion of what a sprite is. The Button control Unity "also provides a Button control, which comes in two" variants. The basic "Button control which only supports a single click, whereas RepeatButton supports "holding down the button. They are both instantiated the same way by using an if statement to capture "when the button is clicked, as shown in the following script: void OnGUI() {if (GUI.Button(new Rect(25, 40, 120, 30), "Button")){   //The button has clicked, holding does nothing}if (GUI.RepeatButton(new Rect(170, 40, 170, 30), "   "RepeatButton")){   //The button has been clicked or is held down}} Each will result in a simple button on screen as follows:   The controls also support using a Texture for the button content as well by providing a texture value for the second parameter as follows: public Texture2D myTexture;void Start() {myTexture = new Texture2D(125, 15);}void OnGUI() {if (GUI.Button(new Rect(25, 40, 120, 30), myTexture)){ }} Like the Label, the "font of the text can be altered using GUIStyle or" by setting the guiText property of the GameObject. It also supports using textures in the same "way as the Label. The easiest way to look at this control is that it is a Label that "can be clicked. The Text control Just as "there is a need to display text, there is also a need for a "user to be able to enter text, and the legacy GUI provides the following controls to do just that: Control Description TextField This" is a basic text box, it supports a single line of text, no new lines (although, if the text contains end of line characters, it will draw the extra lines down). TextArea This "is an extension of TextField that supports entering of multiple lines of text; new lines will be added when the user "hits the enter key. PasswordField This is" a variant of TextField; however, it won't display "the value entered, it will replace each character with a replacement character. Note, the password itself is still stored in text form and if you are storing users' passwords, you should encrypt/decrypt the actual password when using it. Never store characters in plain text.  Using the TextField control is simple, as it returns the final state of the value that has been entered and you have to pass that same variable as a parameter for the current text to be displayed. To use the controls, you apply them in script as follows: string textString1 = "Some text here";string textString2 = "Some more text here";string textString3 = "Even more text here";void OnGUI() {textString = GUI.TextField(new Rect(25, 100, 100, 30), "   textString1);textString = GUI.TextArea(new Rect(150, 100, 200, 75), "   textString2);textString = GUI.PasswordField(new Rect(375, 100, 90, 30), "   textString3, '*');} A note about strings in Unity scripts Strings are immutable. Every time you change their value they create a new string in memory by having the textString variable declared at the class level it is a lot more memory efficient. If you declare the textString variable in the OnGUI method, "it will generate garbage (wasted memory) in each frame. "Worth keeping in mind. When" displayed, the textbox by default looks like this:   As with" the Label and Button controls, the font of the text displayed can be altered using either a GUIStyle or guiText GameObject property. Note that text will overflow within the field if it is too large for the display area, but it will not be drawn outside the TextField dimensions. The same goes for multiple lines. The Box control In the "midst of all the controls is a generic purpose control that "seemingly "can be used for a variety of purposes. Generally, it's used for drawing a "background/shaded area behind all other controls. The Box control implements most of the features mentioned in the controls above controls (Label, Texture, and Text) in a single control with the same styling and layout options. It also supports text and images as the other controls do. You can draw it with its own content as follows: void OnGUI() {GUI.Box (new Rect (350, 350, 100, 130), "Settings");} This gives you the following result:   Alternatively, you" can use it to decorate the background" of other controls, "for example: private string textString = "Some text here";void OnGUI() {GUI.Box (new Rect (350, 350, 100, 130), "Settings");GUI.Label(new Rect(360, 370, 80, 30), "Label");textString = GUI.TextField(new Rect(360, 400, 80, 30), "   textString);if (GUI.Button (new Rect (360, 440, 80, 30), "Button")) {}} Note that using the Box control does not affect any controls you draw upon it. It is drawn as a completely separate control. This statement will be made clearer when you look at the Layout controls later in this article. The example will draw the box background and the Label, Text, and Button controls on top of the box area and looks like this:   The box control can" be useful to highlight groups of controls or providing "a simple background (alternatively, you can use an image instead of just text and color). As with the other controls, the Box control supports styling using GUIStyle. The Toggle/checkbox control If checking "on / checking off is your thing, then the legacy" GUI also has a checkbox control for you, useful for those times when" you need to visualize the on/off state "of an option. Like the TextField control, you "pass the variable that manages Togglestate as "a parameter and it returns the new value (if it changes), so it is applied in code "as follows: bool blnToggleState = false;void OnGUI() {blnToggleState = GUI.Toggle(new Rect(25, 150, 250, 30),blnToggleState, "Toggle");} This results in the following on-screen result: As with the Label and Button controls, the font of the text displayed can be altered using either a GUIStyle or guiText GameObject property. Toolbar panels The basic "GUI controls also come with some very basic" automatic layout panels: "the first handles an arrangement of buttons, however these buttons are grouped "and only one can be selected at a time. As with other controls, the style of the button can be altered using a GUIStyles definition, including fixing the width of the buttons and spacing. There are two layout options available, these are: The Toolbar control The Selection grid control The Toolbar control The Toolbar control "arranges buttons in a horizontal "pattern (vertical is "not supported). Note that it is possible to fake a vertical toolbar by using a selection grid with just one item per row. See the Selection grid section later in this article for more details. As with other controls, the Toolbar returns the index of the currently selected button in the toolbar. The buttons are also the same as the base button control so it also offers options to support either text or images for the button content. The RepeatButton control however is not supported. To implement the toolbar, you specify an array of the content you wish to display in each button and the integer value that controls the selected button, as follows: private int toolbarInt;private string[] toolbarStrings ;Void Start() {toolbarInt = 0;toolbarStrings = { "Toolbar1", "Toolbar2", "Toolbar3" };}void OnGUI() {toolbarInt = GUI.Toolbar(new Rect(25, 200, 200, 30), "   toolbarInt, toolbarStrings);} The main" difference between the preceding controls is that you" have to pass the currently selected index value in to the control and it then returns the new value. The Toolbar when drawn looks as follows:   As stated, you can also pass an array of textures as well and they will be displayed instead of the text content. The SelectionGrid control The "SelectionGrid control is a customization "of the previous standard Toolbar control, it is able to arrange the buttons in a grid layout and resize the buttons appropriately to fill the target area. In code, SelectionGrid is used in a very similar format to the Toolbar code shown previously, for example: private int selectionGridInt ;private string[] selectionStrings;Void Start() {selectionGridInt = 0;selectionStrings = { "Grid 1", "Grid 2", "Grid 3", "Grid 4" };}void OnGUI() {selectionGridInt = GUI.SelectionGrid(new Rect(250, 200, 200, 60), selectionGridInt, selectionStrings, 2);} The main difference between SelectionGrid and Toolbar in code is that with SelectionGrid you can specify the number of items in a single row and the control will automatically lay out the buttons accordingly (unless overridden using GUIStyle). The preceding code will result in the following layout:   On "their own, they provide a little more flexibility" than just using buttons alone. The Slider/Scrollbar controls When "you need to control a range in your games. GUI or add "a handle to control moving properties between two values, like" moving an object around in your scene, this is "where the Slider and Scrollbar controls come in. They provide two similar out-of-the-box implementations that give a scrollable region and a handle to control the value behind the control. Here, they are presented side by side:   The slimmer Slider and chunkier Scrollbar controls can both work in either horizontal or vertical modes and have presets for the minimum and maximum values allowed. Slider control code In code, the "Slider control code is represented as follows: private float fltSliderValue = 0.5f;void OnGUI() {fltSliderValue = GUI.HorizontalSlider(new Rect(25, 250, 100,30), "   fltSliderValue, 0.0f, 10.0f);fltSliderValue = GUI.VerticalSlider(new Rect(150, 250, 25, 50), "   fltSliderValue, 10.0f, 0.0f);} Scrollbar control code In code, the "Scrollbar control code is represented as follows: private float fltScrollerValue = 0.5f;void OnGUI() {fltScrollerValue = GUI.HorizontalScrollbar(new Rect(25, 285,"   100, 30), fltScrollerValue, 1.0f, 0.0f, 10.0f);fltScrollerValue = GUI.VerticalScrollbar(new Rect(200, 250, 25,"   50), fltScrollerValue, 1.0f, 10.0f, 0.0f);} Like Toolbar and SelectionGrid, you are required to pass in the current value and it will return the updated value. However, unlike all the other controls, you actually have two style points, so you can style the bar and the handle separately, giving you a little more freedom and control over the look and feel of the sliders. Normally, you would only use the Slider control; The main reason the Scrollbar is available is that it forms the basis for the next control, the ScrollView control. The ScrollView control The last" of the displayable controls is the ScrollView control, "which gives you masking abilities over GUI elements with" optional horizontal and vertical Scrollbars. Simply put, it allows you to define an area larger for controls behind a smaller window on the screen, for example:   Example list implementation using a ScrollView control Here we have a list that has many items that go beyond the drawable area of the ScrollView control. We then have the two scrollbars to move the content of the scroll viewer up/down and left/right to change the view. The background content is hidden behind a viewable mask that is the width and height of the ScrollView control's main window. Styling the "control is a little different as there is no base style" for it; it relies on the currently set default GUISkin. You can however set separate GUIStyles for each of the sliders but only over the whole slider, not its individual parts (bar and handle). If you don't specify styles for each slider, it will also revert to the base GUIStyle. Implementing a ScrollView is fairly easy, for example: Define the visible area along with a virtual background layer where the controls will be drawn using a BeginScrollView function. Draw your controls in the virtual area. Any GUI drawing between the ScrollView calls is drawn within the scroll area. Note that 0,0 in the ScrollView is from the top-left of the ScrollView area and not the top-left-hand corner of the screen. Complete it by closing the control with the EndScrollView function. For example, the previous example view was created with the following code: private Vector2 scrollPosition = Vector2.zero;private bool blnToggleState = false;void OnGUI(){scrollPosition = GUI.BeginScrollView(" new Rect(25, 325, 300, 200), " scrollPosition, " new Rect(0, 0, 400, 400)); for (int i = 0; i < 20; i++){   //Add new line items to the background   addScrollViewListItem(i, "I'm listItem number " + i);}GUI.EndScrollView();} //Simple function to draw each list item, a label and checkboxvoid addScrollViewListItem(int i, string strItemName){GUI.Label(new Rect(25, 25 + (i * 25), 150, 25), strItemName);blnToggleState = GUI.Toggle(" new Rect(175, 25 + (i * 25), 100, 25), " blnToggleState, "");} In this "code, we define a simple function (addScrollViewListItem) to "draw a list item (consisting of a label and checkbox). We then begin the ScrollView control by passing the visible area (the first Rect parameter), the starting scroll position, and finally the virtual area behind the control (the second Rect parameter). Then we "use that to draw 20 list items inside the virtual area of the ScrollView control using our helper function before finishing off and closing the control with the EndScrollView command. If you want to, you can also nest ScrollView controls within each other. The ScrollView control also has some actions to control its use like the ScrollTo command. This command will move the visible area to the coordinates of the virtual layer, bringing it into focus. (The coordinates for this are based on the top-left position of the virtual layer; 0,0 being top-left.) To use the ScrollTo function on ScrollView, you must use it within the Begin and End ScrollView commands. For example, we can add a button in ScrollView to jump to the top-left of the virtual area, for example: public Vector2 scrollPosition = Vector2.zero;void OnGUI(){scrollPosition = GUI.BeginScrollView(" new Rect(10, 10, 100, 50)," scrollPosition, " new Rect(0, 0, 220, 10)); if (GUI.Button(new Rect(120, 0, 100, 20), "Go to Top Left"))   GUI.ScrollTo(new Rect(0, 0, 100, 20));GUI.EndScrollView();} You can also additionally turn on/off the sliders on either side of the control by specifying "the BeginScrollView statement "using the alwayShowHorizontal and alwayShowVertical properties; these are highlighted here in an updated "GUI.BeginScrollView call: Vector2 scrollPosition = Vector2.zero;bool ShowVertical = false; // turn off vertical scrollbarbool ShowHorizontal = false; // turn off horizontal scrollbarvoid OnGUI() {scrollPosition = GUI.BeginScrollView(new Rect(25, 325, 300, 200),scrollPosition,new Rect(0, 0, 400, 400),ShowHorizontal,ShowVertical);GUI.EndScrollView ();}GUI.EndScrollView ();} Rich Text Formatting Now" having just plain text everywhere would not look" that great and would likely force developers to create images for all the text on their screens (granted a fair few still do so for effect). However, Unity does provide a way to enable richer text display using a style akin to HTML wherever you specify text on a control (only for label and display purposes; getting it to" work with input fields is not recommended). In this HTML style of writing text, we have the following tags we can use to liven up the text displayed. This gives" text a Bold format, for example: The <b>quick</b> brown <b>Fox</b> jumped over the <b>lazy Frog</b> This would result in: The quick brown Fox jumped over the lazy Frog Using this tag "will give text an Italic format, for example: The <b><i>quick</i></b> brown <b>Fox</b><i>jumped</i> over the <b>lazy Frog</b> This would result in: The quick brown Fox jumped over the lazy Frog As you can "probably guess, this tag will alter the Size of the text it surrounds. For reference, the default size for the font is set by the font itself. For example: The <b><i>quick</i></b> <size=50>brown <b>Fox</b></size> <i>jumped</i> over the <b>lazy Frog</b> This would result in: Lastly, you can specify different colors for text surrounded by the Color tag. "The color itself is" denoted using its 8-digit RGBA hex value, for example: The <b><i>quick</i></b> <size=50><color=#a52a2aff>brown</color> <b>Fox</b></size> <i>jumped</i> over the <b>lazy Frog</b> Note that the "color is defined using normal RGBA color space notation (http://en.wikipedia.org/wiki/RGBA_color_space) in hexadecimal form with two characters per color, for example, RRGGBBAA. Although the color property does also support the shorter RGB color space, which is the same notation "but without the A (Alpha) component, for example,. RRGGBB The preceding code would result in: (If you are reading this in print, the previous word brown is in the color brown.) You can also use a color name to reference it but the pallet is quite limited; for more "details, see the Rich Text manual reference page at http://docs.unity3d.com/Manual/StyledText.html. For text meshes, there are two additional tags: <material></material> <quad></quad> These only apply when associated to an existing mesh. The material is one of the materials assigned to the mesh, which is accessed using the mesh index number "(the array of materials applied to the mesh). When applied to a quad, you can also specify a size, position (x, y), width, and height to the text. The text mesh isn't well documented and is only here for reference; as we delve deeper into the new UI system, we will find much better ways of achieving this. Summary So now we have a deep appreciation of the past and a glimpse into the future. One thing you might realize is that there is no one stop shop when it comes to Unity; each feature has its pros and cons and each has its uses. It all comes down to what you want to achieve and the way you want to achieve it; never dismiss anything just because it's old. In this article, we covered GUI controls. It promises to be a fun ride, and once we are through that, we can move on to actually building some UI and then placing it in your game scenes in weird and wonderful ways. Now stop reading this and turn the page already!! Resources for Article: Further resources on this subject: What's Your Input? [article] Parallax scrolling [article] Unity Networking – The Pong Game [article]
Read more
  • 0
  • 0
  • 23149

article-image-implementing-a-rag-enhanced-cookbot-part-1
Bahaaldine Azarmi, Jeff Vestal
15 Jan 2024
10 min read
Save for later

Implementing a RAG-enhanced CookBot - Part 1

Bahaaldine Azarmi, Jeff Vestal
15 Jan 2024
10 min read
Dive deeper into the world of AI innovation and stay ahead of the AI curve! Subscribe to our AI_Distilled newsletter for the latest insights. Don't miss out – sign up today!This article is an excerpt from the book, Vector Search for Practitioners with Elastic, by Bahaaldine Azarmi and Jeff Vestal. Optimize your search capabilities in Elastic by operationalizing and fine-tuning vector search and enhance your search relevance while improving overall search performanceIntroductionEveryone knows the age-old culinary dilemma, “What can I cook with the ingredients I have?” Many people need help when faced with an array of ingredients but a lack of inspiration or knowledge to whip up a dish. This everyday issue was the spark for our idea—CookBot.CookBot is not just any AI. It’s conceived as an advanced culinary assistant that not only suggests recipes based on the available ingredients but also understands the nuances of user queries, adapts to individual dietary preferences and restrictions, and generates insightful culinary recommendations.Our objective was to infuse CookBot with RAG, ELSER, and RRF technologies. These technologies are designed to enhance the semantic understanding of queries, optimize information retrieval, and generate relevant, personalized responses. By harnessing the capabilities of these advanced tools, we aimed for CookBot to be able to provide seamless, context-aware culinary assistance tailored to each user’s unique needs.Figure: CookBot powered by ElasticDataset overview – an introduction to the Allrecipes.com datasetThe Allrecipes.com dataset, in its raw CSV format, is a treasure trove of diverse and detailed culinary information. Thus, it is the perfect foundation to train our CookBot. It houses an extensive range of recipes, each encapsulated in a unique entry brimming with an array of information.You can find and download the dataset here, as it will be used later in the chapter:https://www.kaggle.com/datasets/nguyentuongquang/all-recipesTo illustrate the richness of this dataset, let’s consider a single entry:"group","name","rating","n_rater","n_ reviewer","summary","process","ingredient" "breakfast-and-brunch.eggs.breakfast-burritos","Ham and Cheese Breakfast Tortillas",0,0,44,"This is great for a special brunch or even a quick and easy dinner. Other breakfast meats can be used, but the deli ham is the easiest since it is already fully cooked.","prep: 30 mins,total: 30 mins,Servings: 4,Yield: 4 servings","12 eggs + <U+2153> cup milk + 3 slices cooked ham, diced + 2 green onions, minced + salt and pepper to taste + 4 ounces Cheddar cheese, shredded + 4 (10 inch) flour tortillas + cup salsa" Each entry in the dataset represents a unique recipe and encompasses various fields:group: This is the categorization of the recipe. It provides a general idea about the type and nature of the dish.name: This is the title or name of the recipe. This field straightforwardly tells us what the dish is. • rating and n_rater: These fields indicate the popularity and approval of the dish among users.n_reviewer: This is the number of users who have reviewed the recipe.summary: This is a brief overview or description of the dish, often providing valuable context about its taste, usage, or preparation.process: This field outlines crucial details such as preparation time, total cooking time, servings, and yield.ingredient: This is a comprehensive list of all the ingredients required for the recipe, along with their quantities.The detailed information offered by each field gives us a broad and varied information space, aiding the retriever in navigating the data and ensuring the generator can accurately respond to a diverse range of culinary queries. As we move forward, we will discuss how we indexed this dataset using Elasticsearch, the role of ELSER and RRF in effectively retrieving data, and how the GPT-4 model generates relevant, personalized responses based on the retrieved data.Preparing data for RAG-enhanced searchTo transform the Allrecipes.com data into a searchable database, we first need to parse the CSV file and subsequently create an Elasticsearch index where data will be stored and queried. Let’s walk through this process implemented as part of the Python code.Connecting to ElasticsearchFirst, we need to establish a connection with our Elasticsearch instance. This connection is handled by the Elasticsearch object from the Elasticsearch Python module:from elasticsearch import Elasticsearch es = Elasticsearch()In this case, we assume that our Elasticsearch instance runs locally with default settings. If it doesn’t, we will need to provide the appropriate host and port information to the Elasticsearch class.Defining the indexThe next step is to define an index where our recipes will be stored. An index in Elasticsearch is like a database in traditional database systems. In this case, we’ll call our index recipes:index_name = 'recipes'Creating the mappingNow, we need to create a mapping for our index. A mapping is like a schema in a SQL database and defines the types of each field in the documents that will be stored in the index. We will define a mapping as a Python dictionary:mapping = { "mappings": { "properties": { "group": { "type": "text" }, "name": { "type": "text" }, "rating": { "type": "text" }, "n_rater": { "type": "text" }, "n_reviewer": { "type": "text" }, "summary": { "type": "text", "analyzer": "english" }, "process": { "type": "text" }, "ingredient": { "type": "text", }, "ml.tokens": { "type": "rank_features" } } } } Here, all fields are defined as text, which means they are full-text searchable. We also specify that the summary field should be analyzed using the English analyzer, which will help to optimize searches in English text by taking into account things such as stemming and stop words. Finally, we create the field that ELSER will use to create the token set, which is the result of the expansion happening based on the terms passed to ELSER.Creating the indexOnce we’ve defined our mapping, we can create the index in Elasticsearch with the following:es.indices.create(index=index_name, body=mapping)This sends a request to Elasticsearch to create an index with the specified name and mapping.Reading the CSV fileWith our index ready, we can now read our dataset from the CSV file. We’ll use pandas, a powerful data manipulation library in Python, to do this:import pandas as pd with open('recipe_dataset.csv', 'r', encoding='utf-8', errors='ignore') as file: df = pd.read_csv(file)This code opens the CSV file and reads it into a pandas dataframe, a two-dimensional tabular data structure that’s perfect for manipulating structured data.Converting to dictionariesTo index the data into Elasticsearch, we need to convert our dataframe into a list of dictionaries, where each dictionary corresponds to a row (i.e., a document or recipe) in the dataframe:recipes = df.to_dict('records') print(f"Number of documents: {len(recipes)}")At this point, we have our dataset ready to index in Elasticsearch. However, considering the size of the dataset, it is advisable to use the bulk indexing feature for efficient data ingestion. This will be covered in the next section.Bulk indexing the dataLet’s look into the step-by-step process of bulk indexing your dataset in Elasticsearch.Defining the preprocessing pipelineBefore we proceed to bulk indexing, we need to set up a pipeline to preprocess the documents. Here, we will use the elser-v1-recipes pipeline, which utilizes the ELSER model for semantic indexing. The pipeline is defined as follows:[ { "inference": { "model_id": ".elser_model_1", "target_field": "ml", "field_map": { "ingredient": "text_field" }, "inference_config": { "text_expansion": { "results_field": "tokens" } } } } ]The pipeline includes an inference processor that uses the ELSER pre-trained model to perform semantic indexing. It maps the ingredient field from the recipe data to the text_field object of the ELSER model. The output (the expanded tokens from the ELSER model) is stored in the tokens field under the ml field in the document.Creating a bulk indexing sequenceGiven the size of the Allrecipes.com dataset, it’s impractical to index each document individually. Instead, we can utilize Elasticsearch’s bulk API, which allows us to index multiple documents in a single request. First, we need to generate a list of dictionaries, where each dictionary corresponds to a bulk index operation:bulk_index_body = [] for index, recipe in enumerate(recipes): document = { "_index": "recipes", "pipeline": "elser-v1-recipes", "_source": recipe } bulk_index_body.append(document)In this loop, we iterate over each recipe (a dictionary) in our recipes list and then construct a new dictionary with the necessary information for the bulk index operation. This dictionary specifies the name of the index where the document will be stored (recipes), the ingest pipeline to be used to process the document (elser-v1-recipes), and the document source itself (recipe).Performing the bulk index operationWith our bulk_index_body array ready, we can now perform the bulk index operation:try: response = helpers.bulk(es, bulk_index_body, chunk_size=500, request_timeout=60*3) print ("\nRESPONSE:", response) except BulkIndexError as e: for error in e.errors: print(f"Document ID: {error['index']['_id']}") print(f"Error Type: {error['index']['error']['type']}") print(f"Error Reason: {error['index']['error']['reason']}")We use the helpers.bulk() function from the Elasticsearch library to provide our Elasticsearch connection (es)—the bulk_index_body array we just created—with a chunk_size value of 500 (which specifies that we want to send 500 documents per request) and a request_timeout value of 180 seconds, which specifies that we want to allow each request to take up to 3 minutes before timing out because the indexing could take a long time with ELSER.The helpers.bulk() function will return a response indicating the number of operations attempted and the number of errors, if any.If any errors occur during the bulk index operation, these will be raised as BulkIndexError. We can catch this exception and iterate over its errors attribute to get information about each individual error, including the ID of the document that caused the error, the type of error, and the reason for it.At the end of this process, you will have successfully indexed your entire Allrecipes.com dataset in Elasticsearch, ready for it to be retrieved and processed by your RAG-enhanced CookBot.ConclusionIn closing, the infusion of RAG, ELSER, and RRF technologies into CookBot elevates culinary exploration. With Elasticsearch indexing and the Allrecipes.com dataset, CookBot transcends traditional kitchen boundaries, offering personalized, context-aware assistance. This journey signifies the convergence of cutting-edge AI and the rich tapestry of culinary possibilities. As CookBot orchestrates flavor symphonies, the future of cooking is redefined, promising a delightful harmony of technology and gastronomy for every user. Embrace the evolution—where CookBot's intelligence transforms mere ingredients into a canvas for culinary innovation.Author BioBahaaldine Azarmi, Global VP Customer Engineering at Elastic, guides companies as they leverage data architecture, distributed systems, machine learning, and generative AI. He leads the customer engineering team, focusing on cloud consumption, and is passionate about sharing knowledge to build and inspire a community skilled in AI.Jeff Vestal has a rich background spanning over a decade in financial trading firms and extensive experience with Elasticsearch. He offers a unique blend of operational acumen, engineering skills, and machine learning expertise. As a Principal Customer Enterprise Architect, he excels at crafting innovative solutions, leveraging Elasticsearch's advanced search capabilities, machine learning features, and generative AI integrations, adeptly guiding users to transform complex data challenges into actionable insights.
Read more
  • 0
  • 0
  • 23129
article-image-chaos-conf-2018-recap-chaos-engineering-hits-maturity-as-community-moves-towards-controlled-experimentation
Richard Gall
12 Oct 2018
11 min read
Save for later

Chaos Conf 2018 Recap: Chaos engineering hits maturity as community moves towards controlled experimentation

Richard Gall
12 Oct 2018
11 min read
Conferences can sometimes be confusing. Even at the most professional and well-planned conferences, you sometimes just take a minute and think what's actually the point of this? Am I learning anything? Am I meant to be networking? Will anyone notice if I steal extra food for the journey home? Chaos Conf 2018 was different, however. It had a clear purpose: to take the first step in properly forging a chaos engineering community. After almost a decade somewhat hidden in the corners of particularly innovative teams at Netflix and Amazon, chaos engineering might feel that its time has come. As software infrastructure becomes more complex, less monolithic, and as business and consumer demands expect more of the software systems that have become integral to the very functioning of life, resiliency has never been more important but more challenging to achieve. But while it feels like the right time for chaos engineering, it hasn't quite established itself in the mainstream. This is something the conference host, Gremlin, a platform that offers chaos engineering as a service, is acutely aware of. On the hand it's actively helping push chaos engineering into the hands of businesses, but on the other its growth and success, backed by millions of VC cash (and faith), depends upon chaos engineering becoming a mainstream discipline in the DevOps and SRE worlds. It's perhaps this reason that the conference felt so important. It was, according to Gremlin, the first ever public chaos engineering conference. And while it was relatively small in the grand scheme of many of today's festival-esque conferences attended by thousands of delegates (Dreamforce, the Salesforce conference, was also running in San Francisco in the same week), the fact that the conference had quickly sold out all 350 of its tickets - with more hoping on waiting lists - indicates that this was an event that had been eagerly awaited. And with some big names from the industry - notably Adrian Cockcroft from AWS and Jessie Frazelle from Microsoft - Chaos Conf had the air of an event that had outgrown its insider status before it had even began. The renovated cinema and bar in San Francisco's Mission District, complete with pinball machines upstairs, was the perfect container for a passionate community that had grown out of the clean corporate environs of Silicon Valley to embrace the chaotic mess that resembles modern software engineering. Kolton Andrus sets out a vision for the future of Gremlin and chaos engineering Chaos Conf was quick to deliver big news. They keynote speech, by Gremlin co-founder Kolton Andrus launched Gremlin's brand new Application Level Fault Injection (ALFI) feature, which makes it possible to run chaos experiments at an application level. Andrus broke the news by building towards it with a story of the progression of chaos engineering. Starting with Chaos Monkey, the tool first developed by Netflix, and moving from infrastructure to network, he showed how, as chaos engineering has evolved, it requires and faciliates different levels of control and insight on how your software works. "As we're maturing, the host level failures and the network level failures are necessary to building a robust and resilient system, but not sufficient. We need more - we need a finer granularity," Andrus explains. This is where ALFI comes in. By allowing Gremlin users to inject failure at an application level, it allows them to have more control over the 'blast radius' of their chaos experiments. The narrative Andrus was setting was clear, and would ultimately inform the ethos of the day - chaos engineering isn't just about chaos, it's about controlled experimentation to ensure resiliency. To do that requires a level of intelligence - technical and organizational - about how the various components of your software work, and how humans interact with them. Adrian Cockcroft on the importance of historical context and domain knowledge Adrian Cockcroft (@adrianco) VP at AWS followed Andrus' talk. In it he took the opportunity to set the broader context of chaos engineering, highlighting how tackling system failures is often a question of culture - how we approach system failure and think about our software. Developers love to learn things from first principles" he said. "But some historical context and domain knowledge can help illuminate the path and obstacles." If this sounds like Cockcroft was about to stray into theoretical territory, he certainly didn't. He offered plenty of practical frameworks for thinking through potential failure. But the talk wasn't theoretical - Cockcroft offered a taxonomy of failure that provides a useful framework for thinking through potential failure at every level. He also touched on how he sees the future of resiliency evolving, focusing on: Observability of systems Epidemic failure modes Automation and continuous chaos The crucial point Cockcroft makes is that cloud is the big driver for chaos engineering. "As datacenters migrate to the cloud, fragile and manual disaster recovery will be replaced by chaos engineering" read one of his slides. But more than that, the cloud also paves the way for the future of the discipline, one where 'chaos' is simply an automated part of the test and deployment pipeline. Selling chaos engineering to your boss Kriss Rochefolle, DevOps engineer and author of one of the best selling DevOps books in French, delivered a short talk on how engineers can sell chaos to their boss. He takes on the assumption that a rational proposal, informed by ROI is the best way to sell chaos engineering. He suggests instead that engineers need to play into emotions, and presenting chaos engineer as a method for tackling and minimizing the fear of (inevitable failure. Follow Kriss on Twitter: @crochefolle Walmart and chaos engineering Vilas Veraraghavan, the Director of Engineering was keen to clarify that Walmart doesn't practice chaos. Rather it practices resiliency - chaos engineering is simply a method the organization uses to achieve that. It was particularly useful to note the entire process that Vilas' team adopts when it comes to resiliency, which has largely developed out of Vilas' own work building his team from scratch. You can learn more about how Walmart is using chaos engineering for software resiliency in this post. Twitter's Ronnie Chen on diving and planning for failure Ronnie Chen (@rondoftw) is an engineering manager at Twitter. But she didn't talk about Twitter. In fact, she didn't even talk about engineering. Instead she spoke about her experience as a technical diver. By talking about her experiences, Ronnie was able to make a number of vital points about how to manage and tackle failure as a team. With mortality rates so high in diving, it's a good example of the relationship between complexity and risk. Chen made the point that things don't fail because of a single catalyst. Instead, failures - particularly fatal ones - happen because of a 'failure cascade'. Chen never makes the link explicit, but the comparison is clear - the ultimate outcome (ie. success or failure) is impacted by a whole range of situational and behavioral factors that we can't afford to ignore. Chen also made the point that, in diving, inexperienced people should be at the front of an expedition. "If you're inexperienced people are leading, they're learning and growing, and being able to operate with a safety net... when you do this, all kinds of hidden dependencies reveal themselves... every undocumented assumption, every piece of ancient team lore that you didn't even know you were relying on, comes to light." Charity Majors on the importance of observability Charity Majors (@mipsytipsy), CEO of Honeycomb, talked in detail about the key differences between monitoring and observability. As with other talks, context was important: a world where architectural complexity has grown rapidly in the space of a decade. Majors made the point that this increase in complexity has taken us from having known unknowns in our architectures, to many more unknown unknowns in a distributed system. This means that monitoring is dead - it simply isn't sophisticated enough to deal with the complexities and dependencies within a distributed system. Observability, meanwhile, allows you to to understand "what's happening in your systems just by observing it from the outside." Put simply, it lets you understand how your software is functioning from your perspective - almost turning it inside out. Majors then linked the concept to observability to the broader philosophy of chaos engineering - echoing some of the points raised by Adrian Cockcroft in his keynote. But this was her key takeaway: "Software engineers spend too much time looking at code in elaborately falsified environments, and not enough time observing it in the real world." This leads to one conclusion - the importance of testing in production. "Accept no substitute." Tammy Butow and Ana Medina on making an impact Tammy Butow (@tammybutow) and Ana Medina  (@Ana_M_Medina) from Gremlin took us through how to put chaos engineering into practice - from integrating it into your organizational culture to some practical tests you can run. One of the best examples of putting chaos into practice is Gremlin's concept of 'Failure Fridays', in which chaos testing becomes a valuable step in the product development process, to dogfood it and test out how a customer experiences it. Another way which Tammy and Ana suggested chaos engineering can be used was as a way of testing out new versions of technologies before you properly upgrade in production. To end, their talk, they demo'd a chaos battle between EKS (Kubernetes on AWS) and AKS (Kubernetes on Azure), doing an app container attack, a packet loss attack and a region failover attack. Jessie Frazelle on how containers can empower experimentation Jessie Frazelle (@jessfraz) didn't actually talk that much about chaos engineering. However, like Ronnie Chen's talk, chaos engineering seeped through what she said about bugs and containers. Bugs, for Frazelle, are a way of exploring how things work, and how different parts of a software infrastructure interact with each other: "Bugs are like my favorite thing... some people really hate when they get one of those bugs that turns out to be a rabbit hole and your kind of debugging it until the end of time... while debugging those bugs I hate them but afterwards, I'm like, that was crazy!" This was essentially an endorsement of the core concept of chaos engineering - injecting bugs into your software to understand how it reacts. Jessie then went on to talk about containers, joking that they're NOT REAL. This is because they're made up of  numerous different component parts, like Cgroups, namespaces, and LSMs. She contrasted containers with Virtual machines, zones and jails, which are 'first class concepts' - in other worlds, real things (Jessie wrote about this in detail last year in this blog post). In practice what this means is that whereas containers are like Lego pieces, VMs, zones, and jails are like a pre-assembled lego set that you don't need to play around with in the same way. From this perspective, it's easy to see how containers are relevant to chaos engineering - they empower a level of experimentation that you simply don't have with other virtualization technologies. "The box says to build the death star. But you can build whatever you want." The chaos ends... Chaos Conf was undoubtedly a huge success, and a lot of credit has to go to Gremlin for organizing the conference. It's clear that the team care a lot about the chaos engineering community and want it to expand in a way that transcends the success of the Gremlin platform. While chaos engineering might not feel relevant to a lot of people at the moment, it's only a matter of time that it's impact will be felt. That doesn't mean that everyone will suddenly become a chaos engineer by July 2019, but the cultural ripples will likely be felt across the software engineering landscape. But without Chaos Conf, it would be difficult to see chaos engineering growing as a discipline or set of practices. By sharing ideas and learning how other people work, a more coherent picture of chaos engineering started to emerge, one that can quickly make an impact in ways people wouldn't have expected six months ago. You can watch videos of all the talks from Chaos Conf 2018 on YouTube.
Read more
  • 0
  • 0
  • 23119

article-image-the-new-websocket-inspector-will-be-released-in-firefox-71
Fatema Patrawala
17 Oct 2019
4 min read
Save for later

The new WebSocket Inspector will be released in Firefox 71

Fatema Patrawala
17 Oct 2019
4 min read
On Tuesday,  Firefox DevTools team announced that the new WebSocket (WS) inspector will be available in Firefox 71. It is currently ready for developers to use in Firefox Developer Edition. The WebSocket API is used to create a persistent connection between a client and server. Because the API sends and receives data at any time, it is used mainly in applications requiring real-time communication. Although it is possible to work directly with the WS API, some existing libraries come in handy and help save time. These libraries can help with connection failures, proxies, authentication and authorization, scalability, and much more. The WS inspector in Firefox DevTools currently supports Socket.IO and SockJS, and more support is still a work in progress. Key features included in Firefox WebSocket Inspector The WebSocket Inspector is part of the existing Network panel UI in DevTools. It was possible to filter the content for opened WS connections in the panel, but now you can see the actual data transferred through WS frames. The WS UI now offers a fresh new Messages panel that can be used to inspect WS frames sent and received through the selected WS connection. There are Data and Time columns visible by default, and you can customize the interface to see more columns by right-clicking on the header. The WS inspector currently supports the following WS protocols: Plain JSON Socket.IO SockJS SignalR and WAMP will be supported soon 5. You can use the pause/resume button in the Network panel toolbar to stop intercepting WS traffic. Firefox team is still working on a few things for this release for example, binary payload viewer, indicating closed connections, more protocols like SignalR and WAMP and exporting WS frames and more. For developers, this is a major improvement and the community is really happy with this news. One of them comments on Reddit, “Finally! Have been stuck rolling with Chrome whenever I'm debugging websocket issues until now, because it's just so damn useful to see the exact messages sent and received.” Another user commented, “This came at the most perfect time... trying to interface with a Socket.IO server from a Flutter app is difficult without tools to really look at the internals and see what’s going on” Some of them also feel that with such improvements in Firefox it will soon replace the current Chromium dominance. The comment reads, “I hope that in improving its dev tooling with things like WS inspection, Firefox starts to turn the tide from the Chromium's current dominance. Pleasing webdevs seems to be the key to winning browser wars. The general pattern is, the devs switch to their preferred browser. When building sites, they do all their build testing against their favourite browser, and only make sure it functions on other browsers (however poorly) as an afterthought. Then everyone else switches to suit, because it's a better experience. It happened when IE was dominant (partly becuse of dodgy business practices, but also partly because ActiveX was more powerful than early JS). But then Firefox was faster and had [better] devtools and add-ons, so the devs switched to Firefox and everyone followed suit. Then Chrome came onto the scene as a faster browser with even better devtools, and now Chromium+Forks is over three quarters of the browser market share. A browser monopoly is bad for the web ecosystem, no matter what browser happens to be dominant.” To know more about this news, check out the official announcement on the Firefox blog. Firefox 69 allows default blocking of third-party tracking cookies and cryptomining for all users Cloudflare and Google Chrome add HTTP/3 and QUIC support; Mozilla Firefox soon to follow suit Mozilla brings back Firefox’s Test Pilot Program with the introduction of Firefox Private Network Beta  
Read more
  • 0
  • 0
  • 23119

article-image-7-web-design-trends-and-predictions-for-2019
Guest Contributor
12 Jan 2019
6 min read
Save for later

7 Web design trends and predictions for 2019

Guest Contributor
12 Jan 2019
6 min read
Staying updated about web design trends is very crucial. The latest norm today may change tomorrow with shifting algorithms, captivating visuals and introduction of best practices. Remaining on top by frequently reforming your website is thus quintessential to avoid looking like a reminiscent of an outdated website. 2019 will be all about engaging website designs focusing on flat designs, captivating structures & layouts, speed, mobile performance and so on. Here are 7 web design predictions which we think will be trending in 2019 #1 Website speed You would have come across this pivotal aspect of web design. It is strongly recommended for the loading time of websites to be necessarily less than three seconds to have a lasting impact on visitors. Having your visitors waiting for more than this duration would result in a high bounce rate. Based on a survey by Aberdeen Group, 5% of organizations found that website visitors abandoned their website in a second of delay. Enthralling website design with overloaded data slowing your page speed could eat up on your revenue in a huge way. Google Speed updates which came into effect from July 2018 emphasize the need to focus on the page loading time. Moreover, Google prioritizes and ranks faster loading websites. Though the need for videos and images still exists in web design, the need in 2019 will be to reduce the page loading time without compromising on the look of the website. #2 Mobile first phenomenon With user preferences inclined greatly towards mobile devices, the need for the “mobile first” web design has become the need of the hour. This is not only to rank higher on SERP but also to boost the quality of customer experiences on the device. Websites need to be exclusively designed for mobile devices in the first place. The mobile first web design is a completely focused conceptualization of the website on mobile taking into consideration parameters like a responsive and user-friendly design. Again, 2019 will need more of optimization inclined towards voice search. Users are impatient to get hold of information in the fastest way possible. Voice search on mobile will include: Focusing on long tail keywords, conversational and natural spoken language. Appropriate usage of schema metadata Emphasize on semantics Optimization based on local listing This is yet another unmissable trend of 2019. #3 Flat designs Clutter-free, focused websites have always been in demand. Flat design is all about minimalism and improved usability. This kind of design helps to focus on the important parts of the website using bright colors, clean-edged designs and a lot of free space. There are two reasons for website owners to opt for flat designs in 2019. They contain lesser components which are data-light, and are fast- loading, improving the website speed and optimization quotient. Also, it enhances customer experience with a quick loading website on both the mobile and desktop versions. So by adapting to flat designs, websites can stay back longer on user favorite lists, in turn, churning out elevated conversion rates. #4 Micro-animations Micro animations may seem like minute features on a webpage but they do add great value. A color change when you click the submit button conveys that the action has been performed. An enlarged list when you point the mouse on a particular product makes your presence felt. Such animations communicate to the user about actions accomplished. Again, visuals are always captivating, be it a background video or a micro animation. Such micro animations do impact by creating a visual hierarchy and compelling users towards conversion points. So micro animations are definitely here to stay back in 2019. #5 Chatbots Chatbots have become much more common as they help bridge communication gaps. This is because these chatbots have emerged smarter with improved Artificial Intelligence and machine learning techniques. They can improve response time, personalize communication and automate repetitive tasks. Chatbots understand our data based on previous chat history, predict what we might be looking for and give us auto recommendations about products. Chatbots can sense our interest and provide us with personalized ad content thereby enhancing customer satisfaction. Chatbots serve as crucial touch points. They can intelligently handle customer service while collecting sensitive customer data for the sales team. This way you can analyze your customer base even before initiating a first cut discussion with them. 2019 will be a year which will see many more such interactions being incorporated in websites. #6 Single page designs Simple, clutter-free and single page design is going to be a buzzword of 2019. When we say single page design it literally means a single page without extra links leading to blogs or detailed services. The next question would be about SEO optimization based on keywords and content. To begin with, a single page designed websites have a neatly siloed hierarchy. As they do not have aspects that slow down your website, they are easily compatible across devices. The page-less design has minimal HTML and JavaScript which improves customer experience, in turn, helping to earn a higher keyword ranking on SEO. Also, with way lesser elements on the page, they can be managed easily. Frequent updates and changes based on customer expectations and trends can be done at regular intervals adding greater value to the website. This is yet another aspect to watch in 2019. #7 Shapes incorporated Incorporating simple geometric shapes on your website could do wonders with its appearance. They are easily loadable and are also engaging. Shapes are similar to colors which throw an impact on the mood of the visitors. Rectangles showcase stability, circles represent unity and triangles are supposed to reflect dynamism. Using shapes based on your aesthetic sense either sparingly or liberally can definitely catch the attention of your visitors. You could place them in areas you want to seek attention and create a visual hierarchy. Implementing geometric shapes on your website will drive traffic and affect your potential sales in a huge way. Staying on top of the competition is all about presenting fresh ideas without compromising on the quality of services and user experience. Emerge as a pacesetter on par with upcoming trends and differentiate your services in the current milieu to reap maximum benefits. Author Bio Swetha S. is adept at creating customer-centered marketing strategies focused on augmenting brand presence. She is currently the Digital Marketing Manager for eGrove Systems and Elite Site optimizer, contributing towards the success of the organization.
Read more
  • 0
  • 0
  • 23094
article-image-microsoft-open-sources-infer-net-its-popular-model-based-machine-learning-framework
Melisha Dsouza
08 Oct 2018
3 min read
Save for later

Microsoft open sources Infer.NET, it’s popular model-based machine learning framework

Melisha Dsouza
08 Oct 2018
3 min read
Last week, Microsoft open sourced Infer.NET, the cross-platform framework used for model-based machine learning. This popular machine learning engine used in Office, Xbox and Azure, will be available on GitHub under the permissive MIT license for free use in commercial applications. Features of  Infer.NET The team at Microsoft Research in Cambridge initially envisioned Infer.NET as a research tool and released it for academic use in 2008. The framework has served as a base to publish hundreds of papers across a variety of fields, including information retrieval and healthcare. The team then started using the framework as a machine learning engine within a wide range of Microsoft products. A model-based approach to machine learning Infer.NET allows users to incorporate domain knowledge into their model. The framework can be used to build bespoke machine learning algorithms directly from their model. To sum it up, this framework actually constructs a learning algorithm for users based on the model they have provided. Facilitates interpretability Infer.NET also facilitates interpretability. If users have designed the model themselves and the learning algorithm follows that model, they can understand why the system behaves in a particular way or makes certain predictions. Probabilistic Approach In Infer.NET, models are described using a probabilistic program. This is used to describe real-world processes in a language that machines understand. Infer.NET compiles the probabilistic program into high-performance code for implementing something cryptically called deterministic approximate Bayesian inference. This approach allows a notable amount of scalability. For instance, it can be used in a system that automatically extracts knowledge from billions of web pages, comprising petabytes of data. Additional Features The framework also supports the ability of the system to learn as new data arrives. The team is also working towards developing and growing it further. Infer.NET will become a part of ML.NET (the machine learning framework for .NET developers). They have already set up the repository under the .NET Foundation and moved the package and namespaces to Microsoft.ML.Probabilistic.  Being cross platform, Infer.NET supports .NET Framework 4.6.1, .NET Core 2.0, and Mono 5.0. Windows users get to use Visual Studio 2017, while macOS and Linux folks have command-line options, which could be incorporated into the code wrangler of their choice. Download the framework to learn more about Infer.NET. You can also check the documentation for a detailed User Guide. To know more about this news, head over to Microsoft’s official blog. Microsoft announces new Surface devices to enhance user productivity, with style and elegance Neural Network Intelligence: Microsoft’s open source automated machine learning toolkit Microsoft’s new neural text-to-speech service lets machines speak like people
Read more
  • 0
  • 0
  • 23092

article-image-how-pytorch-is-bridging-the-gap-between-research-and-production-at-facebook-pytorch-team-at-f8-conference
Vincy Davis
04 Dec 2019
7 min read
Save for later

How PyTorch is bridging the gap between research and production at Facebook: PyTorch team at F8 conference

Vincy Davis
04 Dec 2019
7 min read
PyTorch, the machine learning library which was originally developed as a research framework by a Facebook intern in 2017, has now grown into a popular deep learning workflow. One of the most loved products by Facebook, PyTorch is free, open source, and used for applications like computer vision and natural language processing (NLP).  At the F8 conference held this year, the PyTorch team consisting of Joe Spisak, the project manager for PyTorch at Facebook AI and Dmytro Dzhulgakov, the tech lead at Facebook AI gave a talk on how Facebook is developing and scaling AI experiences with PyTorch.  Spisak describes PyTorch as an eager and graph-based execution that is defined by ‘run’. This means that when a user executes a Python code, it generates a graph on the fly. It is dynamic in nature and allows the compilation of the static graph. The dynamic neural networks are accessible, thus, allowing the user to change the parameters very quickly. This feature comes in handy for applications like control flow in NLP. Another important feature of PyTorch, according to Spisak, is the ability to generate accurately distributed training models that possess close to billion parameters, including the cutting-edge ones. It also has a simple and easy API that is very intuitive by nature. This is one of the qualities of PyTorch which has endeared many developers, claims Spisak.  Become a pro at Deep Learning with PyTorch! If you want to become an expert in building and training neural network models with high speed and flexibility in text, vision, and advanced analytics using PyTorch 1.x, read our book Deep Learning with PyTorch 1.x - Second Edition written by Sri. Yogesh K., Laura Mitchell, et al.  It will give you an insight into solving real-world problems using CNNs, RNNs, and LSTMs, along with discovering state-of-the-art modern deep learning architectures, such as ResNet, DenseNet, and Inception. How PyTorch is bridging the gap between research and production at Facebook Dzhulgakov points out how general advances in AI are driven by innovative research in the fields of academia or industry and why it’s necessary to bridge this big lag between research and production. He says, “If you have a new idea and you want to take it all the way through to deployment, you usually need to go through multiple steps - figure out what the approach is and then find the training data maybe prepare massage it a little bit. Actually, build and train your model after that and then there is this painful step of transferring your model to a production environment which often historically involved reimplementation of a lot of code so you can actually take and deploy it and scale-up.”  According to Dzhulgakov, PyTorch is trying to minimize this big gap by encouraging advances and experimentations in the field, so that the research is brought into production in a few days, instead of months. Challenges in bringing research to production Following are the various classes of challenges associated with bringing research to production, according to the PyTorch team. Hardware efficiency: In case of a tight latency constraint environment, users are required to fit all the hardware into the performance budget. On the other hand, an underused hardware environment can lead to an increase in cost. Scalability: In Facebook’s recent work, Dzhulgakov says, they have trained on billions of public images, thus indicating significant accuracy gains as compared to regular datasets like imageNet. Similarly, when models are taken to inference, it means that billions of inferences per second are running with multiple diverse models sharing the same hardware. Cross-platform: Neural networks are mostly not isolated as they need to be deployed inside their target application. It has a lot of interdependence with the surrounding code and application, thus posing different constraints like the user will not be able to run Python code or the user will have to work on very constrained computer capabilities if running a mobile device, and more. Reliability: A lot of PyTorch jobs run for multiple weeks on hundreds of GPUs, hence it is important to design a reliable software which can tolerate hardware failures and deliver results.  How PyTorch is tackling these challenges In order to tackle the above-listed challenges, Dzhulgakov says Facebook develops systems that can take up a training job and perform optimizations focused on performance for the performance-critical pieces. The system also applies “recipes for reliability” so that the developer written modeling code is automatically transformed. The Jit package comes into the picture here and acts like a key factor that is built to capture the structure of the Python program with minimal changes. The main goal of the Jit package is to make this process almost seamless. He asserts that PyTorch has been successful since it feels like regular programming in Python and most of its users start developing in traditional PyTorch mode (eager mode) just by writing and prototyping in the program. “For the subset of promising models which shall show what results you need to bring to production either scale up, so you can apply techniques provided by Jit to exist mental codes and annotated in order to run in so-called script code.”   The Jit is like a subset of Python with a thread list of request semantics, which allows the user to apply transparent transformations for the eager mode to the user. The annotations include adding a few lines of Python code on top of the function in such a way that it can be done incrementally on function by function or module by module fashion. This hybrid fashion ensures that the model works along the way. Such powerful PyTorch tools permit the user to share the same code base between research and production environments.  Next, Dzhulgakov deduces that the common factor between research and production is that both teams of developers work on the same code base built on top of PyTorch. Thus, they share the codes among the teams that have a common domain like text classification or object detection or reinforcement learning. These developers prototype models, train new algorithms and address new tasks for quickly transitioning this functionality to the opposite environment. Watch the full talk to see Dzhulgakov’s examples of PyTorch bridging the gap between research and production at Facebook. If you want to become an expert at implementing deep learning applications in PyTorch, check out our latest book Deep Learning with PyTorch 1.x - Second Edition written by Sri. Yogesh K., Laura Mitchell, and Et al. This book will show you how to apply neural networks to domains such as computer vision and NLP. It will also guide you to build, train, and scale a model with PyTorch and cover complex neural networks such as GANs and autoencoders for producing text and images. NVIDIA releases Kaolin, a PyTorch library to accelerate research in 3D computer vision and AI Introducing ESPRESSO, an open-source, PyTorch based, end-to-end neural automatic speech recognition (ASR) toolkit for distributed training across GPUs Facebook releases PyTorch 1.3 with named tensors, PyTorch Mobile, 8-bit model quantization, and more Transformers 2.0: NLP library with deep interoperability between TensorFlow 2.0 and PyTorch, and 32+ pretrained models in 100+ languages PyTorch announces the availability of PyTorch Hub for improving machine learning research reproducibility
Read more
  • 0
  • 0
  • 23083
Modal Close icon
Modal Close icon