Tech Guides - Data

If you've been looking into the world of Machine Learning lately you might have heard about a mysterious thing called “Deep Learning”. But just what is Deep Learning, and what does it mean for the world of Machine Learning as a whole? Take less than two minutes out of your day to find out and fully realize the awesome potential Deep Learning has with this video today.

Managing and earning cryptocurrency is a lot of hassle and losing it is a lot like losing yourself. While security of this blockchain based currency is a major concern, here is what you can do to secure your crypto fortune. With the ever fluctuating crypto-rates, every time, it’s now or never. While Bitcoin climbed up to $17,900 in the past, the digital currency frenzy is always in-trend and its security is crucial. No crypto geek wants to lose their currency due to malicious activities, negligence or any other reason. Before we delve into securing our crypto currencies, lets discuss the structure and strategy of this crypto vault that ensures the absolute security of a blockchain based digital currency. Why blockchains are secure, at least, in theory Below are the three core elements that contribute in making blockchain a fool proof digital technology.        Public key cryptography        Hashing        Digital signatures Public Key Cryptography This cryptography involves two distinctive keys i.e., private and public keys. Both keys decrypt and encrypt data asymmetrically. Both have simultaneous dependency of data which is encrypted by a private key and can only be decrypted with the public key. Similarly, data decrypted by public key can only be decrypted by a private key. Various cryptography schemes including TLS (Transport Layer Security protocol) and SSL (Secure Sockets Layer) have this system at its core. The strategy works well with you putting in your public key into the world of blockchain and keeping your private key confidential, not revealing it on any platform or place. Hashing Also called a digest, the hash of a message gets calculated on the basis of the contents of a message. The hashing algorithm generates a hash that is created deterministically. Data of an arbitrary length acts an input to the hashing algorithm. The outcome of this complex process is known as a calculated amount of hash with a predefined length. Due to its deterministic nature, the input and output are the same. Considering mathematical calculations, it’s easy to convert a message into hash but when it comes to obtaining an original message from hash, it is tediously difficult. Digital Signatures A digital signature is an encrypted form of hash of a message and is an outcome of a private key. Anyone who has the access to the public key can break into the digital signature by decrypting it and this can be used to get the original hash. Anyone who can read the message can calculate the hash of a message on its own. The independently calculated hash can be compared with the decrypted hash to ensure both the hashes are the same. If they both match, it is a confirmation that the message remains unaltered from creation to reception. Additionally, it is a sign of a relating private key digitally signing the message. A hash is extracted from a message and if a message gets altered, it will produce a different type of hash. Note that it is complex to reverse the process to find the message of a hash but it’s easy to compute the hash of a message. A hash that is encrypted by a private key is known as digital signature. Anyone having a public key can decrypt a digital signature and they have the ability to compare the digital signature with a calculated hash of the message. If the value of an original message is active and the message is signed by the entity having the private key, it means that the hashes are identical. What are Crypto wallets and transactions Every crypto-wallet is a combined collection of single or more wallets. A crypto-wallet is a private key and it can create a public key too. By using a public key, a public wallet address can be easily created. This makes a cryptocurrency wallet a set of private keys. To enable sharing wallet address with the public, they are converted into QR codes eliminated the need to maintain secrecy. One can always show QR codes to the world without any hesitation and anyone can send cryptocurrency using that wallet address. However, a cryptocurrency transaction needs a private key and currency sent into a wallet is owned by the owner of the wallet. In order to transact using cryptocurrency, a transaction is created that is public information. A transaction of crypto currency is a collection of information a blockchain needs. The only needed data for a transaction is the destination wallet’s address and the desired amount to be transferred. While anyone can transact in cryptocurrency, the transactions are only permitted by the blockchain if it is assured by multiple members in the network. A transaction should be digitally signed by a private key in order to get a valid status or else, it would be treated as invalid. In other words, one signs a transaction with the private key and then it gets to the blockchain. Once the blockchain accepts the key by confirming the public key data, it gets included in the blockchain that validates the transaction. Why you should guard your private key An attack on your private key is an attempt to steal your cryptocurrency. By using your private keys, an attacker attempts to digitally sign transactions from your wallet address to their address. Moreover, an attacker can destroy your private keys thus ending your access to your crypto wallet. What are some risk factors involved in owning a crypto wallet Before we move on to creating a security wall around our crypto currency, it is important to know from whom we are protecting our digital currency or who can prove to be a threat for our crypto wallets. If you lose the access to your crypto currency, you have lost it all as there isn’t any ledger with a centralized authority and once you lose the access, you can't regain it by any means. Since a crypto wallet is paired by a private and public key, losing the private key means losing your wallet. In other words, you don’t own any cryptocurrency. This is the very first and foremost threat. The next in line threat is what we hear often. Attackers, hackers or attempters who want to gain access to our cryptocurrency. The malfunctions may be opportunist or they may have their private intentions. Threats for your cryptocurrency Opportunist hackers are low profile attackers who get access to your laptop for transacting money to their public wallet address. Opportunist hackers doesn’t attack or target a person specifically, but if they get access to your crypto currency, they won’t shy away from taking your digital cash. Dedicated attackers, on the other hand, target single handedly or they may be in a group of hackers who work together for a sole purpose that is – stealing cryptocurrency. Their targets include every individual, crypto trader or even a crypto exchange. They initiate phishing campaigns and before executing the attack, they get well-versed with their target by conducting a pre-research. Level 2 attackers go for a broader approach and write malicious code that may steal private keys from a system if it gets attacked or infected. Another kind of hackers are backed by nation states. They are a collective group of people with top level coordination and established financials. They are motivated by gaining access to finances or their will. The crypto currency attacks by Lazarus Group, backed by the North Korea, are an example. How to Protect Your crypto wallet Regardless of the kind of threat, it is you and your private key that needs to be secured. Here’s how to ensure maximum security of your cryptocurrency. Throw away your access keys and you will lose your cryptocurrency forever. Obviously, you won’t do it ever and since the aforementioned thought came into your mind after reading the phrase, here are some other ways to secure your cryptocurrency fortune.       Go through the complete password recovery process. This means going through the process of forgetting the password and creating a multi-factor token. These measures should be taken while setting up a new hosted wallet or else, be prepared to lose it all.       No matter how fast the tech world progresses, basics will remain the same. You should have a printed paper backup of your keys and they should be placed in a secure location such as a bank’s locker or in a personal safe vault. Don’t forget to wipe out the printer’s memory after you are done with printing as printed files can be restored and re used to hack your digital money.       Do not keeps those keys with you nor should you be hiding those keys in a closet that can get damaged due to fire, theft, etc.       If your wallet has multi-signature enabled on it and has two public or private keys for the authorization of transactions, make it to three keys. While the third key will be controlled by an entrusted party, it will help you in the absence of a second person. About Author Tahha Ashraf is a Digital Content Producer at Cubix, a mobile app development company. He is a Certified Hubspot inbound and content marketer. He loves talking about brands, tech, blockchain and content marketing. Along with writing for the online fraternity on a variety of topics, he is fond of creativity and writes poetry in his free time. Cryptocurrency-based firm, Tron acquires BitTorrent Can Cryptocurrency establish a new economic world order? Akon is planning to create a cryptocurrency city in Senegal    

Around 80% of the time in data analysis is spent on cleaning and preparing data for analysis. This is, however, an important task, and is a prerequisite to the rest of the data analysis workflow, including visualization, analysis, and reporting. Although, being an important task given its nature, there are certain myths associated with data wrangling which developers should be cautious of. In this post, we will discuss four such misconceptions. Myth #1: Data wrangling is all about writing SQL query There was a time when data processing needed data to be presented in a relational manner so that SQL queries could be written. Today, there are many other types of data sources in addition to the classic static SQL databases, which can be analyzed. Often, an engineer has to pull data from diverse sources such as web portals, Twitter feeds, sensor fusion streams, police or hospital records. Static SQL query can help only so much in those diverse domains. A programmatic approach, which is flexible enough to interface with myriad sources and is able to parse the raw data through clever algorithmic techniques and use of fundamental data structures (trees, graphs, hash tables, heaps), will be the winner. Myth #2: Knowledge of statistics is not required for data wrangling Quick statistical tests and visualizations are always invaluable to check the ‘quality’ of the data you sourced. These tests can help detect outliers and wrong data entry, without running complex scripts. For effective data wrangling, you don’t need to have knowledge of advanced statistics. However, you must understand basic descriptive statistics and know how to execute them using built-in Python libraries. Myth #3: You have to be a machine learning expert to do great data wrangling Deep knowledge of machine learning is certainly not a pre-requisite for data wrangling. It is true that the end goal of data wrangling is often to prepare the data so that it can be used in a machine learning task downstream. As a data wrangler, you do not have to know all the nitty-gritties of your project’s machine learning pipeline. However, it is always a good idea to talk to the machine learning expert who will use your data and understand the data structure interface and format he/she needs to run the model fast and accurately. Myth #4: Deep knowledge of programming is not required for data wrangling As explained above, the diversity and complexity of data sources require that you are comfortable with deep notions of fundamental data structures and how a programming language paradigm handles them. Increasing deep knowledge of the programming framework (Python for example) will surely help you to come up with innovative methods for dealing with data source interfacing and data cleaning issues. The speed and efficiency of your data processing pipeline can often be benefited from using advanced knowledge of basic algorithms e.g. search, sort, graph traversal, hash table building, etc. Although built-in methods in standard libraries are optimized, having this knowledge gives you an edge for any situation. You read a guest post from Tirthajyoti Sarkar and Shubhadeep Roychowdhury, the authors of Data Wrangling with Python. We hope that these misconceptions would help you realize that data wrangling is not as difficult as it seems. Have fun wrangling data! About the authors Dr. Tirthajyoti Sarkar works as a Sr. Principal Engineer in the semiconductor technology domain where he applies cutting-edge data science/machine learning techniques for design automation and predictive analytics. Shubhadeep Roychowdhury works as a Sr. Software Engineer at a Paris based Cyber Security startup. He holds a Master Degree in Computer Science from West Bengal University Of Technology and certifications in Machine Learning from Stanford. Don’t forget to check out Data Wrangling with Python to learn the essential basics of data wrangling using Python. 30 common data science terms explained Python, Tensorflow, Excel and more – Data professionals reveal their top tools How to create a strong data science project portfolio that lands you a job

Google Cloud Platform (GCP) is considered to be one of the Big 3 cloud platforms among Microsoft Azure and AW. GCP is widely used cloud solutions supporting AI capabilities to design and develop smart models to turn your data into insights at a cheap, affordable cost. The following excerpt is taken from the book 'Cloud Analytics with Google Cloud Platform' authored by Sanket Thodge. GCP offers many machine learning APIs, among which we take a look at the 3 most popular APIs: Cloud Speech API A powerful API from GCP! This enables the user to convert speech to text by using a neural network model. This API is used to recognize over 100 languages throughout the world. It can also support filter of unwanted noise/ content from a text, under various types of environments. It supports context-awareness recognition, works on any device, any platform, anywhere, including IoT. It has features like Automatic Speech Recognition (ASR), Global Vocabulary, Streaming Recognition, Word Hints, Real-Time Audio support, Noise Robustness, Inappropriate Content Filtering and supports for integration with other APIs of GCP.  The architecture of the Cloud Speech API is as follows: In other words, this model enables speech to text conversion by ML. The components used by the Speech API are: REST API or Google Remote Procedure Call (gRPC) API Google Cloud Client Library JSON API Python Cloud DataLab Cloud Data Storage Cloud Endpoints The applications of the model include: Voice user interfaces Domotic appliance control Preparation of structured documents Aircraft / direct voice outputs Speech to text processing Telecommunication It is free of charge for 15 seconds per usage, up to 60 minutes per month. More than that will be charged at $0.006 per usage. Now, as we have learned about the concepts and the applications of the model, let's learn some use cases where we can implement the model: Solving crimes with voice recognition: AGNITIO, A voice biometrics specialist partnered with Morpho (Safran) to bring Voice ID technology into its multimodal suite of criminal identification products. Buying products and services with the sound of your voice: Another most popular and mainstream application of biometrics, in general, is mobile payments. Voice recognition has also made its way into this highly competitive arena. A hands-free AI assistant that knows who you are: Any mobile phone nowadays has voice recognition software in the form of AI machine learning algorithms. Cloud Translation API Natural language processing (NLP) is a part of artificial intelligence that focuses on Machine Translation (MT). MT has become the main focus of NLP group for many years. MT deals with translating text from the source language to text in the target language. Cloud Translation API provides a graphical user interface to translate an inputted string of a language to targeted language, it’s highly responsive, scalable and dynamic in nature. This API enables translation among 100+ languages. It also supports language detection automatically with accuracy. It provides a feature to read a web page contents and translate to another language, and need not be text extracted from a document. The Translation API supports various features such as programmatic access, text translation, language detection, continuous updates and adjustable quota, and affordable pricing. The following image shows the architecture of the translation model:  In other words, the cloud translation API is an adaptive Machine Translation Algorithm. The components used by this model are: REST API Cloud DataLab Cloud data storage Python, Ruby Clients Library Cloud Endpoints The most important application of the model is the conversion of a regional language to a foreign language. The cost of text translation and language detection is $20 per 1 million characters. Use cases Now, as we have learned about the concepts and applications of the API, let's learn two use cases where it has been successfully implemented: Rule-based Machine Translation Local Tissue Response to Injury and Trauma We will discuss each of these use cases in the following sections. Rule-based Machine Translation The steps to implement rule-based Machine Translation successfully are as follows: Input text Parsing Tokenization Compare the rules to extract the meaning of prepositional phrase Find word of inputted language to word of the targeted language Frame the sentence of the targeted language Local tissue response to injury and trauma We can learn about the Machine Translation process from the responses of a local tissue to injuries and trauma. The human body follows a process similar to Machine Translation when dealing with injuries. We can roughly describe the process as follows: Hemorrhaging from lesioned vessels and blood clotting Blood-borne physiological components, leaking from the usually closed sanguineous compartment, are recognized as foreign material by the surrounding tissue since they are not tissue-specific Inflammatory response mediated by macrophages (and more rarely by foreign-body giant cells) Resorption of blood clot Ingrowth of blood vessels and fibroblasts, and the formation of granulation tissue Deposition of an unspecific but biocompatible type of repair (scar) tissue by fibroblasts Cloud Vision API Cloud Vision API is powerful image analytic tool. It enables the users to understand the content of an image. It helps in finding various attributes or categories of an image, such as labels, web, text, document, properties, safe search, and code of that image in JSON. In labels field, there are many sub-categories like text, line, font, area, graphics, screenshots, and points. How much area of graphics involved, text percentage, what percentage of empty area and area covered by text, is there any image partially or fully mapped in web are included web contents. The document consists of blocks of the image with detailed description, properties show that the colors used in image is visualized. If any unwanted or inappropriate content is removed from the image through safe search. The main features of this API are label detection, explicit content detection, logo and landmark detection, face detection, web detection, and to extract the text the API used Optical Character Reader (OCR) and is supported for many languages. It does not support face recognition system. The architecture for the Cloud Vision API is as follows: We can summarize the functionalities of the API as extracting quantitative information from images, taking the input as an image and the output as numerics and text. The components used in the API are: Client Library REST API RPC API OCR Language Support Cloud Storage Cloud Endpoints Applications of the API include: Industrial Robotics Cartography Geology Forensics and Military Medical and Healthcare Cost: Free of charge for the first 1,000 units per month; after that, pay as you go. Use cases This technique can be successfully implemented in: Image detection using an Android or iOS mobile device Retinal Image Analysis (Ophthalmology) We will discuss each of these use cases in the following topics. Image detection using Android or iOS mobile device Cloud Vision API can be successfully implemented to detect images using your smartphone. The steps to do this are simple: Input the image Run the Cloud Vision API Executes methods for detection of Face, Label, Text, Web and Document properties Generate the response in the form of phrase or string Populate the image details as a text view Retinal Image Analysis – ophthalmology Similarly, the API can also be used to analyze retinal images. The steps to implement this are as follows: Input the images of an eye Estimate the retinal biomarkers Do the process to remove the effected portion without losing necessary information Identify the location of specific structures Identify the boundaries of the object Find similar regions in two or more images Quantify the image with retinal portion damage You can learn a lot more about the machine learning capabilities of GCP on their official documentation page. If you found the above excerpt useful, make sure you check out our book 'Cloud Analytics with Google Cloud Platform' for more information on why GCP is a top cloud solution for machine learning and AI. Read more Google announces Cloud TPUs on the Cloud Machine Learning Engine (ML Engine) How machine learning as a service is transforming cloud Google announce the largest overhaul of their Cloud Speech-to-Text  

The data science tech market is buzzing with new and interesting Machine Learning libraries and tools almost everyday. In an increasingly growing market, it becomes difficult to choose the right tool or set of tools. More importantly, Artificial Intelligence and Deep Learning based projects require a different approach than traditional programming which makes things tricky to zero-in on one library or a framework. The choice of a framework is largely based upon the type of problem, one is expecting to solve. But there are other considerations too. Speed is one such factor that more or less would always play an important role in decision making. Other reasons could be how open-ended it is, architecture, functions, complexity of use, support for algorithms, and so on. Here, we present to you six Java libraries for your next Deep Learning and Artificial Intelligence project you shouldn’t miss if you are a Java loyalist or simply a web developer who wants to enter the world of deep learning. DeepLearning4j (DL4J) One of the first, commercial grade, and most popular deep learning frameworks developed in Java. It also supports other JVM languages (Java, Clojure, Scala). What’s interesting about the DL4J, is that it comes with an in-built GPU support for the training process. It also supports Hadoop YARN for distributed application management. It is popular for solving problems related to image recognition, fraud detection and NLP. MALLET Mallet (Machine Learning for Language Toolkit) is an open source Java Machine Learning toolkit. It supports NLP, clustering, modelling, and classification. The most important capability of Mallet is its support for a wide variety of algorithms such as Naive Bayes and Decision Trees. Another useful feature it has is topic modelling toolkit. Topic models are useful when analyzing large collections of unlabelled texts.   Massive Online Analysis (MOA) MOA is an open source data streaming and mining framework for real time analytics. It has a strong and growing community and is similar and related to Weka. It also has the ability to deal with massive data streams. Encog This framework supports a wide array of algorithms and neural networks such as Artificial Neural Network, Bayesian Network, Genetic Programming and algorithms. Neuroph Neuroph as the name suggests offers great simplicity when working on neural networks. The main USP of Neuroph is its incredibly useful GUI (Graphical User Interface) tool that helps in creating and training neural networks. Neuroph is a good choice of framework when you have a quick project on hand and you don’t want to spend hours learning the theory. Neuroph helps you quickly set up and running in putting neural networks to work for your project. Java Machine Learning Library The Java Machine Learning Library offers a great set of reference implementation of algorithms that you can’t miss for your next Machine Learning project. Some of the key highlights are support vector machines and clustering algorithms. These are a few key frameworks and tools you might want to consider when working on your next research work. The Java ML library ecosystem is vast with many tools and libraries to support, and we just touched the tip of that iceberg in this article. One particular tool that deserve an honourable mention is Environment for Developing KDD-Applications Supported by Index-Structure (ELKI). It is designed particularly with researchers and research students kept in mind. The main focus of ELKI is its broad coverage of data algorithms which makes it a natural fit for research work. What’s really important while choosing any of the above or tools outside of the list is a good understanding of the requirements and the problems you intend to solve. To reiterate, some of the key considerations to bear in mind before zeroing in on a tool would be - support for algorithms, implementation of neural networks, dataset size (small, medium, large), and speed.

Massive changes are happening around the way IT services are consumed and delivered. Cloud-based infrastructure is being tied together and instrumented by DevOps processes, while microservices-driven apps are replacing monolithic architectures. This evolution is driving the need for greater monitoring and better analysis of data than we have ever seen before. This need is compounded by the fact that an application today may be instrumented with the help of sensors and devices providing users with critical input in making decisions. Why is there a need for monitoring and analysis? The placement of sensors on practically every available surface in the material world – from machines to humans – is a reality today. Almost anything that is capable of giving off a measurable metric or recorded event can be instrumented, in the virtual world as well as the physical world, and has the need for monitoring. Metrics involve the consistent measurement of characteristics, such as CPU usage, while events are something that is triggered, such as temperature reaching above a threshold. The right instrumentation, observation and analytics are required to create business insight from the myriad of data points coming from these instruments. In the virtual world, monitoring and controlling software components that drive business processes is critical. Data monitoring in software is an important aspect of visualizing what systems are doing – what activities are happening, and precisely when – and how well the applications and services are performing. There is, of course, a business justification for all this monitoring of constant streams of metrics and events data. Companies want to become more data-driven, they want to apply data insights to be better situationally aware of business opportunities and threats. A data-driven organization is able to predict outcomes more effectively than relying on historical information, or on gut instinct. When vast amounts of data points are monitored and analyzed, the organization can find interesting “business moments” in the data. These insights help identify emerging opportunities and competitive advantages. How to develop a Data monitoring strategy Establishing an overall IT monitoring strategy that works for everyone across the board is nearly impossible. But it is possible to develop a monitoring strategy which is uniquely tailored to specific IT and business needs. At a high level, organizations can start developing their Data monitoring strategy by asking these five fundamental questions: #1 Have we considered all stakeholder needs? One of the more common mistakes DevOps teams make is focusing the monitoring strategies on the needs of just a few stakeholders and not addressing the requirements of stakeholders outside of IT operations, such as line of business (LOB) owners, application developers and owners, and other subgroups within operations, such as network operations (NOC) or communications teams. For example, an app developer may need usage statistics around application performance while the network operator might be interested in network bandwidth usage by that app’s users. #2 Will the data capture strategy meet future needs? Organizations, of course, must key on the data capture needs of today at the enterprise level, but at the same time, must consider the future. Developing a long-term plan helps in future-proofing the overall strategy since data formats and data exchange protocols always evolve. The strategy should also consider future needs around ingestion and query volumes. Planning for how much data will be generated, stored and archived will help establish a better long-term plan. #3 Will the data analytics satisfy my organization’s evolving needs? Data analysis needs always change over time. Stakeholders will ask for different types of analysis and planning ahead for those needs, and opting for a flexible data analysis strategy will help ensure that the solution is able to support future needs. #4 Is the presentation layer modular and embeddable? A flexible user interface that addresses the needs of all stakeholders is important for meeting the organization’s overarching goals. Solutions which deliver configurable dashboards that enable users to specify queries for custom dashboards meet this need for flexibility. Organizations should consider a plug-and-play model which allows users to choose different presentation layers as needed. #5 Does architecture enable smart actions? The ability to detect anomalies and trigger specific actions is a critical part of a monitoring strategy. A flexible and extensible model should be used to meet the notification preferences of diverse user groups. Organizations should consider self-learning models which can be trained to detect undefined anomalies from the collected data. Monitoring solutions which address the broader monitoring needs of the entire enterprise are preferred. What are purpose-built monitoring platforms Devising an overall IT monitoring strategy that meets these needs and fundamental technology requirements is a tall order. But new purpose-built monitoring platforms have been created to deal with today’s new requirements for monitoring and analyzing these specific metrics and events workloads – often called time-series data – and provide situational awareness to the business. These platforms support ingesting millions of data points per second, can scale both horizontally and vertically, are designed from the ground up to support real-time monitoring and decision making, and have strong machine learning and anomaly detection functions to aid in discovering interesting business moments. In addition, they are resource-aware, applying compression and down-sampling functions to aid in optimal resource utilization, and are built to support faster time to market with minimal dependencies. With the right strategy in mind, and tools in place, organizations can address the evolving monitoring needs of the entire organization. About the Author Mark Herring is the CMO of InfluxData. He is a passionate marketeer with a proven track record of generating leads, building pipeline, and building vibrant developer and open source communities. Data-driven marketeer with proven ability to define the forest from the trees, improve performance, and deliver on strategic imperatives. Prior to InfluxData, Herring was vice president of corporate marketing and developer marketing at Hortonworks where he grew the developer community by over 40x. Herring brings over 20 years of relevant marketing experience from his roles at Software AG, Sun, Oracle, and Forte Software. TensorFlow announces TensorFlow Data Validation (TFDV) to automate and scale data analysis, validation, and monitoring. How AI is going to transform the Data Center. Introducing TimescaleDB 1.0, the first OS time-series database with full SQL support.

Artificial Intelligence is making it easier for people to do things every day. You can schedule your day, search for photos of loved ones, type emails on the go, or get things done with the virtual assistant. AI also provides innovative ways of tackling existing problems, from healthcare to advancing scientific discovery. According to Gartner’s Top 10 Strategic Technology Trends for 2018, the next few years will see every app, application, and service incorporating AI at some level. With major companies like Google, Amazon, IBM investing in AI and incorporating AI in their products, this statement, instead of a prediction is becoming a fact. Apple’s IPhone X comes with a Facial Recognition System, Samsung’s Bixby, Amazon’s Alexa, Google’s Google Assistant, and the recently launched Android Pie. Android Pie learns your preferences based on your usage patterns and gets better over time. It even provides you a breakdown of the time you spend on your phone. AI comes with endless possibilities, things that we used to dream of are now becoming a part of our day to day life. So, I have listed here, in no particular order, some of those innovative applications: Microsoft’s Seeing AI - Eye for the visually impaired Source: Microsoft Seeing AI is a perfect example of how technology is improving our lives. It is an intelligent camera app that uses computer vision to audibly help blind and visually impaired people to know about their surroundings. It comes with functionalities like reading out short text and documents for you, giving you description about a person, identifies currencies, colour, handwriting, light and even images in other apps using the device's camera. A data scientist named Anirudh Koul started this project (called Deep Vision earlier) to help his grandfather who was gradually losing his vision. Two breakthroughs by the Microsoft researchers facilitated him to further his idea: vision-to-language and image classification. To make the app this advance and real-time, they used the idea of making servers communicate with Microsoft Cognitive Services. This app brings in four technologies together to provide users with an array of functionalities: OCR, barcode scanner, facial recognition, and scene recognition. Check out this YouTube tutorial to understand how it works. Download App Store Ada - Healthcare in your hand Source: Digital Health Ada, with a very simple and conversational UI, helps you understand what could be wrong if you or someone you care about is not feeling well. Just like any doctor’s appointment, it starts with your basic details, then does an assessment, in which it asks several personalized questions related to the symptoms, and then gives a report. The report consists of a summary, possible causes, and less-likely causes. It also allows you to share the report as a PDF. After training over several years using real world cases, Ada has become a handy health advisor. Its platform is powered by a sophisticated Artificial Intelligence engine combined with large medical knowledge base covering many thousands of conditions, symptoms and findings. In every medical assessment, Ada takes all of a patient’s information into account, including past medical history, symptoms, risk factors and more. Using machine learning and multiple closed feedback loops, Ada becomes more intelligent. Download App Store Google Play Store Plume Air Report - An air pollution monitor Source: Plume Labs Blog Industrialization and urbanization definitely comes with their side effects, the main being air pollution. It has become inevitable to keep yourself safe from the pollution, but now at least you can be aware of the air pollution levels in your area. Plume Air Report forecasts how air quality will evolve hour by hour over the next 24 hours similar to weather forecast. You can also easily compare the air quality between cities. It gives you insight on all pollutants (PM2.5, PM10, O3, NO2), with absolute concentration levels and your local air quality scale. It uses machine learning and atmospheric sciences to deliver real-time and hourly forecast air quality data. First, latest pollution levels is collected from over 12,000 monitoring stations and 80 public agencies around the world and then filtered for errors. Local atmospheric data (wind, temperature, atmosphere, etc.) is sourced to track their influence on pollution levels in your city. A team of data scientists analyzes local specifics such as geographical features and human activities. Finally, AI algorithms and atmospheric models are developed that turn this giant amount of data into hourly forecasts. Download App Store Google Play Store Aura - Mindfulness meets AI Source: Popular Science In this fast life, slow down a little and give yourself a time out with Aura. Aura is a new kind of mindfulness app that learns about you and simplifies your learning through guided meditations. It helps in reducing stress and increases positivity through 3-minute meditations, personalized by Artificial Intelligence. Aura is an intelligent app that leverages machine learning to give you a unique experience. After every exercise, you can rate your experience and Aura will learn how to provide more tailored meditations according to your needs. You can even track your mood and learn your mood patterns. Download App Store Google Play Store Replika - An emotive chatbot as a friend for life Source: Medium Want to be friends with someone who is always there to listen to you, talk to you, and never judges you? Then Replika is for you! It helps you make a real connection with an unreal friend. The idea of building Replika came from a very tragic background. The founder of the software company, Luka, Eugenia Kudya, lost her best friend in an accident in November 2015. She used to go through their messenger texts to bring back their memories. This is how she got this idea to develop a chatbot making it learn from the sample texts sent by her best friend. In her own words, “Most of the companies try to build an app that talks, but we tried to build an app that could listen well”. The chatbot uses neural network facilitating more natural one-on-one conversation with its user, and over time, learn how to speak like them. The source code is freely available for developers under the name CakeChat. It comes with a pre-trained model that you can use as is to run a chatbot that maintains a conversation in a certain emotional state. You can also build a variety of other conversational agents by using your own dataset, for example, persona-based model, emotional chatting machine, topic-centric model. To know more about the background and evolution of Replika, check out this amazing YouTube video. Download App Store Google Play Store Google Assistant - Your personal Google Source: Google Assistant When talking of AI-powered apps, voice assistants probably come first in your mind. Google Assistant makes your life easier and helps in organizing your day better. You can manage your little tasks, plan your day, enjoy entertainment, and get answers. It can also sync to your other devices including Google Home, smart TVs, laptops, and more. To give users smart assistance, Google Assistant relies on Artificial Intelligence technologies such as natural language processing, natural language understanding, and machine learning to understand what the user is saying, and to make suggestions or act on that language input. Download App Store Google Play Store Hound - Say it, Get it Source: Android Apps In an array of virtual assistants to choose from, Hound understands your voice commands better. You do not need to give “search query” like commands and can have a more natural conversation. Hound can be used for variety of tasks, some of them are: search, discover, and play music, set alarms, timers, and reminders, call, text, navigate hands-free, get the weather forecast. Hound’s speed and accuracy comes from their powerful Houndify platform. This platform combines Speech Recognition and Natural Language Understanding into a single step, which is called Speech-to-Meaning. Download App Store Google Play Store Picai - An app that picks filters for your pics, keeping you looking your best always Source: Google Play Store Picai with the help of Artificial Intelligence, recommends picture-perfect filters by analyzing the scene. It automatically analyzes the scene and with the help of object recognition detects the type of the object, for example, a plant, a girl, etc. It then uses a proprietary deep learning model to recommend two optimum filters from 100+ filters. What makes this app stand out is the split-screen filter selection, which makes the filter selection easier for the users. When using this app be warned of the picture quality and app size (76 MB), but it is definitely worth trying! Download Google Play Store Microsoft Pix - The pro photographer Source: MSPoweuser Named one of the 50 Best Apps of the Year by Time Magazine, Microsoft Pix helps you take better photos without the extra effort! It solves the problem of “not living in the moment”. It comes with some amazing features like, hyperlapse, live images Microsoft Pix Comix, artistic styles to transform your photos, smart settings that automatically checks scene and lighting between each shutter tap, and updates settings between each shot, and more. Microsoft Pix uses Artificial Intelligence to improve the image, such as cropping edges, enhancing color and tone, and sharpening focus. It includes enhanced deep-learning capabilities around image understanding. It captures a burst of 10 frames with each shutter click and uses AI to select three best shots. Before the remaining photos are deleted, it uses data from the entire burst to remove noise. These best, enhanced images are ready in about a second. The app also detects whether your eyes are open or not using the facial recognition technology. Download App Store ELSA - Your machine learning English teacher Source: TechCrunch ELSA (English Language Speech Assistant)  helps you in learning English and bettering your pronunciation every day. It provides you a curriculum tailored just, regular feedback, progress tracking, common phrases used in daily life. You can practice in a relaxed environment and improve your speaking skills to prepare for the TOEFL, IELTS, TOEIC ELSA coaches you in improving your English pronunciations by using speech recognition, deep learning, and Artificial Intelligence. Download App Store Google Play Store Socratic - Homework in a snap Source: Google Play Store Socratic is your new helper, apart from your parents, in completing those complex Math problems. You just need to take a photo of your homework and can get explanations, videos, step-by-step help, instantly. Also, these resources are jargon-free, helping you understand the concepts better. It supports all subjects including Math (Algebra, Calculus, Statistics, Graphing, etc), Science, Chemistry, History, English, Economics, and more. Socratic uses Artificial Intelligence to figure out the concepts you need to learn in order to answer it. For this it combines cutting-edge computer vision technologies, which read questions from images, with machine learning classifiers. These classifiers are built using millions of sample homework questions, to accurately predict which concepts will help you solve your question. Download App Store Google Play Store Recent News - Stay informed Source: Recent News Recent News is an app that will provide you customized news. Some of the features that it comes with to give you the daily dose of news include one-minute news summary with very quick load time, hot news, local news, and personalized recommendations, instantly share news on Facebook, Twitter, and other social networks, and many more. It uses Artificial Intelligence to learn about your interests, suggest relevant articles, and propose topics you might like to follow. So, the more you use it the better it becomes! The app is surely innovative and saves time, but I do wish the developers applied some innovation in the app’s name as well :P Download App Store Google Play Store And that’s the end of my list. People say, “Smartphones and apps are becoming smarter, and we are becoming dumber”. But I would like to say that these apps, with the right usage, empower us to become smarter. Agree? 7 Popular Applications of Artificial Intelligence in Healthcare 5 examples of Artificial Intelligence in Web apps What Should We Watch Tonight? Ask a Robot, says Matt Jones from OVO Mobile [Interview]

Budget and demand forecasting are important aspects of any finance team. Budget forecasting is the outcome, and demand forecasting is one of its components. In this article, we understand the Markov model for forecasting and budgeting in finance.   This article is an excerpt from a book written by Harish Gulati titled SAS for Finance. Understanding problem of budget and demand forecasting While a few decades ago, retail banks primarily made profits by leveraging their treasury office, recent years have seen fee income become a major source of profitability. Accepting deposits from customers and lending to other customers is one of the core functions of the treasury. However, charging for current or savings accounts with add-on facilities such as breakdown cover, mobile, and other insurances, and so on, has become a lucrative avenue for banks. One retail bank has a plain vanilla classic bank account, mid-tier premier, and a top-of-the-range, benefits included a platinum account. The classic account is offered free and the premier and platinum have fees of $10 and $20 per month respectively. The marketing team has just relaunched the fee-based accounts with added benefits. The finance team wanted a projection of how much revenue could be generated via the premier and the platinum accounts. Solving with Markovian model approach Even though we have three types of account, the classic, premier, and the platinum, it doesn't mean that we are only going to have nine transition types possible as in Figure 4.1. There are customers who will upgrade, but also others who may downgrade. There could also be some customers who leave the bank and at the same time there will be a constant inflow of new customers. Let's evaluate the transition states flow for our business problem: In Figure 4.2, we haven't jotted down the transition probability between each state. We can try to do this by looking at the historical customer movements, to arrive at the transitional probability. Be aware that most business managers would prefer to use their instincts while assigning transitional probabilities. They may achieve some merit in this approach, as the managers may be able to incorporate the various factors that may have influenced the customer movements between states. A promotion offering 40% off the platinum account (effective rate $12/month, down from $20/month) may have ensured that more customers in the promotion period opted for the platinum account than the premier offering ($10/month). Let's examine the historical data of customer account preferences. The data is compiled for the years 2008 – 2018. This doesn't account for any new customers joining after January 1, 2008 and also ignores information on churned customers in the period of interest. Figure 4.3 consists of customers who have been with the bank since 2008: Active customer counts (Millions) Year Classic (Cl) Premium (Pr) Platinum (Pl) Total customers 2008 H1 30.68 5.73 1.51 37.92 2008 H2 30.65 5.74 1.53 37.92 2009 H1 30.83 5.43 1.66 37.92 2009 H2 30.9 5.3 1.72 37.92 2010 H1 31.1 4.7 2.12 37.92 2010 H2 31.05 4.73 2.14 37.92 2011 H1 31.01 4.81 2.1 37.92 2011 H2 30.7 5.01 2.21 37.92 2012 H1 30.3 5.3 2.32 37.92 2012 H2 29.3 6.4 2.22 37.92 2013 H1 29.3 6.5 2.12 37.92 2013 H2 28.8 7.3 1.82 37.92 2014 H1 28.8 8.1 1.02 37.92 2014 H2 28.7 8.3 0.92 37.92 2015 H1 28.6 8.34 0.98 37.92 2015 H2 28.4 8.37 1.15 37.92 2016 H1 27.6 9.01 1.31 37.92 2016 H2 26.5 9.5 1.92 37.92 2017 H1 26 9.8 2.12 37.92 2017 H2 25.3 10.3 2.32 37.92 Figure 4.3: Active customers since 2008 Since we are only considering active customers, and no new customers are joining or leaving the bank, we can calculate the number of customers moving from one state to another using the data in Figure 4.3: Customer movement count to next year (Millions) Year Cl-Cl Cl-Pr Cl-Pl Pr-Pr Pr-Cl Pr-Pl Pl-Pl Pl-Cl Pl-Pr Total customers 2008 H1 - - - - - - - - - - 2008 H2 30.28 0.2 0.2 5.5 0 0.23 1.1 0.37 0.04 37.92 2009 H1 30.3 0.1 0.25 5.1 0.53 0.11 1.3 0 0.23 37.92 2009 H2 30.5 0.32 0.01 4.8 0.2 0.43 1.28 0.2 0.18 37.92 2010 H1 30.7 0.2 0 4.3 0 1 1.12 0.4 0.2 37.92 2010 H2 30.7 0.2 0.2 4.11 0.35 0.24 1.7 0 0.42 37.92 2011 H1 30.9 0 0.15 4.6 0 0.13 1.82 0.11 0.21 37.92 2011 H2 30.2 0.8 0.01 3.8 0.1 0.91 1.29 0.4 0.41 37.92 2012 H1 30.29 0.4 0.01 4.9 0.01 0.1 2.21 0 0 37.92 2012 H2 29.3 0.9 0.1 5.3 0 0 2.12 0 0.2 37.92 2013 H1 29.2 0.1 0 6.1 0.1 0.2 1.92 0 0.3 37.92 2013 H2 28.6 0.3 0.4 6.5 0 0 1.42 0.2 0.5 37.92 2014 H1 28.7 0.1 0 7.2 0.1 0 1.02 0 0.8 37.92 2014 H2 28.7 0 0.1 8.1 0 0 0.82 0 0.2 37.92 2015 H1 28.6 0 0.1 8.3 0 0 0.88 0 0.04 37.92 2015 H2 28.3 0 0.3 8 0.1 0.24 0.61 0 0.37 37.92 2016 H1 27.6 0.8 0 8.21 0 0.16 1.15 0 0 37.92 2016 H2 26 1 0.6 8.21 0.5 0.3 1.02 0 0.29 37.92 2017 H1 25 0.5 1 8 0.5 1 0.12 0.5 1.3 37.92 2017 H2 25.3 0.1 0.6 9 0 0.8 0.92 0 1.2 37.92 Figure 4.4: Customer transition state counts In Figure 4.4, we can see the customer movements between various states. We don't have the movements for the first half of 2008 as this is the start of the series. In the second half of 2008, we see that 30.28 out of 30.68 million customers (30.68 is the figure from the first half of 2008) were still using a classic account. However, 0.4 million customers moved away to premium and platinum accounts. The total customers remain constant at 37.92 million as we have ignored new customers joining and any customers who have left the bank. From this table, we can calculate the transition probabilities for each state: Year Cl-Cl Cl-Pr Cl-Pl Pr-Pr Pr-Cl Pr-Pl Pl-Pl Pl-Cl Pl-Pr 2008 H2 98.7% 0.7% 0.7% 96.0% 0.0% 4.0% 72.8% 24.5% 2.6% 2009 H1 98.9% 0.3% 0.8% 88.9% 9.2% 1.9% 85.0% 0.0% 15.0% 2009 H2 98.9% 1.0% 0.0% 88.4% 3.7% 7.9% 77.1% 12.0% 10.8% 2010 H1 99.4% 0.6% 0.0% 81.1% 0.0% 18.9% 65.1% 23.3% 11.6% 2010 H2 98.7% 0.6% 0.6% 87.4% 7.4% 5.1% 80.2% 0.0% 19.8% 2011 H1 99.5% 0.0% 0.5% 97.3% 0.0% 2.7% 85.0% 5.1% 9.8% 2011 H2 97.4% 2.6% 0.0% 79.0% 2.1% 18.9% 61.4% 19.0% 19.5% 2012 H1 98.7% 1.3% 0.0% 97.8% 0.2% 2.0% 100.0% 0.0% 0.0% 2012 H2 96.7% 3.0% 0.3% 100.0% 0.0% 0.0% 91.4% 0.0% 8.6% 2013 H1 99.7% 0.3% 0.0% 95.3% 1.6% 3.1% 86.5% 0.0% 13.5% 2013 H2 97.6% 1.0% 1.4% 100.0% 0.0% 0.0% 67.0% 9.4% 23.6% 2014 H1 99.7% 0.3% 0.0% 98.6% 1.4% 0.0% 56.0% 0.0% 44.0% 2014 H2 99.7% 0.0% 0.3% 100.0% 0.0% 0.0% 80.4% 0.0% 19.6% 2015 H1 99.7% 0.0% 0.3% 100.0% 0.0% 0.0% 95.7% 0.0% 4.3% 2015 H2 99.0% 0.0% 1.0% 95.9% 1.2% 2.9% 62.2% 0.0% 37.8% 2016 H1 97.2% 2.8% 0.0% 98.1% 0.0% 1.9% 100.0% 0.0% 0.0% 2016 H2 94.2% 3.6% 2.2% 91.1% 5.5% 3.3% 77.9% 0.0% 22.1% 2017 H1 94.3% 1.9% 3.8% 84.2% 5.3% 10.5% 6.2% 26.0% 67.7% 2017 H2 97.3% 0.4% 2.3% 91.8% 0.0% 8.2% 43.4% 0.0% 56.6% Figure 4.5: Transition state probability In Figure 4.5, we have converted the transition counts into probabilities. If 30.28 million customers in 2008 H2 out of 30.68 million customers in 2008 H1 are retained as classic customers, we can say that the retention rate is 98.7%, or the probability of customers staying with the same account type in this instance is .987. Using these details, we can compute the average transition between states across the time series. These averages can be used as the transition probabilities that will be used in the transition matrix for the model: Cl Pr Pl Cl 98.2% 1.1% 0.8% Pr 2.0% 93.2% 4.8% Pl 6.3% 20.4% 73.3% Figure 4.6: Transition probabilities aggregated The probability of classic customers retaining the same account type between semiannual time periods is 98.2%. The lowest retain probability is for platinum customers as they are expected to transition to another customer account type 26.7% of the time. Let's use the transition matrix in Figure 4.6 to run our Markov model. Use this code for Data setup: DATA Current; input date CL PR PL; datalines; 2017.2 25.3 10.3 2.32 ; Run; Data Netflow; input date CL PR PL; datalines; 2018.1 0.21 0.1 0.05 2018.2 0.22 0.16 0.06 2019.1 .24 0.18 0.08 2019.2 0.28 0.21 0.1 2020.1 0.31 0.23 0.14 ; Run; Data TransitionMatrix; input CL PR PL; datalines; 0.98 0.01 0.01 0.02 0.93 0.05 0.06 0.21 0.73 ; Run; In the current data set, we have chosen the last available data point, 2017 H2. This is the base position of customer counts across classic, premium, and platinum accounts. While calculating the transition matrix, we haven't taken into account new joiners or leavers. However, to enable forecasting we have taken 2017 H2 as our base position. The transition matrix seen in Figure 4.6 has been input as a separate dataset. Markov model code PROC IML; use Current; read all into Current; use Netflow; read all into Netflow; use TransitionMatrix; read all into TransitionMatrix; Current = Current [1,2:4]; Netflow = Netflow [,2:4]; Model_2018_1 = Current * TransitionMatrix + Netflow [1,]; Model_2018_2 = Model_2018_1 * TransitionMatrix + Netflow [1,]; Model_2019_1 = Model_2018_2 * TransitionMatrix + Netflow [1,]; Model_2019_2 = Model_2019_1 * TransitionMatrix + Netflow [1,]; Model_2020_1 = Model_2019_2 * TransitionMatrix + Netflow [1,]; Budgetinputs = Model_2018_1//Model_2018_2//Model_2019_1//Model_2019_2//Model_2020_1; Create Budgetinputs from Budgetinputs; append from Budgetinputs; Quit; Data Output; Set Budgetinputs (rename=(Col1=Cl Col2=Pr Col3=Pl)); Run; Proc print data=output; Run; Figure 4.7: Model output The Markov model has been run and we are able to generate forecasts for all account types for the requested five periods. We can immediately see that there is an increase forecasted for all the account types. This is being driven by the net flow of customers. We have derived the forecasts by essentially using the following equation: Forecast = Current Period * Transition Matrix + Net Flow Once the 2018 H1 forecast is derived, we replace the Current Period with the 2018 H1 forecasted number while trying to forecast the 2018 H2 numbers. We are doing this as, based on the 2018 H1 customer counts, the transition probabilities will determine how many customers move across states. This will help generate the forecasted customer count for the required period. Understanding transition probability Now, since we have our forecasts let's take a step back and revisit our business goals. The finance team wants to estimate the revenues from the revamped premium and platinum customer accounts for the next few forecasting periods. As we have seen, one of the important drivers of the forecasting process is the transition probability. This transition probability is driven by historical customer movements, as shown in Figure 4.4. What if the marketing team doesn't agree with the transitional probabilities calculated in Figure 4.6? As we discussed, 26.7% of platinum customers aren't retained in this account type. Since we are not considering customer churn out of the bank, this means that a large proportion of platinum customers downgrade their accounts. One of the reasons the marketing teams revamped the accounts is due to this reason. The marketing team feels that it will be able to raise the retention rates for platinum customers and want the finance team to run an alternate forecasting scenario. This is, in fact, one of the pros of the Markov model approach as by tweaking the transition probabilities we can run various business scenarios. Let's compare the base and the alternate scenario forecasts generated in Figure 4.8: A change in the transition probabilities of how platinum customers moved to various states has brought about a significant change in the forecast for premium and platinum customer accounts. For classic customers, the change in the forecast between the base and the alternate scenario is negligible, as shown in the table in Figure 4.8. The finance team can decide which scenario is best suited for budget forecasting: Cl Pr Pl Cl 98.2% 1.1% 0.8% Pr 2.0% 93.2% 4.8% Pl 5.0% 15.0% 80.0% Figure 4.8: Model forecasts and updated transition probabilities To summarize, we learned the Markov model methodology and learned Markov models for forecasting and imputation. To know more about how to use the other two methodologies such as ARIMA and MCMC for generating forecasts for various business problems, you can check out the book SAS for Finance. Read more How to perform regression analysis using SAS Performing descriptive analysis with SAS Akon is planning to create a cryptocurrency city in Senegal

“..what we want is a machine that can learn from experience..” ~Alan Turing, 1947 Thanks to artificial intelligence, Turing’s vision is coming true. Machines are learning, from others’ experience (using training datasets) and from their own as well.  Machines can now play chess, Go, and other games, they can help predict cancer, manage your day, summarize today’s news for you, edit your essays, identify your face, and even mimic dance moves and facial expressions. Come to think of it, every job role and career demands that you learn from experience, improve over time and explore new ways to do things.  Yes, machines are very effective at the former two, but humans still have an edge when it comes to innovative thinking. Imagine what you could achieve if you put together your mind with that of an efficient learning algorithm! You might think that artificial intelligence and machine learning are a dense and impenetrable field limited to research labs and textbooks. Does that mean only software engineers and researchers can dream of making it into this fascinating field? Not quite. We’ll unpick machine learning in the following sections and present our case for why it makes sense for everyone to understand this field better. Machine learning is, potentially, a first-class ticket to an exciting career, whether you are starting off fresh from college or are considering a career switch. Beyond the artificial intelligence and machine learning hype Artificial intelligence is simply an area of computing that solves complex real-world problems. Yes, research still happens in universities, and yes, data scientists are still exploring the limits of artificial intelligence in forward-thinking businesses, but it's much more than that. AI is so pervasive - and mysterious - that its applications hide in plain sight. Look around you carefully. From Netflix recommending personalized content to its 130 million viewers, to Youtube’s video search and automatic captions in videos, to Amazon’s shopping recommendations, to Instagram hashtags, Snapchat filters, spam filters on your Gmail and virtual assistants like Siri on our smartphones, artificial intelligence, and machine learning techniques are in action everywhere. This means as a user you are at some level already impacted by algorithms every day. The question then is should you be the person who’s career is limited by algorithms or the one whose career is propelled by algorithms. Why get into artificial intelligence development as a non-programmer? Artificial Intelligence is a perfect blend of knowledge, high salary, and some really great opportunities. Your non-programming field does not have to deter your growth in the AI field. In fact, your background can give you an edge over the traditional software developers and data scientists in terms of domain awareness and better understanding what the system should do, what it should look for, and make the users feel. Below are some reasons proving why you should make the jump in AI. Machine learning can help you be better at your current job How? You may ask. Take a news reporter or editor’s job for example. They must possess a blend of research/analysis centric capabilities, a creative set of skills and speed to come up with timely, quality articles on topics of interest to their readers. A data journalist or a writer with machine learning experience could quickly find great topics to write on with the help of machine learning based web scraping apps. Also, they could let the data lead them to unique stories that are emerging before traditional news reporters find their way to them. They could further also get a quick summary of multiple perspectives on a given topic using custom-built news feed algorithms. Then could they also find further research resources by tweaking their search parameters, even adding quality filters on top to only allow for high-quality citations. This kind of writer has cut down on the time they spent finding and understanding topics - which means more time to actually write compelling pieces and to connect with real sources for further insight. Algorithms can also find and correct language issues in writing now. This means editors can spend more time improving the content quality from a scope perspective. You can quickly start to see how artificial intelligence can complement the work you do and help you grow in your career. Yes, all this sounds lovely in theory, but is it really happening in practice? There are others like you who are successfully exploring machine learning Don’t believe me? Mason Fish, a software Engineer at Docker, Inc was earlier a musician. He had done his bachelor’s and masters from two different music conservatories. After graduating, he worked for five years as a professional musician. But, today he helps build and maintain services for Docker, a tool used by software engineers all over the world! This was just one case of a non-programmer diving into the computer science world. When musicians can learn to code and get core developer jobs in cutting-edge tech companies, it is not far fetched to say they can also learn to build machine learning models. Below are some examples of non-programmers of varied experience levels who are exploring the Machine Learning world. Per Harald Borgen, an economics graduate was able to boost the sales at his workplace Xeneta using machine learning algorithms, an accomplishment that helped accelerate his career. You can read his blog to see how he transformed from a machine learning newbie to a seasoned practitioner. Another example is a 14-year-old Tanmay Bakshi, who started a youtube channel at just 7 years of age where he teaches coding, algorithms, AI and machine learning concepts. Similarly, Sean Le Van created an AI chatbot when he was 14 years old using ML algorithms.   Rosebud Anwuri is another great example as she switched from chemical engineering to Data science. “My first exposure to Data Science was from a book that had nothing to do with Data Science,” writes Anwuri on her blog. She created her first Data Science learning path from an answer on Quora, last year. Fast forward to this year, she has been invited to speak at Stanford’s Women in Data Science Conference in Nigeria and has facilitated a workshop at The Women in Machine Learning and Data Science among others. She also writes on Machine Learning and Data Science on her blog.   Like Anwuri, Sce Pike dreamed of being an artist or singer in college and did her major in fine arts and anthropology. Pike went from art to web design to “human factors design,” which involves human-machine interactions, for the telecommunications giant Qualcomm. In addition to that, Pike started her own company IOTAS, that offers smart-home services to renters and homeowners. “I have had to approach my work with logic, research, and great design. Looking back, I’m amazed where I am now,” says Sce Pike. Read also: Data science for non-techies: How I got started (Part 1) Adapt or perish in the oncoming job automation wave of the fourth industrial revolution Ok, so maybe you’re happy with how you are growing anyway in your career. Be warned though, your job may not look the same even in the next few years. Automation is expected to replace up to 30% of jobs in the next 10 years, so upskilling to machine learning is a wise choice. Last month, Bank of England’s Chief Economist warned that 15 million jobs in Britain could be at stake because of artificial intelligence. Machine learning as a skill could help you stay relevant in the future and prepare for what’s being called, “the third machine age”. You can develop machine learning apps with no to minimal coding experience Thanks to great advancements by big tech companies and open source projects, machine learning today is accessible to people with varying degrees of programming experience - from new developers and even those who have never written a line of code in their life. So, whether you’re a curious web/UX designer, a news reporter, an artist, a school student, a filmmaker or an NGO worker, you will find good use of machine learning in your field. There are tools for machine learning for users with varying levels of experience. In fact, there are certain Machine Learning Applications that you can build even today. Some examples are Image and text classification with Neural Network, Facial recognition, Gaming bots, music generation, object detection, etc. Machine learning skills are highly rewarded Machine learning is a nascent field where demand far outweighs supply. According to research done by Indeed.com, the number one job requirement in AI is that of a Machine Learning Engineer, with data scientist jobs taking the second spot. In fact, AI researchers can earn more than 1 million dollar per year and the AI geniuses at Elon Musk’s OpenAI are a living proof for this. OpenAI paid its top AI researcher, Ilya Sutskever, more than  $1.9 million, back in 2016. Another leading researcher, Ian Goodfellow, in OpenAI was paid more than $800,000. Machine Learning is not hard to learn. It might seem intimidating at first, but once you get the basics right, the rest of the ML journey becomes easier. If you’re convinced that ML is for you, but are confused about how to get started then don’t worry, we’ve got you covered. To help you get started, here is a non-programmer’s guide to learning Machine Learning. So, yes, it doesn’t matter if you’re a non-programmer, musician, a librarian, or a student, the future is AI-driven so don’t be afraid to make that dive into Machine Learning. As Robert Frost said, “Two roads diverged in a wood, and I took the one less traveled by, And that has made all the difference”. 8 Machine learning best practices [Tutorial] Google introduces Machine Learning courses for AI beginners Top languages for Artificial Intelligence development

Last month, a team of researchers from the University of Washington and the Allen Institute for Artificial Intelligence, published a paper titled ‘Defending Against Neural Fake News’. The goal of this paper is to reliably detect “neural fake news”, so that its harm can be minimized. With this regard, the researchers have built a model named ‘GROVER’. This works as a generator of fake news, which can also spot its own generated fake news articles, as well as those generated by other AI models. GROVER (Generating aRticles by Only Viewing mEtadata Records) models can generate an efficient yet controllable news article, with not only the body, but also the title, news source, publication date, and author list. The researchers affirm that the ‘best models for generating neural disinformation are also the best models at detecting it’. The framework for GROVER represents fake news generation and detection as an adversarial game: Adversary This system will generate fake stories that match specified attributes: generally, being viral or persuasive. The stories must be realistic to read for both human users as well as the verifier. Verifier This system will classify news stories as real or fake. A verifier will have access to unlimited real news stories and few fake news stories from a specific adversary. The dual objective of these two systems suggest an escalating ‘arms race’ between attackers and defenders. It is expected that as the verification systems get better, the adversaries too will follow. Modeling Conditional Generation of Neural Fake News using GROVER GROVER adopts a language modeling framework which allows for flexible decomposition of an article in the order of p(domain, date, authors, headline, body). During inference time, a set of fields are set as ‘F’ for context, with each field ‘f ‘ containing field-specific start and end tokens. During training, the inference is simulated by randomly partitioning an article’s fields into two disjoint sets F1 and F2. The researchers also randomly drop out individual fields with probability 10%, and drop out all but the body with probability 35%. This allows the model to learn how to perform unconditional generation. For Language Modeling, two evaluation modes are considered: unconditional, where no context is provided and the model must generate the article body; and conditional, in which the full metadata is provided as context. The researchers evaluate the quality of disinformation generated by their largest model, GROVER-Mega, using p=.96. The articles are classified into four classes: human-written articles from reputable news websites (Human News), GROVER-written articles conditioned on the same metadata (Machine News), human-written articles from known propaganda websites (Human Propaganda), and GROVER-written articles conditioned on the propaganda metadata (Machine Propaganda). Image Source: Defending Against Neural Fake News When rated by qualified workers on Amazon Mechanical Turk, it was found that though the quality of GROVER-written news is not as high as human-written news, it is very skilled at rewriting propaganda. The overall trustworthiness score of propaganda increases from 2.19 to 2.42 (out of 3) when rewritten by GROVER. Neural Fake News Detection using GROVER The role of the Verifier is to mitigate the harm of neural fake news by classifying articles as Human or Machine written. The neural fake news detection is framed in a semi-supervised method. The neural verifier (or discriminator) will have access to many human-written news articles from March 2019 and before, i.e., the entire RealNews training set. However, it will   have limited access to generations, and more recent news articles. For example, using 10k news articles from April 2019, for generating article body text; another 10k articles are used as a set of human-written news articles, it is split in a balanced way, with 10k for training, 2k for validation, and 8k for testing. It is evaluated using two modes: In the unpaired setting, a verifier is provided single news articles, which must be classified independently as Human or Machine.  In the paired setting, a model is given two news articles with the same metadata, one real and one machine-generated. The verifier must assign the machine-written article a higher Machine probability than the human-written article. Both the modes are evaluated in terms of accuracy. Image Source: Defending Against Neural Fake News It was found that the paired setting appears significantly easier than the unpaired setting across the board, suggesting that it is often difficult for the model to calibrate its predictions. Second, model size is highly important in the arms race between generators and discriminators. Using GROVER to discriminate GROVER’s generations results in roughly 90% accuracy across the range of sizes. If a larger generator is used, accuracy slips below 81%; conversely, if the discriminator is larger, accuracy is above 98%. Lastly, other discriminators perform worse than GROVER overall. This suggests that effective discrimination requires having a similar inductive bias, as the generator. Thus it has been found that GROVER can rewrite propaganda articles, with humans rating the rewritten versions as more trustworthy. At the same time, GROVER can also defend these models. The researchers are of the opinion that an ensemble of deep generative model, such as GROVER should be used to analyze the content of a text. Obviously the working of the GROVER model has caught many people’s attention. https://twitter.com/str_t5/status/1137108356588605440 https://twitter.com/currencyat/status/1137420508092391424 While some are finding this to be an interesting mechanism to combat fake news, others point out that, it doesn't matter if GROVER can identify its own texts, if it can't identify the texts generated by other models. Releasing a model like GROVERcan turn out to be extremely irresponsible rather than defensive. A user on Reddit says that “These techniques for detecting fake news are fundamentally misguided. You cannot just train a statistical model on a bunch of news messages and expect it to be useful in detecting fake news. The reason for this should be obvious: there is no real information about the label ('fake' vs 'real' news) encoded in the data. Whether or not a piece of news is fake or real depends on the state of the external world, which is simply not present in the data. The label is practically independent of the data.” Another user on Hacker News comments that “Generative neural networks these days are both fascinating and depressing - feels like we're finally tapping into how subsets of human thinking & creativity work. But that knocks us off our pedestal, and threatens to make even the creative tasks we thought were strictly a human specialty irrelevant; I know we're a long way off from generalized AI, but we seem to be making rapid progress, and I'm not sure society's mature enough or ready for it. Especially if the cutting edge tools are in the service of AdTech and such, endlessly optimizing how to absorb everybody's spare attention. Perhaps there's some bright future where we all just relax and computers and robots take care of everything for us, but can't help feeling like some part of the human spirit is dying.” Few users feel that this ‘generating and detecting its own fake news’, kind of model is going to be unnecessary in the future. It’s just a matter of time that the text written by algorithms will be exactly similar to a human written text. At that point, there will be no way to distinguish between such articles. A user suggests that “I think to combat fake news, especially algorithmic one, we'll need to innovate around authentication mechanism that can effectively prove who you are and how much effort you put into writing something. Digital signatures or things like that.” For more details about the GROVER model, head over to the research paper. Worried about Deepfakes? Check out the new algorithm that manipulate talking-head videos by altering the transcripts Speech2Face: A neural network that “imagines” faces from hearing voices. Is it too soon to worry about ethnic profiling? OpenAI researchers have developed Sparse Transformers, a neural network which can predict what comes next in a sequence

Earlier this month, Logic Magazine, a print magazine about technology, hosted a discussion about the past, present, and future of the tech worker movement. This event was co-sponsored by solidarity groups like the Tech Worker Coalition, Coworker.org, NYC-DSA Tech Action Working Group, and Science for the people. Among the panelists were Joan Greenbaum, who was involved in organizing tech workers in the mainframe era and was part of Computer People for Peace. Meredith Whittaker is a research scientist at New York University and co-founder of the AI Now Institute, Google Open Research group, and one of the organizers of Google Walkout. Liz Fong-Jones, the Developer Advocate at Google Cloud Platform was also present, who recently tweeted that she will be leaving the company in February, because of Google’s lack of leadership in response to the demands made by employees during the Google walkout in November 2018. Also in the attendance were Emma Quail representing Unite Here and Patricia Rosa, a Facebook food service worker, who was inspired to fight for the union after watching a pregnant friend lose her job because she took one day off for a doctor’s appointment. The discussion was held in New York, hosted by Ben Tarnoff, the co-founder of Logic Magazine. It lasted for almost an hour, after which the Q&A session started. You can see the full discussion at Logic’s Facebook page. The rise of tech workers organizing In recent years, we have seen tech workers coming together to stand against any unjust decision taken by their companies. We saw tech workers at companies like Google, Amazon, and Microsoft raising their voices against contracts, with government agencies like ICE and Pentagon, which are just “profit-oriented” and can prove harmful to humanity. For instance, there was a huge controversy around Google’s Project Maven, which was focused on analyzing drone footage and could have been eventually used to improve drone strikes on the battlefield. More than 3,000 Google employees signed a petition against this project that led to Google deciding not to renew its contract with the U.S. Department of Defense in 2019. In December 2018, Google workers launched an industry-wide effort focusing on the end of forced arbitration, which affects at least 60 million workers in the US alone. In June, Amazon employees demanded Jeff Bezos to stop selling Rekognition, Amazon's facial recognition technology, to law enforcement agencies and to discontinue partnerships with companies that work with U.S. Immigration and Customs Enforcement (ICE). We also saw workers organizing campaigns demanding safer workplaces, free from sexual harassment and gender discrimination, better working conditions, retirement plans, professionalism standards, and fairness in equity compensation. In November, there was a massive Google Walkout with 20,000 Google employees from all over the world to protest against how Google handled sexual harassment cases. This backlash was triggered when it came into light that Google paid millions of dollars as exit packaged to its male executives who were accused of sexual misconduct. Let’s look at some of the highlights from this discussion: What do these issues ranging from controversial contracts, workplace issues, better benefits, a safe equitable workplace have to do with one another? Most companies today are motivated by profits they make, which also shows in the technology they produce. These technologies benefit a small fraction of users while affecting a larger predictable demographic of people, for instance, black and brown people. Meredith Whittaker remarks, “These companies are acting like parallel states right now.” The technologies that they produce have a significant impact over a number of domains that we are not even aware of. Liz Fong-Jones feels that it is also about us as tech workers taking responsibility for what we build. We are feeding into the profit motive these companies have if we keep participating in building systems that can have bad implications for users or not speaking up for the workers working alongside us. To hold these companies accountable and to ensure that all workers are being used for good and people are treated fairly, we all need to come together no matter in what part of the company we are working in. Joan Greenbaum also believes that these types of movement cannot be successful without forming alliances. Any alliance work between tech workers and different roles? Emma Quail shared that there have been many collaborations between engineers, tech employees, cafeteria workers, and other service workers in the fights against companies treating their employees differently. These collaborations are important as tech workers and engineers are much more privileged in these companies. “They have more voice, their job is taken more seriously,” said Emma Quail. Patricia Rosa sharing her experience said, “When some of the tech workers came to one of our negotiations and spoke on our behalf, the company got nervous, and they finally gave them the contract.” Liz Fong-Jones mentions that the main challenge to eliminate this discrimination is that employers want to keep their workers separate. As an example to this, she added, “Google prohibits its cafeteria workers from being on campus when they are not on shift, it prohibits them from holding leadership positions and employee resource groups.” These companies resort to these policies because they do not want their “valuable employees” to find out about the working conditions of other workers. In the last few years, the tech worker movement saw a huge boost in catching the attention of society, but this did not happen overnight. How did we get to this moment? Liz Fong-Jones attributes the Me Too movement as one of the turning points. This movement made workers realize that they are not alone and there are people who share the same concerns. Another thing that Liz Fong-Jones thinks led us to this movement was, management coming with proposals that can have negative implications on people and asked employees to keep secrets. But now tech workers are more informed about what exactly they are building. In the last few years,  tech companies have come under the attention and scrutiny of the public because of the many tech scandals whether it is related to data, software, or workplace, rights. One of the root cause of this was an endless growth requirement. Meredith Whittaker shares, “Over the last few years, we saw series of relentless embarrassing answers to substantially serious questions. They cannot keep going like this.” What’s in the future? Joan Greenbaum rightly mentions that tech companies should actually, “look to work with people what the industry calls users.” They should adopt participatory design instead of user-centered design. Participatory design is basically an approach in which all stakeholders, from employees, partners to local business owners, customers are involved in the design process. Meredith Whittaker remarks, “The people who are getting harmed by these technologies are not the people who are going to get a paycheck from these companies. They are not going to check tech power or tech culture unless we learn how to know each other and form alliances that also connect corporate.” Once we all come together and form alliances, we will be able to pinpoint these companies about the updates and products these companies are building to know about their implications. So, the future basically is in doing our homework, knowing how these companies work, building relationships and coming together against any unjust decisions by these companies. Liz Fong-Jones adds, “The Google Walkout was just the beginning. The labor movement will spread into other companies and also having more visible effects beyond a walkout.” Emma Quail believes that companies will need to address issues related to housing, immigration, rights for people. Patricia Rosa shared that for the future we need to work towards spreading awareness among other workers that there are people who care about their rights and how they are being treated at the workplace. If they are aware that there are people to support them they will not be scared to speak up as Patricia was when she started her journey. Some of the questions asked in the Q&A session were: What's different politically about tech than any other industry? How was the Google walkout organized?  I was a tech contractor and didn't hear about it until it happened. Are there any possibilities of creating a single union of all tech workers no matter what their roles are? Is that a desirable far goal? How tech workers working in one state can relate to the workers working internationally? Watch the full discussion at Logic’s Facebook page. Tech Workers Coalition volunteers talk unionization and solidarity in Silicon Valley Sally Hubbard on why tech monopolies are bad for everyone: Amazon, Google, and Facebook in focus How far will Facebook go to fix what it broke: Democracy, Trust, Reality

We have come a long way from what ecommerce looked like two decades ago. From a non-existent entity, it has grown into a world-devouring business model that is a real threat to the traditional retail industry. It has moved from a basic static web page with limited product listings to a full grown virtual marketplace where anyone can buy or sell anything from anywhere at anytime at the click of a button. At the heart of this transformation are two things: customer experience and technology. This is what Jeff Bezos, founder & CEO of Amazon, one of the world’s largest ecommerce sites believes: “We see our customers as invited guests to a party, and we are the hosts. It's our job every day to make every important aspect of the customer experience a little bit better.” Now with the advent of AI, the retail space especially e-commerce is undergoing another major transformation that will redefine customer experiences and thereby once again change the dynamics of the industry. So, how is AI-powered ecommerce actually changing the way shoppers shop? AI-powered ecommerce makes search easy, accessible and intuitive Looking for something? Type it! Say it!...Searching for a product you can’t name? No worries. Just show a picture. "A lot of the future of search is going to be about pictures instead of keywords." - Ben Silbermann, CEO of Pinterest We take that statement with a pinch of salt. But we are reasonably confident that a lot of product search is going to be non-text based. Though text searches are common, voice and image searches in e-commerce are now gaining traction. AI makes it possible for the customer to move beyond simple text-based product search and search more easily and intuitively through voice and visual product searches. This also makes search more accessible. It uses Natural Language Processing to understand the customer’s natural language, be it in text or speech to provide more relevant search results. Visual product searches are made possible through a combination of computer vision, image recognition, and reverse image search algorithms.   Amazon Echo, a home-automated speaker has a voice assistant Alexa that helps customers to buy products online by having simple conversations with Alexa. Slyce, uses a visual search feature, wherein the customer can scan a barcode, a catalog, and even a real image; just like Amazon’s in-app visual feature. Clarifai helps developers to build applications that detect images and videos and searches related content. AI-powered ecommerce makes personalized product recommendations   When you search for a product, the AI underneath recommends further options based on your search history or depending on what other users who have similar tastes found interesting. Recommendations engines employ one or a combination of the three types of recommendation algorithms: content-based filtering, collaborative filtering, and complementary products. The relevance and accuracy of the results produced depend on various factors such as the type of recommendation engine used, the quantity and quality of data used to train the system, the data storage and retrieval strategies used amongst others. For instance, Amazon uses DSSTNE (Deep Scalable Sparse Tensor Network Engine, pronounced as Destiny) to make customized product recommendations to their customers. The customer data collected and stored is used by DSSTNE to train and generate predictions for customers. The data processing itself takes place on CPU clusters whereas the training and predictions take place on GPUs to ensure speed and scalability. Virtual Assistants as your personal shopping assistants   Now, what if we said you can have all the benefits we have discussed above without having to do a lot of work yourself? In other words, what if you had a personal shopping assistant who knows your preferences, handles all the boring aspects of shopping (searching, comparing prices, going through customers reviews, tracking orders etc.) and brought you products that were just right with the best deals? Mona, one such personal shopper, can do all of the above and more. It uses a combination of artificial intelligence and big data to do this. Virtual assistants can either be fully AI driven or a combination of AI-human collaboration. Chatbots also assist shoppers but within a more limited scope. They can help resolve customer queries with zero downtime and also assist in simple tasks such as notify the customer of price changes, place and track orders etc. Dominos has a facebook messenger Bot that enables customers to order food. Metail, an AI-powered ecommerce website, take in your body measurements. With this, you can actually see how a clothing would look on you. Botpress helps developers to build their own chatbots consuming lesser time. Maximizing CLV (customer lifetime value) with AI-powered CRM AI-powered ecommerce in CRM aims to help businesses predict CLV and sell the right product to the right customer at the right time, every time leveraging the machine learning and predictive capabilities of AI. It also helps businesses provide the right level of customer service and engagement. In other words, by combining the predictive capabilities and automated 1-1 personalization, an AI backed CRM can maximize CLV for every customer!    Salesforce Einstein, IBM Watson are some of the frontrunners in this space. IBM Watson, with its cognitive touch, helps ecommerce sites analyze their mountain of customer data and glean useful insights to predict a lot of things like what customers are looking for, the brands that are popular, and so on.  It can also help with dynamic pricing of products by predicting when to discount and when to increase the price based on analyzing demand and competitions’ pricing tactics. It is clear that AI not only has the potential to transform e-commerce as we know it but that it has already become central to the way leading ecommerce platforms such as Amazon are functioning. Intelligent e-commerce is here and now. The near future of ecommerce is omnicommerce driven by the marriage between AI and robotics to usher in the ultimate customer experience - one that is beyond our current imagination.

MongoDB, founded in 2007 with more than 15 million downloads, excels at supporting real-time analytics for big data applications. Rather than storing data in tables made out of individual rows, MongoDB stores it in collections made out of JSON documents. But, why use MongoDB? How does it work? What issues should you pay attention to? Let’s answer these questions in this post. MongoDB, a NoSQL database MongoDB is a NoSQL database, and NoSQL == Not Only SQL. The data structure is combined with keyvalue, like JSON. The data type is very flexible, but flexibility can be a problem if it’s not defined properly. Here are some good reasons you should use MongoDB: If you are a front-end developer, MongoDB is much easier to learn than mySQL, because the MongoDB base language is JavaScript and JSON. MongoDB works well for big data, because for instance, you can de-normalize and flatten 6 tables into just 2 tables. MongoDB is document-based. So it is good to use if you have a lot of single types of documents. So, now let’s examine how MongoDB works, starting with installing MongoDB: Download MongoDB from https://www.mongodb.com/download-center#community. De-zip your MongoDB file. Create a folder for the database, for example, Data/mydb. Open cmd to the MongoDB path, $ mongod –dbpath ../data/mydb. $ mongo , to make sure that it works. $ show dbs, and you can see two databases: admin and local. If you need to shut down the server, use $db.shutdownServer(). MongoDB basic usage Now that you have MongoDB on your system, let’s examine some basic usage of MongoDB, covering insertion of a document, removal of a document, and how to drop a collection from MongoDB. To insert a document, use the cmd call. Here we use employee as an example to insert a name, an account, and a country. You will see the data shown in JSON: To remove the document: db.collection.remove({ condition }), justOne) justOne: true | false, set to remove the first data, but if you want to remove them all, use db.employee.remove({}). To drop a collection (containing multiple documents) from the database, use: db.collection.drop() For more commands, please look at the MongoDB documentation. What to avoid Let’s examine some points that you should note when using MongoDB: Not easy to change to another database: When you choose MongoDB, it isn’t like other RDBMSes. It can be difficult to change, for example, from MongoDB to Couchbase. No support for ACID: ACID (Atomicity, Consistency, Isolation, Durability) is the basic item of transactions, but most NoSQL databases don’t guarantee ACID, so you need more technical skills in order to do this. No support for JOIN: Since the NoSQL database is non-relational, it does not support JOIN. Document limited: MongoDB uses stock data in documents, and these documents are in JSON format. Because of this, MongoDB has a limited data size, and the latest version supports up to 16 MB per document. Filter search has to correctly define lowercase/uppercase: For example: db.people.find（{name: ‘Russell’}） and db.people.find（{name: ‘ russell’}） are different. You can filter by regex, such as db.people.find（{name:/Russell/i}）, but this will affect performance. I hope this post has provided you with some important points about MongoDB which will help you decide if you have a big data solution that is a good fit for using this NoSQL database. About the author  Tess Hsu is a UI design and front-end programmer. He can be found on GitHub.

As artificial intelligence and big data have become a ubiquitous part of our everyday lives, cloud-based machine learning services are part of a rising billion-dollar industry. Among the several services currently available in the market, Amazon Machine Learning stands out for its simplicity. In this article, we will look at Amazon Machine Learning, MLaaS, and other related concepts. This article is an excerpt taken from the book 'Effective Amazon Machine Learning' written by Alexis Perrier. Machine Learning as a Service Amazon Machine Learning is an online service by Amazon Web Services (AWS) that does supervised learning for predictive analytics. Launched in April 2015 at the AWS Summit, Amazon ML joins a growing list of cloud-based machine learning services, such as Microsoft Azure, Google prediction, IBM Watson, Prediction IO, BigML, and many others. These online machine learning services form an offer commonly referred to as Machine Learning as a Service or MLaaS following a similar denomination pattern of other cloud-based services such as SaaS, PaaS, and IaaS respectively for Software, Platform, or Infrastructure as a Service. Studies show that MLaaS is a potentially big business trend. ABI Research, a business intelligence consultancy, estimates machine learning-based data analytics tools and services revenues to hit nearly $20 billion in 2021 as MLaaS services take off as outlined in this business report  Eugenio Pasqua, a Research Analyst at ABI Research, said the following: "The emergence of the Machine-Learning-as-a-Service (MLaaS) model is good news for the market, as it cuts down the complexity and time required to implement machine learning and thus opens the doors to an increase in its adoption level, especially in the small-to-medium business sector." The increased accessibility is a direct result of using an API-based infrastructure to build machine-learning models instead of developing applications from scratch. Offering efficient predictive analytics models without the need to code, host, and maintain complex code bases lowers the bar and makes ML available to smaller businesses and institutions. Amazon ML takes this democratization approach further than the other actors in the field by significantly simplifying the predictive analytics process and its implementation. This simplification revolves around four design decisions that are embedded in the platform: A limited set of tasks: binary classification, multi-classification, and regression A single linear algorithm A limited choice of metrics to assess the quality of the prediction A simple set of tuning parameters for the underlying predictive algorithm That somewhat constrained environment is simple enough while addressing most predictive analytics problems relevant to business. It can be leveraged across an array of different industries and use cases. Let's see how! Leveraging full AWS integration The AWS data ecosystem of pipelines, storage, environments, and Artificial Intelligence (AI) is also a strong argument in favor of choosing Amazon ML as a business platform for its predictive analytics needs. Although Amazon ML is simple, the service evolves to greater complexity and more powerful features once it is integrated into a larger structure of AWS data related services. AWS is already a major factor in cloud computing. Here's what an excerpt from The Economist, August  2016 has to say about AWS (http://www.economist.com/news/business/21705849-how-open-source-software-and-cloud-computing-have-set-up-it-industry): AWS shows no sign of slowing its progress towards full dominance of cloud computing's wide skies. It has ten times as much computing capacity as the next 14 cloud providers combined, according to Gartner, a consulting firm. AWS's sales in the past quarter were about three times the size of its closest competitor, Microsoft's Azure. This gives an edge to Amazon ML, as many companies that are using cloud services are likely to be already using AWS. Adding simple and efficient machine learning tools to the product offering mix anticipates the rise of predictive analytics features as a standard component of web services. Seamless integration with other AWS services is a strong argument in favor of using Amazon ML despite its apparent simplicity. The following architecture is a case study taken from an AWS January 2016 white paper titled Big Data Analytics Options on AWS (http://d0.awsstatic.com/whitepapers/Big_Data_Analytics_Options_on_AWS.pdf), showing a potential AWS architecture for sentiment analysis on social media. It shows how Amazon ML can be part of a more complex architecture of AWS services: Comparing performances in Amazon ML services Keeping systems and applications simple is always difficult, but often worth it for the business. Examples abound with overloaded UIs bringing down the user experience, while products with simple, elegant interfaces and minimal features enjoy widespread popularity. The Keep It Simple mantra is even more difficult to adhere to in a context such as predictive analytics where performance is key. This is the challenge Amazon took on with its Amazon ML service. A typical predictive analytics project is a sequence of complex operations: getting the data, cleaning the data, selecting, optimizing and validating a model and finally making predictions. In the scripting approach, data scientists develop codebases using machine learning libraries such as the Python scikit-learn library or R packages to handle all these steps from data gathering to predictions in production. As a developer breaks down the necessary steps into modules for maintainability and testability, Amazon ML breaks down a predictive analytics project into different entities: datasource, model, evaluation, and predictions. It's the simplicity of each of these steps that makes AWS so powerful to implement successful predictive analytics projects. Engineering data versus model variety Having a large choice of algorithms for your predictions is always a good thing, but at the end of the day, domain knowledge and the ability to extract meaningful features from clean data is often what wins the game. Kaggle is a well-known platform for predictive analytics competitions, where the best data scientists across the world compete to make predictions on complex datasets. In these predictive competitions, gaining a few decimals on your prediction score is what makes the difference between earning the prize or being just an extra line on the public leaderboard among thousands of other competitors. One thing Kagglers quickly learn is that choosing and tuning the model is only half the battle. Feature extraction or how to extract relevant predictors from the dataset is often the key to winning the competition. In real life, when working on business-related problems, the quality of the data processing phase and the ability to extract meaningful signal out of raw data is the most important and time-consuming part of building an effective predictive model. It is well known that "data preparation accounts for about 80% of the work of data scientists" (http://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/). Model selection and algorithm optimization remains an important part of the work but is often not the deciding factor when the implementation is concerned. A solid and robust implementation that is easy to maintain and connects to your ecosystem seamlessly is often preferred to an overly complex model developed and coded in-house, especially when the scripted model only produces small gains when compared to a service-based implementation. Amazon's expertise and the gradient descent algorithm Amazon has been using machine learning for the retail side of its business and has built a serious expertise in predictive analytics. This expertise translates into the choice of algorithm powering the Amazon ML service. The Stochastic Gradient Descent (SGD) algorithm is the algorithm powering Amazon ML linear models and is ultimately responsible for the accuracy of the predictions generated by the service. The SGD algorithm is one of the most robust, resilient, and optimized algorithms. It has been used in many diverse environments, from signal processing to deep learning and for a wide variety of problems, since the 1960s with great success. The SGD has also given rise to many highly efficient variants adapted to a wide variety of data contexts. We will come back to this important algorithm in a later chapter; suffice it to say at this point that the SGD algorithm is the Swiss army knife of all possible predictive analytics algorithm. Several benchmarks and tests of the Amazon ML service can be found across the web (Amazon, Google, and Azure: https://blog.onliquid.com/machine-learning-services-2/ and Amazon versus scikit-learn: http://lenguyenthedat.com/minimal-data-science-2-avazu/). Overall results show that the Amazon ML performance is on a par with other MLaaS platforms, but also with scripted solutions based on popular machine learning libraries such as scikit-learn. For a given problem in a specific context and with an available dataset and a particular choice of a scoring metric, it is probably possible to code a predictive model using an adequate library and obtain better performances than the ones obtained with Amazon ML. But what Amazon ML offers is stability, an absence of coding, and a very solid benchmark record, as well as a seamless integration with the Amazon Web Services ecosystem that already powers a large portion of the Internet. Amazon ML service pricing strategy As with other MLaaS providers and AWS services, Amazon ML only charges for what you consume. The cost is broken down into the following: An hourly rate for the computing time used to build predictive models A prediction fee per thousand prediction samples And in the context of real-time (streaming) predictions, a fee based on the memory allocated upfront for the model The computational time increases as a function of the following: The complexity of the model The size of the input data The number of attributes The number and types of transformations applied At the time of writing, these charges are as follows: $0.42 per hour for data analysis and model building fees $0.10 per 1,000 predictions for batch predictions $0.0001 per prediction for real-time predictions $0.001 per hour for each 10 MB of memory provisioned for your model These prices do not include fees related to the data storage (S3, Redshift, or RDS), which are charged separately. During the creation of your model, Amazon ML gives you a cost estimation based on the data source that has been selected. The Amazon ML service is not part of the AWS free tier, a 12-month offer applicable to certain AWS services for free under certain conditions. To summarize, we presented a simple introduction to the Amazon ML service. Amazon ML is built on a solid ground, with a simple yet very efficient algorithm driving its predictions. If you found this post useful, be sure to check out the book  'Effective Amazon Machine Learning' to learn about predictive analytics and other concepts in AWS machine learning. Integrate applications with AWS services: Amazon DynamoDB & Amazon Kinesis [Tutorial] AWS makes Amazon Rekognition, its image recognition AI, available for Asia-Pacific developers AWS Elastic Load Balancing: support added for Redirects and Fixed Responses in Application Load Balancer

On the latest Recode Decode episode, Kara Swisher (co-founder) interviewed DuckDuckGo CEO, Gabriel Weinberg on data tracking and why it’s time for Congress to act now as federal legislation is necessary in the current scenario of constant surveillance. DuckDuckGo is an Internet search engine that emphasizes on protecting searchers' privacy. Its market share in the U.S. is about 1%, as compared to more than 88% share owned by Google. Given below are some of the key highlights of the interview. On how DuckDuckGo is different from Google DuckDuckGo which is a internet privacy company, helps users’ to “escape the creepiness and tracking on the internet”. DuckDuckGo has been an alternative to Google since 11 years. It has about a billion searches a month and is the fourth-largest search engine in the U.S. Weinberg states that “Google and Facebook are the largest traders of trackers”, and claims that his company blocks trackers from hundreds of companies. DuckDuckGo also enables more encryption as they force users to go to the unencrypted version of a website. This prevents Internet Service Providers(ISPs)  from tracking the user. When asked the reason for settling into the ‘search business’, Weinberg replied that being from a tech background (tech policy from MIT), he has always been interested in search. After developing this business, he got many privacy queries. It's then that he realized that, “One, searches are essentially the most private thing on the internet. You just type in all your deepest, darkest secrets and search, right? The second thing is, you don’t need to actually track people to make money on search,” so he realized that this would be a “better user experience, and just made the decision not to track people.” Read More: DuckDuckGo chooses to improve its products without sacrificing user privacy The switch from contextual advertising to behavioral advertising From the time internet started working till mid-2000s, the kind of advertising used is called as contextual advertising. It had a very simple routine, “sites used to sell their own ads, they would put advertising based on the content of the article”. Post mid-2000, the working shifted to behavioral advertising. It includes the “creepy ads, the ones that kind of follow you around the internet.” Weinberg added that when website publishers in the Google Network of content sites used to sell their biggest inventory, banner advertising was done at the top of the page. To explore more money, the bottom of the pages was sold to ad networks, to target the site content and audience. These advertisements are administered, sorted, and maintained by Google, under the name AdSense. This helped Google to get all the behavioral data. So if a user searched for something, Google can follow them around with that search. As these advertisements became more lucrative, publishers ceded most of their page over to this behavioral advertising. There has been “no real regulation in tech” to prevent this. Through these trackers, companies like Google and Facebook and many others get user information and browsing history, including purchase history, location history, browsing history, search history, and even user location. Read More: Ireland’s Data Protection Commission initiates an inquiry into Google’s online Ad Exchange services Read More: Advocacy groups push FTC to fine Facebook and break it up for repeatedly violating the consent order and unfair business practices Weinberg informs that, “when you go to, now, a website that has advertising from one of these networks, there’s a real-time bidding against you, as a person. There’s an auction to sell you an ad based on all this creepy information you didn’t even realize people captured” People do ‘care about privacy’ Weinberg says that “before you knew about it, you were okay with it because you didn’t realize it was so invasive, but after Cambridge Analytica and all the stories about the tracking, that number just keeps going up and up and up.” He also explained about the setting “do not track”, which is available in most of the privacy settings of the browser. He says “People are like, ‘No one ever goes into settings and looks at privacy.’ That’s not true. Literally, tens of millions of Americans have gone into their browser settings and checked this thing. So, people do care!”. Weinberg believes ‘do not track’ is a better mechanism for privacy laws, because once the user makes the setting, no more popups will be allowed i.e., no more sites can track you. He also hopes that the ‘do not track’ mechanism is passed by Congress as it will allow all the people in the country to not being tracked. On challenging Google One main issue faced by DuckDuckGo is that not many people are aware of it. Weinberg says, “There’s 20 percent of people that we think would be interested in switching to DuckDuckGo, but it’s hard to convey all these privacy concepts.” He also claimed that companies like Google are altering people’s searches through ‘filter bubble’. As an example, he added, “when you search, you expect to get the results right? But we found that it varies a lot by location”. Last year, DuckDuckGo had accused Google, that their search personalization contributes to “filter bubbles”. In 2012, DuckDuckGo ran a study showing Google's filter bubble may have significantly influenced the 2012 U.S. Presidential election by inserting tens of millions of more links for Obama than for Romney in the run-up to that election. Read More: DeepMind researchers provide theoretical analysis on recommender system, ‘echo chamber’ and ‘filter bubble effect’ How to prevent online tracking Other than using DuckDuckGo and not using say, any of Google’s internet home devices, Swisher asked Weinberg, what are other ways to protect ourselves from being tracked online. To this, Weinberg says there are plenty of other options available. He suggested, “For Google, there are actually alternatives in every category.” For emails, he suggested ProtonMail, FastMail as options. When asked about Facebook, he admitted that “there aren’t great alternatives to it” and added cheekily, “Just leave it”. He further added that there are a bunch of privacy settings available in the devices themselves. He also mentioned about DuckDuckGo blog spreadprivacy.com which provides advice tips. Also there are things which users can do, like turning off ad tracking in the device or to use an end-to-end encryption. On Facial recognition system Weinberg says “Facial recognition is hard”. A person can wear any minor thing to avoid getting caught on the camera. He admits, “you’re going to need laws” to regulate the use of it and thinks San Francisco started a great trend in banning the technology. Many other points were also discussed by Swisher and Weinberg, which included the Communications Decency Act 230 to control sensitive data on the internet. Weinberg also asserted that there’s a need for a national bill like GDPR in the U.S. There were also questions raised on Amazon’s growing advertisements through Google and Facebook. Weinberg also dismissed the probability of having a DuckDuckGo for YouTube anytime soon. Many users agree with Gabriel Weinberg that we should opt into data tracking and it is time to make ‘Do not track’ the norm. A user on Hacker News commented, “Discounting Internet by axing privacy is a nasty idea. Privacy should be available by default without any added price tags.” Another user added, “In addition to not stalking you across the web, DDG also does not store data on you even when using their products directly. For me that is still cause for my use of DDG.” However, as mentioned by Weinberg, there are still people who do not mind being tracked online. It can be because they are not aware of the big trades that takes place behind a user’s one click. A user on Reddit has given an apt basis for this,  “Privacy matters to people at home, but not online, for some reason. I think because it hasn't been transparent, and isn't as obvious as a person looking in your windows. That slowly seems to be changing as more of these concerns are making the news, more breaches, more scandals. You can argue the internet is "wandering outside", which is true to some degree, but it doesn't feel that way. It feels private, just you and your computer/phone, but it's not. What we experience is not matching up with reality. That is what's dangerous/insidious about the whole thing. People should be able to choose when to make themselves "public", and you largely can't because it's complicated and obfuscated.” For more details about their conversation, check out the full interview. Speech2Face: A neural network that “imagines” faces from hearing voices. Is it too soon to worry about ethnic profiling? ‘Facial Recognition technology is faulty, racist, biased, abusive to civil rights; act now to restrict misuse’ say experts to House Oversight and Reform Committee GDPR complaint in EU claim billions of personal data leaked via online advertising bids