Tech News - Data

Facebook AI team announced a new tool, called Aroma, last week. Aroma is a code-to-code search and recommendation tool that makes use of machine learning (ML) to simplify the process of gaining insights from big codebases. Aroma allows engineers to find common coding patterns easily by making a search query without any need to manually browse through code snippets. This, in turn, helps save time in their development workflow. So, in case a developer has written code but wants to see how others have implemented the same code, he can run the search query to find similar code in related projects. After the search query is run, results for codes are returned as code ‘recommendations’. Each code recommendation is built from a cluster of similar code snippets that are found in the repository. Aroma is a more advanced tool in comparison to the other traditional code search tools. For instance, Aroma performs the search on syntax trees. Instead of looking for string-level or token-level matches, Aroma can find instances that are syntactically similar to the query code. It can then further highlight the matching code by cutting down the unrelated syntax structures. Aroma is very fast and creates recommendations within seconds for large codebases. Moreover, Aroma’s core algorithm is language-agnostic and can be deployed across codebases in Hack, JavaScript, Python, and Java. How does Aroma work? Aroma follows a three-step process to make code recommendations, namely, Feature-based search, re-ranking and clustering, and intersecting. For feature-based search, Aroma indexes the code corpus as a sparse matrix. It parses each method in the corpus and then creates its parse tree. It further extracts a set of structural features from the parse tree of each method. These features capture information about variable usage, method calls, and control structures. Finally, a sparse vector is created for each method according to its features and then the top 1,000 method bodies whose dot products are highest are retrieved as the candidate set for the recommendation. Aroma In the case of re-ranking and clustering, Aroma first reranks the candidate methods by their similarity to the query code snippet. Since the sparse vectors contain only abstract information about what features are present, the dot product score is an underestimate of the actual similarity of a code snippet to the query. To eliminate that, Aroma applies ‘pruning’ on the method syntax trees. This helps to discard the irrelevant parts of a method body and helps retain all the parts best match the query snippet. This is how it reranks the candidate code snippets by their actual similarities to the query. Further ahead, Aroma runs an iterative clustering algorithm to find clusters of code snippets similar to each other and consist of extra statements useful for making code recommendations. In the case of intersecting, a code snippet is taken first as the “base” code and then ‘pruning’ is applied iteratively on it with respect to every other method in the cluster. The remaining code after the pruning process is the code which is common among all methods, making it a code recommendation. “We believe that programming should become a semiautomated task in which humans express higher-level ideas and detailed implementation is done by the computers themselves”, states Facebook AI team. For more information, check out the official Facebook AI blog. How to make machine learning based recommendations using Julia [Tutorial] Facebook AI open-sources PyTorch-BigGraph for faster embeddings in large graphs Facebook AI research and NYU school of medicine announces new open-source AI models and MRI dataset

Yesterday, Deep Learning For Java (DL4J) released their new beta version, DL4J 1.0.0-beta4. The main highlight of this version is the full multi-datatype support for ND4J and DL4J, unlike past releases. The previous version deeplearning4j-1.0.0-beta3 was released last year. This 1.0.0-beta4 version also includes the addition of MKL-DNN support, new attention layers, and many more along with optimizations and bug fixes. What’s new in DL4J 1.0.0-beta4? Full multi-datatype support In past releases, all N-Dimensional arrays in ND4J were limited to a single datatype, set globally. Now, arrays of all datatypes may be used simultaneously. The supported datatypes are Double, Float, Half, Long, Int, Short, Ubyte, Byte, Bool and UTF8. CUDA Support CUDA 10.1 support has been added and CUDA 9.0 support has been dropped. DL4J 1.0.0-beta4 also supports CUDA versions 9.2, 10.0 and 10.1. Mac (OSX) CUDA binaries are no longer provided. However, support for Linux and Windows CUDA, and OSX CPU(x86_64) is still available. Memory Management Changes In DL4J 1.0.0-beta4, the periodic garbage collection is disabled by default; instead, garbage collection (GC) will be called only when it is required to reclaim memory from arrays that are allocated outside of workspaces. Deeplearning4J: Bug Fixes and Optimizations cuDNN helpers will no longer attempt to fall back on built-in layer implementations if an out-of-memory exception is thrown. Batch normalization global variance reparameterized to avoid underflow and zero/negative variance in some cases during distributed training. A bug where dropout instances were incorrectly shared between layers when using transfer learning with dropout has been fixed. An issue where tensor Along Dimension could result in an incorrect array order for edge cases and hence exceptions in LSTMs has been fixed. ND4J and SameDiff Features and Enhancements Removed reliance on periodic garbage collection calls for handling memory management of out-of-workspace (detached) INDArrays. New additions - TensorFlowImportValidator tool, INDArray.close() method, Nd4j.createFromNpzFile method, support for importing BERT models into SameDiff, SameDiff GraphTransformUtil, etc have been added. Evaluation, RegressionEvaluation etc. now support 4d (CNN segmentation) data formats. Bug Fixes and Optimizations The bug with InvertMatrix.invert() with [1,1] shape matrices has been fixed. Edge case bug for Updater instances with length 1 state arrays has been fixed. In SameDiff, ‘gradients’ are now no longer defined for non-floating-point variables, and variables that aren’t required to calculate loss or parameter gradients, thus the gradient calculation performance has improved. To know more about the release, check the detailed release notes. Deeplearning4j 1.0.0-alpha arrives! 7 things Java programmers need to watch for in 2019 Deep Learning Indaba presents the state of Natural Language Processing in 2018

[box type="note" align="aligncenter" class="" width=""]The following is an excerpt from the book Scala and Spark for Big Data Analytics, Chapter 9, Stream me up, Scotty - Spark Streaming written by Md. Rezaul Karim and Sridhar Alla. It explores the big three stream processing paradigms that are in use today. [/box] In today's world of interconnected devices and services, it is hard to spend even a few hours a day without our smartphone to check Facebook, or hail an Uber ride, or tweet something about the burger we just bought, or check the latest news or sports updates on our favorite team. We depend on our phones and Internet, for a lot of things, whether it is to get work done, or just browse, or e-mail a friend. There is simply no way around this phenomenon, and the number and variety of applications and services will only grow over time. As a result, the smart devices are everywhere, and they generate a lot of data all the time. This phenomenon, also broadly referred to as the Internet of Things, has changed the dynamics of data processing forever. Whenever you use any of the services or apps on your iPhone, or Droid or Windows phone, in some shape or form, real-time data processing is at work. Since so much depends on the quality and value of the apps, there is a lot of emphasis on how the various startups and established companies are tackling the complex challenges of SLAs (Service Level Agreements), and usefulness and also the timeliness of the data. One of the paradigms being researched and adopted by organisations and service providers is the building of very scalable, near real-time or real-time processing frameworks on  cutting-edge platforms or infrastructure. Everything must be fast and also reactive to changes and failures. You won’t like it if your Facebook updated once every hour or if you received email only once a day; so, it is imperative that data flow, processing, and the usage are all as close to real time as possible. Many of the systems we are interested in monitoring or implementing, generate a lot of data as an indefinite continuous stream of events. As in any data processing system, we have the same fundamental challenges of data collection, storage, and data processing. However, the additional complexity is due to the real-time needs of the platform. In order to collect such indefinite streams of events and then subsequently process all such events to generate actionable insights, we need to use highly scalable specialized architectures to deal with tremendous rates of events. As such, many systems have been built over the decades starting from AMQ, RabbitMQ, Storm, Kafka, Spark, Flink, Gearpump, Apex, and so on. Modern systems built to deal with such large amounts of streaming data come with very flexible and scalable technologies that are not only very efficient but also help realize the business goals much better than before. Using such technologies, it is possible to consume data from a variety of data sources and then use it in a variety of use cases almost immediately or at a later time as needed. Let us talk about what happens when you book an Uber ride on your smartphone to go to the airport. With a few touches on the smartphone screen, you're able to select a point, choose the credit card, make the payment, and book the ride. Once you're done with your transaction, you then get to monitor the progress of your car real-time on a map on your phone. As the car is making its way toward you, you're able to monitor exactly where the car is and you can also make a decision to pick up coffee at the local Starbucks while you're waiting for the car to pick you up. You could also make informed decisions regarding the car and the subsequent trip to the airport by looking at the expected time of arrival of the car. If it looks like the car is going to take quite a bit of time picking you up, and if this poses a risk to the flight you are about to catch, you could cancel the ride and hop in a taxi that just happens to be nearby. Alternatively, if it so happens that the traffic situation is not going to let you reach the airport on time, thus posing a risk to the flight you are due to catch, you also get to make a decision regarding rescheduling or canceling your flight. Now in order to understand how such real-time streaming architectures such as Uber’s Apollo work to provide such invaluable information, we need to understand the basic tenets of streaming architectures. On the one hand, it is very important for a real-time streaming architecture to be able to consume extreme amounts of data at very high rates while, on the other hand, also ensuring reasonable guarantees that the data that is getting ingested is also processed. The following diagram shows a generic stream processing system with a producer putting events into a messaging system while a consumer is reading from the messaging system. Processing of real-time streaming data can be categorized into the following three essential paradigms: At least once processing At most once processing Exactly once processing Let's look at what these three stream processing paradigms mean to our business use cases. While exactly once processing of real-time events is the ultimate nirvana for us, it is very difficult to always achieve this goal in different scenarios. We have to compromise on the property of exactly once processing in cases where the benefit of such a guarantee is outweighed by the complexity of the implementation. Stream Processing Paradigm 1: At least once processing The at least once processing paradigm involves a mechanism to save the position of the last event received only after the event is actually processed and results persisted somewhere so that, if there is a failure and the consumer restarts, the consumer will read the old events again and process them. However, since there is no guarantee that the received events were not processed at all or partially processed, this causes a potential duplication of events as they are fetched again. This results in the behavior that events get processed at least once. At least once is ideally suitable for any application that involves updating some instantaneous ticker or gauge to show current values. Any cumulative sum, counter, or dependency on the accuracy of aggregations (sum, groupBy, and so on) does not fit the use case for such processing simply because duplicate events will cause incorrect results. The sequence of operations for the consumer are as follows: Save results Save offsets Below is an illustration of what happens if there is a failure and consumer restarts. Since the events have already been processed but the offsets have not been saved, the consumer will read from the previous offsets saved, thus causing duplicates. Event 0 is processed twice in the following figure: Stream Processing Paradigm 2: At most once processing The at-most-once processing paradigm involves a mechanism to save the position of the last event received before the event is actually processed and results persisted somewhere so that, if there is a failure and the consumer restarts, the consumer will not try to read the old events again. However, since there is no guarantee that the received events were all processed, this causes potential loss of events as they are never fetched again. This results in the behavior that the events are processed at most once or not processed at all. At most once is ideally suitable for any application that involves updating some instantaneous ticker or gauge to show current values, as well as any cumulative sum, counter, or other aggregation, provided accuracy is not mandatory or the application needs absolutely all events. Any events lost will cause incorrect results or missing results. The sequence of operations for the consumer are as follows: Save offsets Save results Below is an illustration of what happens if there are a failure and the consumer restarts. Since the events have not been processed but offsets are saved, the consumer will read from the saved offsets, causing a gap in events consumed. Event 0 is never processed in the following figure: Stream Processing Paradigm 3: Exactly once processing The Exactly once processing paradigm is similar to the at least once paradigm, and involves a mechanism to save the position of the last event received only after the event has actually been processed and the results persisted somewhere so that, if there is a failure and the consumer restarts, the consumer will read the old events again and process them. However, since there is no guarantee that the received events were not processed at all or were partially processed, this causes a potential duplication of events as they are fetched again. However, unlike the at least once paradigm, the duplicate events are not processed and are dropped, thus resulting in the exactly once paradigm. Exactly once processing paradigm is suitable for any application that involves accurate counters, aggregations, or which, in general, needs every event processed only once and also definitely once (without loss). The sequence of operations for the consumer are as follows: Save results Save offsets The following is illustration shows what happens if there are a failure and the consumer restarts. Since the events have already been processed but offsets have not saved, the consumer will read from the previous offsets saved, thus causing duplicates. Event 0 is processed only once in the following figure because the consumer drops the duplicate event 0: How does the exactly once paradigm drop duplicates? There are two techniques which can help here: Idempotent updates Transactional updates Idempotent updates involve saving results based on some unique ID/key generated so that, if there is a duplicate, the generated unique ID/key will already be in the results (for instance, a database) so that the consumer can drop the duplicate without updating the results. This is complicated as it's not always possible or easy to generate unique keys. It also requires additional processing on the consumer end. Another point is that the database can be separate for results and offsets. Transactional updates save results in batches that have a transaction beginning and a transaction commit phase within so that, when the commit occurs, we know that the events were processed successfully. Hence, when duplicate events are received, they can be dropped without updating results. This technique is even more complicated than the idempotent updates as now we need some transactional data store. Another point is that the database must be the same for results and offsets. You should look into the use case you're trying to build and see if ‘at least once processing’, or ‘at most once processing’, can be reasonably wide and still achieve an acceptable level of performance and accuracy. If you enjoyed this excerpt, be sure to check out the book Scala and Spark for Big Data Analytics it appears in. You will also like this exclusive interview on why Spark is ideal for stream processing with Romeo Kienzler, Chief Data Scientist in the IBM Watson IoT worldwide team and author of Mastering Apache Spark, 2nd Edition.

A pair of researchers, Ruohan Gao, University of Texas and Kristen Grauman, Facebook AI research came out with a method, earlier this month, that can teach an AI system the conversion of ordinary mono sounds into binaural sounds. The researchers have termed this concept as “2.5D visual sound” and it uses a video to generate synthetic 3D sounds. Background According to the researchers, binaural audio provides a listener with the 3D sound sensation that allows a rich experience of the scene. However, these recordings are not easily available and require expertise and equipment to obtain.  Researchers state that humans generally determine the direction of a sound with the help of visual cues. So, they have used a similar technique, where a machine learning system is provided with a video involving a scene and mono sound recording. Using this video, the ML system then figures out the direction of the sounds and further distorts the “interaural time and level differences” to generate the effect of a 3D sound for the listener. Researchers mention that they have devised a deep convolutional neural network which is capable of learning how to decode the monaural (single-channel) soundtrack into its binaural counterpart. Visual information about object and scene information is injected within the CNN during the whole process. “We call the resulting output 2.5D visual sound—the visual stream helps “lift” the flat single channel audio into spatialized sound. In addition to sound generation, we show the self-supervised representation learned by our network benefits audio-visual source separation”, say researchers. Training method used For the training process, researchers first created a database of examples of the effect that it wants the machine learning system to learn. Grauman and Gao created a database using binaural recordings of over 2,265 musical clips which they had also converted into videos. The researchers mention in the paper, “Our intent was to capture a variety of sound-making objects in a variety of spatial contexts, by assembling different combinations of instruments and people in the room. We post-process the raw data into 10s clips. In the end, our BINAURAL-MUSIC-ROOM dataset consists of 2,265 short clips of musical performances, totaling 6.3 hours”. The equipment used for this project involved a 3Dio Free Space XLR binaural microphone, a GoPro HERO6 Black camera, and a Tascam DR-60D recorder as an audio pre-amplifier. The GoPro camera was mounted on top of the 3Dio binaural microphone to mimic a person seeing and hearing, respectively. The GoPro camera records videos at 30fps with stereo audio. Researchers then used these recordings from the dataset for training a machine-learning algorithm which could recognize the direction of sound from a video of the scene. Once the machine learning system learns this behavior, it is then capable of watching a video and distorting a monaural recording to simulate where the sound is ought to be coming from. Results The video shows the performance results of the research which is quite good. In the video, the results of 2.5D recordings are compared against monaural recording.                                                     2.5D Visual Sound However, it is not capable of generating a complete 3D sound and there certain situations that the algorithm finds difficult to deal with. Other than that, the ML system cannot consider any sound source that is not visible in the video, and the ones that it has not been trained on. Researchers say that this method works best for music videos and they have plans to extend its applications. “Generating binaural audio for off-the-shelf video could potentially close the gap between transporting audio and visual experiences, and will be useful for new applications in VR/AR. As future work, we plan to explore ways to incorporate object localization and motion, and explicitly model scene sounds”, say the researchers. For more information, check out the official research paper. Italian researchers conduct an experiment to prove that quantum communication is possible on a global scale Stanford researchers introduce DeepSolar, a deep learning framework that mapped every solar panel in the US Researchers unveil a new algorithm that allows analyzing high-dimensional data sets more effectively, at NeurIPS conference

GitLab announced yesterday that it's being deployed across multiple clouds via Crossplane, an open source multi-cloud control plane sponsored by Upbound. Yesterday, the Crossplane community also demonstrated the entire process of GitLab deployment across multi-cloud. During early December, last year, GitLab announced it had been chosen as the first complex app to be deployed on Crossplane. Crossplane follows established Kubernetes patterns such as persistent volume claims for supporting a clean separation of concerns between application and infrastructure owners. It also provides a self-service model for managed services entirely within the Kubernetes API. With Crossplanes, real-world application deployments from kubectl are easily accessible with enhanced support for composing external fully-managed services including Redis, PostgreSQL, and object storage. “We’ve been working with GitLab to validate our approach and are proud to unveil the deployment of GitLab to multiple clouds entirely with kubectl using Crossplane, including the use of fully-managed services offered by the respective cloud providers”, the official Crossplane blog mentions. Deployment of GitLab with external managed services using Kubectl Crossplane extends the Kubernetes API by adding resource claims and resource classes to support composability of managed service dependencies in Kubernetes, similar to persistent volume claims and storage classes. Crossplane can be easily added to any existing Kubernetes cluster and neatly layers on top of clusters provisioned by Anthos, EKS, AKS, and OpenShift. Cluster administrators install Crossplane on a Kubernetes cluster, set cloud credentials, and specify which managed services they want to make available for self-service provisioning within the cluster. Policies guide binding to specific managed service offerings configured by the cluster administrator. With this, application owners can consume and compose these managed services on-demand with the familiar Kubernetes patterns, without having to know about the infrastructure details or having to manage credentials. For production deployments, GitLab recommends using external managed services for Redis, PostgreSQL, and object storage. Crossplane supports composability of both out-of-cluster public cloud managed services (GCP, AWS, Azure) and in-cluster managed services like those provided by Rook, a storage orchestrator for in-cluster cloud-native storage including Ceph, Minio, and Cassandra. Bassam Tabbara, CEO of Upbound and maintainer on Crossplane said, “We’re showing a real-world example of the future of multi-cloud today. GitLab is a production application that relies on multiple fully-managed services, so by abstracting these services and integrating them with the declarative Kubernetes API, we are demonstrating the ability to standardize on a single declarative API to manage it all.” To know more about Crossplane in detail and also the steps to deploy GitLab to multiple clouds using Crossplane, head over to Crossplane’s official website. Atlassian Bitbucket, GitHub, and GitLab take collective steps against the Git ransomware attack GitLab 11.10 releases with enhanced operations dashboard, pipelines for merged results and much more! Attackers wiped many GitHub, GitLab, and Bitbucket repos with ‘compromised’ valid credentials leaving behind a ransom note

In a paper, published last week, a member from the US Air Force talks about a model for artificial general intelligence (AGI). The author of the paper, A Model for General Intelligence, is Paul Yaworsky, Information Directorate of the US Air Force Research Laboratory. There have been many efforts in the past to model intelligence in machines, but with little progress in terms of real cognitive intelligence like those of humans. What is artificial general intelligence? Currently, the way AI systems work is not understood completely. Also, AI systems are good at performing narrow tasks but not complex cognitive problems. Artificial general intelligence aims to covers the gap between lower level and higher level work in AI—to try and make sense of the abstract general nature of intelligence. Three basic aspects of artificial intelligence need to be understood to bridge this gap. To realize the general order and the nature of intelligence at a high level. Understand what these realizations mean with respect to the overall intelligence process. Describe these realizations as clearly as possible. The paper proposes a hierarchical model to help capture and exploit the order within intelligence. The underlying order contains patterns of signals that become organized, stored and then activated in space and time. The hierarchical model The paper portrays intelligence as an orderly, organized process using a simple hierarchy as shown: Source: A Model for General Intelligence The real world has order and organization. The human brain understands this and forms an internal model based on that understanding. This model enables learning, which further gives way to decision making, movement, and communication. The flow of input signals and learning within the shown model is bottom-up which is in contrast to the top-down of learned signal representations. The paper says that external order and organization can be modeled internally in the brain in the form of various hierarchies. The hierarchies discussed are temporal, spatial, and general. Impact and concerns When computers continue to improve and cooperation increases between humans and computers, people themselves will become more productive in information processing. A point to remember as the paper states is that computers work for humans. Yaworsky also talks about concerns with AI and it taking over the world. Problems like those are heard today due to sketchy predictions involving intelligence he says. It is difficult to make good scientific predictions in itself but when the predictions have to be done in intelligence it is almost impossible to get them right. This is because a proper understanding of intelligence itself is not good enough to be able to make accurate predictions. Do you buy this explanation or fear the US Air Force working on killer drones that may one day go rampant like in Terminator 2?! Either way, the conclusion is that intelligence involves multiple levels of abstraction. Human intelligence has high processing levels—abstract, general, etc. Majority of the work done in AI currently is at lower levels of abstraction. There is a lot needed for the current AI to become real AI. A high-level hierarchical model for artificial intelligence as explored in the paper addresses this. For more details, you can read the research paper. The ethical dilemmas developers working on Artificial Intelligence products must consider Technical and hidden debts in machine learning – Google engineers’ give their perspective Sex robots, artificial intelligence, and ethics: How desire shapes and is shaped by algorithms

In the annual MongoDB World event that took place last week, MongoDB announced the general availability of the new MongoDB 4.0 release, along with several significant beta features related to the cloud. With 4.0, MongoDB aims to attack two domains at once - the traditional database support as well as the ease of database development on the cloud. Some major features of MongoDB 4.0 There were talks of MongoDB 4.0 including support for ACID transactions earlier in the year, extending to multiple documents. However, this support will be restricted only to a single replica for now, with distributed support across multiple clusters to be incorporated with the 4.2 release that is expected to be out later in the year. The 4.0 release also includes a public beta release for MongoDB Charts, a new in-database visualization feature, which can be used to visualize JSON documents. Some of the other important features include: MongoDB Stitch, a serverless compute environment designed for developers, now generally available A mobile-embedded version of MongoDB that runs on smartphones, tablets and IoT devices are now released as a private beta release in 4.0 A new multi-region capability for MongoDB Atlas, letting you deploy a cloud instance that is distributed across different regions throughout the world A beta version of Kubernetes Operator, which co-ordinates orchestration between Kubernetes and MongoDB Ops Manager is released A key milestone for MongoDB with 4.0 release MongoDB’s 4.0 is a quite significant release. It addresses the needs of modern businesses, who base their processes around cloud and serverless operations. Not just that, going multi-platform with support for mobile devices will be seen as a welcome addition by many users. Incorporating relational capabilities puts MongoDB on par with many popular relational databases as well. The ability to combine the power of business-critical transactional processes while offering the speed and ease of using non-relational databases can prove to be a real game-changer for MongoDB. Read more Why MongoDB is the most popular NoSQL database today 4 must-know levels in MongoDB security Connecting your data to MongoDB using PyMongo and PHP

Yesterday, (on 20th November) Facebook and its subsidiary services- Instagram and Whatsapp- went down for users around UK, Europe, and the US. This is the second time Facebook faced an outage this month.  According to Facebook's site for developers, the outage started around 6 a.m. Eastern time. The outage lasted for 13 hours. 48% Facebook users reported a total blackout, while 35% faced issue with login, and 16% users had issues viewing pictures. In the case of Instagram, 53% of users had problems with their newsfeed, while 33% had issues with login, and 13% with the website. Facebook responded to this outage on Twitter, assuring users that they were working towards resolving the issue. https://twitter.com/facebook/status/1064905103755247621 Facebook's Ads Manager, the tool that lets users create advertisements on its social network, also crashed, just days before businesses use Facebook and Instagram to promote Black Friday sales. This left would-be advertisers unable to create new ad campaigns. One of Facebook’s representatives confirmed the outage to Bloomberg, and stated that people and companies can't create new ads or change their existing campaigns due to the issue. The representative added that advertisements previously launched through the system are still running on Facebook. Many advertisement creators that use Facebook to publish ads were in a confused state once this news was out. https://twitter.com/KayaWhatley/status/1064934308220149760 https://twitter.com/stevekatasi/status/1064977427405901825 Once the service was up and running, Facebook tweeted: https://twitter.com/facebook/status/1065036077486944256 In spite of the official announcement that ‘Facebook was 100 percent up for everyone’, many users complained that the site was not in a completely functional state. Amongst persisting issuers were:  pictures not loading, gifs not loading all the way, broken links in posts, messenger app not functioning properly, pages not loading, accounts not been restored and many more issues. Facebook has not yet reverted back to these complaints. Talking of Facebook issues, Zuckerberg is still on the defensive a report in The New York Times that threw the company under scrutiny for its security breaches from Russian accounts ahead of the 2016 U.S. presidential election and how it deals with controversies it faces.  In an exclusive interview with CNN yesterday, Zuckerberg confirmed that Sheryl Sandberg will continue working for the company and that he won’t be stepping down as chairman. GitHub October 21st outage RCA: How prioritizing ‘data integrity’ launched a series of unfortunate events that led to a day-long outage Why skepticism is important in computer security: Watch James Mickens at USENIX 2018 argue for thinking over blindly shipping code Facebook GEneral Matrix Multiplication (FBGEMM), high performance kernel library, open sourced, to run deep learning models efficiently  

Just weeks after the announcement of Splunk IAI (Industrial Asset Intelligence), Splunk has revealed it will be enhancing machine learning across many of its products. This includes Splunk Enterprise, IT Service Intelligence, and User Behavior Analytics. Clearly, the company are using Spring 2018 as a period to build a solid foundation to future-proof their products. Splunk has also added an 'Experiment Management Interface' to its Machine Learning Toolkit. This is a crucial update that will make tracking machine learning and AI 'experiments' much easier. It means that monitoring a range of issues will become much easier. Splunk's goal here is to ensure a reduction in what it calls "event noise." The machine learning and AI algorithms will help to cut through the amount of data and information at users' disposal. It will allow them to identify the issues that are most business-critical. It's about more than just analytics - it's about the additional dimension that makes prioritization much more straightforward. That's what distinguishes what Splunk are doing compared to competitors. Typically, machine learning in BI software allows users to monitor issues, but doesn't have the capacity to place issues in a wider business context. There are a wide range of applications for this technology. It could be used to identify security issues within a given system, application performance, or even operational management. Tim Tully, CTO, had this to say: "Our latest wave of innovation is intended to arm customers with the tools needed to translate AI into actionable intelligence. While AI and machine learning often seem like unattainable and expensive pipe dreams, Splunk Cloud and Splunk Enterprise now make it easier and more affordable to monitor, analyze and visualize machine data in real time" Of course, while Tully's words contain an element of marketing-speak, made for a press release, it's worth noting that the goal here from Splunk's perspective is all about making AI and machine learning more accessible. Clearly the company knows what their customers want. This suggests, then, that for all the discussion around the machine learning revolution, there are still many businesses that regard machine learning as a considerable challenge.

Researchers at OpenAI have developed a system trained with reinforcement learning algorithms which is dexterous in-hand manipulation. Termed as Dactyl, this system can solve object orientation tasks entirely in a simulation without any human input. After the system’s training phase, it was able to work on a real robot without any fine-tuning. Using humanoid hand systems to manipulate objects has been a long-standing challenge in robotic control. Current techniques remain limited in their ability to manipulate objects in the real world. Although robotic hands have been available for quite some time, they were largely unable to utilize complex end-effectors to perform dexterous manipulation tasks. The Shadow Dexterous Hand, for instance, has been available since 2005 with five fingers and 24 degrees of freedom. However, it did not see large-scale adoption because of the difficulty of controlling such complex systems. Now OpenAI researchers have developed a system that trained control policies allowing a robot hand to perform complex in-hand manipulations. This systems shows unprecedented levels of dexterity and discovers different hand grasp types found in humans, such as the tripod, prismatic, and tip pinch grasps. It is also able to display dynamic behaviors such as finger gaiting, multi-finger coordination, the controlled use of gravity, and application of translational and torsional forces to the object. How does the OpenAI system work? First, they used a large distribution of simulations with randomized parameters to collect data for the control policy and vision-based pose estimator. The control policy receives observed robot states and rewards from the distributed simulations. It then learns to map observations to actions using RNN and reinforcement learning. The vision-based pose estimator renders scenes collected from the distributed simulations. It then learns to predict the pose of the object from images using a CNN, trained from the control policy. The object pose is predicted from 3 camera feeds with the CNN. These cameras measure the robot fingertip locations using a 3D motion capture system and give them to the control policy to produce an action for the robot. OpenAI blog You can place a block in the palm of the Shadow Dexterous hand and the Dactyl can reposition it into different orientations. For example, it can rotate the block to put a new face on top. OpenAI blog According to OpenAI, this project completes a full cycle of AI development that OpenAI has been pursuing for the past two years. “We’ve developed a new learning algorithm, scaled it massively to solve hard simulated tasks, and then applied the resulting system to the real world.” You can read more about Dactyl on OpenAI blog. You can also read the research paper for further analysis. AI beats human again – this time in a team-based strategy game OpenAI charter puts safety, standards, and transparency first Introducing Open AI’s Reptile: The latest scalable meta-learning Algorithm on the block

Amazon has announced its plans of acquiring ‘Eero’, the startup that is focussed on mesh home routers.  Eero makes use of a mesh network to produce wireless routers and extenders that provide better coverage for home Wi-Fi networks and makes it easy to have fast and reliable Wi-Fi all over the house. Eero routers are designed to overcome coverage and dead zone issues encountered through traditional routers. Multiple access points are used to provide coverage to an entire home or apartment with a strong Wi-Fi signal. Amazon says that this deal will “help customers better connect smart home devices.” It will make it easier to set up Alexa-compatible gadgets if Amazon also controls the router technology. Amazon SVP Dave Limp said in a press release that “We are incredibly impressed with the Eero team and how quickly they invented a WiFi solution that makes connected devices just work. We have a shared vision that the smart home experience can get even easier, and we’re committed to continuing innovating on behalf of customers.” While the deal is good news for Amazon investors, many Eero users have expressed their disapproval of the deal. Amazon has faced criticism about how Alexa listens in people’s homes, and can be a threat to user privacy. Existing Eero users have voiced their concerns along the same lines: https://twitter.com/steveriggins/status/1095081742736605184 https://twitter.com/TimSchmitz/status/1095103321407397888 https://twitter.com/DerekWallace/status/1095088112554921984 Eero support has tried to put customers worry to rest with a tweet, saying, “Eero does not track customers’ internet activity and this policy will not change with the acquisition”. Eero is not the first router startup to be acquired by Amazon. Amazon has acquired startups like  Ring and Blink, in recent years, with a vision to launch its own in-house Alexa smart home ecosystem. Details of the deal have yet to be disclosed. Head over to Techcrunch for more insights on this news. “Amazon wants to make all the rules and weaken democracy in NYC”: Brad Lander on Amazon’s HQ2 deal Aurora, a self-driving startup, secures $530 million in funding from Amazon, Sequoia, and T. Rowe Price among others Amazon faces increasing public pressure as HQ2 plans go under the scanner in New York  

On Thursday, the team at Facebook open sourced F14, an algorithm for faster and memory-efficient hash tables. F14 helps the hash tables provide a faster way for maintaining a set of keys or map keys to values, even if the keys are objects, like strings. The team at Facebook aimed at simplifying the process of selecting the right hash table with the help of F14. The algorithm F14 focuses on the 14-way probing hash table within Folly, Facebook’s open source library of C++ components. These F14 hash tables are better in performance as compared to the previous tools the team had. According to Facebook F14 is can be used as default for all sorts of use cases. There are many factors that need to be considered while choosing a C++ hash table. These factors could be about keeping long-lived references or pointers to the entries, the size of the keys, the size of the tables, etc. The team at Facebook suggests if the developers aren’t planning to keep long-lived references to entries then they may start with the option- folly::F14FastMap/Set or, they can opt for folly::F14NodeMap/Set. Challenges with the existing hash table algorithms Usually the hash tables start by computing a numeric hash code for each key and uses that number for indexing into an array. The hash code for a key remains the same but the hash codes for different keys are different. These keys in a hash table are distributed randomly across the slots in the array. So there are chances of collisions between the keys that map to the same array. Using Chaining method Most of the hash table algorithms handle these collisions by using the chaining method. The chaining method uses a secondary data structure such as a linked list for storing all the keys for a slot. This helps in storing the keys directly in the main array and further checking new slots if there is a collision. If the number of keys are divided by the size of the main array, we get a number called the load factor which the measure of the hash table’s fullness. By decreasing the load factor and making the main array larger, it’s possible to reduce the number of collisions. The problem with this method is that it will waste memory. Using STL method The next method is using the standard template library (STL) for C++ that provides hash tables via std::unordered_map and std::unordered_set. The method does guarantee reference stability which means the references and pointers to the keys and values in the hash table remains valid until the corresponding key is removed. Thus the entries must be indirect and individually allocated but that would add a substantial CPU overhead. Folly comes with a fast C++ class without reference stability and a slower C++ class that allocates each entry in a separate node. According to Facebook, the node-based version is not fully standard compliant, but it is still compatible with the standard version for all the codes. F14 reduces collisions with vector instructions F14 provide practical improvements for both performance and memory by using the vector instructions available on modern CPUs. F14 uses the hash code for mapping the keys to a block of slots instead of to a single slot and then searches within the chunk in parallel. For this intra-chunk search, F14 uses vector instructions (SSE2 or NEON) that filters all the slots of the chunk at the same time. The team at Facebook has named this algorithm as F14 because it filters 14 slots at once. This F14  algorithm performs collision resolution in case a chunk overflows or if two keys both pass the filtering step. With F14 there is a low probability of collision taking place within instruction pipelining. The team used Chunking strategy for lowering the collision rate. To explain this better, the chance that 15 of the table’s keys would map to a chunk with 14 slots is quite lower than the chance that two keys would map to one slot. For instance, imagine you are in a room with 180 people. The chance that one other person has the same birthday as you is about 50 percent, but the chance that there are 14 people who were born in the same fortnight as you is much lower than 1 percent. Chunking keeps the collision rate low even for load factors above 80 percent. Even if there were 300 people in the room, the chance of a fortnight “overflow” is still less than 5 percent. F14 uses reference-counted tombstones for empty slots Most of the strategies for reducing the collisions keep looking for an empty slot until they find one but that is a little difficult to execute. In this case, the algorithm should either leave a tombstone, an empty slot that doesn’t terminate the probe search or it has to slide down the later keys in the probe sequence. This is again very complex and difficult to execute. In workloads that mix insert and erase, the tombstones can get accumulated. And accumulated tombstones increase the load factor from the performance perspective. F14 uses a strategy that acts similar to reference-counted tombstones. This strategy is based on an auxiliary bit for each slot suggested by Amble and Knuth in their 1974 article “Ordered hash tables.” A bit is set whenever their insertion routine passes a slot that is already full and the bit records that a slot has overflowed. A tombstone corresponds to an empty slot with the overflow bit set. The overflow bit makes searches faster as the search for a key can be stopped at a full slot where the overflow bit is clear, even if the following slot is not empty. The team at Facebook has worked towards having the count the number of active overflows. The overflow bits are set when a displaced key is inserted rather than when the key that did the displacing is removed. This makes it easy to keep track of the number of keys relying on an overflow bit. Each of the F14 chunks uses 1 byte of metadata for counting the number of keys that wanted to be placed in the chunk but are currently stored in a different chunk. And when a key gets erased, it decrements the overflow counter on all the chunks that are on its probe sequence by cleaning them up. F14 optimizes memory effectively It is important to reduce memory waste for improving performance and allowing more of a program’s data to fit in the cache. The two common strategies used for hash table memory layouts are indirect which uses a pointer stored in the main array and direct which uses a memory of the keys and values incorporated directly into the main hash array. F14 uses pointer-indirect (F14Node) and direct storage (F14Value) versions, and index-indirect (F14Vector). The Facebook team uses STL container std::unordered_set that never wastes any data space as it waits until the last moment to allocate nodes. The F14NodeSet executes a separate memory allocation for every value, like std::unordered_set. It also stores pointers to the values in chunks and uses F14’s probing collision resolution to ensure that there are no chaining pointers and no per-insert metadata. The  F14ValueSet stores the values inline and has lesser data waste due to using a higher maximum load factor. F14ValueSet hence achieves memory-efficiency easily. While testing, the team encountered unit test failures and production crashes. Though the team has randomized the code for debug builds. F14 now randomly chooses among all the empty slots in a chunk when inserting a new key. Also, the entry order isn’t completely randomized and according to the team, the shuffle is good enough to catch a regressing test with only a few runs. To know more about this news, check out Facebook’s post. New York AG opens investigation against Facebook as Canada decides to take Facebook to Federal Court for repeated user privacy violations Facebook shareholders back a proposal to oust Mark Zuckerberg as the board’s chairperson Facebook sets aside $5 billion in anticipation of an FTC penalty for its “user data practices”

Spotify announced earlier this week that it has filed an antitrust complaint against Apple with the European Commission (EC), claiming that Apple’s rules on the App Store ‘limits choice’ and ‘ stifle innovation’ at the expense of user experience. Spotify states that Apple receives a 30% tax on purchases made through Apple’s payment system from Spotify and other similar digital services. “If we pay this tax, it would force us to artificially inflate the price of our Premium membership well above the price of Apple Music. And to keep our price competitive for our customers, that isn’t something we can do”, states Daniel Ek, CEO, Spotify in a blog post. Ek mentions that in case Spotify decides not to use Apple’s payment system, then a series of technical and experience limited restrictions are applied by Apple. For instance, Customer communication on the app is restricted including its outreach efforts beyond the app. Also, they are not able to send emails to Apple customers in certain cases. Apart from that, Apple also blocks the experience enhancing upgrades on the app on a routine basis. Spotify is not the only one to have stood up against Apple tax. Many companies have chastised and spoken against Apple tax. For instance, companies like Netflix Inc. and video game makers Epic Games Inc. and Valve Corp also complained about the cost of Tax that Apple charges, last year. However, Spotify is the first company to file a complaint against Apple tax that is registered with the EU, a regulatory body that ensures fair competition across the market. Ek states that Spotify is not looking for any special treatment and wants Apple to treat them like other apps on the App Store, including  Uber or Deliveroo, who are neither imposed any Apple tax nor do these apps have the same restrictions as Spotify. Ek also mentioned a list of requests to bring about a change in rules laid out by Apple for the app store. These are as follows: Apps should compete fairly on the merits, and not based on who owns the App Store. All apps should be subject to the same fair set of restrictions (including Apple music). Consumers should be offered a real choice of payment systems. They should not be forced to use systems that have discriminatory tariffs associated with them like Apple’s. App stores should not dictate communications between different app services and users, by placing unfair restrictions on marketing. Spotify has also launched a separate ‘Time to Play Fair microsite’ that is dedicated to making users aware of Apple’s ‘anti-competitive behavior’.  The site illustrates the complaints by Spotify extensively. Horacio Gutierrez, General Counsel, and VP, Business & Legal Affairs at Spotify, confirmed that Spotify has submitted an “economic analysis” to the European Commission, that demonstrates how Apple’s policies have impacted its business. “We are confident that the evidence will show that even though we’ve been successful as a company, and have grown our business, we could have been even more successful if it were not for the restrictions that Apple has placed on our business,” said Guiterrez. Apple’s response to the complaint Apple has responded back to Spotify, addressing the claims of the complaint on a Press release published, yesterday. Apple states that Spotify wants to leverage all the benefits of the App Store without making any Contributions to Marketplace, and it wouldn’t be the business that it is today without the App Store. “We share Spotify’s love of music and their vision of sharing it with the world. Where we differ is how you achieve that goal. Underneath the rhetoric, Spotify’s aim is to make more money off others’ work”, states Apple. It also states that the only contribution that it requires is for the digital goods and services that are purchased inside the app with their secure in-app purchase system. Apple agrees that the revenue share is 30 percent but it further states that this is only applicable for the first year of an annual subscription and then drops to 15 percent in the years after. “The only time we have requested adjustments is when Spotify has tried to sidestep the same rules that every other app follows. Just this week, Spotify sued music creators after a decision.. required Spotify to increase its royalty payments. This isn’t just wrong, it represents a real, meaningful and damaging step backward for the music industry”, says Apple. Public reaction the news is varied with some sympathizing with Spotify, while others tweeting against Spotify and in support of Apple: https://twitter.com/cassiusdarrow/status/1106504786340253696 https://twitter.com/AvramNate/status/1106042615592570880 https://twitter.com/TonyXLR2/status/1105814168664506368 https://twitter.com/IsleJoseph/status/1105819045218144257 Rene Ritchie, a Canadian blogger, Youtuber, and editor in chief of iMore, also posted a video on his YouTube channel, where he took a deep dive into the Spotify action against Apple in the EU, and called Spotify’s move as “kinda victimy”. Spotify acquires Gimlet and Anchor to expand its podcast services Spotify releases Chartify, a new data visualization library in python for easier chart creation Spotify has “one of the most intricate uses of JavaScript in the world,” says former engineer

Yesterday, Google introduced a new Tensorflow-based framework named Dopamine, which aims to provide flexibility, stability, and reproducibility for both new and experienced RL researchers. This release also includes a set of colabs that clarify how to use the Dopamine framework. Dopamine is inspired by one of the main components in reward-motivated behavior in the brain. It also reflects a strong historical connection between neuroscience and reinforcement learning research. Its main aim is to enable a speculative research that drives radical discoveries. Dopamine framework feature highlights Ease of Use The two key considerations in Dopamine’s design are its clarity and simplicity. Its code is compact (about 15 Python files) and is well-documented. This is achieved by focusing on the Arcade Learning Environment (a mature, well-understood benchmark), and four value-based agents: DQN, C51, A carefully curated simplified variant of the Rainbow agent, and The Implicit Quantile Network agent, which was presented last month at the International Conference on Machine Learning (ICML). Reproducibility Google has provided the Dopamine code with full test coverage. These tests also serve as an additional form of documentation. Dopamine follows the recommendations given by Machado et al. (2018) on standardizing empirical evaluation with the Arcade Learning Environment. Benchmarking It is important for new researchers to be able to quickly benchmark their ideas against established methods. Following this, Google has provided the full training data of the four provided agents, across the 60 games supported by the Arcade Learning Environment. They have also provided a website where one can quickly visualize the training runs for all provided agents on all 60 games. Given below is a snapshot showcasing the training runs for the 4 agents on Seaquest, one of the Atari 2600 games supported by the Arcade Learning Environment. The x-axis represents iterations, where each iteration is 1 million game frames (4.5 hours of real-time play); the y-axis is the average score obtained per play. The shaded areas show confidence intervals from 5 independent runs. The Google community aims to empower researchers to try out new ideas, both incremental and radical with Dopamine ’s flexibility and ease-of-use. It is actively being used in Google’s research, giving them the flexibility to iterate quickly over many ideas. To know more about Dopamine in detail visit the Google AI blog. You can also check out its GitHubrepo. Build your first Reinforcement learning agent in Keras [Tutorial] Reinforcement learning model optimizes brain cancer treatment, reduces dosing cycles and improves patient quality of life OpenAI builds a reinforcement learning based system giving robots hhuman-likedexterity

Be it a tweet about taking the company private or smoking weed on a radio show, Elon Musk has been in news for all the wrong reasons recently and he is in news again but this time for what he is best admired as a modern day visionary. As per reports the Tesla and SpaceX founder is working on a 'superhuman' product that will connect your brain to a computer. We all know Musk along with eight others founded a company called Neuralink two years ago. The company has been developing implantable brain–computer interfaces, better known as BCIs. While in the short-term the company’s aim is to use the technology to treat brain diseases, Musk’s eventual goal is human enhancement, which he believes will make us more intelligent and powerful than even AI.  According to hints he gave a week ago, Neuralink may soon be close to announcing a product unlike anything we have seen: A brain computer interface. Appearing on the Joe Rogan Experience podcast last week, Musk stated that he’ll soon be announcing a new product – Neuralink – which will connect your brain to a computer, thus making you superhuman. When asked about Neuralink, Musk said "I think we’ll have something interesting to announce in a few months that’s better than anyone thinks is possible. Best case scenario, we effectively merge with AI. It will enable anyone who wants to have superhuman cognition. Anyone who wants. How much smarter are you with a phone or computer or without? You’re vastly smarter, actually. You can answer any question pretty much instantly. You can remember flawlessly. Your phone can remember videos [and] pictures perfectly. Your phone is already an extension of you. You’re already a cyborg. Most people don’t realise you’re already a cyborg. It’s just that the data rate, it’s slow, very slow. It’s like a tiny straw of information flow between your biological self and your digital self. We need to make that tiny straw like a giant river, a huge, high-bandwidth interface." If we visualize what Musk said, it feels like a scene straight from a Hollywood movie. However, many of the creations, from a decade ago, that were thought to belong solely in the world of science-fiction, have become a  reality now. Musk argues that through our over-dependence on smartphones, we have already taken the first step towards our cyborg future. Neuralink is an attempt to just accelerate the process by leaps and bounds. That's not all, Elon Musk was also quoted saying on CNBC. "If your biological self dies, you can upload into a new unit. Literally, with our Neuralink technology". Read the full news on CNBC. About Author Sandesh Deshpande is currently working as a System Administrator for Packt Publishing. He is highly interested in Artificial Intelligence and Machine Learning. Tesla is building its own AI hardware for self-driving cars Elon Musk’s tiny submarine is a lesson in how not to solve problems in tech. DeepMind, Elon Musk, and others pledge not to build lethal AI