How-To Tutorials

Last week, Capital One revealed it was subject to a major data breach due to a configuration vulnerability in its firewall to access its Amazon S3 database, affecting 106 million users in the US and Canada. A week after the breach, not only Capital One, but GitHub and Amazon are also facing scrutiny for their inadvertent role in the breach. Capital One and GitHub sued in California Last week, the law firm Tycko & Zavareei LLP filed a lawsuit in California's federal district court on behalf of their plaintiffs Seth Zielicke and Aimee Aballo. Both plaintiffs claim Capital One and GitHub were unable to protect user’s personal data. The complaint highlighted that Paige A. Thompson, the alleged hacker stole the data in March, posted about the theft on GitHub in April. According to the lawsuit, “As a result of GitHub’s failure to monitor, remove, or otherwise recognize and act upon obviously-hacked data that was displayed, disclosed, and used on or by GitHub and its website, the Personal Information sat on GitHub.com for nearly three months.” The law firm also alleged that with the help of computer logs, Capital One should have known about the data breach when the information was first stolen in March. They “criticized Capital One for not taking action to respond to the breach until last month,” The Hill reports. The lawsuit also alleges that GitHub “encourages (at least) friendly hacking." "GitHub had an obligation, under California law, to keep off (or to remove from) its site Social Security numbers and other Personal Information," the lawsuit further mentions. According to Newsweek, GitHub also violated the federal Wiretap Act, "which permits civil recovery for those whose 'wire, oral, or electronic communication' has been 'intercepted, disclosed, or intentionally used' in violation of, inter alia, the Wiretap Act." A GitHub spokesperson told Newsweek, "GitHub promptly investigates content, once it's reported to us, and removes anything that violates our Terms of Service." "The file posted on GitHub in this incident did not contain any Social Security numbers, bank account information, or any other reportedly stolen personal information. We received a request from Capital One to remove content containing information about the methods used to steal the data, which we took down promptly after receiving their request," the spokesperson further added. On 30th July, New York Attorney General, Letitia James also announced that her office is opening an investigation into the Capital One data breach. “My office will begin an immediate investigation into Capital One’s breach, and will work to ensure that New Yorkers who were victims of this breach are provided relief. We cannot allow hacks of this nature to become every day occurrences,” James said in a statement. Many are confused about why a lawsuit was filed against GitHub as they believe that GitHub is not at fault. Tony Webster, a journalist, and a public records researcher tweeted, “I genuinely can't tell if this lawsuit is incompetence or malice. GitHub owed no duty to CapitalOne customers. This would be like suing a burglar's landlord because they didn't detect and stop their tenant from selling your stolen TV from their apartment.” https://twitter.com/rickhholland/status/1157658909563379713 https://twitter.com/NSQE/status/1157479467805057024 https://twitter.com/xxdesmus/status/1157679112699277312 A user on HackerNews writes, “This is incredible: they're suggesting that, in the same way that YouTube has content moderators, GitHub should moderate every repository that has a 9-digit sequence. They also say that GitHub "promotes hacking" without any nuance regarding modern usage of the word, and they claim that GitHub had a "duty" to put processes in place to monitor submitted content, and that by not having such processes they were in violation of their own terms of service. I hope that this gets thrown out. If not, it could have severe consequences for any site hosting user-generated content.” Read the lawsuit to know more about this news in detail. U.S. Senator’s letter to Amazon CEO raises questions on the security of AWS products Yesterday, Senator Ron Wyden wrote to Amazon’s CEO, Jeff Bezos “requesting details about the security of Amazon’s cloud service”, the Wall Street Journal reports. The letter has put forth questions to understand how the configuration error occurs and what measures is Amazon taking to protect its customers. The Journal reported, “more than 800 Amazon users were found vulnerable to a similar configuration error, according to a partial scan of cloud users, conducted in February by a security researcher.” According to the Senator’s letter, “When a major corporation loses data on a hundred million Americans because of a configuration error, attention naturally focuses on that corporation’s cybersecurity practices.” “However, if several organizations all make similar configuration errors, it is time to ask whether the underlying technology needs to be made safer and whether the company that makes it shares responsibility for the breaches,” the letter further mentions. Jeff Bezos has been asked to reply to these questions by August 13, 2019. “Amazon has said that its cloud products weren’t the cause of the breach and that it provides tools to alert customers when data is being improperly accessed,” WSJ reports. Capital One did not comment on this news. Read the complete letter to know more in detail. U.S. Senator introduces a bill that levies jail time and hefty fines for companies violating data breaches Facebook fails to fend off a lawsuit over data breach of nearly 30 million users Equifax breach victims may not even get the promised $125; FTC urges them to opt for 10-year free credit monitoring services

Last week, the Facebook AI research team published a progress report on dialogue research that is fundamentally building more engageable and personalized AI systems. According to the team, “Dialogue research is a crucial component of building the next generation of intelligent agents. While there’s been progress with chatbots in single-domain dialogue, agents today are far from capable of carrying an open-domain conversation across a multitude of topics. Agents that can chat with humans in the way that people talk to each other will be easier and more enjoyable to use in our day-to-day lives — going beyond simple tasks like playing a song or booking an appointment.” In their blog post, they have described new open source data sets, algorithms, and models that improve five common weaknesses of open-domain chatbots today. The weaknesses identified are maintaining consistency, specificity, empathy, knowledgeability, and multimodal understanding. Let us look at each one in detail: Dataset called Dialogue NLI introduced for maintaining consistency Inconsistencies are a common issue for chatbots partly because most models lack explicit long-term memory and semantic understanding. Facebook team in collaboration with their colleagues at NYU, developed a new way of framing consistency of dialogue agents as natural language inference (NLI) and created a new NLI data set called Dialogue NLI, used to improve and evaluate the consistency of dialogue models. The team showcased an example in the Dialogue NLI model, where in they considered two utterances in a dialogue as the premise and hypothesis, respectively. Each pair was labeled to indicate whether the premise entails, contradicts, or is neutral with respect to the hypothesis. Training an NLI model on this data set and using it to rerank the model’s responses to entail previous dialogues — or maintain consistency with them — improved the overall consistency of the dialogue agent. Across these tests they say they saw 3x lesser contradictions in the sentences. Several conversational attributes were studied to balance specificity As per the team, generative dialogue models frequently default to generic, safe responses, like “I don’t know” to some query which needs specific responses. Hence, the Facebook team in collaboration with Stanford’s AI researcher Abigail See, studied how to fix this by controlling several conversational attributes, like the level of specificity. In one experiment, they conditioned a bot on character information and asked “What do you do for a living?” A typical chatbot responds with the generic statement “I’m a construction worker.” With control methods, the chatbots proposed more specific and engaging responses, like “I build antique homes and refurbish houses." In addition to specificity, the team mentioned, “that balancing question-asking and answering and controlling how repetitive our models are make significant differences. The better the overall conversation flow, the more engaging and personable the chatbots and dialogue agents of the future will be.” Chatbot’s ability to display empathy while responding was measured The team worked with researchers from the University of Washington to introduce the first benchmark task of human-written empathetic dialogues centered on specific emotional labels to measure a chatbot’s ability to display empathy. In addition to improving on automatic metrics, the team showed that using this data for both fine-tuning and as retrieval candidates leads to responses that are evaluated by humans as more empathetic, with an average improvement of 0.95 points (on a 1-to-5 scale) across three different retrieval and generative models. The next challenge for the team is that empathy-focused models should perform well in complex dialogue situations, where agents may require balancing empathy with staying on topic or providing information. Wikipedia dataset used to make dialogue models more knowledgeable The research team has improved dialogue models’ capability of demonstrating knowledge by collecting a data set with conversations from Wikipedia, and creating new model architectures that retrieve knowledge, read it, and condition responses on it. This generative model has yielded the most pronounced improvement and it is rated by humans as 26% more engaging than their knowledgeless counterparts. To engage with images, personality based captions were used To engage with humans, agents should not only comprehend dialogue but also understand images. In this research, the team focused on image captioning that is engaging for humans by incorporating personality. They collected a data set of human comments grounded in images, and trained models capable of discussing images with given personalities, which makes the system interesting for humans to talk to. 64% humans preferred these personality-based captions over traditional captions. To build strong models, the team considered both retrieval and generative variants, and leveraged modules from both the vision and language domains. They defined a powerful retrieval architecture, named TransResNet, that works by projecting the image, personality, and caption in the same space using image, personality, and text encoders. The team showed that their system was able to produce captions that are close to matching human performance in terms of engagement and relevance. And annotators preferred their retrieval model’s captions over captions written by people 49.5% of the time. Apart from this, Facebook team has released a new data collection and model evaluation tool, a Messenger-based Chatbot game called Beat the Bot, that allows people to interact directly with bots and other humans in real time, creating rich examples to help train models. To conclude, the Facebook AI team mentions, “Our research has shown that it is possible to train models to improve on some of the most common weaknesses of chatbots today. Over time, we’ll work toward bringing these subtasks together into one unified intelligent agent by narrowing and eventually closing the gap with human performance. In the future, intelligent chatbots will be capable of open-domain dialogue in a way that’s personable, consistent, empathetic, and engaging.” On Hacker News, this research has gained positive and negative reviews. Some of them discuss that if AI will converse like humans, it will do a lot of bad. While other users say that this is an impressive improvement in the field of conversational AI. A user comment reads, “I gotta say, when AI is able to converse like humans, a lot of bad stuff will happen. People are so used to the other conversation partner having self-interest, empathy, being reasonable. When enough bots all have a “swarm” program to move conversations in a particular direction, they will overwhelm any public conversation. Moreover, in individual conversations, you won’t be able to trust anything anyone says or negotiates. Just like playing chess or poker online now. And with deepfakes, you won’t be able to trust audio or video either. The ultimate shock will come when software can render deepfakes in realtime to carry on a conversation, as your friend but not. As a politician who “said crazy stuff” but really didn’t, but it’s in the realm of believability. I would give it about 20 years until it all goes to shit. If you thought fake news was bad, realtime deepfakes and AI conversations with “friends” will be worse.  Scroll Snapping and other cool CSS features come to Firefox 68 Google Chrome to simplify URLs by hiding special-case subdomains Lyft releases an autonomous driving dataset “Level 5” and sponsors research competition

Update: Jim Watkins, the owner of 8chan has spoken against the ongoing backlash in a defensive video statement on uploaded 6th August on YouTube. "My company takes a firm stand in helping law enforcement and within minutes of these two tragedies, we were working with FBI agents to find out what information we could to help in their investigations. There are about 1 million users of 8chan. 8chan is an empty piece of paper for writing on it is disturbing to me that it can be so easily shut down. Over the weekend the domain name service for 8chan was abruptly terminated by the provider Cloudflare.", he states in the video. He adds, "First of all the El Paso shooter posted on Instagram, not 8chan. Later someone uploaded a manifesto; however, that manifesto was not uploaded by the Walmart shooter. It is unfortunate that this place of free speech has temporarily been removed we are working to restore service. It is clearly a political move to remove 8chan from CloudFlare; it has dispersed a peacefully assembled group of people. " Watkins went on to call Cloudflare's decision 'cowardly'. He said, "Contrary to the unfounded claim by Mr. Prince of CloudFlare 8-chan is a lawful community abiding by the laws of the United States and enforced in the Ninth Circuit Court. His accusation has caused me tremendous damage. In the meantime, I wish his company the best and hold no animosity towards him or his cowardly and not thought-out actions against 8-chan." Saturday witnessed two horrific mass shooting tragedies, one when a maniac gunman shot at least 20 people at a sprawling Walmart shopping complex in El Paso, Texas. The other in Dayton, Ohio at the entrance of Ned Peppers Bar where ten people were killed, including the perpetrator, and at least 27 others were injured. The gunman in the El Paso shooting has been identified as Patrick Crusius according to CNN sources. He appears to have been inspired by the online forum known as 8chan. 8chan is an online message board which is home to online extremists who share racist and anti-Semitic conspiracy theories. According to police officials, a four-page document was posted to 8chan, 20 minutes before the shootings that they believe was written by Crusius. The post said, "I'm probably going to die today." His post blamed white nationalists and immigrants for taking away jobs and spewed racist hatred towards immigrants and Hispanics. The El Paso post is not the only incident. 8chan has been filled with unmoderated violent and extremist content over time. Nearly the same thing happened on 8chan before the terror attack in Christchurch, New Zealand. In his post, the El Paso shooter referenced the Christchurch incident saying he was inspired by the Christchurch content on 8chan which glorified the previous massacre. The suspected killer in the synagogue shootings in Poway, California also posted a hate-filled “open letter” on 8chan. In March, this year Australian telecom company Telstra denied access to millions of Australians to the websites 4chan, 8chan, Zero Hedge, and LiveLeak as a reaction to the Christchurch mosque shootings. Cloudflare first defends 8chan citing ‘moral obligations’ but later cuts all ties Post this disclosure, Cloudflare, that provides internet infrastructure services to 8chan continued to defend hosting 8chan calling it their 'moral obligation' to provide 8chan their services. Keeping 8chan within its network is a “moral obligation”, said Cloudflare, adding: “We, as well as all tech companies, have an obligation to think about how we solve real problems of real human suffering and death. What happened in El Paso today is abhorrent in every possible way, and it’s ugly, and I hate that there’s any association between us and that … For us, the question is which is the worse evil? Is the worse evil that we kick the can down the road and don’t take responsibility? Or do we get on the phone with people like you and say we need to own up to the fact that the internet is home to many amazing things and many terrible things and we have an absolute moral obligation to deal with that.” https://twitter.com/slpng_giants/status/1158214314198745088 https://twitter.com/iocat/status/1158218861658791937 Cloudflare has been under the spotlight over the past few years for continuing to work with websites that foster hate. Previous to 8chan, in 2017, Cloudflare had to discontinue services to neo-Nazi blog, The Daily Stormer, after the terror at Charlottevelle. However, Daily Stormer continues to run today having moved to a different infrastructure service with allegedly more readers than ever. After an intense public and media backlash over the weekend, Cloudflare announced that it would completely stop providing support for 8chan. Cloudflare is also readying for an initial public offering in September which may have been the reason why they cut ties with 8chan. In a blog post today, they explained the decision to cut off 8chan. "We just sent notice that we are terminating 8chan as a customer effective at midnight tonight Pacific Time. The rationale is simple: they have proven themselves to be lawless and that lawlessness has caused multiple tragic deaths." Cloudflare has also cut off 8chan's access to its DDOS protection service. Although, this will have a short term impact; 8chan can always come up with another cloud partner and resume operations. Cloudflare acknowledges it as well, “While removing 8chan from our network takes heat off of us, it does nothing to address why hateful sites fester online. It does nothing to address why mass shootings occur. It does nothing to address why portions of the population feel so disenchanted they turn to hate. In taking this action we've solved our own problem, but we haven't solved the Internet’s.” The company added, “We feel incredibly uncomfortable about playing the role of content arbiter and do not plan to exercise it often,” adding that this is not “due to some conception of the United States’ First Amendment,” since Cloudflare is a private company (and most of its customers, and more than half of its revenue, are outside the United States). Instead, Cloudflare “will continue to engage with lawmakers around the world as they set the boundaries of what is acceptable in those countries through due process of law. And we will comply with those boundaries when and where they are set.” Founder of 8chan wants the site to be shut off 8chan founder Fredrick Brennan also appreciated Cloudfare’s decision to block the site. Post the gruesome El Paso shootings, he also told the Washington Post that the site’s owners should “do the world a favor and shut it off.” However, he told Buzzfeed News, shutting down 8chan wouldn't stop the extremism we're now seeing entirely, but it would make it harder for them to organize. https://twitter.com/HW_BEAT_THAT/status/1158194175755485191 In a March interview with The Wall Street Journal, he expressed his regrets over his role in the site’s creation and warned that the violent culture that had taken root on 8chan’s boards could lead to more mass shootings. Brennan founded the site in 2011 and announced his departure from the company in July 2016. 8Chan is owned by Jim Watkins and run by his son, Ron. He posted on Twitter that 8chan will be moving to another service ASAP. He has also resisted calls to moderate or shut down the site. On Sunday, a banner at the top of 8chan’s home page read, “Welcome to 8chan, the Darkest Reaches of the Internet.” https://twitter.com/CodeMonkeyZ/status/1158202303096094720 Cloudflare acted too late, too little Cloudflare's decision to simply block 8chan was not seen as an adequate response by some who say Cloudflare should have acted earlier. 8chan has been known for enabling child pornography in 2015 and as a result, was removed from Google Search. Coupled with the Christchurch mosque and the Poway synagogue shootings earlier in the year, there was increased pressure on those providing 8chan's Internet and financial service infrastructures to terminate their support. https://twitter.com/BinaryVixen899/status/1158216197705359360 Laurie Voss, the cofounder of npmjs, called out Cloudflare and subsequently, other content sites (Facebook, Twitter) for shirking responsibility under the guise of them being infrastructure companies and therefore cannot enforce content standards. https://twitter.com/seldo/status/1158204950595420160 https://twitter.com/seldo/status/1158206331662323712 “Facebook, Twitter, Cloudflare, and others pretend that they can't. They can. They just don't want to.” https://twitter.com/seldo/status/1158206867438522374 “I am super, super tired of companies whose profits rely on providing maximum communication with minimum moderation pretending this is some immutable law and not just the business model they picked,” he tweeted. Others also agreed that Cloudflare’s statement eschews responsibility. https://twitter.com/beccalew/status/1158196518983045121 https://twitter.com/slpng_giants/status/1158214314198745088 Voxility, 8chan’s hardware provider also bans the site Web services company Voxility has also banned 8chan and it’s new host Epik, which had been leasing web space from it. Epik’s website remains accessible, but 8chan now returns an error message. “As soon as we were notified of the content that Epik was hosting, we made the decision to totally ban them,” Voxility business development VP Maria Sirbu told The Verge. Sirbu said it was unlikely that Voxility would work with Epik again. “This is the second situation we’ve had with the reseller and this is not tolerable,” she said. https://twitter.com/alexstamos/status/1158392795687575554 Does de-platforming even work? De-platforming or banning people that spread extremist or banning these people is not a solution since they will eventually migrate to other platforms and still able to circulate their ideology. Closing 8chan is not the solution to the bigger problem of controlling racism and extremism. Closing one 8chan will sprout another 20chan. “8chan is no longer a refuge for extremist hate — it is a window opening onto a much broader landscape of racism, radicalization, and terrorism. Shutting down the site is unlikely to eradicate this new extremist culture because 8chan is anywhere. Pull the plug, it will appear somewhere else, in whatever locale will host it. Because there's nothing particularly special about 8chan, there are no content algorithms, hosting technology immaterial. The only thing radicalizing 8chan users are other 8chan users.”, Ryan Broderick from Buzzfeed wrote. A group of users told BuzzFeed that it’s now common for large 4chan threads to migrate over into Discord servers before the 404. After Cloudflare, Amazon is beginning to face public scrutiny as 8chan’s operator Jim Watkins sells audiobooks on Amazon.com and Audible. https://twitter.com/slpng_giants/status/1158213239697747968 Facebook will ban white nationalism, and separatism content in addition to white supremacy content. 8 tech companies and 18 governments sign the Christchurch Call to curb online extremism; the US backs off. How social media enabled and amplified the Christchurch terrorist attack

This year the JVM Language Summit 2019 was held on July 29th – 31st at Santa Clara, California. On the first day of the Summit, Oracle Java language architect Brian Goetz gave a talk on updates to OpenJDK Project Valhalla. He shared details on its progress, challenges being faced and what to expect from Project Valhalla in the future. He also talked about the significance of Project Valhalla’s LW2 phase which was released earlier last month.  OpenJDK Project Valhalla is now ready for developers to use for early-adopter experimentation in data structures and language runtimes, concluded Goetz. The main goal of OpenJDK Project Valhalla is to reboot the Java Virtual Machine (JVM) relationship with data and memory and in particular to enable denser and flatter layouts of object graphs in memory. The major restriction in the development of object layout has been the object identity. Object identity enables mutability, layout polymorphism and locking among others. As all objects do not need object identity and it would be impractical to determine whether an identity is relevant or not, Goetz expects programmers to inform about the whereabouts of a class such that it will make it easier to make a broader range of assumptions about it. Who cares about Value types? Goetz believes that value types are important for many applications and writers who desire a better control of memory layout and like to use memory very wisely. He says that library writers would always prefer Value types as it allows them to use all the traditional abstracts without paying the runtime cost of taking an extra indirection every time somebody uses a particular abstraction. Thus library classes like optional or cursors or better numerix do not have to pay the object tax.  Similarly, compiler writers of non-Java languages use Value types as an efficient substrate for language features like tuples, multiple return, built-in numeric types and wrapped native resources. Thus both library writers and compiler writers and their users pay the object tax.  Value types, in a nutshell, can help programmers make their code run faster.  Erased and specialized generics Currently, OpenJDK Project Valhalla uses erased generics and will eventually have specialized generics. In an erased generics, Valhalla uses the knowable type convention where the erased list of values can be called as Foo<V?>. This will also be moved to specialized generics later on. He also adds that this syntax cannot be used as of now, as the Valhalla team still does not have existing utterances of Foo for spontaneously changing their meaning. Goetz hopes that the migration of generic classes like Array List<T> to specialized generics would be painless.  New top types Project Valhalla needs new top types RefObject and ValObject for references and values as types are used to indicate a programmer’s intent. It helps the object model reflect the new reality, as everything is an object, but every object does not need an identity. There are many benefits of implementing ref-ness and val-ness into the type system such as: Dynamically ask x instanceof Ref object Statically constrain method parameters or return values Restrict type parameters Natural place to hang ref- or val-specific behavior Nullity Nullity is labelled as one of the most controversial issues in Valhalla. As many values use all their bit patterns, Nullity interferes with a number of useful optimizations. On the other hand, if some types are migrated towards values, the existing code will assume nullability. Nullity is expected to be a focus of the L3 investigation. What to expect next in Project Valhalla Lastly, Goetz announces that developers building data structures or compiler runtime libraries can start using Project Valhalla. He also adds that the Project Valhalla team is working hard to validate the current programming model by working on quantifying the costs of equality, covariance, etc and is trying to better the user control experience.  Goetz concluded by stating that OpenJDK Project Valhalla is at an inflection point and is trying to figure out Nullity, Migration, specialized generics and support for Graal in the future builds. You can watch the full talk of Brian Goetz for more details. Getting started with Z Garbage Collector (ZGC) in Java 11 [Tutorial] Storm 2.0.0 releases with Java enabled architecture, new core and streams API, and more Brian Goetz on Java futures at FOSDEM 2019

Last month, at USENIX ATC 2019, many systems researchers presented their work on topics including real-world deployed systems, runtimes, big-data programming models, security, virtualization, and much more. This year it happened from 10-12 July at Renton, WA, USA. The USENIX Annual Technical Conference (ATC) is considered to be one of the most prestigious systems research conferences. It covers all practical facets of systems software and aims to improve and further the knowledge of computing systems of all scales. Along with providing a platform to showcase cutting-edge systems research, it also allows researchers to gain insight into fields like virtualization, system management and troubleshooting, cloud and edge computing, security, and more. Here are some of the remarkable papers presented at this event: Captive - a retargetable system-level DBT hypervisor By: Tom Spink, Harry Wagstaff, and Björn Franke from the University of Edinburgh Why Captive is needed To boot an operating system and execute programs compiled for an Instruction Set Architecture (ISA) other than the host machine, system-level Dynamic Binary Translation (DBT) is used. DBT is a process of translating code for one ISA to another on the fly. Due to their performance-critical nature, DBT frameworks are generally hardcoded and heavily optimized for both their guest and host ISAs. Though this ensures performance gains, it poses high engineering costs for supporting a new architecture or extending an existing one. How Captive works The researchers have devised a novel, retargetable system-level DBT hypervisor called Captive. It includes guest specific modules generated from high-level guest machine specifications, which simplifies retargeting of the DBT and relieves users from low-level implementation effort. Captive enforces aggressive optimizations by combining the offline optimizations of the architecture model with online optimizations performed within the generated Just-In-Time compiler. It reduces the compilation overheads while providing high code quality. Additionally, it operates in a virtual bare-metal environment provided by a VM hypervisor. This allows you to fully exploit the underlying host architecture, especially the system-related and privileged features not accessible to other DBT systems operating as user processes. Here’s a diagram depicting how it works: Source: Usenix ATC The researchers evaluated the DBT based on both targeted micro-benchmarks and standard application benchmarks. They also compared it with the de-facto standard Qemu DBT system. The evaluation revealed that Captive delivers an average speedup of 2.21x over Qemu across SPEC CPU2006 integer benchmarks. In the case of floating-point applications, it shows further speedup reaching a 6.49x average. It also significantly reduces the effort required to support a new ISA, while delivering outstanding performance. To know more about Captive, check out this USENIX ATC '19 lightning talk by the authors: https://www.youtube.com/watch?v=syhmsBA6OpY SILK - a new open-source key-value store derived from RocksDB, designed to prevent latency spikes By: Oana Balmau, Florin Dinu, and Willy Zwaenepoel, University of Sydney; Karan Gupta and Ravishankar Chandhiramoorthi, Nutanix, Inc.; Diego Didona, IBM Research–Zurich Why SILK is needed Latency-critical applications demand data platforms that can provide low latency and predictable throughput. Log-structured merge key-value stores (LSM KVs) were designed to handle such write-heavy workloads and large scale data where working set does not fit in the main memory. Some of the common LSM KVs are RocksDB, LevelD, and Cassandra that are widely adopted in production environments and claim to optimize the heavy workload. Despite these claims, the researchers show that tail latencies in state-of-the-art LSM KVs can be quite poor, particularly in the case of heavy and variable client write loads. How SILK works To address the aforementioned limitations, the researchers have come up with the notion of an I/O scheduler for LSM KVs, which aims to achieve the following three goals: Opportunistically allocating I/O bandwidth to internal operations Prioritizing internal operations at the lower levels of the tree. Preempting compactions. This notion of I/O scheduler is implemented in SILK, a new open-source KV store derived from RocksDB. It is designed to prevent client request latency spikes. It uses this I/O scheduler to manage external client load and internal LSM maintenance work. It was tested on a production workload and synthetic benchmarks and was able to achieve up to two orders of magnitude lower 99th percentile latencies than RocksDB and TRIAD. To know more about SILK, check out this USENIX ATC '19 lightning talk by the authors: https://www.youtube.com/watch?v=6kPYDFT1q8w Transactuations and its implementation, Relacs for building reliable IoT applications By: Aritra Sengupta, Tanakorn Leesatapornwongsa, and Masoud Saeida Ardekani, Samsung Research; Cesar A. Stuardo, University of Chicago Why transactuations and the Relacs runtime are needed IoT applications are responsible for reading sensors, executing application logic, and taking action with actuators accordingly. One of the challenges developers face while building an IoT application is ensuring its correctness and reliability. Though current solutions do offer simple abstractions for reading and actuating, they lack high-level abstractions for writing reliable and fault-tolerant applications. Not properly handling failures can lead to inconsistencies between the physical and application state. How transactuations and Relacs work In this paper, the researchers introduced “transactuations”, which are similar to database transactions. These abstract the complexity of handling various failures and make it easy to maintain soft states so that they are consistent with respect to reads and writes to hard states. You need to specify dependencies among operations on soft and hard states along with sensing or actuating policy that specifies the conditions under which soft states can commit despite failures. Transactuation will then be responsible for preserving this dependence even in cases of hardware and communication failures and ensure isolation among transactuations that are executing concurrently. The researchers have also introduced Relacs, a runtime system that implements the abstraction for a smart home platform. It first transforms an application into a serverless function and executes the application in the cloud while enforcing transactuation specific semantics. The researchers further showed that transactuations are an effective solution for building reliable IoT applications. Using them also significantly reduces lines of code compared to manually handling failures. The Relacs runtime also guarantees reliable execution of transactuations while imposing reasonable overheads over a baseline that does not provide consistency between operations on hard states and soft states. To know more about transactuations, check out this USENIX ATC '19 lightning talk by the authors: https://www.youtube.com/watch?v=aBUYHWsWrhE Browsix-Wasm - Run unmodified WebAssembly-compiled Unix applications inside the browser in a performant way By: Abhinav Jangda, Bobby Powers, Emery D. Berger, and Arjun Guha, University of Massachusetts Amherst Why Browsix-Wasm is needed Major browsers today including Mozilla, Chrome, Safari, and Edge support WebAssembly, a small binary format that promises to bring near-native performance to the web. It serves as a portable compilation target for high-level languages like C, C++, and Rust. One of the key goals of WebAssembly is performance parity with native code. The paper that introduced WebAssembly showed that its performance is competitive with native code. However, the evaluation was limited to a suite of scientific kernels rather than full applications with each consisting of about 100 lines of code. The researchers conducted a comprehensive performance analysis using the established SPEC CPU benchmark suite of large programs. However, using such suites also pose a challenge, that is, currently it is not possible to simply compile a sophisticated native program to WebAssembly. These need operating system services such as a filesystem, synchronous I/O, and processes, which WebAssembly and the browser do not provide. How Browsix-Wasm works As a solution to this challenge, the researchers have built Browsix-Wasm, an extension to Browsix that allows running unmodified WebAssembly-compiled Unix applications directly inside the browser. They used Browsix-Wasm to conduct the very first large-scale evaluation of the performance of WebAssembly vs. native. The evaluation results show a substantial performance gap across the SPEC CPU suite of benchmarks. The applications compiled to WebAssembly were slower by an average of 45% (Firefox) to 55% (Chrome), with peak slowdowns of 2.08$\times$ (Firefox) and 2.5$\times$ (Chrome). Some of the reasons behind this performance degradation were missing optimizations and code generation issues. Here’s a chart showing the comparison between the performance analysis done on the basis of PolyBenchC (previous work) and SPEC CPU benchmarks. Source: Usenix ATC To know more about Browsix-Wasm, check out this USENIX ATC '19 lightning talk by the authors: https://youtu.be/-6HoiVWiHiw Zanzibar - Google’s global authorization system By: Ruoming Pang, Ramon Caceres, Mike Burrows, Zhifeng Chen, Pratik Dave, Nathan Germer, Alexander Golynski, Kevin Graney, and Nina Kang, Jeffrey L. Korn, Christopher D. Richards and Mengzhi Wang, Google; Lea Kissner, Humu, Inc.; Abhishek Parmar, Carbon, Inc. Why Zanzibar is needed Every day online interactions involve the exchange of a lot of personal information. These interactions are authorized to ensure that a user has permission to perform an operation on a digital object. For instance, we have several web-based photo storage services that allow users to share a few photos with friends while keeping other photos private. These services must have checks in place to ensure that photos are shared before another user can view them. There are already many ways of authorization and developers constantly work on making them more robust to guarantee online privacy. How Zanzibar works The researchers have come up with Zanzibar, a system that allows you to store permissions and perform authorization checks based on the stored permissions. Many Google services use it including Calendar, Cloud, Drive, Maps, Photos, and YouTube. Zanzibar takes up two roles: A storage system for Access Control Lists (ACLs) and groups. An authorization engine that interprets permissions. It provides a uniform data model and language to define a wide range of access control policies. While making authorization decisions it takes into account the causal ordering of user actions to provide external consistency amid changes to access control lists and object contents. Here’s a diagram depicting its architecture: Source: Usenix ATC Zanzibar is scalable to trillions of access control lists and millions of authorization requests per second to support services. In more than 3 years of production use at Google, it has maintained 95th-percentile latency of less than 10 milliseconds and availability of greater than 99.999%. To know more about Zanzibar, check out this USENIX ATC '19 lightning talk by the authors: https://www.youtube.com/watch?v=cJ334qJ0jBI These were some of the papers presented at USENIX ATC 2019. You can find other research papers on its official website. International cybercriminals exploited Citrix internal systems for six months using password spraying technique Google Cloud and Nvidia Tesla set new AI training records with MLPerf benchmark results “Don’t break your users and create a community culture”, says Linus Torvalds, Creator of Linux, at KubeCon + CloudNativeCon + Open Source Summit China 2019  

Every year a new edition of the ECMAScript (ES) scripting-language specification standard comes out. This year it is its tenth edition, also known as ES2019 or ES10. The feature set of ES2019 got finalized earlier this year and was published last month. Some of the exciting features this specification brings are Object.fromEntries(), trimStart(), trimEnd(), flat(), flatMap(), description property for symbol objects, optional catch binding, and more. These features have also landed in the latest versions of Firefox and Chrome for developers to try out. Let’s take a look at some of the features in ES2019: Object.fromEntries() In JavaScript, you can easily convert objects into arrays with the Object.entries() method that was introduced in the ES2017 standard. ES2019 introduces the Object.fromEntries() method that enables you to do exactly the opposite. Similar to the dict() function in Python, this method allows you to transform a list of key-value pairs into an object. Array.prototype.flat() and Array.prototype.flatMap() The method Array.prototype.flatten() was renamed to Array.prototype.flat() method in ES2019 after last year it ended up breaking MooTools' implementation of it. It recursively flattens an array up to the specified depth, which defaults to 1. The second method, ‘Array.prototype.flatMap’ performs the mapping of each element and then flattens the result into a new array. trimStart() and trimEnd() The purpose of the new trimStart() and trimEnd() methods proposed in ES2019 is same as the trimLeft() and trimRight() methods. While trimStart() is used to remove whitespace from the beginning of a string, trimEnd() is used to remove whitespace characters from the end of a string. These are introduced to maintain consistency with the padStart/padEnd the standard functions. To maintain web compatibility trimLeft() and trimRight() will be their aliases. Optional catch binding In JavaScript, it is mandatory to specify the catch() parameter when using try...catch, no matter whether you use it or not. However, there are a few use cases where you wouldn’t want to use the parameter or catch binding. Axel Rauschmayer, the author of JavaScript for impatient programmers (ES1–ES2019), lists the following two: If you want to completely ignore the error. You don’t care about the error or you already know what it will be, but you do want to react to it. This new proposal allows you to completely omit the unused catch binding without any syntax errors. Function.toString() Earlier, when you called the toString() method on a function it used to strip all the whitespaces, newlines, and comments from the source code. Now, it will return the function source code exactly as it was defined. Description property for Symbol objects ES2019 introduces a new read-only ‘description’ property for Symbol objects. You can add it to a Symbol object to return a string containing its description for debugging purposes. Well-formed JSON.stringify() According to a JSON RFC, JSON text when shared “outside the scope of a closed ecosystem” should be encoded using UTF-8. However, JSON.Stringify() can sometimes return strings and code points, particularly, the surrogate range (U+D800—U+DFFF), that cannot be represented in UTF-8. This ES2019 proposal prevents JSON.stringify() from returning such ill-formed Unicode strings. Many developers are excited about these new ES2019 proposals. A user on Hacker News commented, “That array.flat() and array.flatMap() stuff is great to see. Always having to rely on lodash and friends to do that type of work. Exciting to see how JS is evolving.” Another user added, “Object.fromEntries will be super useful, surprised it’s taken this long to become a native feature.” Others are waiting for the pattern matching and optional chaining proposals to reach the stage 4 of TC39 process, “Now if we could just get pattern matching and optional chaining, that would really elevate things.” These were some of the features introduced in ES2019. To know more, check out the specification published on the ECMA International website. Introducing QuickJS, a small and easily embeddable JavaScript engine Firefox 67 will come with faster and reliable JavaScript debugging tools Introducing Node.js 12 with V8 JavaScript engine, improved worker threads, and much more  

Yesterday, two members of the Google Project Zero team revealed about six “interactionless” security bugs that can affect iOS by exploiting the iMessage Client. Four of these bugs can execute malicious code on a remote iOS device, without any prior user interaction. Apple released fixes for these bugs in the iOS 12.4 update on July 22. The two Project Zero researchers, Natalie Silvanovich and Samuel Groß, published details and demo proof-of-concept only for five out of the six vulnerabilities. Details of one of the "interactionless" vulnerabilities have been kept private because Apple's iOS 12.4 patch did not completely resolve the bug, according to Natalie Silvanovich. https://twitter.com/natashenka/status/1155941211275956226 4 bugs can perform an RCE via a malformed message Bugs with vulnerability IDs, CVE-2019-8647, CVE-2019-8660, CVE-2019-8662, CVE-2019-8641 (the one whose details are kept private), can execute malicious code on a remote iOS device. The attacker has to simply send a malformed message to the victim’s phone. Once the user opens the message and views it, the malicious code will automatically execute without the user knowing about it. 2 bugs can leak user’s on-device data to a remote device The other two bugs, CVE-2019-8624 and CVE-2019-8646, allow an attacker to leak data from a user’s device memory and read files off a remote device. This execution too can happen without the user knowing. “Apple's own notes about iOS 12.4 indicate that the unfixed flaw could give hackers a means to crash an app or execute commands of their own on recent iPhones, iPads and iPod Touches if they were able to discover it”, BBC reports. Silvanovich will talk about these remote and interactionless iPhone vulnerabilities at this year’s Black Hat security conference held at Las Vegas from August 3 - 8. An abstract of her talk reads, “There have been rumors of remote vulnerabilities requiring no user interaction being used to attack the iPhone, but limited information is available about the technical aspects of these attacks on modern devices.” Her presentation will explore “the remote, interaction-less attack surface of iOS. It discusses the potential for vulnerabilities in SMS, MMS, Visual Voicemail, iMessage and Mail, and explains how to set up tooling to test these components. It also includes two examples of vulnerabilities discovered using these methods." According to ZDNet, “When sold on the exploit market, vulnerabilities like these can bring a bug hunter well over $1 million, according to a price chart published by Zerodium. It wouldn't be an exaggeration to say that Silvanovich just published details about exploits worth well over $5 million, and most likely valued at around $10 million”. For iOS users who haven’t yet updated the latest version, it is advisable to install the iOS 12.4 release without any delay. Early this month, the Google Project Zero team revealed a bug in Apple’s iMessage that bricks iPhone causing a repetitive crash and respawn operations. This bug was patched in iOS 12.3 update. To know more about these five vulnerabilities in detail, visit the Google Project Zero bug report page. Stripe’s API degradation RCA found unforeseen interaction of database bugs and a config change led to cascading failure across critical services Azure DevOps report: How a bug caused ‘sqlite3 for Python’ to go missing from Linux images Is the Npm 6.9.1 bug a symptom of the organization’s cultural problems?

Update: On 28th August, an indictment was filed in a US federal district court, which mentioned Thompson allegedly hacked and stole information from an additional 30 AWS-hosted organizations and will face computer abuse charges. Capital One Financial Corp., one of the largest banks in the United States, has been subject to a massive data breach affecting 100 million customers in the U.S and an additional 6 million in Canada. Capital One said the hacker exploited a configuration vulnerability in its firewall that allowed access to the data. In its official statement released yesterday, Capital One revealed that on July 19, it determined an "unauthorized access by an outside individual who obtained certain types of personal information relating to people who had applied for its credit card products and to Capital One credit card customers." Paige A. Thompson, 33, the alleged hacker who broke into Capital One server, was arrested yesterday and appeared in federal court in Seattle. She was an ex-employee from Amazon's Cloud service (AWS), Amazon confirms. The Capital One hacker, an ex-AWS employee, “left a trail online for investigators to follow” FBI Special Agent Joel Martini wrote in a criminal complaint filed on Monday that a “GitHub account belonging to Thompson showed that, earlier this year, someone exploited a firewall vulnerability in Capital One’s network that allowed an attacker to execute a series of commands on the bank’s servers”, according to Ars Technica. IP addresses and other evidence ultimately showed that Thompson was the person who exploited the vulnerability and posted the data to Github, Martini said. “Thompson allegedly used a VPN from IPredator and Tor in an attempt to cover her tracks. At the same time, Martini said that much of the evidence tying her to the intrusion came directly from things she posted to social media or put in direct messages”, Ars Technica reports. On  July 17, a tipster wrote to a Capital One security hotline, warning that some of the bank’s data appeared to have been “leaked,” the criminal complaint said. According to The New York Times, Thompson “left a trail online for investigators to follow as she boasted about the hacking, according to court documents in Seattle”. She is listed as the organizer of a group on Meetup, a social network, called Seattle Warez Kiddies, a gathering for “anybody with an appreciation for distributed systems, programming, hacking, cracking.” The F.B.I. noticed her activity on Meetup and used it to trace her other online activities, eventually linking her to posts boasting about the data theft on Twitter and the Slack messaging service.  “I’ve basically strapped myself with a bomb vest, dropping capital ones dox and admitting it,” Thompson posted on Slack, prosecutors say. Highly sensitive financial and social insurance data compromised The stolen data was stored in Amazon S3, "An AWS spokesman confirmed that the company’s cloud had stored the Capital One data that was stolen, and said it wasn’t accessed through a breach or vulnerability in AWS systems", Bloomberg reports. Capital One said the largest category of information accessed was information on consumers and small businesses as of the time they applied for one of its credit card products from 2005 through early 2019. The breached data included personal information Capital One routinely collects at the time it receives credit card applications, including names, addresses, zip codes/postal codes, phone numbers, email addresses, dates of birth, and self-reported income. The hacker also obtained customer status data, e.g., credit scores, credit limits, balances, payment history, contact information including fragments of transaction data from a total of 23 days during 2016, 2017 and 2018. For the Canadian credit card customers, approximately 1 million Social Insurance Numbers were compromised in this incident. About 140,000 Social Security numbers of Capital One's credit card customers and about 80,000 linked bank account numbers of our secured credit card customers were compromised. Richard D. Fairbank, Capital One’s chief executive officer, said in a statement, "I am deeply sorry for what has happened. I sincerely apologize for the understandable worry this incident must be causing those affected.” Thompson is charged with computer fraud and faces a maximum penalty of five years in prison and a $250,000 fine. U.S. Magistrate Judge Mary Alice Theiler ordered Thompson to be held. A bail hearing is set for Aug 1. Capital One said, it “will notify affected individuals through a variety of channels. We will make free credit monitoring and identity protection available to everyone affected”. Capital One's justification of "Facts" is unsatisfactory Users are very skeptical about trusting Capital One with their data going ahead. A user on Hacker News writes, “Obviously this person committed a criminal act, however, Capital One should also shoulder responsibility for not securing customer data. I have a feeling we'd be waiting a long time for accountability on C1's part.” Security experts are surprised with Capital One’s stating of “facts that say “no Social Security numbers were breached’ and say this cannot be true. https://twitter.com/zackwhittaker/status/1156027826912428032 https://twitter.com/DavidAns/status/1156014432511643649 https://twitter.com/GossiTheDog/status/1156232048975273986 Similar to Capital One, there were other data breaches in the past where the companies have agreed on a settlement to help the affected customers like the Equifax or have been levied with huge fines like the Marriott International and British Airways. The Equifax data breach that affected 143 million U.S. consumers on September 7, 2017, resulted in a global settlement including up to $425 million to help people affected by the data breach amounting to approximately $125 per affected victim, should they apply for compensation. This global settlement was done with the Federal Trade Commission, the Consumer Financial Protection Bureau, and 50 U.S. states and territories. The Marriott data breach occurred in Marriott’s Starwood guest database that compromised 383 million user data was revealed on November 19, 2018. Recently, the Information Commissioner’s Office (ICO) in the UK announced its plans to impose a fine of more than £99 million ($124 million) under GDPR. The British Airways data breach compromised personal identification information of over 500,000 customers and is believed to have begun in June 2018. Early this month, the ICO also announced it will fine British Airways with more than £183m fine. As a major data breach in one of the largest banks, Capital One could feel the pinch by regulators soon. What sets this case apart from the above breaches is that the affected customers are from the US and Canada and not from the EU. In the absence of regulatory action by the ICO or the EU commission, it is yet to be seen if regulators in the US and Canada will rise to the challenge. Also, now that the alleged hacker has been arrested, does this mean Capital One could slip by without paying any significant fine? Only time can tell if Capital One will pay a huge sum to the regulators for not being watchful of their customers' data in two different states. If the Equifax-FTC case and the Facebook-FTC proceedings are any sign of things to come, Capital One has not much to be concerned about. To know more about this news in detail, read Capital One’s official announcement. Thompson faces additional charges for hacking into the AWS accounts of about 30 organizations On 28th August, an indictment was filed in a US federal district court, where the investigators mentioned they have identified most of the companies and institutions allegedly hit by Thompson. The prosecutors said Thompson wrote software that scanned for customer accounts hosted by a “cloud computing company,” which is believed to be her former employer, AWS or Amazon Web Services. "It is claimed she specifically looked for accounts that suffered a common security hole – specifically, a particular web application firewall misconfiguration – and exploited this weakness to hack into the AWS accounts of some 30 organizations, and siphon their data to her personal server. She also used the hacked cloud-hosted systems to mine cryptocurrency for herself, it is alleged," The Register reports. “The object of the scheme was to exploit the fact that certain customers of the cloud computing company had misconfigured web application firewalls on the servers that they rented or contracted from the cloud computing company,” the indictment reads. The indictment further reads, “The object was to use that misconfiguration in order to obtain credentials for accounts of those customers that had permission to view and copy data stored by the customers on their cloud computing company servers. The object then was to use those stolen credentials in order to access and copy other data stored by the customers.” Thus, she also faces a computer abuse charge over the 30 other AWS-hosted organizations she allegedly hacked and stole information from. Facebook fails to fend off a lawsuit over a data breach of nearly 30 million users US Customs and Border Protection reveal data breach that exposed thousands of traveler photos and license plate images Over 19 years of ANU(Australian National University) students’ and staff data breached

Last week Go programmers had a gala time learning, networking and programming at the Marriott Marquis San Diego Marina as the most awaited event GopherCon 2019 was held starting from 24th July till 27th July. GopherCon this year hit the road at San Diego with some exceptional conferences, and many exciting announcements for more than 1800 attendees from around the world. One of the attendees, Andrea Santillana Fernández, says the Go Community is growing, and doing quite well. She wrote on her blog post on the Source graph website that there are 1 million Go programmers around the world and month on month its membership keeps increasing. Indeed there is a significant growth in the Go community, so what did it have in store for the programmers at this year’s GopherCon 2019: On the road to Go 2 The major milestones for the journey to Go 2 were presented by Russ Coxx on Wednesday last week. He explained the main areas of focus for Go 2, which are as below: Error handling Russ notes that writing a program correctly without errors is hard. But writing a program correctly accounting for errors and external dependencies is much more difficult. He listed down a few errors which led in introducing error handling helpers like an optional Unwrap interface, errors.Is and errors.As in Go 1.13 version. Generics Russ spoke about Generics and said that they started exploring a new design since last year. They are working with programming language theory experts on the problem to help refine the proposal of generics code in Go. In a separate session, Ian Lance Taylor, introduced generics codes in Go. He briefly explained the need, implementation and benefits from generics for the Go language. Next, Taylor reviewed the Go contract design draft which included the addition of optional type parameters to types and functions. Taylor defined generics as “Generic programming which enables the representation of functions and data structures in a generic form, with types factored out.” Generic code is written using types, which are specified later. An unspecified type is called as type parameter. A type parameter offers support only when permitted by contracts. A generic code imparts strong basis for sharing codes and building programs. It can be compiled using an interface-based approach which optimizes time as the package is compiled only once. If a generic code is compiled multiple times, it can carry compile time cost. Ian showed a few sample codes written in Generics in Go. Dependency management In Go 2 the team wants to focus on Dependency management and explicitly refer to dependencies similar to Java. Russ explained this by giving a history of how in 2011 they introduced GOPATH to separate the distribution from the actual dependencies so that users could run multiple different distributions and to separate the concerns of the distribution from the external libraries. Then in 2015, they introduced the go vendor spec to formalize the vendor directory and simplify dependency management implementations. But in practice it did not work well. In 2016, they formed the dependency working group. This team started work on dep: a tool to reshape all the existing tools into one.The problem with dep and the vendor directory was multiple distinct incompatible versions of a dependency were represented by one import path. It is now called as the "Import Compatibility Rule". The team took what worked well and learned from VGo. VGo provides package uniqueness without breaking builds. VGo dictates different import paths for incompatible package versions. The team grouped similar packages and gave these groups a name: Modules. The VGo system is now go modules. It now integrates directly with the Go command. The challenge presented going forward is mostly around updating everything to use modules. Everything needs to be updated to work with the new conventions to work well. Tooling Finally, as a result of all these changes, they distilled and refined the Go toolchain. One of the examples of this is gopls or "Go Please". Gopls aims to create a smoother, standard interface to integrate with all editors, IDEs, continuous integration and others. Simple, portable and efficient graphical interfaces in Go Elias Naur presented Gio, a new open source Go library for writing immediate mode GUI programs that run on all the major platforms: Android, iOS/tvOS, macOS, Linux, Windows. The talk covered Gio's unusual design and how it achieves simplicity, portability and performance. Elias said, “I wanted to be able to write a GUI program in GO that I could implement only once and have it work on every platform. This, to me, is the most interesting feature of Gio.” https://twitter.com/rakyll/status/1154450455214190593 Elias also presented Scatter which is a Gio program for end-to-end encrypted messaging over email. Other features of Gio include: Immediate mode design UI state owned by program Only depends on lowest-level platform libraries Minimal dependency tree to keep things low level as possible GPU accelerated vector and text rendering It’s super efficient No garbage generated in drawing or layout code Cross platform (macOS, Linux, Windows, Android, iOS, tvOS, Webassembly) Core is 100% Go while OS-specific native interfaces are optional Gopls, new tool serves as a backend for Go editor Rebecca Stambler, mentioned in her presentation that the Go community has built many amazing tools to improve the Go developer experience. However, when a maintainer disappears or a new Go release wreaks havoc, the Go development experience becomes frustrating and complicated. To solve this issue, Rebecca revealed the details behind a new tool: gopls (pronounced as 'go please'). The tool is currently in development by the Go team and community, and it will ultimately serve as the backend for your Go editor. Below listed functionalities are expected from gopls: Show me errors, like unused variables or typos autocomplete would be nice function signature help, because we often forget While we're at it, hover-accessible "tooltip" documentation in general Help me jump to a variable that is needed to see An outline of package structure Get started with WebAssembly in Go WebAssembly in Go is here and ready to try! Although the landscape is evolving quickly, the opportunity is huge. The ability to deliver truly portable system binaries could potentially replace JavaScript in the browser. WebAssembly has the potential to finally realize the goal of being platform agnostic without having to rely on a JVM. In a session by Johan Brandhorst who introduces the technology, shows how to get started with WebAssembly and Go, discusses what is possible today and what will be possible tomorrow. As of Go 1.13, there is experimental support for WebAssembly using the JavaScript interface but as it is only experimental, using it in production is not recommended. Support for the WASI interface is not currently available but has been planned and may be available as early as in Go 1.14. Better x86 assembly generation from Go Michael McLoughlin in his presentation made the case for code generation techniques for writing x86 assembly from Go. Michael introduced assembly, assembly in Go, the use cases for when you would want to drop into assembly, and techniques for realizing speedups using assembly. He pointed out that most of the time, pure Go will be enough for 97% of programs, but there are those 3% of cases where it is warranted, and the examples he brought up were crypto, syscalls, and scientific computing. Michael then introduced a package called avo which makes high-performance Go assembly easier to write. He said that writing your assembly in Go will allow you to realize the benefits of a high level language such as code readability, the ability to create loops, variables, and functions, and parameterized code generation all while still realizing the benefits of writing assembly. Michael concluded the talk with his ideas for the future of avo. Use avo in projects specifically in large crypto implementations. More architecture support Possibly make avo an assembler itself (these kinds of techniques are used in JIT compilers) avo based libraries (avo/std/math/big, avo/std/crypto) The audience appreciated this talk on Twitter. https://twitter.com/darethas/status/1155336268076576768 The presentation slides for this are available on the blog. Miniature version of Golang, TinyGo for microcontrollers Ron Evans, creator of GoCV, GoBot and "technologist for hire" introduced TinyGo that can run directly on microcontrollers like Arduino and more. TinyGo uses the LLVM compiler toolchain to create native code that can run directly even on the smallest of computing devices. Ron demonstrated how Go code can be run on embedded systems using TinyGo, a compiler intended for use in microcontrollers, WebAssembly (WASM), and command-line tools. Evans began his presentation by countering the idea that Go, while fast, produces executables too large to run on the smallest computers. While that may be true of the standard Go compiler, TinyGo produces much smaller outputs. For example: "Hello World" program compiled using Go 1.12 => 1.1 MB Same program compiled using TinyGo 0.7.0 => 12 KB TinyGo currently lacks support for the full Go language and Go standard library. For example, TinyGo does not have support for the net package, although contributors have created implementations of interfaces that work with the WiFi chip built into Arduino chips. Support for Go Routines is also limited, although simple programs usually work. Evans demonstrated that despite some limitations, thanks to TinyGo, the Go language can still be run in embedded systems. Salvador Evans, son of Ron Evans, assisted him for this demonstration. At age 11, he has become the youngest GopherCon speaker so far. https://twitter.com/erikstmartin/status/1155223328329625600 There were talks by other speakers on topics like, improvements in VSCode for Golang, the first open source Golang interpreter with complete support of the language spec, Athens Project which is a proxy server in Go and how mobile development works in Go. https://twitter.com/ramyanexus/status/1155238591120805888 https://twitter.com/containous/status/1155191121938649091 https://twitter.com/hajimehoshi/status/1155184796386988035 Apart from these there were a whole lot of other talks which happened at the GopherCon 2019. There were live blogs posted by the attendees on various talks and till now more than 25 blogs are posted by the attendees on the Sourcegraph website. The Go team shares new proposals planned to be implemented in Go 1.13 and 1.14 Go introduces generic codes and a new contract draft design at GopherCon 2019 Is Golang truly community driven and does it really matter?  

Tõnis Tiigi, a software engineer at Docker and also a maintainer of Moby/Docker engine, in his recent post on Medium, explained how users can now leverage Docker’s non-root user privileges with Docker 19.03 release. He explains the Docker engine provides functionalities which are often tightly coupled to that of the Linux Kernel. For instance, to create namespaces in Linux users need privileged capabilities this is because a component of container isolation is based on Linux namespaces. “Historically Docker daemon has always needed to be started by the root user”, Tiigi explains. Docker is looking forward to changing this notion by introducing rootless support. With the help of Moby and BuildKit maintainer, Akihiro Suda, “we added rootless support to BuildKit image builder in 2018 and from February 2019 the same rootless support was merged to Moby upstream and is available for all Docker users to try in experimental mode”, Tiigi mentions. The rootless mode will help reduce the security footprint of the daemon and expose Docker capabilities to systems where users cannot gain root privileges. Rootless Docker and its benefits As the name suggests, a rootless mode in Docker allows a user to run Docker daemon, including the containers, as a non-root user on the host. The benefit to this is, even if it gets compromised the attacker will not be able to gain root access to the host. Akhiro Suda in his presentation, “Hardening Docker daemon with Rootless mode” explains rootless mode does not entirely fix vulnerabilities and misconfigurations but can mitigate attacks. With rootless mode attacker would not be able to access files owned by other users, modify firmware and kernel with an undetectable malware, or perform ARP spoofing. A few Caveats to the rootless Docker mode Docker engineers say the rootless mode cannot be considered a replacement for the complete suite of Docker engine features. Some limitation to the rootless mode include: cgroups resource controls, apparmor security profiles, checkpoint/restore, overlay networks etc. do not work on rootless mode. Exposing ports from containers currently requires manual socat helper process. Only Ubuntu-based distros support overlay filesystems in rootless mode. Rootless mode is currently only provided for nightly builds that may not be as stable as you are used to. As a lot of Linux features that Docker needs require privileged capabilities, the rootless mode takes advantage of user namespaces. “User namespaces map a range of user ID-s so that the root user in the inner namespace maps to an unprivileged range in the parent namespace. A fresh process in user namespace also picks up a full set of process capabilities”, Tiigi explains. Source: Medium In the recent release of the experimental rootless mode on GitHub, engineers mention rootless mode allows running dockerd as an unprivileged user, using user_namespaces(7), mount_namespaces(7), network_namespaces(7). Users need to run dockerd-rootless.sh instead of dockerd. $ dockerd-rootless.sh --experimental As Rootless mode is experimental, users need to always run dockerd-rootless.sh with --experimental. To know more about the Docker rootless mode in detail, read Tõnis Tiigi’s Medium post. You can also check out Akihiro Suda’s presentation Hardening Docker daemon with Rootless mode. Docker announces collaboration with Microsoft’s .NET at DockerCon 2019 Announcing Docker Enterprise 3.0 Public Beta! Are Debian and Docker slowly losing popularity?

Laracon US 2019, probably the biggest Laravel conference, wrapped up yesterday. Its creator, Tylor Otwell kick-started the event by talking about the next major release, Laravel 6. He also showcased a project that he has been working on called Laravel Vapor, a full-featured serverless management and deployment dashboard for PHP/Laravel. https://twitter.com/taylorotwell/status/1154168986180997125 This was a two-day event from July 24-25 hosted at Time Square, NYC. The event brings together people passionate about creating applications with Laravel, the open-source PHP web framework. Many exciting talks were hosted at this game-themed event. Evan You, the creator of Vue, was there presenting what’s coming in Vue.js 3.0. Caleb Porzio, a developer at Tighten Co., showcased a Laravel framework named Livewire that enables you to build dynamic user interfaces with vanilla PHP.  Keith Damiani, a Principal Programmer at Tighten, talked about graph database use cases. You can watch this highlights video compiled by Romega Digital to get a quick overview of the event: https://www.youtube.com/watch?v=si8fHDPYFCo&feature=youtu.be Laravel 6 coming next month Since its birth, Laravel has followed a custom versioning system. It has been on 5.x release version for the past four years now. The team has now decided to switch to semantic versioning system. The framework currently stands at version 5.8, and instead of calling the new release 5.9 the team has decided to go with Laravel 6, which is scheduled for the next month. Otwell emphasized that they have decided to make this change to bring more consistency in the ecosystem as all optional Laravel components such as Cashier, Dusk, Valet, Socialite use semantic versioning. This does not mean that there will be any “paradigm shift” and developers have to rewrite their stuff. “This does mean that any breaking change would necessitate a new version which means that the next version will be 7.0,” he added. With the new version comes new branding Laravel gets a fresh look with every major release ranging from updated logos to a redesigned website. Initially, this was a volunteer effort with different people pitching in to help give Laravel a fresh look. Now that Laravel has got some substantial backing, Otwell has worked with Focus Lab, one of the top digital design agencies in the US. They have together come up with a new logo and a brand new website. The website looks easy to navigate and also provides improved documentation to give developers a good reading experience. Source: Laravel Laravel Vapor, a robust serverless deployment platform for Laravel After giving a brief on version 6 and the updated branding, Otwell showcased his new project named Laravel Vapor. Currently, developers use Forge for provisioning and deploying their PHP applications on DigitalOcean, Linode, AWS, and more. It provides painless Virtual Private Server (VPS) management. It is best suited for medium and small projects and performs well with basic load balancing. However, it does lack a few features that could have been helpful for building bigger projects such as autoscaling. Also, developers have to worry about updating their operating systems and PHP versions. To address these limitations, Otwell created this deployment platform. Here are some of the advantages Laravel Vapor comes with: Better scalability: Otwell’s demo showed that it can handle over half a million requests with an average response time of 12 ms. Facilitates collaboration: Vapor is built around teams. You can create as many teams as you require by just paying for one single plan. Fine-grained control: It gives you fine-grained control over what each team member can do. You can set what all they can do across all the resources Vapor manages. A “vanity URL” for different environments: Vapor gives you a staging domain, which you can access with what Otwell calls a “vanity URL.” It enables you to immediately access your applications with “a nice domain that you can share with your coworkers until you are ready to assign a custom domain,“ says Otwell. Environment metrics: Vapor provides many environment metrics that give you an overview of an application environment. These metrics include how many HTTP requests have the application got in the last 24 hours, how many CLI invocations, what’s the average duration of those things, how much these cost on lambda, and more. Logs: You can review and search your recent logs right from the Vapor UI. It also auto-updates when any new entry comes in the log. Databases: With Vapor, you can provision two types of databases: fixed-sized database and serverless database. The fixed-sized database is the one where you have to pick its specifications like VCPU, RAM, etc. In the serverless one, however, if you do not select these specifications and it will automatically scale according to the demand. Caches: You can create Redis clusters right from the Vapor UI with as many nodes as you want. It supports the creation and management of elastic Redis cache clusters, which can be scaled without experiencing any downtime. You can attach them to any of the team’s projects and use them with multiple projects at the same time. To watch the entire demonstration by Otwell check out this video: https://www.youtube.com/watch?v=XsPeWjKAUt0&feature=youtu.be Laravel 5.7 released with support for email verification, improved console testing Building a Web Service with Laravel 5 Symfony leaves PHP-FIG, the framework interoperability group

Last year, Andrew Godwin, a Django contributor, formulated a roadmap to bring async functionality into Django. After a lot of discussion and amendments, the Django Technical Board approved his DEP 0009: Async-capable Django yesterday. Godwin wrote in a Google group, “After a long and involved vote, I can announce that the Technical Board has voted in favour of DEP 0009 (Async Django), and so the DEP has been moved to the "accepted" state.” The reason why Godwin thinks that this is the right time to bring async-native support in Django is that starting from version 2.1, it supports Python 3.5 and up. These Python versions have async def and similar native support for coroutines. Also, the web is now slowly shifting to use cases that prefer high concurrency workloads and large parallelizable queries. The motivation behind Async in Django The Django Enhancement Proposal (DEP) 0009 aims to address one of the core flaws in Python: inefficient threading. Python is not considered to be a perfect asynchronous language. Its ‘asyncio’ library for writing concurrent code suffers from some core design flaws. There are alternative async frameworks for Python but are incompatible. Django Channels brought some async support to Django but they primarily focus on WebSocket handling. Explaining the motivation, the DEP says, “At the same time, it's important we have a plan that delivers our users immediate benefits, rather than attempting to write a whole new Django-size framework that is natively asynchronous from the start.” Additionally, most developers are unacquainted with developing Python applications that have async support. There is also a lack of proper documentation, tutorials, and tooling to help them. Godwin believes that Django can become a “good catalyst” to help in creating guidance documentation. Goals this DEP outlines to achieve The DEP proposes to bring support for asynchronous Python into Django while maintaining synchronous Python support as well in a backward-compatible way. Here are its end goals, that Godwin listed in his roadmap: Making the blocking parts in Django such as sessions, auth, the ORM, and handlers asynchronous natively with a synchronous wrapper exposed on top where needed to ensure backward compatibility. Keeping familiar models/views/templates/middleware layout intact with very few changes. Ensuring that these updates do not compromise speed and cause significant performance regressions at any stage of this plan. Enabling developers to write fully-async websites if they want to, but not enforcing this as the default way of writing websites. Welcoming new talent into the Djang team to help out on large-scale features. Timeline to achieve these goals Godwin in his "A Django Async Roadmap" shared the following timeline: Django Version Updates 2.1 Current in-progress release. No async work 2.2 Initial work to add async ORM and view capability, but everything defaults to sync by default, and async support is mostly threadpool-based. 3.0 Rewrite the internal request handling stack to be entirely asynchronous, add async middleware, forms, caching, sessions, auth. Start the deprecation process for any APIs that are becoming async-only. 3.1 Continue improving async support, potential async templating changes 3.2 Finish deprecation process and have a mostly-async Django. Godwin posted a summary of the discussion he had with the Django Technical Board in the Google Group. Some of the queries they raised were how the team plans to distinguish async versions of functions/method from sync ones, how this implementation will ensure that there is no performance hit if the user opts out of async mode, and more. In addition to these technical queries, the board also raised a non-technical concern, “The Django project has lost many contributors over the years, is essentially in a maintenance mode, and we likely do not have the people to staff a project like this.” Godwin sees a massive opportunity to lurking in this fundamental challenge - namely to revive the Django project. He adds, “I agree with the observation that things have substantially slowed down, but I personally believe that a project like async is exactly what Django needs to get going again. There's now a large amount of fertile ground to change and update things that aren't just fixing five-year-old bugs.” Read the DEP 0009: Async-capable Django to know more in detail. Which Python framework is best for building RESTful APIs? Django or Flask? Django 2.2 is now out with classes for custom database constraints  

Microsoft has invested $1 billion in OpenAI with the goal of building next-generation supercomputers and a platform within Microsoft Azure which will scale to AGI (Artificial General Intelligence). This is a multiyear partnership with Microsoft becoming OpenAI’s preferred partner for commercializing new AI technologies. Open AI will become a big Azure customer, porting its services to run on Microsoft Azure. The $1 billion is a cash investment into OpenAI LP, which is Open AI’s for-profit corporate subsidiary. The investment will follow a standard capital commitment structure which means OpenAI can call for it, as they need it. But the company plans to spend it in less than five years. Per the official press release, “The companies will focus on building a computational platform in Azure for training and running advanced AI models, including hardware technologies that build on Microsoft’s supercomputing technology. These will be implemented in a safe, secure and trustworthy way and is a critical reason the companies chose to partner together.” They intend to license some of their pre-AGI technologies, with Microsoft becoming their preferred partner. “My goal in running OpenAI is to successfully create broadly beneficial A.G.I.,” Sam Altman, who co-founded Open AI with Elon Musk, said in a recent interview. “And this partnership is the most important milestone so far on that path.” Musk left the company in February 2019, to focus on Tesla and because he didn’t agree with some of what OpenAI team wanted to do. What does this partnership mean for Microsoft and Open AI OpenAI may benefit from this deal by keeping their innovations private which may help commercialization, raise more funds and get to AGI faster. For OpenAI this means the availability of resources for AGI, while potentially allowing founders and other investors with the opportunity to either double-down on OpenAI or reallocate resources to other initiatives However, this may also lead to them not disclosing progress, papers with details, and open source code as much as in the past. https://twitter.com/Pinboard/status/1153380118582054912 As for Microsoft, this deal is another attempt in quietly taking over open source. First, with the acquisition of GitHub and the subsequent launch of GitHub Sponsors, and now with becoming OpenAI’s ‘preferred partner’ for commercialization. Last year at an Investor conference, Nadella said, “AI is going to be one of the trends that is going to be the next big shift in technology. It's going to be AI at the edge, AI in the cloud, AI as part of SaaS applications, AI as part of in fact even infrastructure. And to me, to be the leader in it, it's not enough just to sort of have AI capability that we can exercise—you also need the ability to democratize it so that every business can truly benefit from it. That to me is our identity around AI.” Partnership with OpenAI seems to be a part of this plan. This deal can also possibly help Azure catch up with Google and Amazon both in hardware scalability and Artificial Intelligence offerings. A hacker news user comments, “OpenAI will adopt and make Azure their preferred platform. And Microsoft and Azure will jointly "develop new Azure AI supercomputing technologies", which I assume is advancing their FGPA-based deep learning offering. Google has a lead with TensorFlow + TPUs and this is a move to "buy their way in", which is a very Microsoft thing to do.” https://twitter.com/soumithchintala/status/1153308199610511360 It is also likely that Microsoft is investing money which will eventually be pumped back into its own company, as OpenAI buys computing power from the tech giant. Under the terms of the contract, Microsoft will eventually become the sole cloud computing provider for OpenAI, and most of that $1 billion will be spent on computing power, Altman says. OpenAI, who were previously into building ethical AI will now pivot to build cutting edge AI and move towards AGI. Sometimes even neglecting ethical ramifications, wanting to deploy tech at the earliest which is what Microsoft would be interested in monetizing. https://twitter.com/CadeMetz/status/1153291410994532352 I see two primary motivations: For OpenAI—to secure funding and to gain some control over hardware which in turn helps differentiate software. For MSFT—to elevate Azure in the minds of developers for AI training. - James Wang, Analyst at ARKInvest https://twitter.com/jwangARK/status/1153338174871154689 However, the news of this investment did not go down well with some experts in the field who saw this as a pure commercial deal and questioned whether OpenAI’s switch to for-profit research undermines its claims to be “democratizing” AI. https://twitter.com/fchollet/status/1153489165595504640 “I can't really parse its conversion into an LP—and Microsoft's huge investment—as anything but a victory for capital” - Robin Sloan, Author https://twitter.com/robinsloan/status/1153346647339876352 “What is OpenAI? I don't know anymore.” - Stephen Merity, Deep learning researcher https://twitter.com/Smerity/status/1153364705777311745 https://twitter.com/SamNazarius/status/1153290666413383682 People are also speculating whether creating AGI is really even possible. In a recent survey experts estimated that there was a 50 percent chance of creating AGI by the year 2099. Pet New York Times, most experts believe A.G.I. will not arrive for decades or even centuries. Even Altman admits OpenAI may never get there. But the race is on nonetheless. Then why is Microsoft delivering the $1 billion over five years considering that is neither enough money nor enough time to produce AGI. Although, OpenAI has certainly impressed the tech community with its AI innovations. In April, OpenAI’s new algorithm that is trained to play the complex strategy game, Dota 2, beat the world champion e-sports team OG at an event in San Francisco, winning the first two matches of the ‘best-of-three’ series. The competition included a human team of five professional Dota 2 players and AI team of five OpenAI bots. In February, they released a new AI model GPT-2, capable of generating coherent paragraphs of text without needing any task-specific training. However experts felt that the move signalled towards ‘closed AI’ and propagated the ‘fear of AI’ for its ability to write convincing fake news from just a few words. Github Sponsors: Could corporate strategy eat FOSS culture for dinner? Microsoft is seeking membership to Linux-distros mailing list for early access to security vulnerabilities OpenAI: Two new versions and the output dataset of GPT-2 out!

On Thursday, The Register reported that Laurie Voss, the co-founder and chief data officer of JavaScript package registry, NPM Inc left the company. Voss’s last day in office was 1st July while he officially announced the news on Thursday. Voss joined NPM in January 2014 and decided to leave the company in early May this year. NPM has faced its share of unrest in the company in the past few months. In the month of March  5 NPM employees were fired from the company in an unprofessional and unethical way. Later 3 of those employees were revealed to have been involved in unionization and filed complaints against NPM Inc with the National Labor Relations Board (NLRB).  Earlier this month NPM Inc at the third trial settled the labor claims brought by these three former staffers through the NLRB. Voss’ s resignation will be third in line after Rebecca Turner, former core contributor who resigned in March and Kat Marchan, former CLI and community architect who resigned from NPM early this month. Voss writes on his blog, “I joined npm in January of 2014 as co-founder, when it was just some ideals and a handful of servers that were down as often as they were up. In the following five and a half years Registry traffic has grown over 26,000%, and worldwide users from about 1 million back then to more than 11 million today. One of our goals when founding npm Inc. was to make it possible for the Registry to run forever, and I believe we have achieved that goal. While I am parting ways with npm, I look forward to seeing my friends and colleagues continue to grow and change the JavaScript ecosystem for the better.” Voss also told The Register that he supported unions, “As far as the labor dispute goes, I will say that I have always supported unions, I think they're great, and at no point in my time at NPM did anybody come to me proposing a union,” he said. “If they had, I would have been in favor of it. The whole thing was a total surprise to me.” The Register team spoke to one of the former staffers of NPM and they said employees tend not to talk to management in the fear of retaliation and Voss seemed uncomfortable to defend the company’s recent actions and felt powerless to affect change. In his post Voss is optimistic about NPM’s business areas, he says, “Our paid products, npm Orgs and npm Enterprise, have tens of thousands of happy users and the revenue from those sustains our core operations.” However, Business Insider reports that a recent NPM Inc funding round of the company raised only enough to continue operating until early 2020. https://twitter.com/coderbyheart/status/1152453087745007616 A big question on everyone’s mind currently is the stability of the public Node JS Registry. Most users in the JavaScript community do not have a fallback in place. While the community see Voss’s resignation with appreciation for his accomplishments, some are disappointed that he could not raise his voice against these odds and had to quit. "Nobody outside of the company, and not everyone within it, fully understands how much Laurie was the brains and the conscience of NPM," Jonathan Cowperthwait, former VP of marketing at NPM Inc, told The Register. CJ Silverio, a principal engineer at Eaze who served as NPM Inc's CTO said that it’s good that Voss is out but she wasn't sure whether his absence would matter much to the day-to-day operations of NPM Inc. Silverio was fired from NPM Inc late last year shortly after CEO Bryan Bogensberger’s arrival. “Bogensberger marginalized him almost immediately to get him out of the way, so the company itself probably won’t notice the departure," she said. "What should affect fundraising is the massive brain drain the company has experienced, with the entire CLI team now gone, and the registry team steadily departing. At some point they’ll have lost enough institutional knowledge quickly enough that even good new hires will struggle to figure out how to cope." Silverio also mentions that she had heard rumors of eliminating the public registry while only continuing with their paid enterprise service, which will be like killing their own competitive advantage. She says if the public registry disappears there are alternative projects like the one spearheaded by Silverio and a fellow developer Chris Dickinson, Entropic. Entropic is available under an open source Apache 2.0 license, Silverio says "You can depend on packages from any other Entropic instance, and your home instance will mirror all your dependencies for you so you remain self-sufficient." She added that the software will mirror any packages installed by a legacy package manager, which is to say npm. As a result, the more developers use Entropic, the less they'll need NPM Inc's platform to provide a list of available packages. Voss feels the scale of npm is 3x bigger than any other registry and boasts of an extremely fast growth rate i.e approx 8% month on month. "Creating a company to manage an open source commons creates some tensions and challenges is not a perfect solution, but it is better than any other solution I can think of, and none of the alternatives proposed have struck me as better or even close to equally good." he said. With  NPM Inc. sustainability at stake, the JavaScript community on Hacker News discussed alternatives in case the public registry comes to an end. One of the comments read, “If it's true that they want to kill the public registry, that means I may need to seriously investigate Entropic as an alternative. I almost feel like migrating away from the normal registry is an ethical issue now. What percentage of popular packages are available in Entropic? If someone else's repo is not in there, can I add it for them?” Another user responds, “The github registry may be another reasonable alternative... not to mention linking git hashes directly, but that has other issues.” Other than Entropic another alternative discussed is nixfromnpm, it is a tool in which you can translate NPM packages to Nix expression. nixfromnpm is developed by Allen Nelson and two other contributors from Chicago. Surprise NPM layoffs raise questions about the company culture Is the Npm 6.9.1 bug a symptom of the organization’s cultural problems? Npm Inc, after a third try, settles former employee claims, who were fired for being pro-union, The Register reports

A group of researchers from Intel Labs and Facebook have published a paper titled, “A Study of BFLOAT16 for Deep Learning Training”. The paper presents a comprehensive study indicating the success of Brain Floating Point (BFLOAT16) half-precision format in Deep Learning training across image classification, speech recognition, language modeling, generative networks and industrial recommendation systems. BFLOAT16 has a 7-bit mantissa and an 8-bit exponent, similar to FP32, but with less precision. BFLOAT16 was originally developed by Google and implemented in its third generation Tensor Processing Unit (TPU). https://twitter.com/JeffDean/status/1134524217762951168 Many state of the art training platforms use IEEE-754 or automatic mixed precision as their preferred numeric format for deep learning training. However, these formats lack in representing error gradients during back propagation. Thus, they are not able to satisfy the required  performance gains. BFLOAT16 exhibits a dynamic range which can be used to represent error gradients during back propagation. This enables easier migration of deep learning workloads to BFLOAT16 hardware. Image Source: BFLOAT16 In the above table, all the values are represented as trimmed full precision floating point values with 8 bits of mantissa with their dynamic range comparable to FP32. By adopting to BFLOAT16 numeric format, the core compute primitives such as Fused Multiply Add (FMA) can be built using 8-bit multipliers. This leads to significant reduction in area and power while preserving the full dynamic range of FP32. How Deep neural network(DNNs) is trained with BFLOAT16? The below figure shows the mixed precision data flow used to train deep neural networks using BFLOAT16 numeric format. Image Source: BFLOAT16 The BFLOAT16 tensors are taken as input to the core compute kernels represented as General Matrix Multiply (GEMM) operations. It is then forwarded to the FP32 tensors as output.   The researchers have developed a library called Quantlib, represented as Q in the figure, to implement the emulation in multiple deep learning frameworks. One of the functions of a Quantlib is to modify the elements of an input FP32 tensor to echo the behavior of BFLOAT16. Quantlib is also used to modify a copy of the FP32 weights to BFLOAT16 for the forward pass.   The non-GEMM computations include batch-normalization and activation functions. The  FP32 always maintains the bias tensors.The FP32 copy of the weights updates the step uses to maintain model accuracy. How does BFLOAT16 perform compared to FP32? Convolution Neural Networks Convolutional neural networks (CNN) are primarily used for computer vision applications such as image classification, object detection and semantic segmentation. AlexNet and ResNet-50 are used as the two representative models for the BFLOAT16 evaluation. AlexNet demonstrates that BFLOAT16 emulation follows very near to the actual FP32 run and achieves 57.2% top-1 and 80.1% top-5 accuracy. Whereas in ResNet-50, the BFLOAT16 emulation follows the FP32 baseline almost exactly and achieves the same top-1 and top-5 accuracy. Image Source: BFLOAT16 Similarly, the researchers were able to successfully demonstrate that BFLOAT16 is able to represent tensor values across many application domains including Recurrent Neural Networks, Generative Adversarial Networks (GANs) and Industrial Scale Recommendation System. The researchers thus established that the dynamic range of BFLOAT16 is of the same range as that of FP32 and its conversion to/from FP32 is also easy. It is important to maintain the same range as FP32 since no hyper-parameter tuning is required for convergence in FP32. A hyperparameter is a parameter of choosing a set of optimal hyperparameters in machine learning for a learning algorithm. Researchers of this paper expect to see an industry-wide adoption of BFLOAT16 across emerging domains. Recent reports suggest that Intel is planning to graft Google’s BFLOAT16 onto its processors  as well as on its initial Nervana Neural Network Processor for training, the NNP-T 1000. Pradeep Dubey, who directs the Parallel Computing Lab at Intel and is also one of the researchers of this paper believes that for deep learning, the range of the processor is more important than the precision, which is the inverse of the rationale used for IEEE’s floating point formats. Users are finding it interesting that a BFLOAT16 half-precision format is suitable for deep learning applications. https://twitter.com/kevlindev/status/1152984689268781056 https://twitter.com/IAmMattGreen/status/1152769690621448192 For more details, head over to the “A Study of BFLOAT16 for Deep Learning Training” paper. Intel’s new brain inspired neuromorphic AI chip contains 8 million neurons, processes data 1K times faster Google plans to remove XSS Auditor used for detecting XSS vulnerabilities from its Chrome web browser IntelliJ IDEA 2019.2 Beta 2 released with new Services tool window and profiling tools