Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Events
Videos
Audiobooks
Packt Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

Tech News

3711 Articles
article-image-swarm-ai-that-enables-swarms-of-radiologists-outperforms-specialists-or-ai-alone-in-predicting-pneumonia
Natasha Mathur
14 Sep 2018
4 min read
Save for later

Swarm AI that enables swarms of radiologists, outperforms specialists or AI alone in predicting Pneumonia

Natasha Mathur
14 Sep 2018
4 min read
According to a study conducted by researchers at Stanford University School of Medicine and Unanimous AI, small groups of radiologists moderated by AI algorithms achieve higher diagnostic accuracy against individual radiologists or the machine learning algorithms alone. The technology used for research is called Swarm AI. Swarm AI is a swarm intelligence technology by Unanimous AI that empowers networked groups of humans by combining their individual insights in real-time with the help of AI algorithms to converge on optimal solutions. The research paper was presented on earlier this week at the SIIM Conference on Machine Intelligence in Medical Imaging. How does Swarm AI work? The researchers performed the study with a group of eight radiologists at different locations, connected by Swarm AI algorithms. The radiologists reviewed a set of 50 chest x-rays and for each of the X-ray predicted the likelihood that the patient has pneumonia. After a few seconds of individually assessing the results of the chest x-rays, the group worked together as a “Swarm”, converging on a probabilistic diagnosis to predict the likelihood of a patient having pneumonia. This generated a set of 50 probabilities for the 50 test cases. At the same time, separately, the same set of 50 chest x-rays were run through CheXNet software algorithm, a state-of-the-art 121-layer convolutional neural network, that beat humans last year in predicting which patient suffering from pneumonia. CheXNet has been proved to outperform individual human radiologists in pneumonia screening tasks as per prior studies. These two sets of probabilities were then, further compared using different statistical techniques. Results The performance of the Swarm AI system involving a small group of human radiologists was evaluated against the software-only CheXNet system. These two methods were analyzed across three different performance metrics, namely, binary classification accuracy, Mean Absolute Error, and ROC analysis. Let’s see how these two methods performed. Binary Classification: Fifty-percent was set as the cutoff probability for classifying a positive diagnosis. The CheXNet system achieved 60% diagnostic accuracy across the 50 test cases, while the Swarm AI system achieved 82% accuracy across the same 50 cases. Also, The Swarm AI was more accurate in binary classification as compared to the ML system (p<0.01, μdifference = 21.9%). Mean Absolute Error: MAE is the absolute value of the Ground Truth (checking the classifications that machine learning algorithms make against what they know in reality) minus the Predicted Probability. A bootstrap analysis was performed for calculating MAE which revealed that the Swarm AI had significantly higher probabilistic accuracy than the ML system (p<0.001, μdifference = 21.6%). ROC Analysis: The Swarm AI system and the CheXNet system have different approaches to probabilistic forecasting. This is why a ROC (Receiver operating characteristic) analysis was performed that compared the true positive rate to the false positive rate across different cut-off points. This meant that the higher the ratio the better the classification. Area Under the ROC Curve (AUROC) was measured for both methods. Again, the swarm AI system managed to achieve an AUROC of 0.906, while the ML system achieved 0.708. Swarm AI system produced far more accurate results in the diagnosis of pneumonia than a state-of-the-art ML system, like CheXNet. “Diagnosing pathologies like pneumonia from chest X-rays is extremely difficult, making it an ideal target for AI technologies. The results of this study are very exciting as they point towards a future where doctors and AI algorithms can work together in real-time, rather than human practitioners being replaced by automated algorithms,” says Dr. Matthew Lungren, Assistant Professor of Radiology at Stanford University, in the Unanimous AI blog. This suggests that Swarm algorithms are a powerful tool when it comes to establishing Ground Truth for training use as well as for validating the machine learning systems. “It is likely that the Swarm AI system excels in certain types of cases, while the ML system excels in others. We believe future research should identify these differences, so each method can be applied to those cases which are most appropriate. Additional research is warranted using more definitive Ground Truth and a wider range of cases,” write researchers in the paper. For more information, check out the official research paper. MIT’s Transparency by Design Network: A high performant model that uses visual reasoning for machine interpretability A new Video-to-Video Synthesis model uses Artificial Intelligence to create photorealistic videos Survey reveals how artificial intelligence is impacting developers across the tech landscape
Read more
  • 0
  • 0
  • 9619

article-image-vulkan-memory-model-vulkan-becomes-the-worlds-first-graphics-api-to-include-a-formal-memory-model
Savia Lobo
14 Sep 2018
2 min read
Save for later

Vulkan memory model: Vulkan becomes the world’s first graphics API to include a formal memory model

Savia Lobo
14 Sep 2018
2 min read
Yesterday, the Khronos Group announced that its Vulkan API (modern cross-platform graphics and compute API) has become the world’s first graphics API to include a formal memory model (Vulkan Memory model) for its associated GLSL™ and SPIR-V™ programming languages. This announcement has a number of components that come together to significantly boost the robustness of the Vulkan standard for programming correctness and sophisticated compiler optimizations. The Vulkan memory model Vulkan’s memory model is based on the C++ memory model. However, it adds valuable functionality such as scopes, storage classes, and memory availability and visibility operations. These capabilities can be exploited to reduce the cost of synchronization and thus increase performance. Scopes allow synchronization to be limited to threads in close proximity to each other. Storage classes allow synchronization to be limited to specific types of memory. Availability and visibility operations give control over when and how cache maintenance operations are performed in systems with noncoherent cache hierarchies. Additional memory model materials The Khronos Group has lined up additional memory model materials in provisional form to enable feedback from the C++ community, academics, compiler engineers and software developers throughout the industry with experience in multi-threaded communication and memory usage. The additional memory model materials include: A provisional Vulkan Memory Model Specification: This specification includes extensions for Vulkan, SPIR-V, and GLSL that gives Vulkan developers additional control over how their shaders synchronize access to memory in a parallel execution environment. Memory model extension conformance tests to help shader compilers ensure that they implement the specified memory model synchronization functionality correctly. A formal description of the Vulkan memory model using Alloy, which is a language developed by MIT for describing logical structures and a tool for exploring them. This is the first instance where Khronos has used an Alloy model for its specifications. This is because, Alloy precisely documents the interactions of memory operations between multiple threads and devices, and enables formal modeling and experimentation. To know more about the Vulkan Memory model in detail, visit its GitHub page. macOS gets RPCS3 and Dolphin using Gfx-portability, the Vulkan portability implementation for non-Rust apps AMD open sources V-EZ, the Vulkan wrapper library Think Silicon open sources GLOVE: An OpenGL ES over Vulkan middleware
Read more
  • 0
  • 0
  • 14561

article-image-microsoft-acquires-ai-startup-lobe-a-no-code-visual-interface-tool-to-build-deep-learning-models-easily
Natasha Mathur
14 Sep 2018
4 min read
Save for later

Microsoft acquires AI startup Lobe, a no code visual interface tool to build deep learning models easily

Natasha Mathur
14 Sep 2018
4 min read
Microsoft announced yesterday that it has acquired Lobe, a small San Francisco based AI startup. Lobe is a visual Interface tool that allows people to easily create intelligent apps capable of understanding hand gestures, hear music, read handwriting, and more, without any coding involved. Lobe is aimed at making deep learning simple, understandable and accessible to everyone. With the Lobe’s simple visual interface, anyone can develop deep learning and AI models quickly, without having to write any code. A look at Lobe’s features Drag, drop, learn Lobe lets you build custom deep learning models, train them, and ship them directly in your app without any coding required. You can start by dragging in a folder of training examples from your desktop. This lets you build a custom deep learning model and begin its training. Once you’re done with this, you can export a trained model and ship it directly in your app. Connect together smart lobes There are smart building blocks called lobes in Lobe. These lobes can be connected together allowing you to quickly create custom deep learning models. For instance, you can connect the Hand & Face lobe to let you find the most prominent hand in the image. After this, connect the Detect Features lobe to find the important features in the hand. Finally, you can connect the Generate Labels lobe to predict the emoji in the image. You can also refine your model by adjusting each lobes unique settings or by editing any lobe’s sub-layers. Exploring dataset visually With Lobe, you can have your entire dataset displayed visually. This helps you browse and sort through all your examples. All you have to do is select any icon and see how that example performs in your model. Your dataset gets automatically split into a Lesson which teaches your model during training. There’s also a Test used that evaluates how your model will perform in the real world on examples that have never been seen before. Real-time training results Lobe comes with super fast cloud training that provides real-time results without slowing down your computer.  There are interactive charts which help you monitor the accuracy of your model and understand how the model improves over time. The best accuracy then automatically gets selected and saved. Advanced control over every layer Lobe is built on top of the deep learning frameworks TensorFlow and Keras. This allows you to control every layer of your model. With Lobe, you can tune hyperparameters, add layers, and design new architectures with the help of hundreds of advanced building block lobes. Ship it in your application After you’re done training your model, it can be exported to TensorFlow or CoreML which you can then run directly into your app. There’s also an easy-to-use Lobe Developer API, which lets you host your model in the cloud and integrate it into your app. What could Microsoft’s plans be with this acquisition? This is not the first AI startup acquired by Microsoft. Other than Lobe, Microsoft also acquired Bonsai.ai, a deep reinforcement learning platform, in July to build machine learning models for autonomous systems of all kinds. Similarly, Microsoft acquired Semantic Machines this May to build a conversational AI center of excellence in Berkeley to advance the state of conservational AI. “Over the last few months, we’ve made multiple investments in companies to further this (expanding its growth in AI) goal. These are just two recent examples of investments we have made to help us accelerate the current state of AI development”, says Kevin Scott, EVP, and CTO at Microsoft, in yesterday’s announcement on their official blog. Looks like Microsoft is all set on bringing more AI capabilities to its users. In fact, major tech firms around the world are walking along the same path and acquiring as many technology companies as they can. For instance, Amazon acquired AI cybersecurity startup Sqrrl, Facebook acquired Bloomsbury AI, and Intel acquired Vertex.ai earlier this year. “In many ways though, we’re only just beginning to tap into the full potential AI can provide. This in large part is because AI development and building deep learning models are slow and complex processes even for experienced data scientists and developers. To date, many people have been at a disadvantage when it comes to accessing AI, and we’re committed to changing that” writes Kevin. For more information, check out the official Microsoft Announcement. Say hello to IBM RXN, a free AI Tool in IBM Cloud for predicting chemical reactions Google’s new What-if tool to analyze Machine Learning models and assess fairness without any coding
Read more
  • 0
  • 0
  • 21902

article-image-sapfix-and-sapienz-facebooks-hybrid-ai-tools-to-automatically-find-and-fix-software-bugssapfix-and-sapienz-facebooks-hybrid-ai-tools-to-automatically-find-and-fix-software-bugs
Melisha Dsouza
14 Sep 2018
2 min read
Save for later

SapFix and Sapienz: Facebook’s hybrid AI tools to automatically find and fix software bugs

Melisha Dsouza
14 Sep 2018
2 min read
“Debugging code is drudgery” -Facebook Engineers Yue Jia, Ke Mao and Mark Harman To significantly reduce the amount of time developers spend on debugging code and rolling out new software, Facebook engineers have come up with an ingenious tool called ‘SapFix’. Sapfix, which is still under development, can automatically generate fixes for specific bugs  identified by Sapienz. It will then propose these fixes to engineers for approval and deployment to production. SapFix will eventually be able to operate independently from Sapienz, Facebook’s intelligent automated software testing tool. For now, it is a proof-of-concept that relies on the latter tool to pinpoint bugs. How does SapFix work? This AI hybrid tool will generate bug fixes depending upon the type of bug encountered. For instance: For simpler bugs: SapFix will create patches that revert the code submission that introduced these bugs. For complicated bugs: The tool uses a collection of “templated fixes” that were created by human engineers based on previous bug fixes. If human-designed template fixes aren’t up to the job: The tool attempts a “mutation-based fix,” which works by continuously making small modifications to the code that caused the software to crash, until a solution is found. SapFix generates multiple potential fixes for every bug. This is then submitted to the engineers for evaluation. The fixes are tested in advance so engineers can check if they might cause problems like compilation errors and other crashes. Source: Facebook With an automated end-to-end testing and repair, SapFix is an important milestone in AI hybrid tool deployment. Facebook intends to open source both, SapFix and Sapienz, once additional engineering work has been completed. You can read more about this tool on Facebook’s Blog. Facebook introduces Rosetta, a scalable OCR system that understands text on images using Faster-RCNN and CNN How AI is going to transform the Data Center Facebook Reality Labs launch SUMO Challenge to improve 3D scene understanding and modeling algorithms  
Read more
  • 0
  • 0
  • 18866

article-image-rust-1-29-is-out-with-improvements-to-its-package-manager-cargo
Savia Lobo
14 Sep 2018
2 min read
Save for later

Rust 1.29 is out with improvements to its package manager, Cargo

Savia Lobo
14 Sep 2018
2 min read
Yesterday, the Rust team announced next version release of their systems programming language, Rust 1.29. Users using the previous versions of Rust installed via rustup, can easily get this latest version by a simple command: $ rustup update stable This Rust 1.29 release has lesser features. This is because the 1.29 cycle was spent preparing for the upcoming releases, Rust 1.30 and 1.31 that will have a lot more features in them. What’s new in Rust 1.29? This stable release of Rust has two most important improvements to Cargo, Rust’s package manager. The new improvements include cargo fix can automatically fix your code that has warnings cargo clippy is a bunch of lints to catch common mistakes and improve your Rust code cargo fix Rust 1.29 includes a new subcommand for Cargo known as cargo fix. This is the initial release of cargo fix that fixes a small number of warnings in the compiler. The compiler has an API for this, and it only suggests fixing lints that the team says, recommends correct code. Over time, cargo fix can be expanded based on user suggestions to automatically fix more warnings. cargo clippy Clippy has a large number of additional warnings that users can run against their Rust code. Users can now check out a preview of cargo clippy through Rustup, by the following command: $ rustup component add clippy-preview Clippy has not yet reached 1.0. However, its lints may change. The Rust team will release a clippy component once it has stabilized. Users can’t use clippy with cargo-fix yet, really. This is still in work in progress. Additional updates in Rust 1.29 The Rust version 1.29 also include some library stabilizations. The three APIs stabilized in this release are: Arc<T>::downcast Rc<T>::downcast Iterator::flatten Users can also compare &str and OsString. To have a detailed information on Rust 1.29, read its release notes. Deno, an attempt to fix Node.js flaws, is rewritten in Rust Working with Shared pointers in Rust: Challenges and Solutions [Tutorial] Rust Language Server, RLS 1.0 releases with code intelligence, syntax highlighting and more  
Read more
  • 0
  • 0
  • 11395

article-image-meps-pass-a-resolution-to-ban-killer-robots
Bhagyashree R
14 Sep 2018
2 min read
Save for later

MEPs pass a resolution to ban “Killer robots”

Bhagyashree R
14 Sep 2018
2 min read
On Wednesday the Members of European Parliament (MEPs) passed a resolution on banning autonomous weapon systems. They emphasized that weapons like these, without proper human control over selecting and attacking targets should be banned before it is too late. Reportedly, some countries and industries are developing lethal autonomous weapon systems, which are also known as killer robots, ranging from missiles capable of selective targeting to learning machines with cognitive skills to decide whom, when, and where to fight. These might also include armed quadcopters that can search for and eliminate people meeting certain predefined criteria. According to the MEPs, giving machines so much power raises fundamental ethical and legal questions of human control, in particular with regard to critical functions such as target selection and engagement. They want the EU policy chief Federica Mogherini, the member states, and the Council to agree on a common position on lethal autonomous weapon systems, which will ensure meaningful human control over their critical functions and to speak with one voice in different international forums. Federica said during the debate at the European Parliament: "I know this might look like a debate about some distant future or about science fiction. It's not." A further discussion is scheduled at the United Nations in November, where it is hoped an agreement on an international ban can be reached. AI is growing, and it is growing fast. It has reached to a stage where building such systems is feasible in few years and could put power in wrong hands. This needs to stop because any major military power pushes ahead with AI weapon development, a global arms race is virtually inevitable. UN meetings ended with US & Russia avoiding formal talks to ban AI enabled killer robots 15 millions jobs in Britain at stake with Artificial Intelligence robots set to replace humans at workforce 6 powerful microbots developed by researchers around the world
Read more
  • 0
  • 0
  • 14434
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $19.99/month. Cancel anytime
article-image-googles-mistakenly-deployed-experiment-covertly-activated-battery-saving-mode-on-multiple-phones-today
Melisha Dsouza
14 Sep 2018
2 min read
Save for later

Google’s ‘mistakenly deployed experiment’ covertly activated battery saving mode on multiple phones today

Melisha Dsouza
14 Sep 2018
2 min read
Multiple Google Pixel phones experienced an automatic activation of the ‘Battery Saver mode’ today when their phone was almost fully charged. Reddit saw a surge of comments on how users were taken by surprise due to the automatic turning on of the Battery Saver mode. The Pixel team responded to these comments clarifying that ‘an internal experiment to test battery saving features was mistakenly rolled out to more users than intended.’. Even though the settings have been reverted back to the default on their end, the amount of control that Google has on a user’s phone without their knowledge is scary. In most devices, the battery saver mode automatically kicks in when the phone’s charge level reaches a certain threshold. Users can change the percentage at which the battery saver activates. Apparently, Google manipulated the percentage for the users remotely. Reddit saw multiple users responding to Google’s claims of the “mistakenly deployed experiment”   Source: Reddit This is not the first time that users have felt a loss of control over Google-operated services. In October 2017, Google locked out users from Google docs for a violation of their TOS. While many users affirmed that their content nowhere violated the TOS, Google responded saying that they “mistakenly blocked access” to some of their users’ files, including Google Docs. This was due to a short-lived bug that incorrectly flagged some files as violating their terms of service (TOS). The bug was soon removed and the team restored affected users access to all affected files. In both these cases, the changes took place without any prior notification to the users. These ‘accidental mistakes’ on Google’s part raises concerns of the level of control Google has over user devices and data that users aren’t yet aware of. Incidentally, AndroidPolice also reported yesterday that android handsets running Pie or a Pie beta (like Pixel models, the Essential Phone, Nokia 7 Plus, and the OnePlus 6) also experienced the Battery saver issue. Bloomberg says Google, Mastercard covertly track customers’ offline retail habits via a secret million dollar ad deal Like newspapers, Google algorithms are protected by the First amendment making them hard to legally regulate them Google slams Trump’s accusations, asserts its search engine algorithms do not favor any political ideology  
Read more
  • 0
  • 0
  • 19305

article-image-ubers-marmaray-an-open-source-data-ingestion-and-dispersal-framework-for-apache-hadoop
Natasha Mathur
14 Sep 2018
3 min read
Save for later

Uber’s Marmaray, an Open Source Data Ingestion and Dispersal Framework for Apache Hadoop

Natasha Mathur
14 Sep 2018
3 min read
Uber came out with an open source data ingestion and dispersal framework for Apache Hadoop, called “Marmaray”, yesterday. Marmaray is a plug-in based framework built and designed on top of the Hadoop ecosystem by the Hadoop Platform team. Marmaray helps connect a collection of systems and services in a cohesive manner to be able to perform certain functions. Let’s have a look at these functions. Major Functions Marmaray is capable of producing quality schematized data via Uber’s schema management library and services. It ingests data from multiple data stores into Uber’s Hadoop data lake. It can build pipelines using Uber’s internal workflow orchestration service. This allows it to crunch and process the ingested data along with storing and calculating the business metrics based on this data in Hive. Marmaray serves the processed results from Hive to an online data store. This allows the internal customers to query the data and get close to instant results. Other than that, a majority of the fundamental building blocks and abstractions for Marmaray’s design were inspired by Gobblin, a similar project developed at LinkedIn. Marmaray Architecture There are certain generic components such as DataConverters, WorkUnitCalculator, Metadata Manager, ISourceand ISink in Marmaray that facilitates its overall job flow. Let’s discuss these components.  Marmaray Architecture DataConverters DataConverters are responsible for producing the error records with every transformation. It is important for all the raw data to conform to a schema before it is ingested into Uber’s Hadoop data lake, this is where DataConverts come into picture. It filters out any data that is malformed, missing required fields, or has other issues. WorkUnitCalculator Uber introduced the concept of WorkUnitCalculator in order to measure the amount of data to process. At advanced levels, WorkUnitCalculator analyzes the type of input source and the previously stored checkpoint. It then calculates the next work unit or batch of work. The WorkUnitCalculator also considers throttling information when measuring the next batch of data which needs processing. Metadata Manager The Metadata Manager is responsible to cache job level metadata information. The metadata store is capable of storing any relevant metrics which are useful to track, describe, or collect status on jobs. This helps Marmaray to cache job level metadata information. ISource and ISink The ISource consists of necessary information from the source data required for the appropriate work units, and ISink comprises all the necessary information on writing to the sink. Marmaray’s support for any-source to any-sink data pipelines can be applied to a wide range of use cases both in the Hadoop ecosystem and for data migration. “We hope that Marmaray will serve the data needs of other organizations, and that open source developers will broaden its functionalities,” reads the Uber Blog. For more information, check out the official Uber Blog. Uber open sources its large scale metrics platform, M3 for Prometheus Uber introduces Fusion.js, a plugin-based web development framework for high performance apps Uber’s kepler.gl, an open source toolbox for GeoSpatial Analysis
Read more
  • 0
  • 0
  • 14164

article-image-oracle-releases-open-source-and-commercial-licenses-for-java-11-and-later
Savia Lobo
13 Sep 2018
3 min read
Save for later

Oracle releases open source and commercial licenses for Java 11 and later

Savia Lobo
13 Sep 2018
3 min read
Oracle announced that it will provide JDK releases in two combinations ( an open source license and a commercial license): Under the open source GNU General Public License v2, with the Classpath Exception (GPLv2+CPE) Under a commercial license for those using the Oracle JDK as part of an Oracle product or service, or who do not wish to use open source software. These combinations will replace the historical BCL(Binary Code License for Oracle Java SE technologies), which had a combination of free and paid commercial terms. The BCL has been the primary license for Oracle Java SE technologies for well over a decade. It historically contained ‘commercial features’ that were not available in OpenJDK builds. However, over the past year, Oracle has contributed features to the OpenJDK Community, which include Java Flight Recorder, Java Mission Control, Application Class-Data Sharing, and ZGC. From Java 11 onwards, therefore, Oracle JDK builds and OpenJDK builds will be essentially identical. Minute differences between Oracle JDK 11 and OpenJDK Oracle JDK 11 emits a warning when using the -XX:+UnlockCommercialFeatures option. On the other hand, in OpenJDK builds this option results in an error. This difference remains in order to make it easier for users of Oracle JDK 10 and earlier releases to migrate to Oracle JDK 11 and later. The javac --release command behaves differently for the Java 9 and Java 10 targets. This is because, in those releases the Oracle JDK contained some additional modules that were not part of corresponding OpenJDK releases. Some of them are: javafx.base javafx.controls javafx.fxml javafx.graphics javafx.media javafx.web This difference remains in order to provide a consistent experience for specific kinds of legacy use. These modules are either now available separately as part of OpenJFX, are now in both OpenJDK and the Oracle JDK because they were commercial features which Oracle contributed to OpenJDK (e.g., Flight Recorder), or were removed from Oracle JDK 11 (e.g., JNLP). The Oracle JDK always requires third party cryptographic providers to be signed by a known certificate. The cryptography framework in OpenJDK has an open cryptographic interface. This means it does not restrict which providers can be used.  Oracle JDK 11 will continue to require a valid signature, and Oracle OpenJDK builds will continue to allow the use of either a valid signature or unsigned third party crypto provider. The Oracle JDK has always required third party cryptographic providers to be signed by a known certificate.  The cryptography framework in OpenJDK has an open cryptographic interface, meaning it does not restrict which providers can be used.  Oracle JDK 11 will continue to require a valid signature, and Oracle OpenJDK builds will continue to allow the use of either a valid signature or unsigned third party crypto provider. Read more about this news in detail on Oracle blog. State of OpenJDK: Past, Present and Future with Oracle Oracle announces a new pricing structure for Java Oracle reveals issues in Object Serialization. Plans to drop it from core Java
Read more
  • 0
  • 0
  • 15770

article-image-facebook-launches-logdevice-an-open-source-distributed-data-store-designed-for-logs
Bhagyashree R
13 Sep 2018
3 min read
Save for later

Facebook launches LogDevice: An open source distributed data store designed for logs

Bhagyashree R
13 Sep 2018
3 min read
Yesterday, Facebook open-sourced LogDevice, a distributed data store designed specifically for logs. This initial release includes the toolset that forms the core of LogDevice operational infrastructure. In future, they are planning to open-source the automation tools that they have built to manage LogDevice clusters. It is currently supported only on Ubuntu 18.04 (Bionic Beaver). However, it should be possible to build it on any other Linux distribution without significant challenges. What is LogDevice LogDevice, as the name suggests, is a log system which promises to be scalable and fault tolerant. Unlike filesystems, which store and serve data as organized files, LogDevice stores and delivers data as logs. A log is a record-oriented, append-only, trimmable file. How it works LogDevice uses a placement and delivery scheme, which is great for write availability and handling spiky write workloads: 1. Separating sequencing and storage: First, the ordering of records is decoupled from the actual storage of record copies. For every log in a LogDevice cluster, a sequencer object is executed to issue monotonically increasing sequence numbers as records are appended to that log. Source: Facebook 2. Placement: After a record is stamped with a sequence number, its copies can be potentially stored on any storage in the cluster. 3. Reading: A client who wants to read a particular log contacts all storage nodes that are permitted to store records of that log. These nodes are collectively called node set of the log and usually kept smaller than the total number of nodes in the cluster. The contacted nodes deliver record copies to the client by pushing them into the TCP connections as fast as they can. 4. Metadata history: The node set is part of the replication policy of the log. It can be changed at any time, with an appropriate note in the log’s metadata history. Readers can then consult it in order to find the storage nodes to connect to. 5. Reordering and de-duplication: The reordering and occasional de-duplication of records is done by the LogDevice client library. This is done to ensure that the records are delivered to the reader application in the order of their log sequence number (LSN). What are its common use cases Facebook uses LogDevice for various use cases, which include: Write-ahead logging for durability Transaction logging in a distributed database Event logging Journals of deferred work items Distribution of index updates in large distributed databases Machine learning pipelines Replication pipelines Durable reliable task queues Stream processing pipelines If you want to explore more on LogDevice and contribute to this open-source project, check out its official website and GitHub repository. Why Neo4j is the most popular graph database Why MongoDB is the most popular NoSQL database today Introducing Watermelon DB: A new relational database to make your React and React Native apps highly scalable
Read more
  • 0
  • 0
  • 8822
article-image-has-the-eu-just-ended-the-internet-as-we-know-it
Richard Gall
13 Sep 2018
4 min read
Save for later

Has the EU just ended the internet as we know it?

Richard Gall
13 Sep 2018
4 min read
Yesterday (12 September), the EU Parliament voted through the EU Copyright Directive. This move will, according to critics, put an end to the open internet as know it. We reported on what the EU Copyright Directive means for the developer world earlier this week. Now, that world has been rocked significantly, with engineers, technologists and free speech advocates searching for solutions for what looks like a potentially devastating result. What happened in the EU Copyright Directive vote? Articles 11 and 13 were both crucial issues in this week's vote. They were the reason the directive was rejected back in July. The vote yesterday was on small amendments to these articles that, for the most part, keeps their intent intact. Article 11 has been described as a link tax - it effectively hands publishers control over who can link to their content and how, while article 13 has been criticised for enforcing 'copyright filters' on websites and platforms where users upload content. 438 MEPs voted in favorr of the directive; 226 against it. Why did MEPs vote in favor of the EU Copyright Directive? The EU Parliament press release provides a good indication of the thinking behind the directive. It would seem that the intent is too remove some of the power from large tech platforms - like Google and Facebook - and return some power to media companies and content producers that have been struggling in the digital age. The press release states: "Many of Parliament’s changes to the EU Commission’s original proposal aim to make certain that artists, notably musicians, performers and script authors, as well as news publishers and journalists, are paid for their work when it is used by sharing platforms such as YouTube or Facebook, and news aggregators such as Google News." Alongside this, there are a number of exemptions in the legislation that the EU Parliament argues will ensure none of the consequences its critics have suggested could happen will actually happen. For example: Small and micro platforms are excluded from the directive. Normal hyperlinks won't be impacted by article 11: the press release states that 'merely sharing hyperlinks to articles, together with “individual words” to describe them, will be free of copyright constraints'. Wikipedia and open source platforms like GitHub will be exempt. What happens next? There will be a final vote on the directive in January 2019. However, if this passes, the implementation of the legislation might vary at a national level. Individual EU countries could choose to enact the directive in whichever way they choose. Reaction to the vote The EU Copyright Directive has faced intense criticism since it first appeared back in 2016 - but with the vote yesterday, organizations and individuals have voiced their concern at the result. Julia Reda, MEP and member of the Pirate Party in Germany, who has been a vociferous opponent of the directive in Parliament, called it "a severe blow to the free and open internet." Similarly, the Electronic Frontier Foundation published a forthright post against the EU Parliament's decision: "We suffered a crushing setback today, but it doesn't change the mission. To fight, and fight, and fight, to keep the Internet open and free and fair, to preserve it as a place where we can organise to fight the other fights that matter, about inequality and antitrust, race and gender, speech and democratic legitimacy." The EFF also put together a letter addressed to Antonio Tajani, the President of the European Parliament. It was signed by some of the best known figures in technology, including Tim Berners-Lee, Guido van Rossum, and Jimmy Wales. The letter ends: "We support the consideration of measures that would improve the ability for creators to receive fair remuneration for the use of their works online. But we cannot support Article 13, which would mandate Internet platforms to embed an automated infrastructure for monitoring and censorship deep into their networks. For the sake of the Internet’s future, we urge you to vote for the deletion of this proposal." The fight isn't over yet, but you can sense palpable fear in many quarters about what this means for the future of the internet as we know it.
Read more
  • 0
  • 0
  • 12534

article-image-python-serious-about-diversity-dumps-offensive-master-slave-terms-in-its-documentation
Natasha Mathur
13 Sep 2018
3 min read
Save for later

Python serious about diversity, dumps offensive ‘master’, ‘slave’ terms in its documentation

Natasha Mathur
13 Sep 2018
3 min read
Python is set on changing its “master” and “slave” terminology in its documentation and code. This is to conciliate the people claiming the terminology as offensive. A python developer at Red Hat, Victor Stinner, started a discussion titled “avoid master/slave terminology” on Python bug report, last week. The bug report discusses changing "master" and "slave" in Python documentation to terms such as "parent", "worker", or something similar, based on the complaints received “privately”. “For diversity reasons, it would be nice to try to avoid ‘master’ and ‘slave’ terminology which can be associated to slavery” mentioned Victor Stinner in the bug report. Not every Python developer who participated in this discussion agreed with Victor Stinner. One of the developers in the discussion, Larry Hastings, wrote “I'm a little surprised by this.  It's not like slavery was acceptable when these computer science terms were coined and it's only comparatively recently that they've gone out of fashion. On the other hand, there are some areas in computer software where "master" and "slave" are the exact technical terms (e.g. IDE), and avoiding them would lead to confusion”. Another Python developer, Terry J. Reedy wrote, “To me, there is nothing wrong with the word 'master', as such. I mastered Python to become a master of Python. Purging Python of 'master' seems ill-conceived. Like Larry, I object to action based on hidden evidence”. Python is not the only one who has been under Scrutiny. The Redis community, Django, and Drupal all faced the same issue. Drupal changed the terms "master" and "slave" for "primary" and "replica". Similarly, Django swapped "master" and "slave" for "leader" and "follower". To put an end to this debate about the use of this politically incorrect language, Guido Van Rossum, who resigned as “Benevolent dictator for life” or BDFL in July, but is still active as a core developer, was pulled back in. Guido ended the discussion by saying, “I'm closing this now. Three out of four of Victor's PRs have been merged. The fourth one should not be merged because it reflects the underlying terminology of UNIX ptys. There's a remaining quibble about "pliant children" -> "helpers" but that can be dealt with as a follow-up PR without keeping this discussion open”. The final commit on this is as follows: bpo-34605, pty: Avoid master/slave terms * pty.spawn(): rename master_read parameter to parent_read * Rename pty.slave_open() to pty.child_open(), but keep an pty.slave_open alis to pty.child_open for backward compatibility * os.openpty(), os.forkpty(): rename master_fd/slave_fd to parent_fd/child_fd * Rename internal variables: * Rename master_fd/slave_fd to parent_fd/child_fd * Rename slave_name to child_name For more information on the discussion, be sure to check out the official Python bug report. Why Guido van Rossum quit as the Python chief (BDFL) No new PEPS will be approved for Python in 2018, with BDFL election pending Python comes third in TIOBE popularity index for the first time
Read more
  • 0
  • 0
  • 15475

article-image-net-announcements-preview-2-of-net-core-2-2-and-entity-framework-core-2-2-c-7-3-and-ml-net-0-5
Savia Lobo
13 Sep 2018
5 min read
Save for later

.NET announcements: Preview 2 of .NET Core 2.2 and Entity Framework Core 2.2, C# 7.3, and ML.NET 0.5

Savia Lobo
13 Sep 2018
5 min read
Yesterday, the .NET community announced the second preview of .NET Core 2.2 and the Entity Framework 2.2. They also released C# version 7.3 and ML.NET 0.5. Let’s have a look at the highlights and features of each of these announcements. .NET Core 2.2 Preview 2 .NET Core 2.2 Preview 2 can be used with Visual Studio 15.8, Visual Studio for Mac and Visual Studio Code. Following are two highlights of this release. Tiered Compilation Enabled The tiered compilation is enabled by default. The tiered compilation was available as part of the .NET Core 2.1 release. During that time, one had to enable tiered compilation via application configuration or an environment variable. It is now enabled by default and can be disabled, as needed. In the image below, the baseline is .NET Core 2.1 RTM, running in a default configuration, with tiered compilation disabled. The second scenario has tiered compilation. One can see a significant request-per-second (RPS) throughput benefit with tiered compilation enabled. The numbers in the chart are scaled so that baseline always measures 1.0. Such an approach makes it very easy to calculate performance changes as a percentage. The first two tests are TechEmpower benchmarks and the last one is Music Store, a frequent sample ASP.NET app. Platform Support .NET Core 2.2 is supported on the following operating systems: Windows Client: 7, 8.1, 10 (1607+) Windows Server: 2008 R2 SP1+ macOS: 10.12+ RHEL: 6+ Fedora: 27+ Ubuntu: 14.04+ Debian: 8+ SLES: 12+ openSUSE: 42.3+ Alpine: 3.7+ Read about the .NET Core 2.2 Preview 2 in detail, on Microsoft blog. Entity Framework Core 2.2 Preview 2 This preview includes a large number of bug fixes and two additional important previews, one is a data provider for Cosmos DB and the second one, new spatial extensions for .NET’s  SQL Server and in-memory providers. New EF Core provider for Cosmos DB This new provider enables developers (familiar with the EF programming model) to easily target Azure Cosmos DB as an application database. It also includes global distribution, elastic scalability, ‘always on’ availability, very low latency, and automatic indexing. Spatial extensions for SQL Server and in-memory This implementation picks the NetTopologySuite library that the PostgreSQL provider uses as the source of spatial .NET types. NetTopologySuite is a database-agnostic spatial library that implements standard spatial functionality using .NET idioms like properties and indexers. The extension then adds the ability to map and convert instances of these types to the column types supported by the underlying database, and usage of methods defined on these types in LINQ queries, to SQL functions supported by the underlying database. Read more about the Entity Framework Core 2.2 Preview 2 on the Microsoft blog. C# 7.3 C# 7.3 is the newest point release in the 7.0 family. Along with new compiler options, there are two main themes to the C# 7.3 release One provides features that enable safe code to be as performant as unsafe code. The second provides incremental improvements to existing features. New features that support the theme of better performance for safe code: Access to fixed fields without pinning. Easy Reassign ref local variables. Use of initializers on stackalloc arrays. Easy use of fixed statements with any type that supports a pattern. One can use additional generic constraints. The new compiler options in C# 7.3 are: -publicsign to enable Open Source Software (OSS) signing of assemblies. -pathmap to provide a mapping for source directories. Read more about the C# 7.3 in detail in its documentation notes. ML.NET 0.5 The .NET community released ML.NET version 0.5. ML.NET is a cross-platform, open source machine learning framework for .NET developers. This version release includes two key highlights: Addition of a TensorFlow model scoring transform (TensorFlowTransform) Starting from this version, the community plans to add support for Deep Learning in ML.NET. Following this, they introduced the TensorFlowTransform which enables taking an existing TensorFlow model, either trained by the user or downloaded from somewhere else, and get the scores from the TensorFlow model in ML.NET. This new TensorFlow scoring capability doesn’t require one to have a working knowledge of TensorFlow internal details. The implementation of this transform is based on code from TensorFlowSharp. One can simply add a reference to the ML.NET NuGet packages in thier .NET Core or .NET Framework apps. Under the covers, ML.NET includes and references the native TensorFlow library which allows writing code that loads an existing trained TensorFlow model file for scoring. New ML.NET API proposal exploration The new ML.NET API offers more flexible capabilities than what the current LearningPipeline API offers. The LearningPipeline API will be deprecated when this new API is ready. The new ML.NET API offers attractive features which aren’t possible with the current LearningPipeline API. These include: Strongly-typed API  takes advantage of C# capabilities. This helps errors to be discovered in compilation time along with improved Intellisense in the editors. Better flexibility: This new API provides a decomposable train and predict process, eliminating rigid and linear pipeline execution. Improved usability: This new API makes direct call to the APIs from user’s code. No more scaffolding or insolation layer creating an obscure separation between what the user/developer writes and the internal APIs. Entrypoints are no longer mandatory. Ability to simply score with TensorFlow models: One can also simply load a TensorFlow model and score by using it without needing to add any additional learner and training process. Better visibility of the transformed data: User’s have better visibility of the data while applying transformers. As this API inclusion will be a significant change in ML.NET, the community has started an open discussion where users can provide their feedback and help shape the long-term API for ML.NET. Users can share their feedback on ML.NET GitHub repo. Read more about ML.NET 0.5 in detail, on Microsoft blog. Task parallel library for easy multi-threading in .NET Core [Tutorial] Use App Metrics to analyze HTTP traffic, errors & network performance of a .NET Core app [Tutorial] Microsoft’s .NET Core 2.1 now powers Bing.com  
Read more
  • 0
  • 0
  • 15012
article-image-pay-your-respects-inbox-googles-email-innovation-getting-discontinued
Sunith Shetty
13 Sep 2018
4 min read
Save for later

Pay your respects to Inbox, Google’s email innovation is getting discontinued

Sunith Shetty
13 Sep 2018
4 min read
Yesterday, Google announced that they will be retiring Inbox early next year invoking deep emotions ranging from gratefulness to anger and disappointment among existing users of the app. Inbox by Gmail is an email service which was developed by the Google team, first purely on the basis of limited invitation-only on October 2014. However, the general availability to the public was made in May 2015. Interestingly the same group of people who developed Gmail built Inbox--which served as a completely different type of inbox offering a more personalized email app designed to focus on what really matters. The platform is available on all major internet media such as Web, and mobile operating systems such as Android and iOS. It provides improved email productivity through several key features. However, lately, new approaches added to the Inbox haven’t been quite novel when compared to stellar Gmail upgrades, including the big redesign feature in April 2018. Why is Google sunsetting Inbox? Inbox has always been Google’s most bold and exciting email application. If we look at the evolution of Gmail and Inbox, Gmail has stayed the same over the years while Inbox was always full of exciting unique features. Inbox worked as a sort of incubator for trying out new approaches to email service. But recently, Gmail has been getting all the attention in terms of redesigning and the feature support on Android such as customizing swipes to mark as read, delete, snooze, and more. This has made many suspect the eminent demise of Inbox. Google, now officially announced that they will be discontinuing the Inbox by the end of March 2019. According to the product manager Matthew Izatt, the company wants a more focused approach to email, leaving Gmail as the lone survivor. This is a surprising move by the Google, especially after the announcement which was made after the big Gmail redesign. They said with respect to the upcoming new features in Gmail, there are no changes to Inbox by Gmail. It is a great product for users with specific workflows and is one which allows them to test innovative features for email. Seems like Google used Inbox to beta test the features before rolling them out to Gmail. Apparently, the beta test is now over. Makes one wonder if Google losing the advantage to experiment in the future with new email functionalities that’s tough to do with a billion-user base. Moving on to Gmail Most of the Inbox initial set of features like Smart Compose, Smart Reply, organized folder bundles, and Follow-ups are already available in Gmail. You might not lose any major functionalities when opting for Gmail instead of Inbox. After the redesign, with large buttons and a friendlier look, the interface for Gmail has become somewhat Inbox-like. Google gave Gmail a major overhaul this year which absorbed a lot of features that made Inbox special. All these features on Gmail ensure it provides a better user experience when dealing with their emails. Since there are still a few months left until Inbox disappears, Google team has provided a transition guide for Inbox users to move to the new Gmail easily. What’s next? Google is, of course, no stranger to discontinuing apps and services. If you have been using Inbox by Gmail for all these years, you will surely not be happy with this news. However, Gmail has been slowly and steadily adopting many of the Inbox’s features. You will be surprised to see how Gmail has changed over the years. But people who don't like Gmail and are looking for an official alternative will now have to find a new email app to satisfy their expectations. Read more How to send email Notifications using SendGrid Machine learning based Email-sec-360°surpasses 60 antivirus engines in detecting malicious emails Wireshark for analyzing issues and malicious emails in POP, IMAP, and SMTP [Tutorial]
Read more
  • 0
  • 0
  • 9253

article-image-survey-reveals-how-artificial-intelligence-is-impacting-developers-across-the-tech-landscape
Richard Gall
13 Sep 2018
2 min read
Save for later

Survey reveals how artificial intelligence is impacting developers across the tech landscape

Richard Gall
13 Sep 2018
2 min read
The hype around artificial intelligence has reached fever pitch. It has captured the imagination - and stoked the fears - of the wider public, reaching beyond computer science departments and research institutions. But when artificial intelligence dominates the international conversation, it's easy to forget that it's not simply a thing that exists and develops itself. However intelligent machines are, and however adept they are at 'learning', it's essential to remember they are things that are engineered - things that are built by developers. That's the thinking behind this year's AI Now survey. To capture the experiences and perspectives of developers and to better understand the impact of artificial intelligence on their work and lives. Key findings from Packt's artificial intelligence survey Launched in August, and receiving 2,869 responses from developers working in every area, from cloud to cybersecurity, the survey had some interesting findings. These include... 69% of developers aren’t currently using AI enabling tools in their day to day role. But 75% of respondents said they were planning on learning AI enabling software in the next 12 months. TensorFlow is the tool defining AI development - 27% of respondents listed it as their top tool in their to-learn list. 75% of developers believe automation will have either a positive or significant positive impact on their career. 47% of respondents believe AGI will be a reality within the next 30 years The biggest challenges for developers in terms of AI are having the time to learn new skills and knowing which frameworks and tools to learn Internal data literacy is the biggest challenge for AI implementation As well as as quantitative results, the survey also produced  qualitative insights from developers. This provided some useful and unique perspectives on artificial intelligence. One developer, talking about bias in AI, said that: “As a CompSci/IT professional I understand this is a more subtle manifestation of ‘Garbage In/Garbage Out”. As an African American, I have significant concerns about say, well documented bias in say criminal sentencing being legitimized because ‘the algorithm said so’.” To read the report click here. To coincide with the release of survey results Packt is also running a $10 sale on all eBooks and videos across their website throughout September. Visit the Packt eCommerce store to start exploring.
Read more
  • 0
  • 0
  • 16001
Modal Close icon
Modal Close icon