Tech News - Cloud & Networking

Machine learning experts are increasingly becoming interested in researching on how machine learning can be used to reduce greenhouse gas emissions and help society adapt to a changing climate. For example, Machine Learning can be used to regulate cloud data centres that manage an important asset, ‘Data’ as these data centres typically comprise tens to thousands of interconnected servers and consume a substantial amount of electrical energy. Researchers from Huawei published a paper in April 2015, estimating that by 2030 data centres will use anywhere between 3% and 13% of global electricity At the ICT4S 2019 conference held in Lappeenranta, Finland, from June 10-15, researchers from the University of Bristol, UK, introduced their research on a low carbon scheduling policy for the open-source Kubernetes container orchestrator. “Low Carbon Kubernetes Scheduler” can provide demand-side management (DSM) by migrating consumption of electric energy in cloud data centres to countries with the lowest carbon intensity of electricity. In their paper the researchers highlight, “All major cloud computing companies acknowledge the need to run their data centres as efficiently as possible in order to address economic and environmental concerns, and recognize that ICT consumes an increasing amount of energy”. Since the end of 2017, Google Cloud Platform runs its data centres entirely on renewable energy. Also, Microsoft has announced that its global operations have been carbon neutral since 2012. However, not all cloud providers have been able to make such an extensive commitment. For example, Oracle Cloud is currently 100% carbon neutral in Europe, but not in other regions. The Kubernetes Scheduler selects compute nodes based on the real-time carbon intensity of the electric grid in the region they are in. Real-time APIs that report grid carbon intensity is available for an increasing number of regions, but not exhaustively around the planet. In order to effectively demonstrate the schedulers ability to perform global load balancing, the researchers have evaluated the scheduler based on its ability to the metric of solar irradiation. “While much of the research on DSM focusses on domestic energy consumption there has also been work investigating DSM by cloud data centres”, the paper mentions. Demand side management (DSM) refers to any initiatives that affect how and when electricity is being required by consumers. Source: CEUR-WS.org Existing schedulers work with consideration to singular data centres rather than taking a more global view. On the other hand, the Low Carbon Scheduler considers carbon intensity across regions as scaling up and down of a large number of containers that can be done in a matter of seconds. Each national electric grid contains electricity generated from a variable mix of alternative sources. The carbon intensity of the electricity provided by the grid anywhere in the world is a measure of the amount of greenhouse gas released into the atmosphere from the combustion of fossil fuels for the generation of electricity. Significant generation sites report the volume of electricity input to the grid in regular intervals to the organizations operating the grid (for example the National Grid in the UK) in real-time via APIs. These APIs typically provide the retrieval of the production volumes and thus allow to calculate the carbon intensity in real-time. The Low carbon scheduler collects the carbon intensity from the available APIs and ranks them to identify the region with the lowest carbon intensity. [box type="shadow" align="" class="" width=""]For the European Union, such an API is provided by the European Network of Transmission System Operators for Electricity (www.entsoe.eu) and for the UK this is the Balancing Mechanism Reporting Service (www.elexon.co.uk).[/box] Why Kubernetes for building a low carbon scheduler Kubernetes can make use of GPUs4 and has also been ported to run on ARM architecture 5. Researchers have also said that Kubernetes has to a large extent won the container orchestration war. It also has support for extendability and plugins which makes it the “most suitable for which to develop a global scheduler and bring about the widest adoption, thereby producing the greatest impact on carbon emission reduction”. Kubernetes allows schedulers to run in parallel, which means the scheduler will not need to re-implement the pre-existing, and sophisticated, bin-packing strategies present in Kubernetes. It need only to apply a scheduling layer to complement the existing capabilities proffered by Kubernetes. According to the researchers, “Our design, as it operates at a higher level of abstraction, assures that Kubernetes continues to deal with bin-packing at the node level, while the scheduler performs global-level scheduling between data centres”. The official Kubernetes documentation describes three possible ways of extending the default scheduler (kube-scheduler): adding these rules to the scheduler source code and recompiling, implementing one’s own scheduler process that runs instead of, or alongside kube-scheduler, or implementing a scheduler extender. Evaluating the performance of the low carbon Kubernetes scheduler The researchers recorded the carbon intensities for the countries that the major cloud providers operate data centers between 18.2.2019 13:00 UTC and 21.4.2019 9:00 UTC. Following is a table showing countries where the largest public cloud providers operate data centers, as of April 2019. Source: CEUR-WS.org They further ranked all countries by the carbon intensity of their electricity in 30-minute intervals. Among the total set of 30-minute values, Switzerland had the lowest carbon intensity (ranked first) in 0.57% of the 30-minute intervals, Norway 0.31%, France 0.11% and Sweden in 0.01%. However, the list of the least carbon intense countries only contains countries in central Europe locations. To justify Kubernetes’ ability or globally distributed deployments the researchers chose to optimize placement to regions with the greatest degree of solar irradiance termed a Heliotropic Scheduler. This scheduler is termed ‘heliotropic’ in order to differentiate it from a ‘follow-the-sun’ application management policy that relates to meeting customer demand around the world by placing staff and resources in proximity to those locations (thereby making them available to clients at lower latency and at a suitable time of day). A ‘heliotropic’ policy, on the other hand, goes to where sunlight, and by extension solar irradiance, is abundant. They further evaluated the Heliotropic Scheduler implementation by running BOINC jobs on Kubernetes. BOINC (Berkeley Open Infrastructure for Network Computing) is a software platform for volunteer computing that allows users to contribute computational capacity from their home PCs towards scientific research. Einstein@Home, SETI@home and IBM World Community Grid are some of the most widely supported projects. Researchers say: “Even though many cloud providers are contracting for renewable energy with their energy providers, the electricity these data centres take from the grid is generated with release of a varying amount of greenhouse gas emissions into the atmosphere. Our scheduler can contribute to moving demand for more carbon intense electricity to less carbon intense electricity”. While the paper concludes that wind-dominant, solar-complementary strategy is superior for the integration of renewable energy sources into cloud data centres’ infrastructure, the Low Carbon Scheduler provides a proof-of-concept demonstrating how to reduce carbon intensity in cloud computing. To know more about this implementation for lowering carbon emissions read the research paper. Machine learning experts on how we can use machine learning to mitigate and adapt to the changing climate VMware reaches the goal of using 100% renewable energy in its operations, a year ahead of their 2020 vision Deep learning models have massive carbon footprints, can photonic chips help reduce power consumption?

Yesterday, the Linux Foundation announced that they are joining hands with the RISC-V Foundation to drive the open source development and adoption of the RISC-V instruction set architecture (ISA). https://twitter.com/risc_v/status/1067553703685750785 The RISC-V Foundation is a non-profit corporation, which is responsible for directing the future development of the RISC-V ISA. Since its formation, the RISC-V Foundation has quickly grown and now includes more than 100 member organizations. With this collaboration, the foundations aim to further grow this RISC-V ecosystem and provide improved support for the development of new applications and architectures across all computing platforms. Rick O’Connor, the executive director of the RISC-V Foundation, said, “With the rapid international adoption of the RISC-V ISA, we need increased scale and resources to support the explosive growth of the RISC-V ecosystem. The Linux Foundation is an ideal partner given the open source nature of both organizations. This joint collaboration with the Linux Foundation will enable the RISC-V Foundation to offer more robust support and educational tools for the active RISC-V community, and enable operating systems, hardware implementations and development tools to scale faster.” The Linux Foundation will provide governance, best practices for open source development, and resources such as training programs and infrastructure tools. Along with this, they will also help RISC-V in community outreach, marketing, and legal expertise. Jim Zemlin, the executive director at the Linux Foundation believes that RISC-V has great potential seeing its popularity in areas like AI, machine learning, IoT, and more. He said, “RISC-V has great traction in a number of markets with applications for AI, machine learning, IoT, augmented reality, cloud, data centers, semiconductors, networking and more. RISC-V is a technology that has the potential to greatly advance open hardware architecture. We look forward to collaborating with the RISC-V Foundation to advance RISC-V ISA adoption and build a strong ecosystem globally.” The two foundations have already started working on a pair of getting started guides for running Zephyr, a small, scalable open source real-time operating system (RTOS) optimized for resource-constrained devices. They are also conducting RISC-V Summit, a 4-day event starting from December 3-6 in Santa Clara. This summit will include sessions on RISC-V ISA architecture, commercial and open-source implementations, software and silicon, vectors and security, applications and accelerators, and much more. Read the complete announcement on the Linux Foundation’s official website. Uber becomes a Gold member of the Linux Foundation The Ceph Foundation has been launched by the Linux Foundation to support the open source storage project Google becomes new platinum member of the Linux foundation

Yesterday, the team at Elastic released Elastic Stack 6.7 a group of open source products from Elastic designed to help users take data from any type of source and visualize that data in real time. What’s new in Elastic Stack 6.7? Elastic Maps Elastic Maps is a new dedicated solution used for mapping, querying, and visualizing geospatial data in Kibana. They expand on existing geospatial visualization options in Kibana with features such as visualization of multiple layers and data sources in the same map. It also includes features like dynamic data-driven styling on vector layers on maps, mapping of both aggregate and document-level data and much more. Elastic Maps also embeds the query bar with autocomplete for real-time ad-hoc search. Elastic Uptime This release comes with Elastic Uptime, that makes it easy to detect when application services are down or they are responding slowly. It notifies users about problems way before those services are called by the application. Cross Cluster Replication (CCR) Cross Cluster Replication (CCR) now has a variety of use cases that include cross-datacenter and cross-region replication and it is generally available. Index Lifecycle Management (ILM) With this release, Index lifecycle management (ILM) is now generally available and also ready for production use. ILM helps Elasticsearch admins with defining and automating lifecycle management policies, such as how data is to be managed and moved between phases like hot, warm, cold, and deletion phases while it ages. Elasticsearch SQL Elasticsearch SQL, helps users with interacting and querying their Elasticsearch data using SQL. Elasticsearch SQL functionality includes the JDBC and ODBC clients, which allows third-party tools to connect to Elasticsearch as a backend datastore. With this release, Elasticsearch SQL gets generally available. Canvas Canvas that helps users to showcase and present live data from Elasticsearch with pixel-perfect precision, becomes generally available with this release. Kibana localization In this release, Kibana’s first localization, which is now available in simplified Chinese. Kibana also introduces a new localization framework that provides support for additional languages. Functionbeat Functionbeat is a Beat that deploys as a function in serverless computing frameworks, as well as streams, cloud infrastructure logs, and metrics into Elasticsearch. The Functionbeat is now generally available and it supports the AWS Lambda framework and can stream data from CloudWatch Logs, SQS, and Kinesis. Upgrade Assistant The Upgrade Assistant in this release will help users in preparing their existing Elastic Stack environment for the upgrade to 7.0. The Upgrade Assistant includes both APIs and UIs and works as an important cluster checkup tool to help plan the upgrade. It also helps in identifying things like deprecation warnings to enable a smoother upgrade experience. To know more about this release, check out Elastic’s blog post. Microsoft brings PostgreSQL extension and SQL Notebooks functionality to Azure Data Studio Core CPython developer unveils a new project that can analyze his phone’s ‘silent connections’ How to handle backup and recovery with PostgreSQL 11 [Tutorial]  

On Tuesday, Red Hat announced the general availability of RHEL (Red Hat Enterprise Linux) 7.6. RHEL 7.6 is a consistent hybrid cloud foundation for enterprise IT. It is built on an open source innovation, designed to enable organizations to match the pace with emerging cloud-native technologies. It also supports IT operations across enterprise IT’s four footprints. Just three months back the beta version of RHEL 7.6 was released. Red Hat Enterprise Linux 7.6  addresses a range of IT challenges, emphasizes security and compliance, management and automation, and Linux container innovations. Features in RHEL 7.6 RHEL 7.6 solves security concerns IT security has always been a key challenge for many IT departments as it does not get easier in complex hybrid and multi-cloud environments. Red Hat Enterprise Linux 7.6 is the answer to this problem as it introduces a Trusted Platform Module (TPM) 2.0 hardware modules as part of Network Bound Disk Encryption (NBDE). NBDE provides security across networked environments whereas, TPM works on-premise to add an additional layer of security, tying disks to specific physical systems. These two layers of security for hybrid cloud operations help keep information on disks physically more secure. RHEL 7.6 also makes it easier to manage firewalls with improvements to nftables, a packet filtering framework. It also simplifies the configuration of counter-intrusion measures. Updated cryptographic algorithms delivered for RSA and elliptic-curve cryptography (ECC) are enabled by default with RHEL 7.6. This helps the organizations handling sensitive information to match their pace with Federal Information Processing Standards (FIPS) compliance and standards bodies like the National Institute of Standards and Technology (NIST). Management and automation get better Red Hat Enterprise Linux 7.6 helps in making Linux adoption easier for the users as it brings enhancements to the Red Hat Enterprise Linux Web Console, which provides a graphical overview of Red Hat system health and status. RHEL 7.6 has made it easier to find updates on the system summary page. It also provides automated configuration of single sign-on for identity management and a firewall control interface. This makes it easier for security administrators. RHEL 7.6 comes with the Extended Berkeley Packet Filter (eBPF), which provides a safer and efficient mechanism for monitoring activities within the kernel. Soon, it will help in enabling additional performance monitoring and network tracing tools. Red Hat Enterprise Linux 7.6 also provides support for Red Hat Enterprise Linux System Roles which is a collection of Ansible modules. These modules are designed to provide a consistent way to automate and remotely manage Red Hat Enterprise Linux deployments. Each of these modules provides a ready-made automated workflow for handling common and complex tasks, involved in Linux environments. This automation helps to remove the possibilities of human error from these tasks.  This, in turn, frees up the IT teams and lets them focus more on adding business value. Red Hat’s lightweight container toolkit Red Hat Enterprise Linux 7.6 supports the rise of cloud-native technologies by introducing Red Hat’s lightweight container toolkit. This toolkit comprises of CRI-O, Buildah, Skopeo, and now Podman. Each of these tools is built on a fully open source and community-backed technologies. They are based on open standards like the Open Container Initiative (OCI) format. Podman complements Buildah and Skopeo while sharing the same foundations as CRI-O. It enables users to run containers and groups of containers (pods) from a familiar command-line interface, which eliminates the need of a daemon. This, in turn, helps to reduce the complexity in container creation while making it easier for developers to build containers on workstations, in continuous integration/continuous development (CI/CD) systems and within high-performance computing (HPC) or big data scheduling systems. For more information on this release, check out Red Hat’s official website Red Hat Enterprise Linux 7.6 Beta released with focus on security, cloud, and automation Red Hat Enterprise Linux 7.5 (RHEL 7.5) now generally available 4 reasons IBM bought Red Hat for $34 billion

Last week, Amazon Web Services announced the general availability of Amazon Managed Streaming for Apache Kafka (Amazon MSK). Amazon MSK makes it easy for developers to build and run applications based on Apache Kafka without having to manage the underlying infrastructure. It is fully compatible with Apache Kafka that enables customers to easily migrate their on-premises or Amazon Elastic Cloud Compute (Amazon EC2) clusters to Amazon MSK without code changes. Customers can use Apache Kafka for capturing and analyzing real-time data streams from a range of sources, including database logs, IoT devices, financial systems, and website clickstreams. Many customers choose to self-manage their Apache Kafka clusters and they end up spending their time and cost in securing, scaling, patching, and ensuring high availability for Apache Kafka and Apache ZooKeeper clusters. But Amazon MSK offers attributes of Apache Kafka that are combined with the availability, security, and scalability of AWS. Customers can now create Apache Kafka clusters that are designed for high availability that span multiple Availability Zones (AZs) with few clicks. Amazon MSK also monitors the server health and automatically replaces servers when they fail. Customers can now easily scale out cluster storage in the AWS management console to meet changes in demand. Amazon MSK runs the Apache ZooKeeper nodes at no additional cost and provides multiple levels of security for Apache Kafka clusters which include VPC network isolation, AWS Identity and Access Management (IAM), etc. It allows customers to continue to run applications built on Apache Kafka and allow them to use Apache Kafka compatible tools and frameworks. General Manager of Amazon MSK, AWS, Rajesh Sheth, wrote to us in an email, "Customers who are running Apache Kafka have told us they want to spend less time managing infrastructure and more time building applications based on real-time streaming data.” He further added, “Amazon MSK gives these customers the ability to run Apache Kafka without having to worry about managing the underlying hardware, and it gives them an easy way to integrate their Apache Kafka applications with other AWS services. With Amazon MSK, customers can stand up Apache Kafka clusters in minutes instead of weeks, so they can spend more time focusing on the applications that impact their businesses.” Amazon MSK is currently available in the US East (Ohio), US East (N. Virginia), US West (Oregon), EU (Ireland), EU (Frankfurt), EU (Paris), EU (London), Asia Pacific (Singapore), Asia Pacific (Tokyo), and Asia Pacific (Sydney), and will expand to additional AWS Regions in the next year. Amazon rejects all 11 shareholder proposals including the employee-led climate resolution at Annual shareholder meeting Amazon to roll out automated machines for boxing up orders: Thousands of workers’ job at stake Amazon resists public pressure to re-assess its facial recognition business; “failed to act responsibly”, says ACLU

Routine database maintenance is a way of life. Updates keep your business running smoothly and securely. And with a managed service, like Cloud SQL, your databases automatically receive the latest patches and updates, with significantly less downtime. But we get it: Nobody likes downtime, no matter how brief.  That's why we’re pleased to announce that Cloud SQL, our fully managed database service for MySQL, PostgreSQL, and SQL Server, now gives you more control over when your instances undergo routine maintenance. Cloud SQL is introducing maintenance deny period controls. With maintenance deny periods, you can prevent automatic maintenance from occurring during a 90-day time period.  This can be especially useful for the Cloud SQL retail customers about to kick off their busiest time of year, with Black Friday and Cyber Monday just around the corner. This holiday shopping season is a time of peak load that requires heightened focus on infrastructure stability, and any upgrades can put that at risk. By setting a maintenance deny period from mid-October to mid-January, these businesses can prevent planned upgrades from Cloud SQL during this critical time. Understanding Cloud SQL maintenanceBefore describing these new controls, let’s answer a few questions we often hear about the automatic maintenance that Cloud SQL performs. What is automatic maintenance?To keep your databases stable and secure, Cloud SQL automatically patches and updates your database instance (MySQL, Postgres, and SQL Server), including the underlying operating system. To perform maintenance, Cloud SQL must temporarily take your instances offline. What is a maintenance window?Maintenance windows allow you to control when maintenance occurs. Cloud SQL offers maintenance windows to minimize the impact of planned maintenance downtime to your applications and your business.  Defining the maintenance window lets you set the hour and day when an update occurs, such as only when database activity is low (for example, on Saturday at midnight).  Additionally, you can control the order of updates for your instance relative to other instances in the same project (“Earlier” or “Later”). Earlier timing is useful for test instances, allowing you to see the effects of an update before it reaches your production instances.  What are the new maintenance deny period controls?You can now set a single deny period, configurable from 1 to 90 days, each year. During the deny period, Cloud SQL will not perform maintenance that causes downtime on your database instance. Deny periods can be set to reduce the likelihood of downtime during the busy holiday season, your next product launch, end of quarter financial reporting, or any other important time for your business. Paired with Cloud SQL’s existing maintenance notification and rescheduling functionality, deny periods give you even more flexibility and control. After receiving a notification of upcoming maintenance, you can reschedule ad hoc, or if you want to prevent maintenance longer, set a deny period.  Getting started with Cloud SQL’s new maintenance controlReview our documentation to learn more about maintenance deny periods and, when you're ready, start configuring them for your database instances.  What’s next for Cloud SQLSupport for additional maintenance controls continues to be a top request from users. These new deny periods are an addition to the list of existing maintenance controls for Cloud SQL. Have more ideas? Let us know what other features and capabilities you need with our Issue Tracker and by joining the Cloud SQL discussion group. We’re glad you’re along for the ride, and we look forward to your feedback!

Cloudflare’s cloud computing platform Workers doesn’t use containers or virtual machines to deploy computing. Workers allows users to build serverless applications on Cloudflare's data centers. It provides a lightweight JavaScript execution environment to augment existing applications or create entirely new ones without having to configure or maintain infrastructure. Why did Cloudflare create workers? Cloudflare provided limited features and options that developers could build in-house. There was not much flexibility for customers to build features themselves. To enable users to write code on their servers deployed around the world, they had to allow untrusted code to run, with low overhead. This needed to process millions of requests per second and that too at a very fast speed. Customers couldn’t write their own code without the team’s supervision. It would be expensive to use traditional virtualization and container technologies like Kubernetes let alone run thousands of Kubernetes pod at 155 data centers of Cloudflare would be resource intensive. Enter Cloudflare’s ‘Workers’ to solve these issues. Features of Workers #1 ‘Isolates’- Run code from multiple customers ‘Isolates’ is a technology built by Google Chrome team to power the Javascript engine in that browser, V8: Isolates.  These are lightweight contexts that group variables, with the code allowed to mutate them. A single process can run hundreds or thousands of Isolates, while easily  switching between them. Thus, Isolates make it possible to run untrusted code from different customers within a single operating system process. They start real quick (Any given Isolate can start around a hundred times faster than a Node process on a machine) and do not allow one Isolate to access the memory of another. #2 Cold Starts Workers facilitate the concept of ‘cold start’ when a new copy of code has to be started on a machine. In the Lambda world, this means spinning up a new containerized process which can delay requests  for as much as ten seconds ending up in a terrible user experience. A Lambda can only process one single request at a time. A new Lambda has to be cold-started every time an additional concurrent request is recieved. If a Lambda doesn’t get a request soon enough, it will be shut down and it all starts again.  Since Workers don’t have to start a process, Isolates start in 5 milliseconds. It scales and deploys quickly, entirely upgrading existing Serverless technologies. #3 Context Switching A normal context switch performed by an OS can take as much as 100 microseconds. When multiplied by all the Node, Python or Go processes running on average Lambda servers, this leads to a heavy overhead. This splits the CPUs power between running the customer’s code and switching between processes. An Isolate-based system runs all of the code in a single process which means there are no expensive context switches. The machine can invest virtually all of its time running your code. #4 Memory The V8 was designed to be multi-tenant. It runs the code from the many tabs in a user’s browser in isolated environments within a single process. Since memory is often the highest cost of running a customer’s code, V8 lowers it and dramatically changes the cost economics. #5 Security It is not safe to run code from multiple customers within the same process. Testing, fuzzing, penetration testing, and bounties are required to build a truly secure system of that complexity. The open-source nature of V8 helps in creating aanisolation layer that helps Cloudflare take care of the security aspect. Cloudlfare’s Workers also allows users to build responses from multiple background service requests either to the Cloudflare cache, application origin, or third party APIs. They can build conditional responses for inbound requests to assess and subsequently block or reroute malicious or unauthorized requests. All of this at just a third of what AWS costs, remarked an astute Twitter observer. https://twitter.com/seldo/status/1061461318765555713 Running code through WebAssembly One of the disadvantages of using Workers is that, since it is an Isolate-based system, it cannot run arbitrary compiled code. Users have to either write their code in Javascript, or a language which targets WebAssembly (eg. Go or Rust). Also, if a user cannot recompile their processes, they won’t be able to run them in an Isolate. This has been nicely summarised in the above mentioned tweet. He notes that WebAssembly modules are already in the npm registry and it creates the potential for npm to become the dependency management solution for every programming language. He mentions that the “availability of open source libraries to achieve the task at hand is the primary reason people pick a programming language”. This leads us to the question of “How does software development change when you can use any library anytime?” You can head over to the Cloudflare blog to understand more about containerless cloud computing. Cloudflare Workers KV, a distributed native key-value store for Cloudflare Workers Cloudflare’s decentralized vision of the web: InterPlanetary File System (IPFS) Gateway to create distributed websites  

Today, the team at Kong, the creators of the API and service lifecycle management platform for modern architectures announced the release of Kuma, a new open-source project.  Kuma is based on the open-source Envoy proxy that addresses limitations of first-generation service mesh technologies by seamlessly managing services on the network. The first-generation meshes didn't have a mature control plane, and later on, when they provided a control plane, it wasn’t easy to use them as they were hard to deploy. Kuma is easy to use and enables rapid adoption of mesh. Also Read: Kong CTO Marco Palladino on how the platform is paving the way for microservices adoption [Interview] Features of Kuma Runs on all the platforms Kuma can run on any platform including Kubernetes, containers, virtual machines, and legacy environments. It also includes a fast data plane as well as an advanced control plane that makes it easier to use.  It is reliable The initial service mesh solutions were not flexible and it was difficult to use them. Kuma ensures reliability by automating the process of securing the underlying network.  Support for all the environments Kuma has support for all the environments in the organization, so the existing applications can still be used in their traditional environments. This provides comprehensive coverage across an organization. Couples a fast data plane using control plane Kuma couples a fast data plane with a control plane that helps users to set permissions, routing rules and expose metrics with just a few commands. Tracing and logging Kuma helps users to implement tracing and logging and analyze metrics for rapid debugging. Routing and Control  Kuma provides traffic control capabilities including circuit breakers and health checks in order to enhance L4 (Layer 4) routing. Marco Palladino, CTO and co-founder of Kong, said, “We now have more microservices talking to each other and connectivity between them is the most unreliable piece: prone to failures, insecure and hard to observe.”  Palladino further added, “It was important for us to make Kuma very easy to get started with on both Kubernetes and VM environments, so developers can start using service mesh immediately even if their organization hasn’t fully moved to Kubernetes yet, providing a smooth path to containerized applications and to Kubernetes itself. We are thrilled to be open-sourcing Kuma and extending the adoption of Envoy, and we will continue to contribute back to the Envoy project like we have done in the past. Just as Kong transformed and modernized API Gateways with open-source Kong, we are now doing that for service mesh with Kuma.” The Kuma platform will be on display during the second annual Kong Summit, which is to be held on October 2-3, 2019. Other interesting news in Cloud and Networking  Kubernetes releases etcd v3.4 with better backend storage, improved raft voting process, new raft non-voting member and more VMworld 2019: VMware Tanzu on Kubernetes, new hybrid cloud offerings, collaboration with multi cloud platforms and more! The Accelerate State of DevOps 2019 Report: Key findings, scaling strategies and proposed performance & productivity models

On Tuesday, Google announced of being one of the founding members of the newly-formed Continuous Delivery Foundation (CDF). As a part of its membership, Google will be contributing to two projects namely Spinnaker and Tekton. About Continuous Delivery Foundation The formation of CDF was announced at the Linux Foundation Open Source Leadership Summit on Tuesday. CDF will act as a “vendor-neutral home” for some of the most important open source projects for continuous delivery and specifications to speed up the release pipeline process. https://twitter.com/linuxfoundation/status/1105515314899492864 The existing CI/CD ecosystem is heavily fragmented, which makes it difficult for developers and companies to decide on particular tooling for their projects. Also, DevOps practitioners often find it very challenging to gather guidance information on software delivery best practices. CDF was formed to make CI/CD tooling easier and define the best practices and guidelines that will enable application developers to deliver better and more secure software at speed. CDF is currently hosting some of the most popularly used CI/CD tools including Jenkins, Jenkins X, Spinnaker, and Tekton. The foundation is backed by 20+ founding members which include Alauda, Alibaba, Anchore, Armory.io, Atos, Autodesk, Capital One, CircleCI, CloudBees, DeployHub, GitLab, Google, HSBC, Huawei, IBM, JFrog, Netflix, Puppet, Rancher, Red Hat, SAP, Snyk, and SumoLogic. Why Google joined CDF? Google as a part of this foundation will be working on Spinnaker and Tekton. Originally created by Netflix and jointly led by Netflix and Google, Spinnaker is an open source, multi-cloud delivery platform. It comes with various features for making continuous delivery reliable including support for advanced deployment strategies, an open source canary analysis service named Kayenta, and more. The Spinnaker’s user community has great experience in the continuous delivery domain, and by joining CDF Google aims to share that expertise with the broader community. Tekton is a set of shared, open source components for building CI/CD systems. It allows you to build, test, and deploy applications across multiple environments such as virtual machines, serverless, Kubernetes, or Firebase. In the next few months, we can expect to see support for results and event triggering in Tekton. Google is also planning to work with CI/CD vendors to build an ecosystem of components that will allow users to use Tekton with existing tools like Jenkins X, Kubernetes native, and others. Dan Lorenc, Staff Software Engineer at Google Cloud, sharing Google’s motivation behind joining CDF said, “Continuous Delivery is a critical part of modern software development, but today space is heavily fragmented. The Tekton project addresses this problem by working with the open source community and other leading vendors to collaborate on the modernization of CI/CD infrastructure.” Kim Lewandowski, Product Manager at Google Cloud, said, “The ability to deploy code securely and as fast as possible is top of mind for developers across the industry. Only through best practices and industry-led specifications will developers realize a reliable and portable way to take advantage of continuous delivery solutions. Google is excited to be a founding member of the CDF and to work with the community to foster innovation for deploying software anywhere.” To know more, check out the official announcement at the Google Open Source blog. Google Cloud Console Incident Resolved! Cloudflare takes a step towards transparency by expanding its government warrant canaries Google to acquire cloud data migration start-up ‘Alooma’

On Tuesday, Google engineer Brendan Higgins announced an experimental set of 31 patches by introducing KUnit as a new Linux kernel unit testing framework to help preserve and improve the quality of the kernel's code. KUnit is a lightweight unit testing and mocking framework designed for the Linux kernel. Unit tests necessarily have finer granularity, they are able to test all code paths easily solving the classic problem of difficulty in exercising error handling code. KUnit is heavily inspired by JUnit, Python's unittest.mock, and Googletest/Googlemock for C++. KUnit provides facilities for defining unit test cases, grouping related test cases into test suites, providing common infrastructure for running tests, mocking, spying, and much more. Brenden writes, "It does not require installing the kernel on a test machine or in a VM and does not require tests to be written in userspace running on a host kernel. Additionally, KUnit is fast: From invocation to completion KUnit can run several dozen tests in under a second. Currently, the entire KUnit test suite for KUnit runs in under a second from the initial invocation (build time excluded)." When asked if KUnit will replace the other testing frameworks for the Linux Kernel, Brenden denied it,  saying, “Most existing tests for the Linux kernel are end-to-end tests, which have their place. A well tested system has lots of unit tests, a reasonable number of integration tests, and some end-to-end tests. KUnit is just trying to address the unit test space which is currently not being addressed.” To know more about KUnit in detail, read Brendan Higgins’ email threads. What role does Linux play in securing Android devices? bpftrace, a DTrace like tool for Linux now open source Linux drops Code of Conflict and adopts new Code of Conduct

On January 29, Microsoft Cloud services including Microsoft Azure, Office 365, and Dynamics 365 suffered a major outage. This resulted in customers experiencing intermittent access to Office 365 and also deleting several database records. This comes just after a major outage that prevented Microsoft 365 users from accessing their emails for an entire day in Europe. https://twitter.com/AzureSupport/status/1090359445241061376 Users who were already logged into Microsoft services weren’t affected; however, those that were trying to log into new sessions were not able to do so. How did this Microsoft Azure outage happen? According to Microsoft, the preliminary reason behind this outage was a DNS issue with CenturyLink, an external DNS provider. Microsoft Azure’s status page read, “Engineers identified a DNS issue with an external DNS provider”. CenturyLink, in a statement, mentioned that their DNS services experienced disruption due to a software defect, which affected connectivity to a customer’s cloud resources. Along with authentication issues, this outage also caused the deletion of users’ live data stored in Transparent Data Encryption (TDE) databases in Microsoft Azure. TDE databases encrypt information dynamically and decrypt them when customers access it. As the data is stored in encrypted form, it prevents intruders from accessing the database. For encryption, many Azure users store their own encryption keys in Microsoft’s Key Vault encryption key management system. The deletion was triggered by a script that automatically drops TDE database tables when corresponding keys can no longer be accessed in the Key Vault. Microsoft was able to restore the tables from a five-minute snapshot backup. But, those transactions that customers had processed within five minutes of the table drop were expected to raise a support ticket asking for the database copy. Read more about Microsoft’s Azure outage in detail on ZDNet. Microsoft announces Internet Explorer 10 will reach end-of-life by January 2020 Outage in the Microsoft 365 and Gmail made users unable to log into their accounts Microsoft Office 365 now available on the Mac App Store

Cloudflare announced yesterday that it is adding Warp, a free VPN to the 1.1.1.1 DNS resolver app. Cloudflare team states that it began its plans to integrate 1.1.1.1 app with warp performance and security tech, about two years ago. The 1.1.1.1 app was released in November last year for iOS and Android. The mobile app included features such as VPN support that helped move the mobile traffic towards 1.1.1.1 DNS servers, thereby, helping improve speeds. Now with warp integration, 1.1.1.1 app will speed up mobile data using Cloudflare network to resolve DNS queries at a faster pace.  With Warp, all the unencrypted connections are encrypted automatically by default. Also, Warp comes with end-to-end encryption and doesn’t require users to install a root certificate to observe the encrypted Internet traffic. For cases when you browse the unencrypted Internet through Warp, Cloudflare’s network can cache and compress content to improve performance and decrease your data usage and mobile carrier bill. “In the 1.1.1.1 App, if users decide to enable Warp, instead of just DNS queries being secured and optimized, all Internet traffic is secured and optimized. In other words, Warp is the VPN for people who don't know what V.P.N. stands for”, states the Cloudflare team. Apart from that, Warp also offers excellent performance and reliability. Warp is built around a UDP-based protocol that has been optimized for the mobile Internet. Warp also makes use of Cloudflare’s massive global network and allows Warp to connect with servers within milliseconds. Moreover, Warp has been tested to show that it increases internet performance. Another factor is reliability which has also significantly improved. Warp is not as capable of eliminating mobile dead spots, but it is very efficient at recovering from loss. Warp doesn’t increase your battery usage as it is built around WireGuard, a new and efficient VPN protocol. The basic version of Warp has been added as a free option with the 1.1.1.1 app for free. However, Cloudflare team will be charging for Warp+, a premium version of Warp, that will be even faster with Argo technology. A low monthly fee will be charged for Warp+ that will vary based on different regions. Also, the 1.1.1.1 App with Warp will have all the privacy protections launched formerly with the 1.1.1.1 app. Cloudflare team states that 1.1.1.1 app with warp is still under works, and although sign-ups for Warp aren’t open yet, Cloudflare has started a waiting list where you can “claim your place” by downloading the 1.1.1.1 app or by updating the existing app. Once the service is available, you’ll be notified. “Our whole team is proud that today, for the first time, we’ve extended the scope of that mission meaningfully to the billions of other people who use the Internet every day”, states the Cloudflare team. For more information, check out the official Warp blog post. Cloudflare takes a step towards transparency by expanding its government warrant canaries Cloudflare raises $150M with Franklin Templeton leading the latest round of funding workers.dev will soon allow users to deploy their Cloudflare Workers to a subdomain of their choice

Yesterday Twitter went down across major parts of the world including the US and the UK. Twitter users reported being unable to access the platform on web and mobile devices. The outage lasted on the site for approximately an hour. According to DownDetector.com, the site began experiencing major issues at 2:46pm EST, with problems being reported from users attempting to access Twitter through its website, iPhone or iPad app and via Android devices. While the majority of problems being reported from Twitter were website issues (51%), nearly 30% were from iPhone and iPad app usage and another 18% from Android users, as per the outage report. Twitter acknowledged that the platform was experiencing issues on its status page shortly after the first outages were reported online. The company listed the status as “investigating” and noted a service disruption was causing the seemingly global issue. “We are currently investigating issues people are having accessing Twitter,” the statement read. “We will keep you updated on what's happening.” This month has experienced several high-profile outages among social networks. Facebook and Instagram experienced a day-long outage affecting large parts of the world on July 3rd. LinkedIn went down for several hours on Wednesday. Cloudfare suffered two major outages in the span of two weeks this month. One was due to an internal software glitch and another was caused when Verizon accidentally rerouted IP packages after it wrongly accepted a network misconfiguration from a small ISP in Pennsylvania, USA. Reddit was experiencing outages on its website and app earlier in the day, but appeared to be back up and running for most users an hour before Twitter went down, according to DownDetector.com. In March, Facebook and its family of apps experience a 14 hour long outage which was reasoned as server config change issue. Twitter site then began operating normally nearly an hour later at approximately 3:45pm EST. The users on Twitter joked saying they were "all censored for the last hour" when the site eventually was back up and running. On the status page of the outage report Twitter said that the outage was caused due to “an internal configuration change, which we're now fixing.” “Some people may be able to access Twitter again and we're working to make sure Twitter is available to everyone as quickly as possible,” the company said in a follow up statement. https://twitter.com/TwitterSupport/status/1149412158121267200 On Hacker News too users discussed about number of outages in major tech companies and why is this happening. One of the user comments reads, “Ok, this is too many high-profile, apparently unrelated outages in the last month to be completely a coincidence. Hypotheses: 1) software complexity is escalating over time, and logically will continue to until something makes it stop. It has now reached the point where even large companies cannot maintain high reliability. 2) internet volume is continually increasing over time, and periodically we hit a point where there are just too many pieces required to make it work (until some change the infrastructure solves that). We had such a point when dialup was no longer enough, and we solved that with fiber. Now we have a chokepoint somewhere else in the system, and it will require a different infrastructure change 3) Russia or China or Iran or somebody is f*(#ing with us, to see what they are able to break if they needed to, if they need to apply leverage to, for example, get sanctions lifted 4) Just a series of unconnected errors at big companies 5) Other possibilities?” On this comment another user adds, “I work at Facebook. I worked at Twitter. I worked at CloudFlare. The answer is nothing other than #4. #1 has the right premise but the wrong conclusion. Software complexity will continue escalating until it drops by either commoditization or redefining problems. Companies at the scale of FAANG(+T) continually accumulate tech debt in pockets and they eventually become the biggest threats to availability. Not the new shiny things. The sinusoidal pattern of exposure will continue.” Facebook, Instagram and WhatsApp suffered a major outage yesterday; people had trouble uploading and sending media files Facebook family of apps hits 14 hours outage, longest in its history How Verizon and a BGP Optimizer caused a major internet outage affecting Amazon, Facebook, CloudFlare among others

Google Cloud Next conference in San Francisco in July 2018 saw some exciting new developments in the field of serverless technology. The company is giving development teams the ability to build apps without worrying about managing servers with their new serverless technology. Bringing the best of both worlds: Serverless and containers, Google announced that Cloud Functions is now generally available and ready for production use. Here is a list of the all-new features that developers can watch out for- #1 Write Cloud Functions using  Node 8, Python 3.7 With support for async/await and a new function signature, you can now write Cloud Functions using Node 8. Dealing with multiple asynchronous operations is now easier thanks to Cloud Functions that provide data and context. You can use the await keyword to await the results of asynchronous operations. Python 3.7 can also be used to write Cloud Functions.  Similar to Node, you get data and context for background functions, and request for HTTP. Python HTTP functions are based on the popular Flask microframework. Flask allows you to get set up really fast. The requests are based on flask.Request and the responses just need to be compatible with flask.make_response. As with Node, you get data (dict) with Python background functions and context (google.cloud.functions.Context). To signal completion, you just need to return from your function or raise an exception and Stackdriver error handling will kick in. And, similarly to Node (package.json), Cloud Functions will automatically do the installation of all of your Python dependencies (requirements.txt) and build in the cloud. You can have a look at the code differences between Node 6 and Node 8 behavior and at a Flask request on the Google Cloud website. #2 Cloud Functions is now out  for Firebase Cloud Functions for Firebase is also generally available. It has full support for Node 8, including ECMAScript 2017 and async/await. The additional granular controls include support  for runtime configuration options, including region, memory, and timeout. Thus allowing you to refine the behavior of your applications. You can find more details from the Firebase documentation. Flexibility for the application stack now stands improved. Firebase events (Analytics, Firestore, Realtime Database, Authentication) are directly available in the Cloud Functions Console on GCP. You can now trigger your functions in response to the Firebase events directly from your GCP project. #3 Run headless Chrome by accessing system libraries Google Cloud functions have also broadened the scope of libraries available by rebasing the underlying Cloud Functions operating system onto Ubuntu 18.04 LTS. Access to system libraries such as ffmpeg and libcairo2 is now available- in addition to imagemagick- as well as everything required to run headless Chrome. For example, you can now process videos and take web page screenshots in Chrome from within Cloud Functions. #4 Set environment variables You can now pass configuration to your functions by specifying key-value pairs that are bound to a function. The catch being, these pairs don’t have to exist in your source code. Environment variables are set at the deploy time using the --set-env-vars argument. These are then injected into the environment during execution time. You can find more details on the Google cloud webpage. #5 Cloud SQL direct connect Now connect Cloud Functions to Cloud SQL instances through a fully managed secure direct connection.  Explore more from the official documentation. What to expect next in Google Cloud Functions? Apart from these, Google also promises a range of features to be released in the future. These include: 1. Scaling controls This will be used to limit the number of instances on a per-function basis thus limiting traffic. Sudden traffic surge scenarios will , therefore,come under control when Cloud Functions rapidly scales up and overloads a database or general prioritization based on the importance of various parts of your system. 2. Serverless scheduling You’ll be able to schedule Cloud Functions down to one-minute intervals invoked via HTTP(S) or Pub/Sub. This allows you to execute Cloud Functions on a repeating schedule. Tasks like daily report generation or regularly processing dead letter queues will now pick up speed! 3. Compute Engine VM Access Now connect to Compute Engine VMs running on a private network using --connected-vpc option. This provides a direct connection to compute resources on an internal IP address range. 4. IAM Security Control The new Cloud Functions Invoker IAM role allows you to add IAM security to this URL. You can control who can invoke the function using the same security controls as used in Cloud Platform 5. Serverless containers With serverless containers, Google provides the same infrastructure that powers Cloud Functions. Users will now be able to simply provide a Docker image as input. This will allow them to deploy arbitrary runtimes and arbitrary system libraries on arbitrary Linux distributions This will be done while still retaining the same serverless characteristics as Cloud Functions. You can find detailed information about the updated services on Google Cloud’s Official page. Google Cloud Next: Fei-Fei Li reveals new AI tools for developers Google Cloud Launches Blockchain Toolkit to help developers build apps easily Zeit releases Serverless Docker in beta

Kubernetes has announced their first release of 2018: Kubernetes 1.10. This release majorly focuses on stabilizing 3 key areas which include storage, security, and networking. Kubernetes is an open-source system, initially designed by Google and at present is maintained by the Cloud Native Computing Foundation, which helps in automating deployment, scaling, and management of containerized applications. Storage - CSI and Local Storage move to beta: In this version, you will find: The Container Storage Interface (CSI) moves to beta. One can install new volume plugins similar to deploying a pod. This helps third-party storage providers to develop independent solutions outside the core Kubernetes codebase. Local storage management has also progressed to beta, enabling locally attached storage available as a persistent volume source. This assures lower-cost and higher performance for distributed file systems and databases. Security - External credential providers (alpha): Complementing the Cloud Controller Manager feature added in 1.9 Kubernetes has extended its feature with the addition of External credential providers in 1.10. This enables Cloud providers and other platform developers to release binary plugins to handle authentication for specific cloud-provider Identity Access Management services. Networking - CoreDNS as a DNS provider (beta): Kubernetes now provides the ability to switch the DNS service to CoreDNS during installation. CoreDNS is a single process that can now supports more use cases. To get a complete list of additional features of this release visit the Changelog. Check out other related posts: The key differences between Kubernetes and Docker Swarm Apache Spark 2.3 now has native Kubernetes support! OpenShift 3.9 released ahead of planned schedule