Tech Guides

Mixed Reality has become a disruptive force that is bridging the gap between reality and imagination. With AI, it is now poised to change the world as we see it! The global mixed reality market is expected to reach USD 6.86 billion by 2024. Mixed Reality has found application in not just the obvious gaming and entertainment industries but also has great potential in business and other industries ranging from manufacturing, travel, and medicine to advertising. Maybe that is why the biggest names in tech are battling it out to capture the MR market with their devices - Microsoft HoloLens, GoogleGlass 2.0, Meta2 handsets to name a few. Incorporating Artificial Intelligence is their next step towards MR market domination. So what’s all the hype about MR and how can AI take it to the next level? Through the looking glass: Understanding Mixed Reality Mixed reality is essentially a fantastic concoction of virtual reality (a virtual world with virtual objects) and augmented reality (the real world with digital information). This means virtual objects are overlaid in the real world and mixed reality enables the person who is experiencing the MR environment to perceive virtual objects as ”real ones”. While in augmented reality it is easy to break the illusion and recognize that the objects are not real (Hello Pokemon Go!), in Mixed Reality, it is harder to break the illusion as virtual objects behave like real-world objects. So when you lean in close or interact with a virtual object in MR, it gets closer in a way a real object would. The MR experience is made possible with mixed reality devices that are typically lightweight and wearable. They are generally equipped with front-mounted cameras to recognize the distinctive features of the real world (such as objects and walls) and blend them with the virtual reality as seen through the headset. They also include a processor for processing the information relayed by an array of sensors embedded in the headset. These processors run the algorithms used for pattern recognition on a cloud-based server. These devices then use a projector for displaying virtual images in real environments which are finally reflected to the eye with the help of beam-splitting technology. All this sounds magical already, what can AI do for MR to top this? Curiouser and curiouser: The AI-powered Mixed Reality Mixed Reality and Artificial Intelligence are two powerful technology tools. The convergence of the two means a seamless immersive experience for users that blends the virtual and physical reality. Mixed Reality devices already enable interaction of virtual holograms in the physical environment and thereby combine virtual worlds with reality. But most MR devices require a large number of calculations and adjustments to accurately determine the position of a virtual object in a real-world scenario. They then apply rules and logic to those objects to make them behave like real-world objects. As these computations happen on the cloud, the results have perceivable time lag which comes in the way of giving the user a truly immersive experience. Also, user mobility is restricted due to current device limitations. Recently there has been a rise of the AI coprocessor in Mixed Reality devices. The announcement of Microsoft’s HoloLens 2 project, an upgrade to the existing MR device which now includes an AI coprocessor is a case in point. By using AI chips for computing, the above calculations, for example, MR devices will deliver high precision results faster. It means algorithms and calculations can run instantaneously without the need for data to be sent to/from a cloud. Having the data locally on your headset will eliminate time lag, thereby creating more real-time immersive experiences. In other words, as the visual data is analyzed directly on the device and computationally-exhaustive tasks are performed close to the data source, the enhanced processing speed results in quicker performance. Since the data remains on your headset always, fewer computations are needed to be performed on the cloud, hence the data is more secure. Using an AI chip also allows flexible implementation of deep neural networks. They help in automating complex calculations such as depth perception estimations and generally provide a better understanding of the environment to the MR devices. Generative models from Deep Learning can be used to generate believable virtual characters (avatars) in the real world. Images can also be more intelligently compressed with AI techniques. This would enable faster transmission over wireless networks. Motion capture techniques are now employing AI functionalities such as phase-functioned neural networks and self-teaching AI. They use machine learning techniques to combine a vast library of stored movements and combine and fit them into new characters. By using AI-powered Mixed Reality devices, the plan is to provide a more realistic experience which is fast and provides more mobility. The ultimate goal is to build AI-powered mixed reality devices that are intelligent and self-learning. Follow the white rabbit: Applications of AI-power Mixed Reality  Let us look at various sectors where Artificially Intelligent Mixed Reality has started finding traction. Gaming and Entertainment In the field of gaming, procedural content generation techniques allow automatic generation of Mixed Reality games (as opposed to manual creation by game designers) by encoding elements such as individual structures, enemies etc with their relationships. Artificial Intelligence enhances PCG algorithms in object identification and in recognizing other relationships between the real and virtual objects. Deep learning techniques can be used for tasks like super resolution, photo to texture mapping, and texture multiplication. Healthcare and Surgical Procedures AI-powered mixed reality tech has also found its use in the field of Healthcare and surgical operations. Scopis has announced a mixed reality surgical navigation system that uses the Microsoft HoloLens for spinal surgery applications. It employs image recognition and manipulation techniques which allow the surgeon to see both the patient and a superimposed image of the pedicle screws (used for vertebrae fixation surgeries) for surgical procedures. Retail Retail is another sector which is under the spell of this AI infused MR tech. DigitalBridge, a mixed-reality company uses mixed reality, artificial intelligence, and deep learning to create a platform that allows consumers to virtually try on home decor products before buying them. Image and Video Manipulation AI algorithms and MR techniques can also enrich video and image manipulation. As we speak, Microsoft is readying the release of Microsoft Remix 3D service. This software adds “mixed reality” digital images and animations to videos. It keeps the digital content in the same position in relation to the real objects using image recognition, computer vision, and AI algorithms. Military and Defence AI-powered Mixed Reality is also finding use in the defense sector, where MR training simulations controlled by artificially intelligent software combine real people and physical environments with a virtual setup. Construction and Homebuilding Builders can visualize their options in life-sized models with MR devices. With AI, they can leave virtual messages or videos at key locations to keep other technicians and architects up to date when they’re away. Using MR and AI techniques, an architect can call a remote expert into the virtual environment if need be, and virtual assistants can be utilized for further assistance. COINS is a construction and homebuilding organization which uses AI-powered Mixed Reality devices. They are collaborating with Sketchup for virtual messaging and 3D modeling and Microsoft for HoloLens and Skype Chatbot assistance. Industrial Farming Machine learning algorithms can be used to study a sensor-enabled field of crop and record the growth, requirements and anticipate future needs. An MR device can then provide a means to interact with the plants and analyze present conditions all the while adjusting future needs. Infosys’ Plant.IO is one such digital farm which when combined with the power of an MR device can overlay virtual objects over a real-world scenario. Conclusion Through these examples, we can see the rapid adoption of AI-powered Mixed Reality recipes across diverse fields enabled by the rise of AI chips and the employment of more exhaustive computations with complex machine learning and deep learning algorithms. The next milestone is to see the rise of a Mixed Reality environment which is completely immersive and untethered. This would be made possible by the adoption of more complex AI techniques and advances made in the field of AI both in hardware and software. Instead of voice or search based commands in the MR environments, AI techniques will be used to harness eye and body gestures. As MR devices become smaller and mobile, AI-powered mixed reality will also give rise to intelligent application development incorporating mixed reality through the phone’s camera lens. AI technologies would thus help expand the scope of MR not only as an interesting tool, with applications in gaming and entertainment, but also as a practical and useful approach for how we see, interact and learn from our environment.

This post explores the functional aspect of both Elm and TypeScript, providing a better understanding of both programming languages. Elm and TypeScript both use JavaScript as a compile target, but they do have major differences, which we’ll examine here. Elm Elm is a functional language with a static type analysis and strong inferred type system. In other words, the Elm compiler only runs type checks during compilation and can predict types for all expressions without having explicit type annotations. This guarantees the absence of runtime exceptions due to type mismatch. Elm supports generic types and structural type definitions. TypeScript TypeScript is a superset of JavaScript, but unlike Elm it is a multi-paradigm language with a strong imperative part and a significantly weaker functional part. It also has static type checker and supports generic types and structural typing. TypeScript also has type inference, and although it is not as reliable as Elm’s, it’s still quite useful. Functions Let’s see how both languages handle functions. Elm Functions is where Elm shines. Function declarations and lambda expressions are supported. Type definitions are simple and robust. It is worth mentioning that lambda expressions can only have a type definition when used as a return value from a function declaration. -- Function declaration add : Int -> Int -> Int add x y = x + y -- Function expression x y -> x + y -- Type definition for lambda expression add : Int -> Int -> Int add = x y -> x + y TypeScript TypeScript can offer the standard set from JavaScript: function declaration, lambda expression and arrow functions. // Function declaration function add(x: number, y: number): number { return x + y; } // Function expression const add = function (x: number, y: number): number { return x + y; } // Arrow function expresion const add = (x: number, y: number): number => { return x + y; } Structural typings Let’s see how they both stack up. Elm The structural type annotation definition is available for Tuples and Records. Records are primarily used for modelling abstract data types. Type aliases can be used as value constructors of the said data type. type alias User = { name: String , surname: Maybe String } displayName : User -> String displayName { name, surname } = case surname of Just value -> name ++ " " ++ value Nothing -> name displayName (User "John" (Just "Doe")) -- John Doe TypeScript Structural typing is done with interfaces, so it is possible to deconstruct values. It is also worth mentioning that classes can extend interfaces. interface User { name: string; surname?: string; } function displayName ({ name, surname="" }:User ):string { return name + ' ' + surname } console.log(displayName({ name: 'John', surname: 'Doe' })) Union Types Elm and TypeScript both handle union types. Elm Union Types in Elm are the essential tool for defining abstract data structures with dynamic nature. Let’s have a look at the Maybe type. The union describes two possible states—the closest you can get to this in TypeScript is an optional value.  Type variable a points out that a stored value might belong to any data type. type Maybe a = Just a | Nothing This might be useful and make a lot of sense in a functional language. Here is an example of how you might use it. If you want to crash the program explicitly, then there is a logical error in the state of the application. displayName userName = case userName of Just name -> String.toUpper name Nothing -> Debug.crash "Name is missing" displayName (Just "Bond") -- BOND displayName Nothing -- Cause run-time error explicitly TypeScript Union types in TypeScript are currently referred to as Discriminated Union. Here is an example of using union todefine a list of available actions for a dispatcher. interface ActionSendId { name: "ID", data: number } interface ActionSendName { name: "NAME", data: string } function dispatch(action:ActionSendId | ActionSendName):void { switch (action.name) { case "ID": sendId(action.data) break; case "NAME": sendName(action.data) break; default: break; } } Interoperation with JavaScript I will only focus on aspects that are affected by the implementation of a type system. Elm You can pass values once from JavaScript to Elm during the initialization process. There is a special type of program for that called programWithFlags. The Elm application can inter-operate with JavaScript directly using special interfaces, called ports. It implementsa Signal pattern. Sending a value of an unexpected type will cause an error. During HTTP communication, you have to decode and encode values using Json.Decode and Json.Encode. During DOM Events, you can use Json.Decode to retrieve values from an Event object. TypeScript Using JavaScript with TypeScript is quite possible, but you will have to specify type definitions for the code. As an option, you can use a special type: any. The any type is a powerful way to work with existing JavaScript, allowing you to gradually optin and optout of typechecking during compilation. As an alternative, you might have to provide typing files. Conclusion Both Elm and TypeScript have their strengths and weaknesses, but despite all of the differences, both type systems give you similar benefits. Elm has the upper hand with type inference, thanks to the purely functional nature of the language and strict inter-operation with the outside world. TypeScript does not guarantee type-error-free runtime, but it’s easier pickup and very intuitive if you have JavaScript or a C# background. About the author Eduard Kyvenko is a frontend lead at Dinero. He has been working with Elm for over half a year and has built a tax return and financial statements app for Dinero. You can find him on GitHub at @halfzebra.

The retail giant Amazon plans to reinvent in-store shopping in much the same way it revolutionized online shopping. Amazon’s cashierless stores (3000 of which you can expect by 2021) give a glimpse into what the future of eCommerce could look like with automation. eCommerce automation involves combining the right eCommerce software with the right processes and the right people to streamline and automate the order lifecycle. This reduces the complexity and redundancy of many tasks that an eCommerce retailer typically faces from the time a customer places her order until the time it is delivered to them. Let’s look at why eCommerce retailers should prioritize automation in 2019. 1. Augmented Customer Experience + Personalization A PwC study titled “Experience is Everything” suggests that 42% of consumers would pay more for a friendly, welcoming experience. Way back in 2015, Gartner predicted that nearly 50% of companies would be implementing changes in their business model in order to augment customer experience. This is especially true for eCommerce, and automation in eCommerce certainly represents one of these changes. Customization and personalization of services are a huge boost for customer experience. A BCG report revealed that retailers who implemented personalization strategies saw a 6-10% boost in their sales. How can automation help? To start with, you can automate your email marketing campaigns and make them more personalized by adding recommendations, discount codes, and more. 2.  Fraud Prevention The scope for fraud on the Internet is huge. According to the October 2017 Global Fraud Index, Account Takeover fraud cost online retailers a whopping $3.3 billion dollars in Q2 2017 alone. Additionally, the average transaction rate for eCommerce fraud has also been on this rise. eCommerce retailers have been using a number of fraud prevention tools such as address verification service, CVN (card verification number), credit history check, and more to verify the buyer’s identity. An eCommerce tool equipped with Machine Learning capabilities such as Shuup, can detect fraudulent activity and effectively run through thousands of checks in the blink of an eye. Automating order handling can ensure that these preliminary checks are carried out without fail, in addition to carrying out specific checks to assess the riskiness of a particular order. Depending on the nature and scale of the enterprise, different retailers would want to set different thresholds for fraud detection, and eCommerce automation makes that possible. If a medium-risk transaction is detected, the system can be automated to notify the finance department for immediate review, whereas high-risk transactions can be canceled immediately. Automating your eCommerce software processes allows you to break the mold of one-size-fits-all coding and make the solution specific to the needs of your organization. 3. Better Customer Service Customer service and support is an essential part of the buying process. Automated customer support does not necessarily mean your customers will get canned responses to all their queries. However,  it means that common queries can be dealt with in a more efficient manner. Live chats and chatbots have become incredibly popular with eCommerce retailers because these features offer convenience to both the customer and the retailer. The retailer’s support staff will not be held up with routine inquiries and the customer can get their queries resolved quickly. Timely responses are a huge part of what constitutes positive customer experience. Live chat can even be used for shopping carts in order to decrease cart abandonment rates. Automating priority tickets as well as the follow-up on resolved and unresolved tickets is another way to automate customer service. With new generation automated CRM systems/ help desk software you can automate the tags that are used to filter tickets. It saves the customer service rep’s time and ensures that priority tickets are resolved quickly. Customer support/service can be made as a strategic asset in your overall strategy. The same Oracle study mentioned above suggests that back in 2011, 50% of customers would give a brand at most one week to respond to a query before they stopped doing business with them. You can imagine what those same numbers for 2018 would be. 4. Order Fulfillment Physical fulfillment of orders is prone to errors as it requires humans to oversee the warehouse selection. With an automated solution, you can set up an order fulfillment to match the warehouse requirements as closely as possible. This ensures that the closest warehouse which has the required item in stock is selected. This guarantees timely delivery of the order to the customer. It could also be set up to integrate with your billing software so as to calculate accurate billing/shipping charges, taxes (for out of state/overseas shipments), etc. Automation can also help manage your inventory effectively. Product availability is updated automatically with each transaction, be it a return or the addition of a new product. If the stock of a particular in-demand item was nearing danger levels, your eCommerce software would send an automated email to the supplier asking to replenish its stock ASAP. Automation ensures that your prerequisites to order fulfillment are fulfilled for successful processing and delivery of an order. Challenges with eCommerce automation The aforementioned benefits are not without some risks, just as any evolving concept tends to be. One of the most important challenges to making automation work for eCommerce smoothly is data accuracy. As eCommerce platforms gear up for greater multichannel/omnichannel retailing strategies, automation is certainly going to help them bolster and enhance their workflows, but only if the right checks are in place. Automation still has a long way to go, so for now, it might be best to focus on automating tasks that take up a lot of time such as updating inventory, reserving products for certain customers, updating customers on their orders, etc. Advanced applications such as fraud detection might still take a few years to be truly ‘automated’ and free from any need for human review. For now, eCommerce retailers still have a whole lot to look forward to. All things considered, automating the key tasks/functions of your eCommerce platform will impart flexibility, agility, and a scope for improved customer experience. Invest rightly in automation solutions for your eCommerce software to stay competitive in the dynamic, unpredictable eCommerce retail scenario. Author Bio Fretty Francis works as a Content Marketing Specialist at SoftwareSuggest, an online platform that recommends software solutions to businesses. Her areas of expertise include eCommerce platforms, hotel software, and project management software. In her spare time, she likes to travel and catch up on the latest technologies. Software developer tops the 100 Best Jobs of 2019 list by U.S. News and World Report IBM Q System One, IBM’s standalone quantum computer unveiled at CES 2019 Announcing ‘TypeScript Roadmap’ for January 2019- June 2019

The tech world has seen a number of languages emerge, grow, and become super popular, but equally it has seen its fair share of failures and things that make you ask yourself “just why?” We initially had the dominant set of languages introduced to us many years ago (in the 80s), which are still popular and widely used; these include C++, C, Fortran, Erlang, Pearl, SQL, Objective C, and so on. There is nothing to suggest these languages will die out completely or even lose their market share, but the world of programming really came to life in the 90s in an era known as the “Internet age” where a new set of languages came to the party. During this period, a set of “special” languages emerged and I personally would go as far as to say they revolutionized the way we programme. These were languages like JavaScript, Python, Java, R, Haskell, Ruby, and PHP. What’s more interesting is that you see a huge demand for these languages currently on the market (even after 20 years!) and you certainly wouldn’t categorize them as new; so why are they still so popular? Has the tech market stalled? Have developers not moved on? Do we have everything we need from these languages? And what’s next for the future? The following image helps explain the introduction and growth of these languages, in terms of use and adoption; it’s based on Redmonks’ analysis which compares the popularity of Stackoverflow tags and Github repositories: This graph shows a number of languages with the movement from left to right as a positive one. It’s apparent that the languages that were introduced in the 90s are at the forefront of programming; there are even few from the 80s, which supports my earlier statement that older languages don’t die out but seem to improve and adapt over time. However with time and the ever changing tech market, new demands always arise and where there are problems, there are developers with solutions. Recently and over the past few years, we have seen the emergence of new programming languages. Interestingly they seem to be very similar to the older ones, but they have that extra oomph that makes them so popular. I would like to introduce you to a set of languages that may be around for many years to come and may even shape the tech world in the future. Could they be the next generation? Scala is a multi-paradigm programming language supports both object-oriented and functional programming. It is a scripting language used to build applications for the JVM. Scala has seen increased adoption from Java developers due to its flexibility, as it provides the ability to carry out functional programming capabilities. Could Scala replace Java? Go, introduced by Google, is a statically-typed programming language with syntax similar to C. It has been compared and seen as a viable alternative to major languages such as Java or C++. However, Go is different thanks to its inherent support for concurrent programming, where it independently executes tasks, and computations are designed to interact with each other that can be run on a single processor or multi-core processors. Swift is Apple’s new programming language, unveiled in June 2014. It is set to replace Objective-C as the lingua franca for developing apps for Apple operating systems. As a multi-paradigm language, it has expressive features familiar to those used to working with modern functional languages, while also keeping the object-oriented features of Objective-C. F# is a multi-paradigm programming language that encompasses object-oriented features but is predominantly focused on functional programming. F# was developed by Microsoft as an alternative to C#, touted as a language that can do everything C# can but better. The language was primarily designed with the intention of applying it to data-driven concepts. One of the greatest benefits of using F# is the interoperability with other .NET languages. This means code written in F# can work with different parts of an application written in C#.  Elixir  is a functional programming language that leverages features of the Erlang VM and has syntax similar to Ruby. Elixir offers concurrency, high scalability, and fault-tolerance, enabling higher levels of productivity and extensibility while maintaining compatibility with Erlang’s tools and ecosystem.  Clojure is a dynamic, general-purpose programming language that runs on the Java Virtual Machine that offers interactive development with the speed and reliable runtime of the JVM. It takes advantage of Java libraries, services, and all of the resources of the JVM ecosystem. Dart, introduced by Google, is a pure object-oriented language with C-style syntax. Developers look at Dart as a JavaScript competitor that offers simplicity, flexibility, better performance, and security. Dart was designed for web development and to scale complex web applications. Juliais an expressive and dynamic multi-paradigm language. It’s as fast as C and it can be used for general programming. It is supposed to be a high level programming language with syntax similar to Matlab and Fortran. The language is predominantly used in the field of data science, and is one to keep an eye out for as it’s tipped to rival R and Python in the future.   D is another multi-paradigm programming language that allows developers to write “elegant” code. There’s demand for D as it’s a genuine improvement over C++, while still offering all the benefits of C++. D can be seen as a solution for developers who build half their application in Ruby/Python and then use C++ to “deal with the bottle-necks”. D lets you have all the benefits of both of these languages. Rust is another multi-paradigm systems language developed by Mozilla. It has been touted as a valuable alternative to C++. Rust combines strong concurrent programming, low-level abstraction, with super-fast performance, making it ideal for high-level projects. Rust’s type system ensures memory errors are minimized, a problem that is common in C++ with memory leaks. However, for the moment Rust isn’t designed to replace C++ but improve on its flaws, yet with advancements in the future you never know… From this list of new languages, it’s clear that the majority of them were created to solve issues of previous languages. They are all subsets of a similar language, but better refined to meet the modern day developer’s needs. There has been an increase in support for functional languages and there’s also been a steep rise of multi-paradigm features, which suggests the need for flexible programming. Whether we’re looking at the new “class” of languages for the future remains to be seen, but one thing is for sure: they were designed to make a difference in an increasingly new and brave tech world. 

Kicked into overdrive by Apple's iOS9 infamously coming with adblocking options for Safari, the content creators of the Internet have woken up to the serious challenge of ad-blocking tech. The AdBlock+ Chrome extension boasts over 50 million active users. I'm one of them. I'm willing to bet that you might be one too. AdBlock use is rising massively and globally and shows no sign of slowing down. Commentators have blamed the web-reading public, have declared web publishers have brought this on themselves and even made worryingly convincing arguments that adblocking is a conspiracy by corporate supergiants to kill the web as we know it. They all agree on one point though - the way we present and consume web content is going to have to evolve or die. So how might adblocking change the web? We All Go Native One of the most proposed and most popular solutions to the adblocking crisis is to embrace "native" advertising. Similar to sponsorship or product placement in other media, native advertising interweaves its sponsor into the body of the content piece. By doing so, an advert is made immune to the traditional scripts and methods that identify and block net ads. This might be a thank you note to a sponsor at the end of a blog, an 'advertorial' upsell of a product or service, or corporate content marketing where a company produces and promotes their own content in a bid to garner your attention for their paid products. (Just like this blog. I'm afraid it's content marketing. Would you like to buy a quality tech eBook? How about the Web Developer's Reference guide - your Bible for everything you need to know about web dev! Help keep this Millennial creative in a Netflix account and pop culture tee-shirts.) The Inevitable Downsides Turns out nobody wants to read sponsored content - only 24% of readers scroll down on a native ad. A 2014 survey by Contently revealed two-thirds of respondents saying they felt deceived by sponsored advertising. We may see this changing. As the practice becomes more mainstream, readers come to realize it does not impact on quality or journalistic integrity. But it's a worrying set of statistics for anyone who hoped direct advertising might save them from the scourge of adblock. The Great App Exodus There's a increasingly popular prediction that adblocking may lead to a great exodus of content from browser-based websites to exist more and more in a scattered app-based ecosystem. We can already see the start of this movement. Every major content site bugs you to download its dedicated app, where ads can live free of fear. If you consume most of your mobile media through Snapchat Discover channels, through Facebook mobile sharing, or even through IM services like Telegram, you'll be reading your web content in that apps dedicated built-in reader. That reader is, of course, free of adblocking extensions. The Inevitable Downsides The issue here is one of corporate monopoly. Some journalists have criticized Facebook Instant (the tech which has Facebook host articles from popular news sites for increased load times) for giving Facebook too much power over the news business. Vox's Matthew Yglesias's predicts restructuring where "instead of digital media brands being companies that build websites, they will operate more like television studios — bringing together teams that collaborate on the creation of content, which is then distributed through diverse channels that are not themselves controlled by the studio." The control that these platforms could exert raises troubling concerns for the future of the Internet as a bastion of free and public speech. User Experience with Added Guilt Alongside adding advertising <script> tags, web developers are increasingly creating sites that detect if you're using AdBlocking software and punishing you accordingly. This can take many forms - from a simple plea to be put on your whitelist in order to keep the servers running, to the cruel and inhuman: Some sites are going as far as actively blocking content for users using adblockers. Trying accessing an article on the likes of Forbes or CityAM with an adblocker turned on. You'll find yourself greeted with an officious note and a scrambled page that refuses to show you the goods unless you switch off the blocker. The Inevitable Downsides No website wants to be in a position where it has to beg or bully their visitors. Whilst your committed readers might be happy to whitelist your URL, antagonizing new users is a surefire way to get them to bounce from the site. Sadly, sabotaging their own sites for ad blocking visitors might be one of the most effective ways for 'traditional' web content providers to survive. After all, most users block ads in order to improve their browsing experience. If the UX of a site on the whitelist is vastly superior to the UX under adblock, it might end up being more pleasant to browse with the extension off. An Uneasy Truce between Adblockers and Content In many ways adblocking was a war that web adverts started. From the pop-up to the autoplaying video, web ad software has been built to be aggressive. The response of adblockers is an indiscriminate and all-or-nothing approach. As Marco Arment, creator of the Peace adblocking app, notes "Today’s web readers [are so] fed up that they disable all ads, or even all Javascript. Web developers and standards bodies couldn’t be more out of touch with this issue, racing ahead to give browsers and Javascript even more capabilities without adequately addressing the fundamental problems that will drive many people to disable huge chunks of their browser’s functionality." Both sides need to learn to trust one another again. The AdBlock+ Chrome extension now comes with an automatic whitelist of sites; 'guilt' website UX works to remind us that a few banner ads might be the vital price we pay to keep our favorite mid-sized content site free and accessible. If content providers work to restore sanity to the web on their ends, then our need for adblocking software as users will diminish accordingly. It's a complex balance that will need a lot of good will from both 'sides' - but if we're going to save the web as we know it, then a truce might be necessary. Building a better web? How about checking out our Essential Web Dev? Get five titles for only $50!  

Previously known as eelo, /e/ is an ‘ethical’ operating system for mobile phones. Leading the project is Gaël Duval who is also the creator of Mandrake Linux. Is it a new OS? Well not exactly, it is a forked version of Lineage OS stripped of Google apps, with a focus on privacy and considered as an ethical OS. What’s so good about /e/? The good thing here is that this is a unique effort for an ethical OS. Something different from the data collection of Android or the expensive devices by Apple. With a functional ROM including all functionalities, Duval seems to be pretty serious about this. An OS that respects user privacy does sound like a very nice thing. However, as pointed out by people on Reddit, this is what Cyanogen was in the beginning. The ethical OS /e/ is not actually a new OS from scratch. Who has the time or funding for that today? You have /e/ services instead of Google services, but ummm can you trust them? Is /e/ a trick… or a treat? We have mixed feelings about this one, it is a commendable effort, the idea is right. But with the recent privacy debates everywhere trusting a new OS is tricky. We’ll reserve judgement till it is out of beta and has a name that you can Google search for.

One of the important things that you can use Wireshark for is application analysis and troubleshooting. When the application slows down, it can be due to the LAN (quite uncommon in wired LAN), the WAN service (common due to insufficient bandwidth or high delay), or slow servers or clients. It can also be due to slow or problematic applications. The purpose of this article is to get into the details of how applications work, and provide relevant guidelines and recipes for isolating and solving these problems. In the first recipe, we will learn how to find out and categorize applications that work over our network. Then, we will go through various types of applications to see how they work, how networks influence their behavior, and what can go wrong. Further, we will learn how to use Wireshark in order to resolve and troubleshoot common applications that are used in an enterprise network. These are Microsoft Terminal Server and Citrix, databases, and Simple Network Management Protocol (SNMP). This is an excerpt from Network Analysis using Wireshark 2 Cookbook - Second Edition written by Nagendra Kumar Nainar, Yogesh Ramdoss, Yoram Orzach. Find out what is running over your network The first thing to do when monitoring a new network is to find out what is running over it. There are various types of applications and network protocols, and they can influence and interfere with each other when all of them are running over the network. In some cases, you will have different VLANs, different Virtual Routing and Forwardings (VRFs), or servers that are connected to virtual ports in a blade server. Eventually, everything is running on the same infrastructure, and they can influence each other. There is a common confusion between VRFs and VLANs. Even though their purpose is quite the same, they are configured in different places. While VLANs are configured in the LAN in order to provide network separation in the OSI layers 1 and 2, VRFs are multiple instances of routing tables to make them coexist in the same router. This is a layer 3 operation that separates between different customer's networks. VRFs are generally seen in service provider environments using Multi-Protocol Label Switching (MPLS) to provide layer 3 connectivity to different customers over the same router's network, in such a way that no customer can see any other customer's network. In this recipe, we will see how to get to the details of what is running over the network, and the applications that can slow it down. The term blade server refers to a server enclosure, which is a chassis of server shelves on the front and LAN switches on the back. There are several different acronyms for it; for example, IBM calls them blade center and HP calls them blade system. Getting ready When you get into a new network, the first thing to do is connect Wireshark to sniff what is running over the applications and protocols. Make sure you follow these points: When you are required to monitor a server, port-mirror it and see what is running on its connection to the network. When you are required to monitor a remote office, port-mirror the router port that connects you to the WAN connection. Then, check what is running over it. When you are required to monitor a slow connection to the internet, port-mirror it to see what is going on there. In this recipe, we will see how to use the Wireshark tools for analyzing what is running and what can cause problems. How to do it... For analyzing, follow these steps: Connect Wireshark using one of the options mentioned in the previous section. You can use the following tools: Navigate to Statistics | Protocol Hierarchy to view the protocols that run over the network and the percentage of the total traffic Navigate to Statistics | Conversations to see who is talking and what protocols are used In the Protocol Hierarchy feature, you will get a window that will help you analyze who is talking over the network. It is shown in the following screenshot: In the preceding screenshot, you can see the protocol distribution: Ethernet: IP, Logical-Link Control (LLC), and configuration test protocol (loopback) Internet Protocol Version 4: UDP, TCP, Protocol Independent Multicast (PIM), Internet Group Management Protocol (IGMP), and Generic Routing Encapsulation  (GRE) If you click on the + sign, all the underlying protocols will be shown. To see a specific protocol throughput, click down to the protocols as shown in the following screenshot. You will see the application average throughput during the capture (HTTP in this example): Clicking on the + sign to the left of HTTP will open a list of protocols that run over HTTP (XML, MIME, JavaScripts, and more) and their average throughput during the capture period. There's more... In some cases (especially when you need to prepare management reports), you are required to provide a graphical picture of the network statistics. There are various sources available for this, for example: Etherape (for Linux): http://etherape.sourceforge.net/ Compass (for Windows): http://download.cnet.com/Compass-Free/3000-2085_4-75447541.html?tag=mncol;1 Analyzing Microsoft Terminal Server and Citrix communications problems Microsoft Terminal Server, which uses Remote Desktop Protocol (RDP) and Citrix metaframe Independent Computing Architecture (ICA) protocols, are widely used for local and remote connectivity for PCs and thin clients. The important thing to remember about these types of applications is that they transfer screen changes over the network. If there, are only a few changes, they will require low bandwidth. If there many changes, they will require high bandwidth. Another thing is that the traffic in these applications is entirely asymmetric. Downstream traffic takes from tens of Kbps up to several Mbps, while the upstream traffic will be at most several Kbps. When working with these applications, don't forget to design your network according to this. In this recipe, we will see some typical problems of these applications and how to locate them. For the convenience of writing, we will refer to Microsoft Terminal Server, and every time we write Microsoft Terminal Server, we will refer to all applications in this category, for example, Citrix Metaframe. Getting ready When suspecting a slow performance with Microsoft Terminal Server, first check with the user what the problem is. Then, connect the Wireshark to the network with port-mirror to the complaining client or to the server. How to do it... For locating a problem when Microsoft Terminal Server is involved, start with going to the users and asking questions. Follow these steps: When users complain about a slow network, ask them a simple question: Do they see the slowness in the data presented on the screen or when they switch between windows? If they say that the switch between windows is very fast, it is not a Microsoft Terminal Server problem. Microsoft Terminal Server problems will cause slow window changes, picture freezes, slow scrolling of graphical documents, and so on. If they say that they are trying to generate a report (when the software is running over Microsoft Terminal Server), but the report is generated after a long period of time, this is a database problem and not Microsoft Terminal Server or Citrix. When a user works with Microsoft Terminal Server over a high-delay communication line and types very fast, they might experience delays with the characters. This is because Microsoft Terminal Server is transferring window changes, and with high delays, these windows changes will be transferred slowly. When measuring the communication line with Wireshark: Use I/O graphs to monitor the line Use filters to monitor the upstream and the downstream directions Configure bits per second on the y axis You will get the following screenshot: In the preceding screenshot, you can see a typical traffic pattern with high downstream and very low upstream traffic. Notice that the Y-Axis is configured to Bits/Tick. In the time between 485s and 500s, you see that the throughput got to the maximum. This is when applications will slow down and users will start to feel screen freezes, menus that move very slowly, and so on. When a Citrix ICA client connects to a presentation server, it uses TCP ports 2598 or 1494. When monitoring Microsoft Terminal Server servers, don't forget that the clients access the server with Microsoft Terminal Server and the servers access the application with another client that is installed on the server. The performance problem can come from Microsoft Terminal Server or the application. If the problem is an Microsoft Terminal Server problem, it is necessary to figure out whether it is a network problem or a system problem: Check the network with Wireshark to see if there are any loads. Loads such as the one shown in the previous screenshot can be solved by simply increasing the communication lines. Check the server's performance. Applications like Microsoft Terminal Server are mostly memory consuming, so check mostly for memory (RAM) issues. How it works... Microsoft Terminal Server, Citrix Metaframe, and applications simply transfer window changes over the network. From your client (PC with software client or thin client), you connect to the terminal server; and the terminal server, runs various clients that are used to connect from it to other servers. In the following screenshot, you can see the principle of terminal server operation: There's more... From the terminal server vendors, you will hear that their applications improve two things. They will say that it improves manageability of clients because you don't have to manage PCs and software for every user; you simply install everything on the server, and if something fails, you fix it on the server. They will also say that traffic over the network will be reduced. Well, I will not get into the first argument. This is not our subject, but I strongly reject the second one. When working with a terminal client, your traffic entirely depends on what you are doing: When working with text/character-based applications, for example, some Enterprise Resource Planning (ERP) screens, you type in and read data. When working with the terminal client, you will connect to the terminal server that will connect to the database server. Depending on the database application you are working with, the terminal server can improve performance significantly or does not improve it at all. We will discuss this in the database section. Here, you can expect a load of tens to hundreds of Kbps. If you are working with regular office documents such as Word, PowerPoint, and so on, it entirely depends on what you are doing. Working with a simple Word document will require tens to hundreds of Kbps. Working with PowerPoint will require hundreds of Kbps to several Mbps, and when you present the PowerPoint file with full screen (the F5 function), the throughput can jump up to 8 to 10 Mbps. Browsing the internet will take between hundreds of Kbps and several Mbps, depending on what you will do over it. High resolution movies over terminal server to the internet-well, just don't do it. Before you implement any terminal environment, test it. I once had a software house that wanted their logo (at the top-right corner of the user window) to be very clear and striking. They refreshed it 10 times a second, which caused the 2 Mbps communication line to be blocked. You never know what you don't test! Analyzing the database traffic and common problems Some of you may wonder why we have this section here. After all, databases are considered to be a completely different branch in the IT environment. There are databases and applications on one side and the network and infrastructure on the other side. It is correct since we are not supposed to debug databases; there are DBAs for this. But through the information that runs over the network, we can see some issues that can help the DBAs with solving the relevant problems. In most of the cases, the IT staff will come to us first because people blame the network for everything. We will have to make sure that the problems are not coming from the network and that's it. In a minority of the cases, we will see some details on the capture file that can help the DBAs with what they are doing. Getting ready When the IT team comes to us complaining about the slow network, there are some things to do just to verify that it is not the case. Follow the instructions in the following section to make sure you avoid the slow network issue. How to do it... In the case of database problems, follow these steps: When you get complaints about the slow network responses, start asking these questions: Is the problem local or global? Does it occur only in the remote offices or also in the center? When the problem occurs in the entire network, it is not a WAN bandwidth issue. Does it happen the same for all clients? If not, there might be a specific problem that happens only with some users because only those users are running a specific application that causes the problem. Is the communication line between the clients and the server loaded? What is the application that loads them? Do all applications work slowly, or is it only the application that works with the specific database? Maybe some PCs are old and tired, or is it a server that runs out of resources? When we are done with the questionnaire, let's start our work: Open Wireshark and start capturing packets. You can configure port-mirror to a specific PC, the server, a VLAN, or a router that connects to a remote office in which you have the clients. Look at the TCP events (expert info). Do they happen on the entire communication link, on specific IP address/addresses, or on specific TCP port number/numbers? This will help you isolate the problem and check whether it is on a specific link, server, or application. When measuring traffic on a connection to the internet, you will get many retransmissions and duplicate ACKs to websites, mail servers, and so on. This is the internet. In an organization, you should expect 0.1 to 0.5 percent of retransmissions. When connecting to the internet, you can expect much higher numbers. But there are some network issues that can influence database behavior. In the following example, we see the behavior of a client that works with the server over a communication line with a round trip delay of 35 to 40 ms. We are looking at the TCP stream number 8 (1) and the connection started with TCP SYN/SYN-ACK/ACK. I've set this as a reference (2). We can see that the entire connection took 371 packets (3): The connection continues, and we can see time intervals of around 35 ms between DB requests and responses: Since we have 371 packets travelling back and forth, 371 x 35 ms gives us around 13 seconds. Add to this some retransmissions that might happen and some inefficiencies, and this leads to a user waiting for 10 to 15 seconds and more for a database query. In this case, you should consult the DBA on how to significantly reduce the number of packets that run over the network, or you can move to another way of access, for example, terminal server or web access. Another problem that can happen is that you will have a software issue that will reflect in the capture file. If you have a look at the following screenshot, you will see that there are five retransmissions, and then a new connection is opened from the client side. It looks like a TCP problem but it occurs only in a specific window in the software. It is simply a software procedure that stopped processing, and this stopped the TCP from responding to the client: How it works... Well, how databases work was always be a miracle to me. Our task is to find out how they influence the network, and this is what we've learned in this section. There's more... When you right-click on one of the packets in the database client to the server session, a window with the conversation will open. It can be helpful to the DBA to see what is running over the network. When you are facing delay problems, for example, when working over cellular lines over the internet or over international connections, the database client to the server will not always be efficient enough. You might need to move to web or terminal access to the database. An important issue is how the database works. If the client is accessing the database server, and the database server is using files shared from another server, it can be that the client-server works great; but the problems come from the database server to the shared files on the file server. Make sure that you know all these dependencies before starting with your tests. And most importantly, make sure you have very professional DBAs among your friends. One day, you will need them! Analyzing SNMP SNMP is a well-known protocol that is used to monitor and manage different types of devices in a network by collecting data and statistics at regular intervals. Beyond just monitoring, it can also be used to configure and modify settings with appropriate authorization given to SNMP servers. Devices that typically support SNMP are switches, routers, servers, workstations, hosts, VoIP Phones, and many more. It is important to know that there are three versions of SNMP: SNMPv1, SNMPv2c, and SNMPv3. Versions v2c and v3, which came later, offer better performance and security. SNMP consists of three components: The device being managed (referred to as managed device). SNMP Agent. This is a piece of software running on the managed device that collects the data from the device and stores it in a database, referred to as the Managed Information Base (MIB) database. As configured, this SNMP agent exports the data/statistics to the server (using UDP port 161) at regular intervals, and also any events and traps. SNMP server, also called Network Management Server (NMS). This is a server that communicates with all the agents in the network to collect the exported data and build a central repository. SNMP server provides access to the IT staff managing network; they can monitor, manage, and configure the network remotely. It is very important to be aware that some of the MIBs implemented in a device could be vendor-specific. Almost all the vendors publicize these MIBs implemented in their devices. Getting ready Generally, the complaints we get from the network management team are about not getting any statistics or traps from a device(s) for a specific interval, or having completely no visibility to a device(s). Follow the instructions in the following section to analyze and troubleshoot these issues. How to do it... In the case of SNMP problems, follow these steps. When you get complaints about SNMP, start asking these questions: Is this a new managed device that has been brought into the network recently? In other words, did the SNMP in the device ever work properly? If this is a new device, talk to relevant device administrator and/or check the SNMP-related configurations, such as community strings. If SNMP configurations looks correct, make sure that the NMS's IP address configured is correct and also check the relevant password credentials. If SNMP v3 is in use, which supports encryption, make sure to check encryption-related settings like transport methods. If the setting and configuration looks valid and correct, make sure the managed devices have connectivity with the NMS, which can be verified by simple ICMP pings. If it is a managed device that has been working properly and didn't report any statistics or alerts for a specific duration: Did the device in discussion have any issues in the control plane or management plane that stopped it from exporting SNMP statistics? Please be aware that for most devices in the network, SNMP is a least-priority protocol, which means that if a device has a higher-priority process to work on, it will hold the SNMP requests and responses in the queue. Is the issue experienced only for a specific device or for multiple devices in the network? Did the network (between managed device and NMS) experience any issue? For example, during any layer 2 spanning-tree convergence, traffic loss could occur between the managed device and SNMP server, by which NMS would lose visibility to the managed devices. As you can see in the following picture, an SNMP Server with IP address 172.18.254.139 is performing SNMP walk with a sequence of GET-NEXT-REQUEST to a workstation with IP address 10.81.64.22, which in turn responds with GET-RESPONSE. For simplicity, the Wireshark filter used for these captures is SNMP. The workstation is enabled with SNMP v2c, with community string public. Let's discuss some of the commonly seen failure scenarios. Polling a managed device with a wrong SNMP version As I mentioned earlier, the workstation is enabled with v2c, but when the NMS polls the device with the wrong SNMP version, it doesn't get any response. So, it is very important to make sure that the managed devices are polled with the correct SNMP version. Polling a managed device with a wrong MIB object ID (OID) In the following example, the NMS is polling the managed device to get a number of bytes sent out on interfaces. The MIB OID for byte count is .1.3.6.1.2.1.2.2.1.16, which is ifOutOctets. The managed device in discussion has two interfaces, mapped to OID .1.3.6.1.2.1.2.2.1.16.1 and .1.3.6.1.2.1.2.2.1.16.2. When NMS polls the device to check the statistics for the third interface (which is not present), it returns a noSuchInstance error. How it works... As you have learned in the earlier sections, SNMP is a very simple and straightforward protocol and all its related information on standards and MIB OIDs is readily available in the internet. There's more... Here are some of the websites with good information about SNMP and MIB OIDs: Microsoft TechNet SNMP: https://technet.microsoft.com/en-us/library/cc776379(v=ws.10).aspx Cisco IOS MIB locator: http://mibs.cloudapps.cisco.com/ITDIT/MIBS/servlet/index We have learned to perform enterprise-level network analysis with real-world examples like Analyzing Microsoft Terminal Server and Citrix communications problems. Get to know more about security and network forensics from our book Network Analysis using Wireshark 2 Cookbook - Second Edition. What’s new in Wireshark 2.6 ? Top 5 penetration testing tools for ethical hackers 5 pen testing rules of engagement: What to consider while performing Penetration testing

If you're on the Packt Hub, it's likely that you already know how to code. But perhaps you don't - and if you don't, it's our job to let you know why you should learn to code. And even if you do know how, evangelizing and explaining to others why they should learn is incredibly important. People often think of writing code as something very specialized and technically demanding. There’s sometimes even a perception of people that do write code as a certain kind of person. But this is, luckily, demonstrably untrue. Writing code is something that can be done by anyone. While there are some incredibly specialized roles that require incredibly detailed levels of programming knowledge, in actual fact a huge number of roles today use software in creative and analytical ways. That’s why learning how to code is so essential. Let’s take a look at 5 reasons why you should learn how to write code. Learn to code to better understand how software works Code is everywhere. Learning how to write it is a way of understanding how so much of the world works. Perhaps you’re a writer. Or maybe a designer. Maybe you work in marketing. Whatever you’re doing, it’s likely that you’re working in a digital environment. Understanding how these are built and how they work puts you at an advantage, in that you have an extra dimension of knowledge of how things work ‘under the hood’. Learning to code will help you better manipulate and use data The world runs on data. Code can help you better understand it and manage it. Excel is a sad part of business life. You don’t have to look had to finding someone complaining about spreadsheets. But if you’ve ever been impressed with someone’s dexterity with Excel, that’s really just a type of code. In the world of data today there are far more sophisticated systems for processing and managing data. But if you’re working with large datasets in any capacity, learning how to write SQL scripts, for example, is a great place to begin your coding adventure. Learn to code simply to have fun Coding is fun and creative! Before talk of spreadsheets and databases turns you off, it’s important to note that writing code and learning to program is fun. If you’re just starting out seeing the effect of your code as you build your first web page is a really neat experience. A lot of people think coding is purely technical - but the technical part is just a small aspect of it. Really, it’s about solving problems and being creative. Even if you’re not sure how learning to code might help you professionally right now, having a creative outlet might just inspire your future. Coding isn't as hard as you think It’s not actually that hard… Writing code isn’t really that hard. It might take some getting used to, but once you’re over those first steps you’ll feel like an expert. True, learning how to write code always involves some ups and downs, and if you’re getting deeper into more complex programming concepts and tasks, it will be challenging. But that doesn’t mean that it’s beyond you. It’s a valuable skill you can actually develop quickly. You can learn to code without spending lots of money It’s a valuable skill you can develop cheaply. Learning how to write code could increase your earning power. Even if it doesn’t transform your career, it could still give you a push to get to the next step. But while development often requires expensive training and certifications that your employer might pay for, learning the basics of coding is something you can do without spending very much money at all. There’s lots of free resources out there. And even the very best resources won’t put you out of pocket. Slow down to learn how to code faster

‘Service mesh’ is a term that is relatively new and has gained visibility in the past year. It’s a configurable infrastructure layer for a microservices application that makes communication between service instances flexible, reliable, and fast. Why are people talking about ‘service meshes’? Modern applications contain a range of (micro)services that allow it to run effectively. Load balancing, traffic management, routing, security, user authentication - all of these things need to work together properly if the application is going to function as intended.. Managing these various services, across a whole deployment of containers, poses a challenge for those responsible for updating and maintaining them. How does a service mesh work? Enter the Service mesh. It works delivering these services from within the compute cluster through a set of APIs. These APIs, when brought together, form the ‘mesh’.. This makes it much easier to manage software infrastructures of particular complexity - hence why organizations like Netflix and Lyft have used them.. Trick or treat? With the service meshes addressing some of the key challenges when it comes to microservices, this is definitely a treat for 2018 and beyond. NGINX Hybrid Application Delivery Controller Platform improves API management, manages microservices and much more! Kong 1.0 launches: the only open source API platform specifically built for microservices, cloud, and serverless OpenFaaS releases full support for stateless microservices in OpenFaaS 0.9.0

The rise of Big Data and Analytics is drastically changing the landscape of many businesses - and the sports industry is one of them. In today’s age of cut-throat competition, data-based strategies are slowly taking the front seat when it comes to crucial decision making - helping teams gain that decisive edge over their competition.Sports Analytics is slowly becoming the next big thing! In the past, many believed that the key to conquering the opponent in any professional sport is to make the player or the team better - be it making them stronger, faster, or more intelligent.  ‘Analysis’ then was limited to mere ‘clipboard statistics’ and the intuition built by coaches on the basis of raw video footage of games. This is not the case anymore. From handling media contracts and merchandising to evaluating individual or team performance on matchday, analytics is slowly changing the landscape of sports. The explosion of data in sports The amount and quality of information available to decision-makers within the sports organization have increased exponentially over the last two decades. There are several factors contributing to this: Innovation in sports science over the last decade, which has been incredible, to say the least. In-depth records maintained by trainers, coaches, medical staff, nutritionists and even the sales and marketing departments Improved processing power and lower cost of storage allowing for maintaining large amounts of historical data. Of late, the adoption of motion capture technology and wearable devices has proved to be a real game-changer in sports, where every movement on the field can be tracked and recorded. Today, many teams in a variety of sports such as Boston Red Sox and Houston Astros in Major League Baseball (MLB), San Antonio Spurs in NBA and teams like Arsenal, Manchester City and Liverpool FC in football (soccer) are adopting analytics in different capacities. Turning sports data into insights Needless to say, all the crucial sports data being generated today need equally good analytics techniques to extract the most value out of it. This is where Sports Analytics comes into the picture. Sports analytics is defined as the use of analytics on current as well as historical sport-related data to identify useful patterns, which can be used to gain a competitive advantage on the field of play. There are several techniques and algorithms which fall under the umbrella of Sports Analytics. Machine learning, among them, is a widely used set of techniques that sports analysts use to derive insights. It is a popular form of Artificial Intelligence where systems are trained using large datasets to give reliable predictions on random data. With the help of a variety of classification and recommendation algorithms, analysts are now able to identify patterns within the existing attributes of a player, and how they can be best optimized to improve his performance. Using cross-validation techniques, the machine learning models then ensure there is no degree of bias involved, and the predictions can be generalized even in cases of unknown datasets. Analytics is being put to use by a lot of sports teams today, in many different ways. Here are some key use-cases of sports analytics: Pushing the limit: Optimizing player performance Right from tracking an athlete’s heartbeats per minute to finding injury patterns, analytics can play a crucial role in understanding how an individual performs on the field. With the help of video, wearables and sensor data, it is possible to identify exactly when an athlete’s performance drops and corrective steps can be taken accordingly. It is now possible to assess a player’s physiological and technical attributes and work on specific drills in training to push them to an optimal level. Developing search-powered data intelligence platforms seems to be the way forward. The best example for this is Tellius, a search-based data intelligence tool which allows you to determine a player’s efficiency in terms of fitness and performance through search-powered analytics. Smells like team spirit: Better team and athlete management Analytics also helps the coaches manage their team better. For example, Adidas has developed a system called miCoach which works by having the players use wearables during the games and training sessions. The data obtained from the devices highlights the top performers and the ones who need rest. It is also possible to identify and improve patterns in a team’s playing styles, and developing a ‘system’ to improve the efficiency in gameplay. For individual athletes, real-time stats such as speed, heart rate, and acceleration could help the trainers plan the training and conditioning sessions accordingly. Getting intelligent responses regarding player and team performances and real-time in-game tactics is something that will make the coaches’ and management’s life a lot easier, going forward. All in the game: Improving game-day strategy By analyzing the real-time training data, it is possible to identify the fitter, in-form players to be picked for the game. Not just that, analyzing opposition and picking the right strategy to beat them becomes easier once you have the relevant data insights with you. Different data visualization techniques can be used not just with historical data but also with real-time data, when the game is in progress. Splashing the cash: Boosting merchandising What are fans buying once they’re inside the stadium? Is it the home team’s shirt, or is it their scarfs and posters? What food are they eating in the stadium eateries? By analyzing all this data, retailers and club merchandise stores can store the fan-favorite merchandise and other items in adequate quantities, so that they never run out of stock. Analyzing sales via online portals and e-stores also help the teams identify the countries or areas where the buyers live. This is a good indicator for them to concentrate sales and marketing efforts in those regions. Analytics also plays a key role in product endorsements and sponsorships. Determining which brands to endorse, identifying the best possible sponsor, the ideal duration of sponsorship and the sponsorship fee - these are some key decisions that can be taken by analyzing current trends along with the historical data. Challenges in sports analytics Although the advantages offered by analytics are there for all to see, many sports teams have still not incorporated analytics into their day-to-day operations. Lack of awareness seems to be the biggest factor here. Many teams underestimate or still don’t understand, the power of analytics. Choosing the right Big Data and analytics tool is another challenge. When it comes to the humongous amounts of data, especially, the time investment needed to clean and format the data for effective analysis is problematic and is something many teams aren’t interested in. Another challenge is the rising demand for analytics and a sharp deficit when it comes to supply, driving higher salaries. Add to that the need to have a thorough understanding of the sport to find effective insights from data - and it becomes even more difficult to get the right data experts. What next for sports analytics? Understanding data and how it can be used in sports - to improve performance and maximize profits - is now deemed by many teams to be the key differentiator between success and failure. And it’s not just success that teams are after - it’s sustained success, and analytics goes a long way in helping teams achieve that. Gone are the days when traditional ways of finding insights were enough. Sports have evolved, and teams are now digging harder into data to get that slightest edge over the competition, which can prove to be massive in the long run. If you found the article to be insightful, make sure you check out our interview on sports analytics with ESPN Senior Stats Analyst Gaurav Sundararaman.

Algorithmic trading relies on computer programs that execute algorithms to automate some, or all, elements of a trading strategy. Algorithms are a sequence of steps or rules to achieve a goal and can take many forms. In the case of machine learning (ML), algorithms pursue the objective of learning other algorithms, namely rules, to achieve a target based on data, such as minimizing a prediction error.  In this article, we have a look at use cases of ML and how it is used in algorithmic trading strategies. These algorithms encode various activities of a portfolio manager who observes market transactions and analyzes relevant data to decide on placing buy or sell orders. The sequence of orders defines the portfolio holdings that, over time, aim to produce returns that are attractive to the providers of capital, taking into account their appetite for risk. This article is an excerpt taken from the book 'Hands-On Machine Learning for Algorithmic Trading' written by Stefan Jansen.  The book explores effective trading strategies in real-world markets using NumPy, spaCy, pandas, scikit-learn, and Keras. Ultimately, the goal of active investment management consists in achieving alpha, that is, returns in excess of the benchmark used for evaluation. The fundamental law of active management applies the information ratio (IR) to express the value of active management as the ratio of portfolio returns above the returns of a benchmark, usually an index, to the volatility of those returns. It approximates the information ratio as the product of the information coefficient (IC), which measures the quality of forecast as their correlation with outcomes, and the breadth of a strategy expressed as the square root of the number of bets. The use of ML for algorithmic trading, in particular, aims for more efficient use of conventional and alternative data, with the goal of producing both better and more actionable forecasts, hence improving the value of active management. Quantitative strategies have evolved and become more sophisticated in three waves: In the 1980s and 1990s, signals often emerged from academic research and used a single or very few inputs derived from the market and fundamental data. These signals are now largely commoditized and available as ETF, such as basic mean-reversion strategies. In the 2000s, factor-based investing proliferated. Funds used algorithms to identify assets exposed to risk factors like value or momentum to seek arbitrage opportunities. Redemptions during the early days of the financial crisis triggered the quant quake of August 2007 that cascaded through the factor-based fund industry. These strategies are now also available as long-only smart-beta funds that tilt portfolios according to a given set of risk factors. The third era is driven by investments in ML capabilities and alternative data to generate profitable signals for repeatable trading strategies. Factor decay is a major challenge: the excess returns from new anomalies have been shown to drop by a quarter from discovery to publication, and by over 50% after publication due to competition and crowding. There are several categories of trading strategies that use algorithms to execute trading rules: Short-term trades that aim to profit from small price movements, for example, due to arbitrage Behavioral strategies that aim to capitalize on anticipating the behavior of other market participants Programs that aim to optimize trade execution, and A large group of trading based on predicted pricing The HFT funds discussed above most prominently rely on short holding periods to benefit from minor price movements based on bid-ask arbitrage or statistical arbitrage. Behavioral algorithms usually operate in lower liquidity environments and aim to anticipate moves by a larger player likely to significantly impact the price. The expectation of the price impact is based on sniffing algorithms that generate insights into other market participants' strategies, or market patterns such as forced trades by ETFs. Trade-execution programs aim to limit the market impact of trades and range from the simple slicing of trades to match time-weighted average pricing (TWAP) or volume-weighted average pricing (VWAP). Simple algorithms leverage historical patterns, whereas more sophisticated algorithms take into account transaction costs, implementation shortfall or predicted price movements. These algorithms can operate at the security or portfolio level, for example, to implement multileg derivative or cross-asset trades. Let's now have a look at different applications in Trading where ML is of key importance. Use Cases of ML for Trading ML extracts signals from a wide range of market, fundamental, and alternative data, and can be applied at all steps of the algorithmic trading-strategy process. Key applications include: Data mining to identify patterns and extract features Supervised learning to generate risk factors or alphas and create trade ideas Aggregation of individual signals into a strategy Allocation of assets according to risk profiles learned by an algorithm The testing and evaluation of strategies, including through the use of synthetic data The interactive, automated refinement of a strategy using reinforcement learning Supervised learning for alpha factor creation and aggregation The main rationale for applying ML to trading is to obtain predictions of asset fundamentals, price movements or market conditions. A strategy can leverage multiple ML algorithms that build on each other. Downstream models can generate signals at the portfolio level by integrating predictions about the prospects of individual assets, capital market expectations, and the correlation among securities. Alternatively, ML predictions can inform discretionary trades as in the quantamental approach outlined above. ML predictions can also target specific risk factors, such as value or volatility, or implement technical approaches, such as trend following or mean reversion. Asset allocation ML has been used to allocate portfolios based on decision-tree models that compute a hierarchical form of risk parity. As a result, risk characteristics are driven by patterns in asset prices rather than by asset classes and achieve superior risk-return characteristics. Testing trade ideas Backtesting is a critical step to select successful algorithmic trading strategies. Cross-validation using synthetic data is a key ML technique to generate reliable out-of-sample results when combined with appropriate methods to correct for multiple testing. The time series nature of financial data requires modifications to the standard approach to avoid look-ahead bias or otherwise contaminate the data used for training, validation, and testing. In addition, the limited availability of historical data has given rise to alternative approaches that use synthetic data. Reinforcement learning Trading takes place in a competitive, interactive marketplace. Reinforcement learning aims to train agents to learn a policy function based on rewards. In this article, we briefly discussed how ML has become a key ingredient for different stages of algorithmic trading strategies. If you want to learn more about trading strategies that use ML, be sure to check out the book  'Hands-On Machine Learning for Algorithmic Trading'. Using machine learning for phishing domain detection [Tutorial] Anatomy of an automated machine learning algorithm (AutoML) 10 machine learning algorithms every engineer needs to know

Isn’t it spooky how Facebook can identify faces of your friends before you manually tag them? Have you been startled by Cortana, Siri or Google Assistant when they instantly recognize and act on your voice as you speak to these virtual assistants? Deep Learning is the driving force behind these uncanny yet innovative applications. The next thing that is all set to dive into deep learning is the music industry. Neural networks not only ease production and generation of songs, but also assist in music recommendation, transcription and classification. Here are some ways that deep learning will elevate music and the listening experience itself: Generating melodies with Neural Nets At the most basic level, a deep learning algorithm follows 3 simple steps for music generation: First, the neural net is trained with a sample data set of songs which are labelled. The labelling is done based on the emotions you want your song to convey (happy, sad, funny, etc). For training, the program converts the speech of the data set in text format and then creates vector for each word.  The training data can also be in the form of MIDI format which is a standard protocol for encoding musical notes. After completing the training, the program is fed with a set of emotions as input. It identifies the associated input vectors and compares them to training vectors. The output is a melody or chords that represent the desired emotions. Long short-term memory (LSTM) architectures are also used for music generation. They take structured input of a music notation. These inputs are then encoded as vectors and fed into an LSTM at each timestep. LSTM then predicts the encoding of the next timestep. Fully connected convolutional layers are utilized to increase the music quality and to represent rich features in the frequency domain. Magenta, the popular art and music project of Google has launched Performance RNN, which is an LSTM-based recurrent neural network. It is designed to produce multiple sounds with expressive timing and dynamics. In other words, Performance RNN determines which notes to play, when to play them, and how hard to strike each note. IBM’s Watson Beat uses a neural network to produce complete tracks by understanding music theory, structure, and emotional intent. According to Richard Daskas, a music composer working on the Watson Beat project, “Watson only needs about 20 seconds of musical inspiration to create a song.” Transcripting music with deep learning Deep learning methods can also be used for arranging a piece of music for a different instrument. LSTM networks are a popular choice for music transcription and modelling. These networks are trained using a large dataset of pre-labelled music transcriptions (expressed with ABC notation). These transcriptions are then used to generate new music transcriptions. In fact, transformed audio data can be used to predict the group of notes currently being played. This can be achieved by treating the transcription model as an image classification problem. For this, an image of an audio is used, called as Spectrogram. A spectrogram displays how the spectrum or frequency content changes over time. A Short Time Fourier Transform (STFT) or a constant Q transform is used to create this spectrogram. The spectrogram is then feeded to a Convolutional Neural network(CNN). The CNN estimates current notes from audio data and determines what specific notes are present by analysing 88 output nodes for each of the piano keys. This network is generally trained using large number of examples from MIDI files spanning several different genres of music. Magenta has developed The NSynth dataset, which is a high-quality multi-note dataset for music transcription. It is inspired by image recognition datasets and has a huge collection of annotated musical notes. Make better music recommendations Neural Nets are also used to make intelligent music recommendations and are a step ahead of the traditional Collaborative filtering networks. Using neural networks, the system can analyse the songs saved by the users, and then utilize those songs to make new recommendations.  Neural nets can also be used to analyze songs based on musical qualities such as pitch, chord progression, bass, etc. Using the similarities between songs having the same traits as each other, neural networks can detect and predict new songs.  Thus providing recommendation based on similar lyrical and musical styles. Convolutional neural networks (CNNs) are utilized for making music recommendations. A time-frequency representation of the audio signal is fed into the network as the input. 3 second audio clips are randomly chosen from the audio samples to train the neural network. The CNNs are then used to predict latent factors from music audio by taking the average of the predictions for consecutive clips. The feature extraction layers and pooling layers permits operation on several timescales. Spotify is working on a music recommendation system with a CNN. This recommendation system, when trained on short clips of songs, can create playlists based on the audio content only. Classifying music according to genre Classifying music according to a genre is another achievement of neural nets. At the heart of this application lies the LSTM network. At the very first stage, convolutional layers are used for feature extraction from the spectrograms of the audio file. The sequence of features so obtained is given as input to the LSTM layer. LSTM evaluates dependencies of the song across both short time period as well as long term structure. After the LSTM, the input is fed into a fully connected, time-distributed layer which essentially gives us a sequence of vectors. These vectors are then used to output the network's evaluation of the genre of the song at the particular point of time. Deepsound uses GTZAN dataset and an LSTM network to create a model for music genre recognition.  On comparing the mean output distribution with the correct genre, the model gives almost 67% of accuracy. For musical pattern extraction, MFCC feature dataset is used for audio analysis. First, the audio signal is extracted in the MFCC format. Next, the input song is modified into an MFCC map. This Map is then split to feed it as the input of the CNN. Supervised learning is used for automatically obtaining musical pattern extractors, considering the song label is provided. The extractors so acquired, are used for restoring high-order pattern-related features. After high-order classification, the result is combined and undergoes a voting process to produce the song-level label. Scientists from Queen Mary University of London trained a neural net with over 6000 songs in a ballad, hip-hop, and dance to develop a neural network that achieves almost 75% accuracy in song classification. The road ahead Neural networks have advanced the state of music to whole new level where one would no longer require physical instruments or vocals to compose music. The future would see more complex models and data representations to understand the underlying melodic structure. This would help models create compelling artistic content on their own. Combination of music with technology would also foster a collaborative community consisting of artists, coders and deep learning researchers, leading to a tech-driven, yet artistic future.  

[box type="note" align="" class="" width=""]This article is a book extract from  Mastering Blockchain, written by Imran Bashir. In the book he has discussed distributed ledgers, decentralization and smart contracts for implementation in the real world.[/box] Today we will look at various challenges that need to be addressed before blockchain becomes a mainstream technology. Even though various use cases and proof of concept systems have been developed and the technology works well for most scenarios, some fundamental limitations continue to act as barriers to mainstream adoption of blockchains in key industries. We’ll start by listing down the main challenges that we face while working on Blockchain and propose ways to over each of those challenges: Scalability This problem has been a focus of intense debate, rigorous research, and media attention for the last few years. This is the single most important problem that could mean the difference between wider adaptability of blockchains or limited private use only by consortiums. As a result of substantial research in this area, many solutions have been proposed, which are discussed here: Block size increase: This is the most debated proposal for increasing blockchain performance (transaction processing throughput). Currently, bitcoin can process only about three to seven transactions per second, which is a major inhibiting factor in adapting the bitcoin blockchain for processing micro-transactions. Block size in bitcoin is hardcoded to be 1 MB, but if block size is increased, it can hold more transactions and can result in faster confirmation time. There are several Bitcoin Improvement Proposals (BIPs) made in favor of block size increase. These include BIP 100, BIP 101, BIP 102, BIP 103, and BIP 109. In Ethereum, the block size is not limited by hardcoding; instead, it is controlled by gas limit. In theory, there is no limit on the size of a block in Ethereum because it's dependent on the amount of gas, which can increase over time. This is possible because miners are allowed to increase the gas limit for subsequent blocks if the limit has been reached in the previous block. Block interval reduction: Another proposal is to reduce the time between each block generation. The time between blocks can be decreased to achieve faster finalization of blocks but may result in less security due to the increased number of forks. Ethereum has achieved a block time of approximately 14 seconds and, at times, it can increase. This is a significant improvement from the bitcoin blockchain, which takes 10 minutes to generate a new block. In Ethereum, the issue of high orphaned blocks resulting from smaller times between blocks is mitigated by using the Greedy Heaviest Observed Subtree (GHOST) protocol whereby orphaned blocks (uncles) are also included in determining the valid chain. Once Ethereum moves to Proof of Stake, this will become irrelevant as no mining will be required and almost immediate finality of transactions can be achieved. Invertible Bloom lookup tables: This is another approach that has been proposed to reduce the amount of data required to be transferred between bitcoin nodes. Invertible Bloom lookup tables (IBLTs) were originally proposed by Gavin Andresen, and the key attraction in this approach is that it does not result in a hard fork of bitcoin if implemented. The key idea is based on the fact that there is no need to transfer all transactions between nodes; instead, only those that are not already available in the transaction pool of the synching node are transferred. This allows quicker transaction pool synchronization between nodes, thus increasing the overall scalability and speed of the bitcoin network. Sharding: Sharding is not a new technique and has been used in distributed databases for scalability such as MongoDB and MySQL. The key idea behind sharding is to split up the tasks into multiple chunks that are then processed by multiple nodes. This results in improved throughput and reduced storage requirements. In blockchains, a similar scheme is employed whereby the state of the network is partitioned into multiple shards. The state usually includes balances, code, nonce, and storage. Shards are loosely coupled partitions of a blockchain that run on the same network. There are a few challenges related to inter-shard communication and consensus on the history of each shard. This is an open area for research. State channels: This is another approach proposed for speeding up the transaction on a blockchain network. The basic idea is to use side channels for state updating and processing transactions off the main chain; once the state is finalized, it is written back to the main chain, thus offloading the time-consuming operations from the main blockchain. State channels work by performing the following three steps: First, a part of the blockchain state is locked under a smart contract, ensuring the agreement and business logic between participants. Now off-chain transaction processing and interaction is started between the participants that update the state only between themselves for now. In this step, almost any number of transactions can be performed without requiring the blockchain and this is what makes the process fast and a best candidate for solving blockchain scalability issues. However, it could be argued that this is not a real on-blockchain solution such as, for example, sharding, but the end result is a faster, lighter, and robust network which can prove very useful in micropayment networks, IoT networks, and many other applications. Once the final state is achieved, the state channel is closed and the final state is written back to the main blockchain. At this stage, the locked part of the blockchain is also unlocked. This technique has been used in the bitcoin lightning network and Ethereum's Raiden. 6. Subchains: This is a relatively new technique recently proposed by Peter R. Rizun which is based on the idea of weak blocks that are created in layers until a strong block is found. Weak blocks can be defined as those blocks that have not been able to be mined by meeting the standard network difficulty criteria but have done enough work to meet another weaker difficulty target. Miners can build sub-chains by layering weak blocks on top of each other, unless a block is found that meets the standard difficulty target. At this point, the subchain is closed and becomes the strong block. Advantages of this approach include reduced waiting time for the first verification of a transaction. This technique also results in a reduced chance of orphaning blocks and speeds up transaction processing. This is also an indirect way of addressing the scalability issue. Subchains do not require any soft fork or hard fork to implement but need acceptance by the community. 7. Tree chains: There are also other proposals to increase bitcoin scalability, such as tree chains that change the blockchain layout from a linearly sequential model to a tree. This tree is basically a binary tree which descends from the main bitcoin chain. This approach is similar to sidechain implementation, eliminating the need for major protocol change or block size increase. It allows improved transaction throughput. In this scheme, the blockchains themselves are fragmented and distributed across the network in order to achieve scalability. Moreover, mining is not required to validate the blocks on the tree chains; instead, users can independently verify the block header. However, this idea is not ready for production yet and further research is required in order to make it practical. Privacy Privacy of transactions is a much desired property of blockchains. However, due to its very nature, especially in public blockchains, everything is transparent, thus inhibiting its usage in various industries where privacy is of paramount importance, such as finance, health, and many others. There are different proposals made to address the privacy issue and some progress has already been made. Several techniques, such as indistinguishability obfuscation, usage of homomorphic encryption, zero knowledge proofs, and ring signatures. All these techniques have their merits and demerits and are discussed in the following sections. Indistinguishability obfuscation: This cryptographic technique may serve as a silver bullet to all privacy and confidentiality issues in blockchains but the technology is not yet ready for production deployments. Indistinguishability obfuscation (IO) allows for code obfuscation, which is a very ripe research topic in cryptography and, if applied to blockchains, can serve as an unbreakable obfuscation mechanism that will turn smart contracts into a black box. The key idea behind IO is what's called by researchers a multilinear jigsaw puzzle, which basically obfuscates program code by mixing it with random elements, and if the program is run as intended, it will produce expected output but any other way of executing would render the program look random and garbage. This idea was first proposed by Sahai and others in their research paper Candidate Indistinguishability Obfuscation and Functional Encryption for All Circuits. Homomorphic encryption: This type of encryption allows operations to be performed on encrypted data. Imagine a scenario where the data is sent to a cloud server for processing. The server processes it and returns the output without knowing anything about the data that it has processed. This is also an area ripe for research and fully homomorphic encryption that allows all operations on encrypted data is still not fully deployable in production; however, major progress in this field has already been made. Once implemented on blockchains, it can allow processing on cipher text which will allow privacy and confidentiality of transactions inherently. For example, the data stored on the blockchain can be encrypted using homomorphic encryption and computations can be performed on that data without the need for decryption, thus providing privacy service on blockchains. This concept has also been implemented in a project named Enigma by MIT's Media Lab. Enigma is a peer-to-peer network which allows multiple parties to perform computations on encrypted data without revealing anything about the data. Zero knowledge proofs: Zero knowledge proofs have recently been implemented in Zcash successfully, as seen in previous chapters. More specifically, SNARKs have been implemented in order to ensure privacy on the blockchain. The same idea can be implemented in Ethereum and other blockchains also. Integrating Zcash on Ethereum is already a very active research project being run by the Ethereum R&D team and the Zcash Company. State channels: Privacy using state channels is also possible, simply due to the fact that all transactions are run off-chain and the main blockchain does not see the transaction at all except the final state output, thus ensuring privacy and confidentiality. Secure multiparty computation: The concept of secure multiparty computation is not new and is based on the notion that data is split into multiple partitions between participating parties under a secret sharing mechanism which then does the actual processing on the data without the need of the reconstructing data on single machine. The output produced after processing is also shared between the parties. MimbleWimble: The MimbleWimble scheme was proposed somewhat mysteriously on the bitcoin IRC channel and since then has gained a lot of popularity. MimbleWimble extends the idea of confidential transactions and Coinjoin, which allows aggregation of transactions without requiring any interactivity. However, it does not support the use of bitcoin scripting language along with various other features of standard Bitcoin protocol. This makes it incompatible with existing Bitcoin protocol. Therefore, it can either be implemented as a sidechain to bitcoin or on its own as an alternative cryptocurrency. This scheme can address privacy and scalability issues both at once. The blocks created using the MimbleWimble technique do not contain transactions as in traditional bitcoin blockchains; instead, these blocks are composed of three lists: an input list, output list, and something called excesses which are lists of signatures and differences between outputs and inputs. The input list is basically references to the old outputs, and the output list contains confidential transactions outputs. These blocks are verifiable by nodes by using signatures, inputs, and outputs to ensure the legitimacy of the block. In contrast to bitcoin, MimbleWimble transaction outputs only contain pubkeys, and the difference between old and new outputs is signed by all participants involved in the transactions. Coinjoin: Coinjoin is a technique which is used to anonymize the bitcoin transactions by mixing them interactively. The idea is based on forming a single transaction from multiple entities without causing any change in inputs and outputs. It removes the direct link between senders and receivers, which means that a single address can no longer be associated with transactions, which could lead to identification of the users. Coinjoin needs cooperation between multiple parties that are willing to create a single transaction by mixing payments. Therefore, it should be noted that, if any single participant in the Coinjoin scheme does not keep up with the commitment made to cooperate for creating a single transaction by not signing the transactions as required, then it can result in a denial of service attack. In this protocol, there is no need for a single trusted third party. This concept is different from mixing a service which acts as a trusted third party or intermediary between the bitcoin users and allows shuffling of transactions. This shuffling of transactions results in the prevention of tracing and the linking of payments to a particular user. Security Even though blockchains are generally secure and make use of asymmetric and symmetric cryptography as required throughout the blockchain network, there still are few caveats that can result in compromising the security of the blockchain. There are two types of Contract Verification that can solve the issue of Security. Why3 formal verification: Formal verification of solidity code is now available as a feature in the solidity browser. First the code is converted into Why3 language that the verifier can understand. In the example below, a simple solidity code that defines the variable z as maximum limit of uint is shown. When this code runs, it will result in returning 0, because uint z will overrun and start again from 0. This can also be verified using Why3, which is shown below: Once the solidity is compiled and available in the formal verification tab, it can be copied into the Why3 online IDE available at h t t p ://w h y 3. l r i . f r /t r y /. The example below shows that it successfully checks and reports integer overflow errors. This tool is under heavy development but is still quite useful. Also, this tool or any other similar tool is not a silver bullet. Even formal verification generally should not be considered a panacea because specifications in the first place should be defined appropriately: Oyente tool: Currently, Oyente is available as a Docker image for easy testing and installation. It is available at https://github.com/ethereum/oyente, and can be quickly downloaded and tested. In the example below, a simple contract taken from solidity documentation that contains a re-entrancy bug has been tested and it is shown that Oyente successfully analyzes the code and finds the bug: This sample code contains a re-entrancy bug which basically means that if a contract is interacting with another contract or transferring ether, it is effectively handing over the control to that other contract. This allows the called contract to call back into the function of the contract from which it has been called without waiting for completion. For example, this bug can allow calling back into the withdraw function shown in the preceding example again and again, resulting in getting Ethers multiple times. This is possible because the share value is not set to 0 until the end of the function, which means that any later invocations will be successful, resulting in withdrawing again and again. An example is shown of Oyente running to analyze the contract shown below and as can be seen in the following output, the analysis has successfully found the re-entrancy bug. The bug is proposed to be handled by a combination of the Checks-Effects-Interactions pattern described in the solidity documentation: We discussed the scalability, security, confidentiality, and privacy aspects of blockchain technology in this article. If you found it useful, go ahead and buy the book, Mastering Blockchain by Imran Bashir to become a professional Blockchain developer.      

Eric Wastl created the website Advent of Code, a website that published a new programming exercise from the first of December until Christmas. I came across this website somewhere in the first few days of December and as I participated in the ACM ICPC in the past, I expected I should be able to solve these problems. I decided it would be a good idea to use Swift to write these solutions. While solving the problems, I came across one problem that I was able to do really well in Swift and I'd like to explain that one in this post. Introduction After reading the problem, I immediately noticed some interesting points: We can model the input as a graph, where each wire is a vertex and each connection in the circuit connects some vertices to another vertex. For example x AND y -> z connect both x and y to vertex z. The example input is ordered in such a way that you can just iterate over the lines from top to bottom and apply the changes. However, the real input does not have this ordering. To get the real input in the correct ordering, one should note that the input is basically a DAG. Or at least it should be, otherwise it cannot be solved. This means we can use topological sorting to sort the vertices of the graph in the order we should walk them. Although in the example input, it seems that AND, OR, NOT, LSHIFT and RSHFT always operated on a wire, this is not the case. They can also operate on a constant value. Implementation Note that I replaced some guard lines with forced unwrapping here. The source code linked at the end contains the original code. First off, we define a Source, which is an element of an operation, i.e. in x AND y both x and y are a Source: enum Source { case Vertex(String) case Constant(UInt16) func valueForGraph(graph: Graph) -> UInt16 { switch self { case let .Vertex(vertex): return graph.vertices[vertex]!.value! case let .Constant(val): return val } } var vertex: String? { switch self { case let .Vertex(v): return v case .Constant(_): return nil } } static func parse(s: String) -> Source { if let i = UInt16(s) { return .Constant(i) } else { return .Vertex(s) } } } A Source is either a Vertex (i.e. a wire) and then it has a corresponding string as identifier, or it is a constant and then it contains some value. We define a function that will return the value for this Source given a Graph (more on this later). For a constant source the whole graph does not matter, but for a wire we should look up the value in the graph. The second function is used to extract the identifier of the vertex of the source, if any. Finally we also have a function that helps us parse a string or integer into a Source. Next up we have an Operation enumeration, which holds all information about one line of input: enum Operation { case Assign(Source) case And(Source, Source) case Or(Source, Source) case Not(Source) case LeftShift(Source, UInt16) case RightShift(Source, UInt16) func applytoGraph(graph: Graph, vertex: String) { let v = graph.vertices[vertex]! switch self { case let .Assign(source1): v.value = source1.valueForGraph(graph) case let .And(source1, source2): v.value = source1.valueForGraph(graph) & source2.valueForGraph(graph) /* etc for other cases */ case let .RightShift(source1, bits): v.value = source1.valueForGraph(graph) >> bits } } static func parseOperation(input: String) -> Operation { if let and = input.rangeOfString(" AND ") { let before = input.substringToIndex(and.startIndex) let after = input.substringFromIndex(and.endIndex) return .And(Source.parse(before), Source.parse(after)) } /* etc for other options */ var sourceVertices: [String] { /* code that switches on self and extracts vertex from each source */ } The operation enum has a static function that allows us to parse a line from the input into an Operation and a function that will allow us to apply it to a graph. Furthermore, it has a computed variable that returns all source vertices for the operation. Now a Vertex is an easy class: class Vertex { var idx: String var outgoing: Set<String> var incoming: Set<String> var operations: [Operation] var value: UInt16? init(idx: String) { self.idx = idx self.outgoing = [] self.incoming = [] self.operations = [] } } It has an ID and keeps track of a set of incoming and outgoing edges (we need both for topological sorting). Furthermore, it has a value (which is initially not set) and a list of operations that has this vertex as target. Because we want to store vertices in a set, we need to let it conform to Equatable and Hashable. Because we have a unique string identifier for each vertex, this is easy: extension Vertex: Equatable {} func ==(lhs: Vertex, rhs: Vertex) -> Bool { return lhs.idx == rhs.idx } extension Vertex: Hashable { var hashValue: Int { return self.idx.hashValue } } The last structure we need is a graph, which basically hold a list of all vertices: class Graph { var vertices: [String: Vertex] init() { self.vertices = [:] } func addVertexIfNotExists(idx: String) { if let _ = self.vertices[idx] { return } self.vertices[idx] = Vertex(idx: idx) } func addOperation(operation: Operation, target: String) { // Add an operation for a given target to this graph self.addVertexIfNotExists(target) self.vertices[target]?.operations.append(operation) let sourceVertices = operation.sourceVertices for v in sourceVertices { self.addVertexIfNotExists(v) self.vertices[target]?.incoming.insert(v) self.vertices[v]?.outgoing.insert(target) } } } We define a helper function that ads a vertex if not already added. We then use this function to define a function that can add operation to the graph, together with all required vertices and edges. Now we need to be able to topologically sort the vertices of the graph, which can be done using Kahn's Algorithm[MA6] . This[MA7]  can be done in Swift almost exactly using the pseudo-code explained there: extension Graph { func topologicalOrder() -> [Vertex] { var L: [Vertex] = [] var S: Set<Vertex> = Set(vertices.values.filter { $0.incoming.count == 0 }) while S.count > 0 { guard let n = S.first else { fatalError("No more nodes in S") } S.remove(n) L.append(n) for midx in n.outgoing { guard let m = self.vertices[midx] else { fatalError("Can not find vertex") } n.outgoing.remove(m.idx) m.incoming.remove(n.idx) if m.incoming.count == 0 { S.insert(m) } } } return L } } Now we are basically done, as we can now write up a function that calculates the value of a given wire in a graph: func getFinalValueInGraph(graph: Graph, vertex: String) -> UInt16? { let topo = graph.topologicalOrder() for vertex in topo { for op in v.operations { op.applytoGraph(graph, vertex: vertex.idx) } } return graph.vertices[vertex]?.value } Conclusions This post (hopefully) gave you some insight intohow I solved one of the bigger Advent of Code problems. As you can see Swift has some really nice features that help in this case, like enums with types and functional methods like filter. If you like these kinds of problems I suggest you go to the Advent of Code website and start solving the problems. There are quite a few that are really easy to get started. The complete code for this blogpost can be found at my GitHub account. About the author Nicky Gerritsen is currently a Software Architect at StreamOne, a small Dutch company specialized in video streaming and storage. In his spare time he loves to code on Swift projects and learn about new things Swift. He can be found on Twitter @nickygerritsen and on GitHub: https://github.com/nickygerritsen/.

Introduced by Apple, CoreML is a machine learning framework that powers the iOS app developers to integrate machine learning technology into their apps. It supports natural language processing (NLP), image analysis, and various other conventional models to provide a top-notch on-device performance with minimal memory footprint and power consumption. This article is an extract taken from the book Machine Learning with Core ML written by Joshua Newnham. In this article, you will get to know the basics of what CoreML is and its typical workflow. With the release of iOS 11 and Core ML, performing inference is just a matter of a few lines of code. Prior to iOS 11, inference was possible, but it required some work to take a pre-trained model and port it across using an existing framework such as Accelerate or metal performance shaders (MPSes). Accelerate and MPSes are still used under the hood by Core ML, but Core ML takes care of deciding which underlying framework your model should use (Accelerate using the CPU for memory-heavy tasks and MPSes using the GPU for compute-heavy tasks). It also takes care of abstracting a lot of the details away; this layer of abstraction is shown in the following diagram: There are additional layers too; iOS 11 has introduced and extended domain-specific layers that further abstract a lot of the common tasks you may use when working with image and text data, such as face detection, object tracking, language translation, and named entity recognition (NER). These domain-specific layers are encapsulated in the Vision and natural language processing (NLP) frameworks; we won't be going into any details of these frameworks here, but you will get a chance to use them in later chapters: It's worth noting that these layers are not mutually exclusive and it is common to find yourself using them together, especially the domain-specific frameworks that provide useful preprocessing methods we can use to prepare our data before sending to a Core ML model. So what exactly is Core ML? You can think of Core ML as a suite of tools used to facilitate the process of bringing ML models to iOS and wrapping them in a standard interface so that you can easily access and make use of them in your code. Let's now take a closer look at the typical workflow when working with Core ML. CoreML Workflow As described previously, the two main tasks of a ML workflow consist of training and inference. Training involves obtaining and preparing the data, defining the model, and then the real training. Once your model has achieved satisfactory results during training and is able to perform adequate predictions (including on data it hasn't seen before), your model can then be deployed and used for inference using data outside of the training set. Core ML provides a suite of tools to facilitate getting a trained model into iOS, one being the Python packaged released called Core ML Tools; it is used to take a model (consisting of the architecture and weights) from one of the many popular packages and exporting a .mlmodel file, which can then be imported into your Xcode project. Once imported, Xcode will generate an interface for the model, making it easily accessible via code you are familiar with. Finally, when you build your app, the model is further optimized and packaged up within your application. A summary of the process of generating the model is shown in the following diagram:   The previous diagram illustrates the process of creating the .mlmodel;, either using an existing model from one of the supported frameworks, or by training it from scratch. Core ML Tools supports most of the frameworks, either internal or as third party plug-ins, including  Keras, Turi, Caffe, scikit-learn, LibSVN, and XGBoost frameworks. Apple has also made this package open source and modular for easy adaption for other frameworks or by yourself. The process of importing the model is illustrated in this diagram: In addition; there are frameworks with tighter integration with Core ML that handle generating the Core ML model such as Turi Create, IBM Watson Services for Core ML, and Create ML. We will be introducing Create ML in chapter 10; for those interesting in learning more about Turi Create and IBM Watson Services for Core ML then please refer to the official webpages via the following links: Turi Create; https://github.com/apple/turicreate IBM Watson Services for Core ML; https://developer.apple.com/ibm/ Once the model is imported, as mentioned previously, Xcode generates an interface that wraps the model, model inputs, and outputs. Thus, in this post, we learned about the workflow of training and how to import a model. If you've enjoyed this post, head over to the book  Machine Learning with Core ML  to delve into the details of what this model is and what Core ML currently supports. Emotional AI: Detecting facial expressions and emotions using CoreML [Tutorial] Build intelligent interfaces with CoreML using a CNN [Tutorial] Watson-CoreML: IBM and Apple’s new machine learning collaboration project