How-To Tutorials

As an application developer, you must be familiar with the complexity of deploying apps to a fleet of servers with minimum down time. AWS introduced a new service called AWS Code Deploy to ease out the deployment of applications to an EC2 instance on the AWS cloud. Before explaining the full process, I will assume that you are using AWS VPC and are having all of your EC2 instances inside VPC, and that each instance is having an IAM role. Let's see how we can deploy a Node.js application to AWS. Install AWS Code Deploy Agent The first thing you need to do is to install aws-codedeploy-agent on each machine that you want your code deployed on. Before installing a client, please make sure that you have trust relationship for codedeploy.us-west-2.amazonaws.com and codedeploy.us-east-1.amazonaws.com added in IAM role that EC2 instance is using. Not sure what it is? Then, click on the top left dropdown with your account name in AWS console, select Security Credentials option you will be redirected to a new page, select Roles from left menu and look for IAM role that EC2 instance is using, click it and scroll to them bottom, and you will see Edit Trust Relationship button. Click this button to edit trust relationship, and make sure it looks like the following. ... "Principal": { "Service": [ "ec2.amazonaws.com", "codedeploy.us-west-2.amazonaws.com", "codedeploy.us-east-1.amazonaws.com" ] } ... Ok we are good to install the AWS Code Deploy Agent, so make sure ruby2.0 is installed. Use the following script to install code deploy agent. aws s3 cp s3://aws-codedeploy-us-east-1/latest/install ./install-aws-codedeploy-agent --region us-east-1 chmod +x ./install-aws-codedeploy-agent sudo ./install-aws-codedeploy-agent auto rm install-aws-codedeploy-agent Ok, hopefully nothing will go wrong and agent will be installed up and running. To check if its running or not, try the following command: sudo service codedeploy-agent status Let's move to the next step. Create Code Deploy Application Login to your AWS account. Under Deployment & Management click on Code Deploy link, on next screen click on the Get Started Now button and complete the following things: Choose Custom Deployment and click the Skip Walkthrough button. Create New Application; the following are steps to create an application. –            Application Name: display name for application you want to deploy. –            Deployment Group Name: this is something similar to environments like LIVE, STAGING and QA. –            Add Instances: you can choose Amazon EC2 instances by name group name etc. In case you are using autoscaling feature, you can add that auto scaling group too. –            Deployment Config: its a way to specify how we want to deploy application, whether we want to deploy one server at-a-time or half of servers at-a-time or deploy all at-once. –            Service Role: Choose the IAM role that has access to S3 bucket that we will use to hold code revisions. –            Hit the Create Application button. Ok, we just created a Code Deploy application. Let's hold it here and move to our NodeJs app to get it ready for deployment. Code Revision Ok, you have written your app and you are ready to deploy it. The most important thing your app need is appspec.yml. This file will be used by code deploy agent to perform various steps during the deployment life cycle. In simple words the deployment process includes the following steps: Stop the previous application if already deployed; if its first time then this step will not exist. Update the latest code, such as copy files to the application directory. Install new packages or run DB migrations. Start the application. Check if the application is working. Rollback if something went wrong. All above steps seem easy, but they are time consuming and painful to perform each time. Let's see how we can perform these steps easily with AWS code deploy. Lets say we have a following appspec.yml file in our code and also we have bin folder in an app that contain executable sh scripts to perform certain things that I will explain next. First of all take an example of appspec.yml: version: 0.0 os: linux files: - source: / destination: /home/ec2-user/my-app permissions: - object: / pattern: "**" owner: ec2-user group: ec2-user hooks: ApplicationStop: - location: bin/app-stop timeout: 10 runas: ec2-user AfterInstall: - location: bin/install-pkgs timeout: 1200 runas: ec2-user ApplicationStart: - location: bin/app-start timeout: 60 runas: ec2-user ValidateService: - location: bin/app-validate timeout: 10 runas: ec2-user It's a way to tell Code Deploy to copy and provide a destination of those files. files: - source: / destination: /home/ec2-user/my-app We can specify the permissions to be set for the source file on copy. permissions: - object: / pattern: "**" owner: ec2-user group: ec2-user Hooks are executed in an order during the Code Deploy life cycle. We have ApplicationStop, DownloadBundle, BeforeInstall, Install, AfterInstall, ApplicationStart and ValidateService hooks that all have the same syntax. hooks: deployment-lifecycle-event-name - location: script-location timeout: timeout-in-seconds runas: user-name location is the relative path from code root to script file that you want to execute. timeout is the maximum time a script can run. runas is an os user to run the script, and some time you may want to run a script with diff user privileges. Lets bundle your app, exclude the unwanted files such as node_modules folder, and zip it. I use AWS CLI to deploy my code revisions, but you can install awscli using PPI (Python Package Index). sudo pip install awscli I am using awscli profile that has access to s3 code revision bucket in my account. Here is code sample that can help: aws --profile simsaw-baas deploy push --no-ignore-hidden-files --application-name MY_CODE_DEPLOY_APP_NAME --s3-location s3://MY_CODE_REVISONS/MY_CODE_DEPLOY_APP_NAME/APP_BUILD --source MY_APP_BUILD.zip Now Code Revision is published to s3 and also the same revision is registered with the Code Deploy application with the name MY_CODE_DEPLOY_APP_NAME (it will be name of the application you created earlier in the second step.) Now go back to AWS console, Code Deploy. Deploy Code Revision Select your Code Deploy application from the application list show on the Code Deploy Dashboard. It will take you to the next window where you can see the published revision(s), expand the revision and click on Deploy This Revision. You will be redirected to a new window with options like application and deployment group. Choose them carefully and hit deploy. Wait for magic to happen. Code Deploy has a another option to deploy your app from github. The process for it will be almost the same, except you need not push code revisions to S3. About the author Ankit Patial has a Masters in Computer Applications, and nine years of experience with custom APIs, web and desktop applications using .NET technologies, ROR and NodeJs. As a CTO with SimSaw Inc and Pink Hand Technologies, his job is to learn and help his team to implement the best practices of using Cloud Computing and JavaScript technologies.

Minecraft has shaped a generation of gamers. It's popular with all ages, but elementary age kids live and breath it. Inevitably, someone starts to wonder whether this is a good thing, whether that be parents, teachers, or the media. But something that is often overlooked is the influence that Minecraft Mods can have on that equation. Minecraft Mods come in many different flavors and varieties. The Minecraft Modding community is perhaps the largest such community to exist, and Minecraft lends itself well to presenting real world problems as part of the game world. Due to the shear number of mods, you can find many examples that incorporate some form of beneficial learning that has real world applications. For example, there are mods that incorporate aspects of engineering, systems design, genetics, logic puzzles, computer programming, and more. Vanilla Minecraft aside, let us take a look at the kinds of challenges and tasks that foster learning in various Minecraft Mods. Many mods include some kind of interactive machine system. Whether it be pipes or wires, they all have specific rules for construction and present the player with the challenge of combining a bunch of simple pieces together to create something more. Usually the end result is a factory for manufacturing more items and blocks to build yet more machinery or a power plant for powering all that machinery. Construction typically requires logical problem solving and spatial comprehension. Running a pipe from one end of your factory to the other can be just as complex a piece of spaghetti as in a real factory. There are many mods that focus on these challenges, including Buildcraft, EnderIO, IndustrialCraft2, PneumaticCraft, and more. These mods are also generally capable of interacting with each other seamlessly for even more creative solutions for your factory floor. But factories aren’t the only logic problems that mods present. There are also many logic puzzles built into mods. My own mod, Railcraft, has a fully functional Train routing and signaling system. It's strongly based on real life examples of railroads and provides many of the same solutions you’ll find real railway engineers using to solve the challenges of a railway. Problems that a budding railway engineer faces include scheduling, routing, best usage of track, avoiding collisions using signal logic, and more. But there are many other mods out there with similar types of puzzles. Some item management and piping systems are very logic driven. Logistics Pipes and Applied Energetics 2 are a couple such mods that take things just a step beyond normal pipe mods, in terms of both the amount of logical thinking required and the system’s overall capabilities. Both mods allow you to intelligently manage supply and demand of items across an entire base using logic modules that you install in the machines and pipes. This is all well and good of course, but there are some mods that take this even further. When it comes to logic, some mods allow you to actually write computer code. ComputerCraft and OpenComputers are two mods that allow you to use LUA to control in-game displays, robots, and more. There are even add-ons to these mods that allow you to control a Railcraft railway network from an in-game computer screen. Robot programming is generally very similar to the old “move the turtle around the screen” introductory programming lessons; ComputerCraft even calls its robots Turtles. You can instruct them to move and interact with the world, creating complex structures or just mining out an entire area for ore. The more complex the task, the more complex the code required. However, while mechanical and logic based problems are great, they are far from all that Minecraft Mods have to offer. Another area that has received a lot of attention from mods is Genetics. The first major mod to pioneer Genetics was IndustrialCraft2 with its Crop Breeding mechanics. As an added bonus, IC2 also provides an interesting power system. However, when most people think of Genetics in Mods, the first mod they think of is the Forestry Mod by SirSengir. In Forestry, you can breed Bees, Trees, and Butterflies. But there are other mods, too, such as Mariculture, which allows you to breed Fish. Genetics systems generally provide a wide range of traits and abilities that can be bred into the organisms: for example, increasing the yields of crops or improving the speed at which your bees work, and even breeding in more interesting traits, such as bees that heal you when you get close to the hive. The systems are generally fairly representative of Mendelian Inheritance: each individual has two sets of genes, each one a random set of genes from each parent. There are dominant and recessive genes and the two sets combined give your individual its specific traits. Punnett Squares are encouraged, just like those taught in school. Speaking of school, no discussion of learning and Minecraft would be complete without at least mentioning MinecraftEdu. TeacherGaming, the company behind MinecraftEdu, partnered with Mojang to provide a version of Minecraft specifically tailored for school programs. Of note is the fact that MinecraftEdu ships ready for use with mods and even recommends some of the mods mentioned in this post, including a special version of ComputerCraft created just for the MinecraftEdu project. Real schools use this stuff in real classrooms, teaching kids lessons using Minecraft and Minecraft Mods. So yes, there are many things that can be learned by playing Minecraft, especially if you play with the right mods. So should we be worried about the current generational obsession with Minecraft? Probably not. There are much less edifying things these kids could be playing. So next time your kid starts lecturing you about Mendelian Inheritance, remember it's quite possible he learned about it while playing Minecraft. About the Author Aaron Mills was born in 1983 and lives in the Pacific Northwest, which is a land rich in lore, trees, and rain. He has a Bachelor's Degree in Computer Science and studied at Washington State University Vancouver. He is best known for his work on the Minecraft Mod, Railcraft, but has also contributed significantly to the Minecraft Mods of Forestry and Buildcraft as well some contributions to the Minecraft Forge project.

Here's another reason why Rust is the present and the future in programming. Few days ago, Microsoft announced that they are going to start exploring Rust and skip their own C languages. This announcement was made by the Principal Security Engineering Manager of Microsoft Security Response Centre (MSRC), Gavin Thomas. Thomas states that ~70% of the vulnerabilities which Microsoft assigns a CVE each year are caused by developers, who accidently insert memory corruption bugs into their C and C++ code. He adds, "As Microsoft increases its code base and uses more Open Source Software in its code, this problem isn’t getting better, it's getting worse. And Microsoft isn’t the only one exposed to memory corruption bugs—those are just the ones that come to MSRC." Image Source: Microsoft blog He highlights the fact that even after having so many security mechanisms (like static analysis tools, fuzzing at scale, taint analysis, many encyclopaedias of coding guidelines, threat modelling guidance, etc) to make a code secure, developers have to invest a lot of time in studying about more tools for training and vulnerability fixes. Thomas states that though C++ has many qualities like fast, mature, small memory and disk footprint, it does not have the memory security guarantee of languages like .NET C#. He believes that Rust is one language, which can provide both the requirements. Thomas strongly advocates that a software security industry should focus on providing a secure environment for developers to work on, rather than turning deaf ear to the importance of security, outdated methods and approaches. He thus concludes by hinting that Microsoft is going to adapt the Rust programming language. As he says that, "Perhaps it's time to scrap unsafe legacy languages and move on to a modern safer system programming language?" Microsoft exploring Rust is not surprising as Rust has been popular with many developers for its simpler syntax, less bugs, memory safe and thread safety. It has also been voted as the most loved programming language, according to the 2019 StackOverflow survey, the biggest developer survey on the internet. It allows developers to focus on their applications, rather than worrying about its security and maintenance. Recently, there have been many applications written in Rust, like Vector, Brave ad-blocker, PyOxidizer and more. Developers couldn't agree more with this post, as all have expressed their love for Rust. https://twitter.com/alilleybrinker/status/1151495738158977024 https://twitter.com/karanganesan/status/1151485485644054528 https://twitter.com/shah_sheikh/status/1151457054004875264 A Redditor says, "While this first post is very positive about memory-safe system programming languages in general and Rust in particular, I would not call this an endorsement. Still, great news!" Visit the Microsoft blog for more details. Introducing Ballista, a distributed compute platform based on Kubernetes and Rust EU Commission opens an antitrust case against Amazon on grounds of violating EU competition rules Fastly CTO Tyler McMullen on Lucet and the future of WebAssembly and Rust [Interview]

(For more resources related to this topic, see here.) But why Mobile First? Why did Bootstrap completely change its course from Desktop First to Mobile First to get into this new way to develop more suitable websites and web applications? Why did the most popular frontend framework embrace this change at a time when responsive web design is continuously growing with better suited and standard techniques such as media-queries, fluid layout, and JavaScript on demand? Mobile browsers are increasing support for the brand new HTML5 and CSS3, with the philosophy to offer, for older browsers, a less stylized but fully functional component, and for capable browsers a rich and full experience that comes from mobiles to larger screens such as TVs. For older browsers (such as IE 8 and IE 9), Bootstrap has functional support, but enhanced features such as rounded corners and a placeholder attribute for tips in input fields are not supported for these browsers. To see the full details on browser support, check the Bootstrap documentation from the Getting started section (http://getbootstrap.com/getting-started/#browsers). We are living at a time when mobile use is increasing at a pace that will soon surpass desktop usage (http://www.businessinsider.com/mobile-will-eclipsedesktop-by-2014-2012-6). Apart from the statistics, one thing we can presume is that the web scenario is changing so fast that we have to embrace the certainty of devices getting better and smarter. In this article, we will explore the main changes in Bootstrap 3. If you are already familiar with Bootstrap 2, check the migration guide (http://getbootstrap.com/getting-started/#migration) to have a practical overview about what has changed. If you're not familiar with Bootstrap, there's nothing that's too difficult for you to understand directly from this article about this new version. The only thing you need to have in mind is the Mobile First approach, which is covered well in this article. You will be guided to design with Mobile First, discover why Mobile First is so important, and how to make Bootstrap a powerful frontend platform to make your site friendly for a wider range of devices. We can take a step further and add to your previous Bootstrap knowledge by thinking of a concrete way to design processes as a continuous layer of capabilities and embrace the constraints and not fight with them. Mobile First with Bootstrap is an elegant solution for frontend development. Combined with server-side techniques, we get a full bag of solutions to get your product better suited to different users and needs in different platforms. This article will cover the following topics: Bootstrap reviewed Desktop to responsive   Bootstrap reviewed In the third era of Bootstrap that is coming, the developers have redesigned the whole framework with a different approach. Let's get started building interface components of small and simple screens, instead of adapting the existent UI components to fit in a constrained environment. From mobile, we will then go to desktop. However, we will not adapt the experience as we usually do with responsive design going from desktop to mobile. Now with Mobile First we will enhance accordingly as we increase the device screens. Why should I do this if my target audience will be using desktops? Going to mobile indirectly benefits desktop users. But how? To better understand this, let's recap Bootstrap history for a while. In 2011, Bootstrap was launched to serve as a live and agnostic style guide that was used by Twitter to create their products. It became an open source framework at that time. It was a time when we worked in pixel-perfect layouts and explored CSS3 animations, and we found in Bootstrap a well-documented and standardized set of features. Bootstrap creates a new design for the browsers because you don't need to define basic interface elements from scratch, such as buttons. At the same time, you have utility elements like badges to cover the most common interface elements. Bootstrap does what a framework is supposed to do: Bootstrapping! The term means the act of taking off a new project; it's like saying, "give me the tools that I will need to start developing my application for different needs". Bootstrap is a toolkit belt with standard conventions from well-defined classes with clean and practical documentation to live code that is ready to use and be customized for your needs. It's not a magic solution to solve the interface element reuse issue, but it's a kick-start. It fits in so many scenarios that developers are increasing its use with their own tools. "CSS moved beyond type, forms and grids. People get tired to create the same stuffs"— Mark Otto, one of Bootstrap's creators, in the Desktop First to Mobile First Bootstrap presentation (https://speakerdeck.com/mdo/desktop-firstto- with-bootstrap) A must-have from this breeding ground of possibilities is the Bootstrap extension font-awesome (http://fortawesome.github.io/Font-Awesome/). It uses font-face, which is widely supported and flexible, instead of sprites for icons. With a single CSS file and font resources used to render the custom fonts, you have a tool that can handle all your icons. This shows the flexibility of Bootstrap tools; for example, font-awesome is independent, works as a standalone project, and is a great fit with Bootstrap. There are a lot of ways to use Bootstrap. You can customize and extend components, from editing the source code in LESS variables or customize via the Bootstrap download page (http://getbootstrap.com/customize/). At the time of this writing, Bootstrap is the most popular project on Github, so it's just one more reason to consider its importance. There's now an official Bootstrap Expo (http://expo.getbootstrap.com/). This is one of the changes in this new version. Bootstrap Expo is the official directory for websites and web applications that are being developed using this framework. A lot of developers get their first touch with the capabilities of HTML5 and CSS3 with this framework. Bootstrap has amazing capabilities such as offering a responsive grid, dozens of JavaScript components, and a customizer in a web interface or through the LESS variables, if you're an experienced developer. It's suitable for any level of developer and designers because it has solutions that suit both scenarios. This is the second of Bootstrap's main philosophies—it's made for everyone. Desktop to responsive With the rise of smart phones, there is a need for responsive content to cover the growing demand. It's possible to add an optional file with media queries and a bunch of CSS code and be adapted to mobile needs. Media queries, a CSS3 module introduced in June 2012, is a basic structure that gives a namespace with a bunch of CSS rules and declarations according to the user resolution, density, and screen capabilities. So, with CSS files, it is possible to manage the ongoing rise of smartphones. It was possible with just one stylesheet file with good support to adapt according with the device and make a website mobile friendly. In Bootstrap Version 2, we used to have an optional file (responsive.less) that used to have all the media queries necessary for Bootstrap to work well with mobiles. Another good news is that we can adapt to tablets as a bonus. We have breakpoints for the most common mobile resolutions—this means we have a range of width (768 px to 979 px) that can represent tablet devices. A breakpoint is the extreme point (minimum and/or maximum) where you can define CSS rules specific to that range and change your layout. This could be achieved with a simple declaration of media queries in your CSS: @media (min-width: 768px) and (max-width: 979px) { ... } But sometimes it's indispensable to rethink some elements—some of those already developed only for desktops—in a pixel-perfect scenario. There's no flexibility in a pixel-width accommodation. No matter how much the screen is different, the website will behave like you were using a desktop when we work with fixed units. This is when we can use a bunch of media queries to get more flexible. Even with this solution, redefining dimensions and CSS rules according to the device using media queries will solve screen flexibility issues but not solve performance issues on mobiles. Performance is one of the main concerns when we go mobile. We have to consider scenarios where the Internet connection is slow and it is a recurrent issue. You will have to perform reverse engineering to make your JavaScript optimize loading, and combine it with server-side solutions. A worse solution would be to just hide content after considering what could be painful for your page load; for example, images have a deep impact on the final performance. Lower page response time is equivalent to more money spent, as we can see in this article about page loading versus user patience (http://blog.kissmetrics.com/loading-time/). One of the curious things this research points to is that mobile Internet users expect their browsing experience in phones to be comparable to what they get on their desktops. We are living at a time when the Web is filled with rich content and we have faster Internet connections. We have to be prepared to offer the closest thing to a fast and optimized loading, at least for our most important content. This does not involve just the use of CSS to hide content and show content depending on the device, as we can do using media queries. It's all about keeping the concepts simple and focused and developing each interface component thoughtfully from scratch—the primary use, with the constraints and its enhanced capabilities. It's not just about adapting, it's exploring the device's capabilities and delivering the best user experience across platforms. Sounds familiar? Yes, for sure, the same concept as progressive enhancement, you might think. You're not wrong. Progressive enhancement was a term widely used at a time when we talked about HTML page dependency on JavaScript to be functional. Progressive enhancement is a strategy for web design that relies on semantic markup and technologies such as JavaScript. Nowadays, progressive enhancement is a longer term for Mobile First because it's not just about JavaScript disabled, as it was vastly talked before. A hundred of articles tried to show its benefits in a no JavaScript environment scenarios. Now progressive enhancement is about to be faster (http://coding.smashingmagazine.com/2013/09/03/progressive-enhancement-is-faster/). Progressive enhancement is one of the three keys of Mobile First, together with responsive design and giving priority to content over navigation. So, these three rationales are at the background of all the details of Bootstrap 3, from your CSS components to your grid structure. Summary In this article we saw the Twitter Bootstrap's latest version Mobile First. We also saw how developers developed this framework. The growing world of smartphones have forced for the need for Mobile First. Resources for Article: Further resources on this subject: Downloading and setting up Bootstrap [Article] Introduction to RWD frameworks [Article] Getting started with using Chef [Article]

Welcome back to our how-to on Building and Running 12 Factor Microservices on Docker. In Part 1, we introduced a very simple python flask application which displayed a list of users from a relational database. Then we walked through the first four of these factors, reworking the example application to follow these guidelines. In Part 2, we'll be introducing a multi-container Docker setup as the execution environment for our application. We’ll continue from where we left off with the next factor, number five. Build, Release, Run. A 12-factor app strictly separates the process for transforming a codebase into a deploy into distinct build, release, and run stages. The build stage creates an executable bundle from a code repo, including vendoring dependencies and compiling binaries and asset packages. The release stage combines the executable bundle created in the build with the deploy’s current config. Releases are immutable and form an append-only ledger; consequently, each release must have a unique release ID. The run stage runs the app in the execution environment by launching the app’s processes against the release. This is where your operations meet your development and where a PaaS can really shine. For now, we’re assuming that we’ll be using a Docker-based containerized deploy strategy. We’ll start by writing a simple Dockerfile. The Dockerfile starts with an ubuntu base image and then I add myself as the maintainer of this app. FROM ubuntu:14.04.2 MAINTAINER codyaray Before installing anything, let’s make sure that apt has the latest versions of all the packages. RUN echo "deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -sc) main universe" >> /etc/apt/sources.list RUN apt-get update Install some basic tools and the requirements for running a python webapp RUN apt-get install -y tar curl wget dialog net-tools build-essential RUN apt-get install -y python python-dev python-distribute python-pip RUN apt-get install -y libmysqlclient-dev Copy over the application to the container. ADD /. /src Install the dependencies. RUN pip install -r /src/requirements.txt Finally, set the current working directory, expose the port, and set the default command. EXPOSE 5000 WORKDIR /src CMD python app.py Now, the build phase consists of building a docker image. You can build and store locally with docker build -t codyaray/12factor:0.1.0 . If you look at your local repository, you should see the new image present. $ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE codyaray/12factor 0.1.0 bfb61d2bbb17 1 hour ago 454.8 MB The release phase really depends on details of the execution environment. You’ll notice that none of the configuration is stored in the image produced from the build stage; however, we need a way to build a versioned release with the full configuration as well. Ideally, the execution environment would be responsible for creating releases from the source code and configuration specific to that environment. However, if we’re working from first principles with Docker rather than a full-featured PaaS, one possibility is to build a new docker image using the one we just built as a base. Each environment would have its own set of configuration parameters and thus its own Dockerfile. It could be something as simple as FROM codyaray/12factor:0.1.0 MAINTAINER codyaray ENV DATABASE_URL mysql://sa:mypwd@mydbinstance.abcdefghijkl.us-west-2.rds.amazonaws.com/mydb This is simple enough to be programmatically generated given the environment-specific configuration and the new container version to be deployed. For the demonstration purposes, though, we’ll call the above file Dockerfile-release so it doesn’t conflict with the main application’s Dockerfile. Then we can build it with docker build -f Dockerfile-release -t codyaray/12factor-release:0.1.0.0 . The resulting built image could be stored in the environment’s registry as codyaray/12factor-release:0.1.0.0. The images in this registry would serve as the immutable ledger of releases. Notice that the version has been extended to include a fourth level which, in this instance, could represent configuration version “0” applied to source version “0.1.0”. The key here is that these configuration parameters aren’t collated into named groups (sometimes called “environments”). For example, these aren’t static files named like Dockerfile.staging or Dockerfile.dev in a centralized repo. Rather, the set of parameters is distributed so that each environment maintains its own environment mapping in some fashion. The deployment system would be setup such that a new release to the environment automatically applies the environment variables it has stored to create a new Docker image. As always, the final deploy stage depends on whether you’re using a cluster manager, scheduler, etc. If you’re using standalone Docker, then it would boil down to docker run -P -t codyaray/12factor-release:0.1.0.0 Processes. A 12-factor app is executed as one or more stateless processes which share nothing and are horizontally partitionable. All data which needs to be stored must use a stateful backing service, usually a database. This means no sticky sessions and no in-memory or local disk-based caches. These processes should never daemonize or write their own PID files; rather, they should rely on the execution environment’s process manager (such as Upstart). This factor must be considered up-front, in line with the discussions on antifragility, horizontal scaling, and overall application design. As the example app delegates all stateful persistence to a database, we’ve already succeeded on this point. However, it is good to note that a number of issues have been found using the standard ubuntu base image for Docker, one of which is its process management (or lack thereof). If you would like to use a process manager to automatically restart crashed daemons, or to notify a service registry or operations team, check out baseimage-docker. This image adds runit for process supervision and management, amongst other improvements to base ubuntu for use in Docker such as obsoleting the need for pid files. To use this new image, we have to update the Dockerfile to set the new base image and use its init system instead of running our application as the root process in the container. FROM phusion/baseimage:0.9.16 MAINTAINER codyaray RUN echo "deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -sc) main universe" >> /etc/apt/sources.list RUN apt-get update RUN apt-get install -y tar git curl nano wget dialog net-tools build-essential RUN apt-get install -y python python-dev python-distribute python-pip RUN apt-get install -y libmysqlclient-dev ADD /. /src RUN pip install -r /src/requirements.txt EXPOSE 5000 WORKDIR /src RUN mkdir /etc/service/12factor ADD 12factor.sh /etc/service/12factor/run # Use baseimage-docker's init system. CMD ["/sbin/my_init"]  Notice the file 12factor.sh that we’re now adding to /etc/service. This is how we instruct runit to run our application as a service. Let’s add the new 12factor.sh file. #!/bin/sh python /src/app.py Now the new containers we deploy will attempt to be a little more fault-tolerant by using an OS-level process manager. Port Binding. A 12-factor app must be self-contained and bind to a port specified as an environment variable. It can’t rely on the injection of a web container such as tomcat or unicorn; instead it must embed a server such as jetty or thin. The execution environment is responsible for routing requests from a public-facing hostname to the port-bound web process. This is trivial with most embedded web servers. If you’re currently using an external web server, this may require more effort to support an embedded server within your application. For the example python app (which uses the built-in flask web server), it boils down to port = int(os.environ.get("PORT", 5000)) app.run(host='0.0.0.0', port=port) Now the execution environment is free to instruct the application to listen on whatever port is available. This obviates the need for the application to tell the environment what ports must be exposed, as we’ve been required to do with Docker. Concurrency. Because a 12-factor exclusively uses stateless processes, it can scale out by adding processes. A 12-factor app can have multiple process types, such as web processes, background worker processes, or clock processes (for cron-like scheduled jobs). As each process type is scaled independently, each logical process would become its own Docker container as well. We’ve already seen building a web process; other processes are very similar. In most cases, scaling out simply means launching more instances of the container. (Its usually not desirable to scale out the clock processes, though, as they often generate events that you want to be scheduled singletons within your infrastructure.) Disposability. A 12-factor app’s processes can be started or stopped (with a SIGTERM) anytime. Thus, minimizing startup time and gracefully shutting down is very important. For example, when a web service receives a SIGTERM, it should stop listening on the HTTP port, allow in-flight requests to finish, and then exit. Similar, processes should be robust against sudden death; for example, worker processes should use a robust queuing backend. You want to ensure the web server you select can gracefully shutdown. The is one of the trickier parts of selecting a web server, at least for many of the common python http servers that I’ve tried.  In theory, shutting down based on receiving a SIGTERM should be as simple as follows. import signal signal.signal(signal.SIGTERM, lambda *args: server.stop(timeout=60)) But often times, you’ll find that this will immediately kill the in-flight requests as well as closing the listening socket. You’ll want to test this thoroughly if dependable graceful shutdown is critical to your application. Dev/Prod Parity. A 12-factor app is designed to keep the gap between development and production small. Continuous deployment shrinks the amount of time that code lives in development but not production. A self-serve platform allows developers to deploy their own code in production, just like they do in their local development environments. Using the same backing services (databases, caches, queues, etc) in development as production reduces the number of subtle bugs that arise in inconsistencies between technologies or integrations. As we’re deploying this solution using fully Dockerized containers and third-party backing services, we’ve effectively achieved dev/prod parity. For local development, I use boot2docker on my Mac which provides a Docker-compatible VM to host my containers. Using boot2docker, you can start the VM and setup all the env variables automatically with boot2docker up $(boot2docker shellinit) Once you’ve initialized this VM and set the DOCKER_HOST variable to its IP address with shellinit, the docker commands given above work exactly the same for development as they do for production. Logs. Consider logs as a stream of time-ordered events collected from all running processes and backing services. A 12-factor app doesn’t concern itself with how its output is handled. Instead, it just writes its output to its `stdout` stream. The execution environment is responsible for collecting, collating, and routing this output to its final destination(s). Most logging frameworks either support logging to stderr/stdout by default or easily switching from file-based logging to one of these streams. In a 12-factor app, the execution environment is expected to capture these streams and handle them however the platform dictates. Because our app doesn’t have specific logging yet, and the only logs are from flask and already to stderr, we don’t have any application changes to make.  However, we can show how an execution environment which could be used handle the logs. We’ll setup a Docker container which collects the logs from all the other docker containers on the same host. Ideally, this would then forward the logs to a centralized service such as Elasticsearch. Here we’ll demo using Fluentd to capture and collect the logs inside the log collection container; a simple configuration change would allow us to switch from writing these logs to disk as we demo here and instead send them from Fluentd to a local Elasticsearch cluster. We’ll create a Dockerfile for our new logcollector container type. For more detail, you can find a Docker fluent tutorial here. We can call this file Dockerfile-logcollector. FROM kiyoto/fluentd:0.10.56-2.1.1 MAINTAINER kiyoto@treasure-data.com RUN mkdir /etc/fluent ADD fluent.conf /etc/fluent/ CMD "/usr/local/bin/fluentd -c /etc/fluent/fluent.conf" We use an existing fluentd base image with a specific fluentd configuration. Notably this tails all the log files in /var/lib/docker/containers/<container-id>/<container-id>-json.log, adds the container ID to the log message, and then writes to JSON-formatted files inside /var/log/docker. <source> type tail path /var/lib/docker/containers/*/*-json.log pos_file /var/log/fluentd-docker.pos time_format %Y-%m-%dT%H:%M:%S tag docker.* format json </source> <match docker.var.lib.docker.containers.*.*.log> type record_reformer container_id ${tag_parts[5]} tag docker.all </match> <match docker.all> type file path /var/log/docker/*.log format json include_time_key true </match> As usual, we create a Docker image. Don’t forget to specify the logcollector Dockerfile. docker build -f Dockerfile-logcollector -t codyaray/docker-fluentd . We’ll need to mount two directories from the Docker host into this container when we launch it. Specifically, we’ll mount the directory containing the logs from all the other containers as well as the directory to which we’ll be writing the consolidated JSON logs. docker run -d -v /var/lib/docker/containers:/var/lib/docker/containers -v /var/log/docker:/var/log/docker codyaray/docker-fluentd Now if you check in the /var/log/docker directory, you’ll see the collated JSON log files. Note that this is on the docker host rather than in any container; if you’re using boot2docker, you can ssh into the docker host with boot2docker ssh and then check /var/log/docker. Admin Processes. Any admin or management tasks for a 12-factor app should be run as one-off processes within a deploy’s execution environment. This process runs against a release using the same codebase and configs as any process in that release and uses the same dependency isolation techniques as the long-running processes. This is really a feature of your app's execution environment. If you’re running a Docker-like containerized solution, this may be pretty trivial. docker run -i -t --entrypoint /bin/bash codyaray/12factor-release:0.1.0.0 The -i flag instructs docker to provide interactive session, that is, to keep the input and output ttys attached. Then we instruct docker to run the /bin/bash command instead of another 12factor app instance. This creates a new container based on the same docker image, which means we have access to all the code and configs for this release. This will drop us into a bash terminal to do whatever we want. But let’s say we want to add a new “friends” table to our database, so we wrote a migration script add_friends_table.py. We could run it as follows: docker run -i -t --entrypoint python codyaray/12factor-release:0.1.0.0 /src/add_friends_table.py As you can see, following the few simple rules specified in the 12 Factor manifesto really allows your execution environment to manage and scale your application. While this may not be the most feature-rich integration within a PaaS, it is certainly very portable with a clean separation of responsibilities between your app and its environment. Much of the tools and integration demonstrated here were a do-it-yourself container approach to the environment, which would be subsumed by an external vertically integrated PaaS such as Deis. If you’re not familiar with Deis, its one of several competitors in the open source platform-as-a-service space which allows you to run your own PaaS on a public or private cloud. Like many, Deis is inspired by Heroku. So instead of Dockerfiles, Deis uses a buildpack to transform a code repository into an executable image and a Procfile to specify an app’s processes. Finally, by default you can use a specialized git receiver to complete a deploy. Instead of having to manage separate build, release, and deploy stages yourself like we described above, deploying an app to Deis could be a simple as git push deis-prod While it can’t get much easier than this, you’re certainly trading control for simplicity. It's up to you to determine which works best for your business. Find more Docker tutorials alongside our latest releases on our dedicated Docker page. About the Author Cody A. Ray is an inquisitive, tech-savvy, entrepreneurially-spirited dude. Currently, he is a software engineer at Signal, an amazing startup in downtown Chicago, where he gets to work with a dream team that’s changing the service model underlying the Internet.

[box type="note" align="" class="" width=""]This article is an excerpt from a book written by Fabio M. Soares and Alan M. F. Souza, titled Neural Network Programming with Java Second Edition. This book covers the current state-of-art in the field of neural network that helps you understand and design basic to advanced neural networks with Java.[/box] Our article explores the power of neural networks in pattern recognition by showcasing how to recognize digits from 0 to 9 in an image. For pattern recognition, the neural network architectures that can be applied are MLPs (supervised) and the Kohonen Network (unsupervised). In the first case, the problem should be set up as a classification problem, that is, the data should be transformed into the X-Y dataset, where for every data record in X there should be a corresponding class in Y. The output of the neural network for classification problems should have all of the possible classes, and this may require preprocessing of the output records. For the other case, unsupervised learning, there is no need to apply labels to the output, but the input data should be properly structured. To remind you, the schema of both neural networks are shown in the next figure: Data pre-processing We have to deal with all possible types of data, i.e., numerical (continuous and discrete) and categorical (ordinal or unscaled).  However, here we have the possibility of performing pattern recognition on multimedia content, such as images and videos. So, can multimedia could be handled? The answer to this question lies in the way these contents are stored in files. Images, for example, are written with a representation of small colored points called pixels. Each color can be coded in an RGB notation where the intensity of red, green, and blue define every color the human eye is able to see. Therefore an image of dimension 100x100 would have 10,000 pixels, each one having three values for red, green and blue, yielding a total of 30,000 points. That is the challenge for image processing in neural networks. Some methods, may reduce this huge number of dimensions. Afterwards an image can be treated as big matrix of numerical continuous values. For simplicity, we are applying only gray-scale images with small dimensions in this article. Text recognition (optical character recognition) Many documents are now being scanned and stored as images, making it necessary to convert these documents back into text, for a computer to apply edition and text processing. However, this feature involves a number of challenges: Variety of text font Text size Image noise Manuscripts In spite of that, humans can easily interpret and read even texts produced in a bad quality image. This can be explained by the fact that humans are already familiar with text characters and the words in their language. Somehow the algorithm must become acquainted with these elements (characters, digits, signalization, and so on), in order to successfully recognize texts in images. Digit recognition Although there are a variety of tools available on the market for OCR, it still remains a big challenge for an algorithm to properly recognize texts in images. So, we will be restricting our application to in a smaller domain, so that we'll face simpler problems. Therefore, in this article, we are going to implement a neural network to recognize digits from 0 to 9 represented on images. Also, the images will have standardized and small dimensions, for the sake of simplicity. Digit representation We applied the standard dimension of 10x10 (100 pixels) in gray scaled images, resulting in 100 values of gray scale for each image: In the preceding image we have a sketch representing the digit 3 at the left and a corresponding matrix with gray values for the same digit, in gray scale. We apply this pre-processing in order to represent all ten digits in this application. Implementation in Java To recognize optical characters, data to train and to test neural network was produced by us. In this example, digits from 0 (super black) to 255 (super white) were considered. According to pixel disposal, two versions of each digit data were created: one to train and another to test. Classification techniques will be used here. Generating data Numbers from zero to nine were drawn in the Microsoft Paint ®. The images have been converted into matrices, from which some examples are shown in the following image. All pixel values between zero and nine are grayscale: For each digit we generated five variations, where one is the perfect digit, and the others contain noise, either by the drawing, or by the image quality. Each matrix row was merged into vectors (Dtrain and Dtest) to form a pattern that will be used to train and test the neural network. Therefore, the input layer of the neural network will be composed of 101 neurons. The output dataset was represented by ten patterns. Each one has a more expressive value (one) and the rest of the values are zero. Therefore, the output layer of the neural network will have ten neurons. Neural architecture So, in this application our neural network will have 100 inputs (for images that have a 10x10 pixel size) and ten outputs, the number of hidden neurons remaining unrestricted. We created a class called DigitExample to handle this application. The neural network architecture was chosen with these parameters: Neural network type: MLP Training algorithm: Backpropagation Number of hidden layers: 1 Number of neurons in the hidden layer: 18 Number of epochs: 1000 Minimum overall error: 0.001 Experiments Now, as has been done in other cases previously presented, let's find the best neural network topology training several nets. The strategy to do that is summarized in the following table:   Experiment Learning rate Activation Functions #1 0.3 Hidden Layer: SIGLOG Output Layer: LINEAR #2 0.5 Hidden Layer: SIGLOG Output Layer: LINEAR #3 0.8 Hidden Layer: SIGLOG Output Layer: LINEAR #4 0.3 Hidden Layer: HYPERTAN Output Layer: LINEAR #5 0.5 Hidden Layer: SIGLOG Output Layer: LINEAR #6 0.8 Hidden Layer: SIGLOG Output Layer: LINEAR #7 0.3 Hidden Layer: HYPERTAN Output Layer: SIGLOG #8 0.5 Hidden Layer: HYPERTAN Output Layer: SIGLOG #9 0.8 Hidden Layer: HYPERTAN Output Layer: SIGLOG The following DigitExample class code defines how to create a neural network to read from digit data: // enter neural net parameter via keyboard (omitted) // load dataset from external file (omitted) // data normalization (omitted) // create ANN and define parameters to TRAIN: Backpropagation backprop = new Backpropagation(nn, neuralDataSetToTrain, LearningAlgorithm.LearningMode.BATCH); backprop.setLearningRate( typedLearningRate ); backprop.setMaxEpochs( typedEpochs ); backprop.setGeneralErrorMeasurement(Backpropagation.ErrorMeasurement.SimpleError); backprop.setOverallErrorMeasurement(Backpropagation.ErrorMeasurement.MSE); backprop.setMinOverallError(0.001); backprop.setMomentumRate(0.7); backprop.setTestingDataSet(neuralDataSetToTest); backprop.printTraining = true; backprop.showPlotError = true; // train ANN: try {    backprop.forward();    //neuralDataSetToTrain.printNeuralOutput();    backprop.train();    System.out.println("End of training");    if (backprop.getMinOverallError() >= backprop.getOverallGeneralError()) {        System.out.println("Training successful!"); } else {        System.out.println("Training was unsuccessful"); }    System.out.println("Overall Error:" + String.valueOf(backprop.getOverallGeneralError()));    System.out.println("Min Overall Error:" + String.valueOf(backprop.getMinOverallError()));    System.out.println("Epochs of training:" + String.valueOf(backprop.getEpoch())); } catch (NeuralException ne) {    ne.printStackTrace(); } // test ANN (omitted) Results After running each experiment using the DigitExample class, excluding training and testing overall errors and the quantity of right number classifications using the test data (table above), it is possible observe that experiments #2 and #4 have the lowest MSE values. The differences between these two experiments are learning rate and activation function used in the output layer. Experiment Training overall error Testing overall error # Right number classifications #1 9.99918E-4 0.01221 2 by 10 #2 9.99384E-4 0.00140 5 by 10 #3 9.85974E-4 0.00621 4 by 10 #4 9.83387E-4 0.02491 3 by 10 #5 9.99349E-4 0.00382 3 by 10 #6 273.70 319.74 2 by 10 #7 1.32070 6.35136 5 by 10 #8 1.24012 4.87290 7 by 10 #9 1.51045 4.35602 3 by 10 The figure above shows the MSE evolution (train and test) by each epoch graphically by experiment #2. It is interesting to notice the curve stabilizes near the 30th epoch: The same graphic analysis was performed for experiment #8. It is possible to check the MSE curve stabilizes near the 200th epoch. As already explained, only MSE values might not be considered to attest neural net quality. Accordingly, the test dataset has verified the neural network generalization capacity. The next table shows the comparison between real output with noise and the neural net estimated output of experiment #2 and #8. It is possible to conclude that the neural network weights by experiment #8 can recognize seven digits patterns better than #2's: Output comparison Real output (test dataset) Digit 0.0 0.0        0.0     0.0     0.0     0.0     0.0     0.0     0.0     1.0 0.0 0.0        0.0     0.0     0.0     0.0     0.0     0.0     1.0     0.0 0.0 0.0        0.0     0.0     0.0        0.0     0.0     1.0     0.0     0.0 0.0 0.0        0.0     0.0     0.0     0.0     1.0     0.0     0.0     0.0 0.0 0.0        0.0     0.0     0.0     1.0     0.0     0.0     0.0     0.0 0.0    0.0        0.0     0.0     1.0     0.0     0.0     0.0     0.0     0.0 0.0 0.0        0.0     1.0     0.0     0.0     0.0     0.0     0.0     0.0 0.0 0.0        1.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0 0.0 1.0        0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0 1.0 0.0        0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0 0 1 2 3 4 5 6 7 8 9 Estimated output (test dataset) – Experiment #2 Digit 0.20   0.26  0.09  -0.09  0.39   0.24  0.35   0.30  0.24   1.02 0.42  -0.23  0.39   0.06  0.11    0.16 0.43   0.25  0.17  -0.26 0.51   0.84  -0.17  0.02  0.16    0.27 -0.15  0.14  -0.34 -0.12 -0.20  -0.05  -0.58  0.20  -0.16     0.27 0.83 -0.56  0.42   0.35 0.24   0.05  0.72  -0.05  -0.25    -0.38 -0.33  0.66  0.05  -0.63 0.08   0.41  -0.21  0.41  0.59     -0.12 -0.54  0.27  0.38  0.00 -0.76  -0.35  -0.09  1.25  -0.78     0.55 -0.22  0.61  0.51  0.27 -0.15   0.11  0.54  -0.53  0.55     0.17 0.09  -0.72  0.03  0.12 0.03   0.41  0.49  -0.44  -0.01    0.05 -0.05 -0.03  -0.32 -0.30 0.63  -0.47  -0.15  0.17  0.38    -0.24 0.58   0.07  -0.16 0.54 0 (OK) 1 (ERR) 2 (ERR) 3 (OK) 4 (ERR) 5 (OK) 6 (OK) 7 (ERR) 8 (ERR) 9 (OK) Estimated output (test dataset) – Experiment #8 Digit 0.10 0.10    0.12 0.10 0.12    0.13 0.13 0.26    0.17 0.39 0.13 0.10    0.11 0.10 0.11    0.10 0.29    0.23 0.32 0.10 0.26 0.38    0.10 0.10 0.12    0.10 0.10 0.17    0.10 0.10 0.10 0.10    0.10 0.10 0.10    0.17 0.39 0.10    0.38 0.10 0.15 0.10    0.24 0.10 0.10    0.10 0.10 0.39    0.37 0.10 0.20 0.12    0.10 0.10 0.37    0.10 0.10 0.10    0.17 0.12 0.10 0.10    0.10 0.39 0.10    0.16 0.11 0.30    0.14 0.10 0.10 0.11    0.39 0.10 0.10    0.15 0.10 0.10    0.17 0.10 0.10 0.25    0.34 0.10 0.10    0.10 0.10 0.10    0.10 0.10 0.39 0.10    0.10 0.10 0.28    0.10 0.27 0.11    0.10 0.21 0 (OK) 1 (OK) 2 (OK) 3 (ERR) 4 (OK) 5 (ERR) 6 (OK) 7 (OK) 8 (ERR) 9 (OK) The experiments showed in this article have taken in consideration 10x10 pixel information images. We recommend that you try to use 20x20 pixel datasets to build a neural net able to classify digit images of this size. You should also change the training parameters of the neural net to achieve better classifications. To summarize, we applied neural network techniques to perform pattern recognition on a series of numbers from 0 to 9 in an image. The application here can be extended to any type of characters instead of digits, under the condition that the neural network should all be presented with the predefined characters. If you enjoyed this excerpt, check out the book Neural Network Programming with Java Second Edition to know more about leveraging the multi-platform feature of Java to build and run your personal neural networks everywhere.    

The most common question that anyone asks when they buy a service is, can it be secured? The answer to that question, in this case, is absolutely yes. In this tutorial, we will learn to secure your connection between virtual machines. On top of the security, Microsoft provides for Azure as a vendor, there are some configurations that you can do at your end to increase the level of security to your virtual network. For a higher level of security, you can use the following: NSG: It is like a firewall that controls the inbound and outbound traffic by specifying which traffic is allowed to flow to/from the NIC/subnet Distributed denial of service (DDoS) protection: It is used to prevent DDoS attacks and at the time of writing is in preview This tutorial is an excerpt from the book, Hands-On Networking with Azure, written by Mohamed Waly.  Network Security Groups (NSG) NSG controls the flow of traffic by specifying which traffic is allowed to enter or exit the network. Creating NSG Creating NSG is a pretty straightforward process. To do it, you need to follow these steps: Navigate to Azure portal, and search for network security groups, as shown in the following screenshot: Figure 2.13: Searching for network security groups Once you have clicked on it, a new blade will be opened wherein all the created NSGs are located, as shown in the following screenshot: Figure 2.12: Network security groups blade Click on Add and a new blade will pop up, where you have to specify the following: Name: The name of the NSG Subscription: The subscription, which will be charged for NSG usage Resource group: The resource group within which the NSG will be located as a resource Location: The region where this resource will be created Figure 2.13: Creating an NSG Once you have clicked on Create, the NSG will be created within seconds. Inbound security rules By default, all the subnets and NICs that are not associated with NSG have all the inbound traffic allowed and once they are associated with an NSG, the following inbound security rules are assigned to them as they are a default part of any NSG: AllowVnetInBound: Allows all the inbound traffic that comes from a virtual network AllowAzureLoadBalancerInBound: Allows all the inbound traffic that comes from Load Balancer DenyAllInbound: Denies all the inbound traffic that comes from any source Figure 2.14: Default inbound security rules As shown in the previous screenshot, the rule consists of some properties, such as PRIORITY, NAME, PORT, and so on. It is important to understand what these properties mean for a better understanding of security rules. So, let's go ahead and explain them: PRIORITY: A number assigned to each rule to specify which rule has a higher priority than the other. The lower the number, the higher the priority. You can specify a priority with any number between 100 and 4096. NAME: The name of the rule. The same name cannot be reused within the same network security group. PORT: The allowed port through which the traffic will flow to the network. PROTOCOL: Specify whether the protocol you are using is TCP or UDP. SOURCE and DESTINATION: The source can be any, an IP address range, or a service tag. You can remove the default rules by clicking on Default rules. You can customize your own inbound rules, by following these steps: On the Inbound security rules blade, click on Add. A new blade will pop up, where you have to specify the following: Source: The source can be Any, an IP address range, or a service tag. It specifies the incoming traffic from a specific source IP address range that will be allowed or denied by this rule. Source port ranges: You can provide a single port, such as 80, a port range, such as 1024 - 65535, or a comma-separated list of single ports and/or port ranges, such as 80, 1024 - 65535. This specifies on which ports traffic will be allowed or denied by this rule. Provide an asterisk (*) to allow traffic on any port. Destination: The destination can be Any, an IP address range, or a virtual network. It specifies the outing traffic to a specific destination IP address range that will be allowed or denied by this rule. Destination port ranges: What applies for the source port ranges, applies for the destination port ranges. Protocol: It can be Any, TCP, or UDP. Action: Whether to Allow the rule or to Deny it. Priority: As mentioned earlier, the lower the number, the higher the priority. The priority number must be between 100 - 4096. Name: The name of the rule. Description: The description of the rule, which will help you to differentiate between the rules. In our scenario, I want to allow all the incoming connections to access a website published on a web server located in a virtual network, as shown in the following screenshot: Figure 2.15: Creating an inbound security rule Once you click on OK, the rule will be created. Outbound security rules Outbound security rules are no different than inbound security rules, except inbound rules are meant for inbound traffic and outbound rules are meant for outbound traffic. Otherwise, everything else is similar. Associating the NSG Once you have the NSG created, you can associate it to either an NIC or a subnet. Associating the NSG to an NIC To associate the NSG to an NIC, you need to follow these steps: Navigate to the Network security groups that you have created and then select Network interfaces, as shown in the following screenshot: Figure 2.16: Associated NICs to an NSG Click on Associate. A new blade will pop up, from which you need to select the NIC that you want to associate with the NSG, as shown in the following screenshot: Figure 2.17: NICs to be associated to the NSG Voila! You are done. Associating the NSG to a subnet To associate the NSG to a subnet, you need to follow these steps: Navigate to the Network security groups that you have created and then select Subnets, as shown in the following screenshot: Figure 2.18: Associated subnets to an NSG Click on Associate. A new blade will pop up, where you have to specify the virtual network within which the subnet exists, as shown in the following screenshot: Figure 2.19: Choosing the VNet within which the subnet exists Then, you need to specify which subnet of the VNet you want to associate the NSG to, as shown in the following screenshot: Figure 2.20: Selecting the subnet to which the NSG will be associated Once the subnet is selected, click on OK, and it will take some seconds to get it associated to the NSG. Azure DDoS protection DDoS attacks have spread out lately, by exhausting the application and making it unavailable for use, and you can expect an attack of that type any time. Recently, Microsoft announced the support of Azure DDoS protection as a service for protecting Azure resources, such as Azure VMs, Load Balancers, and Application Gateways. Azure DDoS protection comes in two flavors: Basic: This type has been around for a while as it is already enabled for Azure services to mitigate DDoS attacks. It incurs no charges. Standard: This flavor comes with more enhancements that mitigate attacks, especially for Azure VNet. At the time of writing this book, Azure DDoS protection standard is in preview and it is not available at the portal. You need to request it by filling out a form that is available here. If you found this post useful, do check out the book Hands on Networking with Azure, to design and implement Azure Networking for Azure VMs. Read More The Microsoft Azure Stack Architecture What is Azure API Management?

In this article, by Reza Toghraee, the author of the book, Learning OpenDayLight, we will: What is and what is not SDN Components of a SDN Difference between SDN and overlay The SDN controllers (For more resources related to this topic, see here.) You might have heard about Software-Defined Networking (SDN). If you are in networking industry this is a topic which probably you have studied initially when first time you heard about the SDN.To understand the importance of SDN and SDN controller, let's look at Google. Google silently built its own networking switches and controller called Jupiter. A home grown project which is mostly software driven and supports such massive scale of Google. The SDN base is There is a controller who knows the whole network. OpenDaylight (ODL), is a SDN controller. In other words, it's the central brain of the network. Why we are going towards SDN Everyone who is hearing about SDN, should ask this question that why are we talking about SDN. What problem is it trying to solve? If we look at traditional networking (layer 2, layer 3 with routing protocols such as BGP, OSPF) we are completely dominated by what is so called protocols. These protocols in fact have been very helpful to the industry. They are mostly standard. Different vendor and products can communicate using standard protocols with each other. A Cisco router can establish a BGP session with a Huawei switch or an open source Quagga router can exchange OSPF routes with a Juniper firewall. Routing protocol is a constant standard with solid bases. If you need to override something in your network routing, you have to find a trick to use protocols, even by using a static route. SDN can help us to come out of routing protocol cage, look at different ways to forward traffic. SDN can directly program each switch or even override a route which is installed by routing protocol. There are high-level benefits of using SDN which we explain few of them as follows: An integrated network: We used to have a standalone concept in traditional network. Each switch was managed separately, each switch was running its own routing protocol instance and was processing routing information messages from other neighbors. In SDN, we are migrating to a centralized model, where the SDN controller becomes the single point of configuration of the network, where you will apply the policies and configuration. Scalable layer 2 across layer 3:Having a layer 2 network across multiple layer 3 network is something which all network architects are interested and till date we have been using proprietary methods such as OTV or by using a service provider VPLS service. With SDN, we can create layer 2 networks across multiple switches or layer 3 domains (using VXLAN) and expand the layer 2 networks. In many cloud environment, where the virtual machines are distributed across different hosts in different datacenters, this is a major requirement. Third-party application programmability: This is a very generic term, isn't it? But what I'm referring to is to let other applications communicate with your network. For example,In many new distributed IP storage systems, the IP storage controller has ability to talk to network to provide the best, shortest path to the storage node. With SDN we are letting other applications to control the network. Of course this control has limitation and SDN doesn't allow an application to scrap the whole network. Flexible application based network:In SDN, everything is an application. L2/L3, BGP, VMware Integration, and so on all are applications running in SDN controller. Service chaining:On the fly you add a firewall in the path or a load balancer. This is service insertion. Unified wired and wireless: This is an ideal benefit, to have a controller which supports both wired and wireless network. OpenDaylight is the only controller which supports CAPWAP protocols which allows integration with wireless access points. Components of a SDN A software defined network infrastructure has two main key components: The SDN Controller (only one, could be deployed in a highly available cluster) The SDN enabled switches(multiple switches, mostly in a Clos topology in a datacenter):   SDN controller is the single brain of the SDN domain. In fact, an SDN domain is very similar to a chassis based switch. You can imagine supervisor or management module of a chassis based switch as a SDN controller and rest of the line card and I/O cards as SDN switches. The main difference between a SDN network and a chassis based switch is that you can scale out the SDN with multiple switches, where in a chassis based switch you are limited to the number of slots in that chassis: Controlling the fabric It is very important that you understand the main technologies involved in SDN. These methods are used by SDN controllers to manage and control the SDN network. In general, there are two methods available for controlling the fabric: Direct fabric programming: In this method, SDN controller directly communicates with SDN enabled switches via southbound protocols such as OpenFlow, NETCONF and OVSDB. SDN controller programs each switch member with related information about fabric, and how to forward the traffic. Direct fabric programming is the method used by OpenDaylight. Overlay:In overlay method, SDN controller doesn't rely on programing the network switches and routers. Instead it builds an virtual overlay network on top of existing underlay network. The underlay network can be a L2 or L3 network with traditional network switches and router, just providing IP connectivity. SDN controller uses this platform to build the overlay using encapsulation protocols such as VXLAN and NVGRE. VMware NSX uses overlay technology to build and control the virtual fabric. SDN controllers One of the key fundamentals of SDN is disaggregation. Disaggregation of software and hardware in a network and also disaggregation of control and forwarding planes. SDN controller is the main brain and controller of an SDN environment, it's a strategic control point within the network and responsible for communicating information to: Routers and switches and other network devices behind them. SDN controllers uses APIs or protocols (such as OpenFlow or NETCONF) to communicate with these devices. This communication is known as southbound Upstream switches, routers or applications and the aforementioned business logic (via APIs or protocols). This communication is known as northbound. An example for a northbound communication is a BGP session between a legacy router and SDN controller. If you are familiar with chassis based switches like Cisco Catalyst 6500 or Nexus 7k chassis, you can imagine a SDN network as a chassis, with switches and routers as its I/O line cards and SDN controller as its supervisor or management module. Infact SDN is similar to a very scalable chassis where you don't have any limitation on number of physical slots. SDN controller is similar to role of management module of a chassis based switch and it controls all switches via its southbound protocols and APIs. The following table compares the SDN controller and a chassis based switch:  SDN Controller Chassis based switch Supports any switch hardware Supports only specific switch line cards Can scale out, unlimited number of switches Limited to number of physical slots in the chassis Supports high redundancy by multiple controllers in a cluster Supports dual management redundancy, active standby Communicates with switches via southbound protocols such as OpenFlow, NETCONF, BGP PCEP Use proprietary protocols between management module and line cards Communicates with routers, switches and applications outside of SDN via northbound protocols such as BGP, OSPF and direct API Communicates with other routers and switches outside of chassis via standard protocols such as BGP, OSPF or APIs. The first protocol that popularized the concept behind SDN was OpenFlow. When conceptualized by networking researchers at Stanford back in 2008, it was meant to manipulate the data plane to optimize traffic flows and make adjustments, so the network could quickly adapt to changing requirements. Version 1.0 of the OpenFlow specification was released in December of 2009; it continues to be enhanced under the management of the Open Networking Foundation, which is a user-led organization focused on advancing the development of open standards and the adoption of SDN technologies. OpenFlow protocol was the first protocol that helped in popularizing SDN. OpenFlow is a protocol designed to update the flow tables in a switch. Allowing the SDN controller to access the forwarding table of each member switch or in other words to connect control plane and data plane in SDN world. Back in 2008, OpenFlow conceptualized by networking researchers at Stanford University, the initial use of OpenFlow was to alter the switch forwarding tables to optimize traffic flows and make adjustments, so the network could quickly adapt to changing requirements. After introduction of OpenFlow, NOX introduced as original OpenFlow controller (still there wasn't concept of SDN controller). NOX was providing a high-level API capable of managing and also developing network control applications. Separate applications were required to run on top of NOX to manage the network.NOX was initially developed by Nicira networks (which acquired by VMware, and finally became part of VMware NSX). NOX introduced along with OpenFlow in 2009. NOX was a closed source product but ultimately it was donated to SDN community which led to multiple forks and sub projects out of original NOX. For example, POX is a sub project of NOX which provides Python support. Both NOX and POX were early controllers. NOX appears an inactive development, however POX is still in use by the research community as it is a Python based project and can be easily deployed. POX is hosted at http://github.com/noxrepo/pox NOX apart from being the first OpenFlow or SDN controller also established a programing model which inherited by other subsequent controllers. The model was based on processing of OpenFlow messages, with each incoming OpenFlow message trigger an event that had to be processed individually. This model was simple to implement but not efficient and robust and couldn't scale. Nicira along with NTT and Google started developing ONIX, which was meant to be a more abstract and scalable for large deployments. ONIX became the base for Nicira (the core of VMware NSX or network virtualization platform) also there are rumors that it is also the base for Google WAN controller. ONIX was planned to become open source and donated to community but for some reasons the main contributors decided to not to do it which forced the SDN community to focus on developing other platforms. Started in 2010, a new controller introduced,the Beacon controller and it became one of the most popular controllers. It born with contribution of developers from Stanford University. Beacon is a Java-based open source OpenFlow controller created in 2010. It has been widely used for teaching, research, and as the basis of Floodlight. Beacon had the first built-in web user-interface which was a huge step forward in the market of SDN controllers. Also it provided a easier method to deploy and run compared to NOX. Beacon was an influence for design of later controllers after it, however it was only supporting star topologies which was one of the limitations on this controller. Floodlight was a successful SDN controller which was built as a fork of Beacon. BigSwitch networks is developing Floodlight along with other developers. In 2013, Beacon popularity started to shrink down and Floodlight started to gain popularity. Floodlight had fixed many issues of Beacon and added lots of additional features which made it one of the most feature rich controllers available. It also had a web interface, a Java-based GUI and also could get integrated with OpenStack using quantum plugin. Integration with OpenStack was a big step forward as it could be used to provide networking to a large pool of virtual machines, compute and storage. Floodlight adoption increased by evolution of OpenStack and OpenStack adopters. This gave Floodlight greater popularity and applicability than other controllers that came before. Most of controllers came after Floodlight also supported OpenStack integration. Floodlight is still supported and developed by community and BigSwitch networks, and is a base for BigCloud Fabric (the BigSwitch's commercial SDN controller). There are other open source SDN controllers which introduced such as Trema (ruby-based from NEC), Ryu (supported by NTT), FlowER, LOOM and the recent OpenMUL. The following table shows the current open source SDN controllers:  Active open source SDNcontroller Non-active open source SDN controllers Floodlight Beacon OpenContrail FlowER OpenDaylight NOX LOOM NodeFlow OpenMUL   ONOS   POX   Ryu   Trema     OpenDaylight OpenDaylight started in early 2013, and was originally led by IBM and Cisco. It was a new collaborative open source project. OpenDaylight hosted under Linux Foundation and draw support and interest from many developers and adopters. OpenDaylight is a platform to provide common foundations and a robust array of services for SDN environments. OpenDaylight uses a controller model which supports OpenFlow as well as other southbound protocols. It is the first open source controller capable of employing non-OpenFlow proprietary control protocols which eventually lets OpenDaylight to integrate with modern and multi-vendor networks. The first release of OpenDaylight in February 2014 with code name of Hydrogen, followed by Helium in September 2014. The Helium release was significant because it marked a change in direction for the platform that has influenced the way subsequent controllers have been architected. The main change was in the service abstraction layer, which is the part of the controller platform that resides just above the southbound protocols, such as OpenFlow, isolating them from the northbound side and where the applications reside. Hydrogen used an API-driven Service Abstraction Layer (AD-SAL), which had limitations specifically, it meant the controller needed to know about every type of device in the network AND have an inventory of drivers to support them. Helium introduced a Model-driven service abstraction layer (MD-SAL), which meant the controller didn't have to account for all the types of equipment installed in the network, allowing it to manage a wide range of hardware and southbound protocols. Helium release made the framework much more agile and adaptable to changes in the applications; an application could now request changes to the model, which would be received by the abstraction layer and forwarded to the network devices. The OpenDaylight platform built on this advancement in its third release, Lithium, which was introduced in June of 2015. This release focused on broadening the programmability of the network, enabling organizations to create their own service architectures to deliver dynamic network services in a cloud environment and craft intent-based policies. Lithium release was worked on by more than 400 individuals, and contributions from Big Switch Networks, Cisco, Ericsson, HP, NEC, and so on, making it one of the fastest growing open source projects ever. The fourth release, Beryllium come out in February of 2016 and the most recent fifth release, Boron released in September 2016. Many vendors have built and developed commercial SDN controller solutions based on OpenDaylight. Each product has enhanced or added features to OpenDaylight to have some differentiating factor. The use of OpenDaylight in different vendor products are: A base, but sell a commercial version with additional proprietary functionality—for example: Brocade, Ericsson, Ciena, and so on. Part of their infrastructure in their Network as a Service (or XaaS) offerings—for example: Telstra, IBM, and so on. Elements for use in their solution—for example: ConteXtream (now part of HP) Open Networking Operating System (ONOS), which was open sourced in December 2014 is focused on serving the needs of service providers. It is not as widely adopted as OpenDaylight, ONOS has been finding success and gaining momentum around WAN use cases. ONOS is backed by numerous organizations including AT&T, Cisco, Fujitsu, Ericsson, Ciena, Huawei, NTT, SK Telecom, NEC, and Intel, many of whom are also participants in and supporters of OpenDaylight. Apart from open source SDN controllers, there are many commercial, proprietary controllers available in the market. Products such as VMware NSX, Cisco APIC, BigSwitch Big Cloud Fabric, HP VAN and NEC ProgrammableFlow are example commercial and proprietary products. The following table lists the commercially available controllers and their relationship to OpenDaylight:  ODL-based ODL-friendly Non-ODL based Avaya Cyan (acquired by Ciena) BigSwitch Brocade HP Juniper Ciena NEC Cisco ConteXtream (HP) Nuage Plexxi Coriant   PLUMgrid Ericsson Pluribus Extreme Sonus Huawei (also ships non-ODL controller) VMware NSX Core features of SDN Regardless of an open source or a proprietary SDN platform, there are core features and capabilities which requires the SDN platform to support. These capabilities include: Fabric programmability:Providing the ability to redirect traffic, apply filters to packets (dynamically), and leverage templates to streamline the creation of custom applications. Ensuring northbound APIs allow the control information centralized in the controller available to be changed by SDN applications. This will ensure the controller can dynamically adjust the underlying network to optimize traffic flows to use the least expensive path, take into consideration varying bandwidth constraints, meet quality of service (QoS) requirements. Southbound protocol support:Enabling the controller to communicate to switches and routers and manipulate and optimize how they manage the flow of traffic. Currently OpenFlow is the most standard protocol used between different networking vendors, while there are other southbound protocols that can be used. A SDN platform should support different versions of OpenFlow in order to provide compatibility with different switching equipments. External API support:Ensuring the controller can be used within the varied orchestration and cloud environments such as VMware vSphere, OpenStack, and so on. Using APIs the orchestration platform can communicate with SDN platform in order to publish network policies. For example VMware vSphere shall talk to SDN platform to extend the virtual distributed switches(VDS) from virtual environment to the physical underlay network without any requirement form an network engineer to configure the network. Centralized monitoring and visualization:Since SDN controller has a full visibility over the network, it can offer end-to-end visibility of the network and centralized management to improve overall performance, simplify the identification of issues and accelerate troubleshooting. The SDN controller will be able to discover and present a logical abstraction of all the physical links in the network, also it can discover and present a map of connected devices (MAC addresses) which are related to virtual or physical devices connected to the network. The SDN controller support monitoring protocols, such as syslog, snmp and APIs in order to integrate with third-party management and monitoring systems. Performance: Performance in a SDN environment mainly depends on how fast SDN controller fills the flow tables of SDN enabled switches. Most of SDN controllers pre-populate the flow tables on switches to minimize the delay. When a SDN enabled switch receives a packet which doesn't find a matching entry in its flow table, it sends the packet to the SDN controller in order to find where the packet needs to get forwarded to. A robust SDN solution should ensure that the number of requests form switches are minimum and SDN controller doesn't become a bottleneck in the network. High availability and scalability: Controllers must support high availability clusters to ensure reliability and service continuity in case of failure of a controller. Clustering in SDN controller expands to scalability. A modern SDN platform should support scalability in order to add more controller nodes with load balancing in order to increase the performance and availability. Modern SDN controllers support clustering across multiple different geographical locations. Security:Since all switches communicate with SDN controller, the communication channel needs to be secured to ensure unauthorized devices doesn't compromise the network. SDN controller should secure the southbound channels, use encrypted messaging and mutual authentication to provide access control. Apart from that the SDN controller must implement preventive mechanisms to prevent from denial of services attacks. Also deployment of authorization levels and level controls for multi-tenant SDN platforms is a key requirement. Apart from the aforementioned features SDN controllers are likely to expand their function in future. They may become a network operating system and change the way we used to build networks with hardware, switches, SFPs and gigs of bandwidth. The future will look more software defined, as the silicon and hardware industry has already delivered their promises for high performance networking chips of 40G, 100G. Industry needs more time to digest the new hardware and silicons and refresh the equipment with new gears supporting 10 times the current performance. Current SDN controllers In this section, I'm putting the different SDN controllers in a table. This will help you to understand the current market players in SDN and how OpenDaylight relates to them:  Vendors/product Based on OpenDaylight? Commercial/open source Description Brocade SDN controller Yes Commercial It's a commercial version of OpenDaylight, fully supported and with extra reliable modules. Cisco APIC No Commercial Cisco Application Policy Infrastructure Controller (APIC) is the unifying automation and management point for the Application Centric Infrastructure (ACI) data center fabric. Cisco uses APIC controller and Nexus 9k switches to build the fabric. Cisco uses OpFlex as main southbound protocol. Erricson SDN controller Yes Commercial Ericsson's SDN controller is a commercial (hardened) version OpenDaylight SDN controller. Domain specific control applications that use the SDN controller as platform form the basis of the three commercial products in our SDN controller portfolio. Juniper OpenContrial /Contrail No Both OpenContrail is opensource, and Contrail itself is a commercial product. Juniper Contrail Networking is an open SDN solution that consists of Contrail controller, Contrail vRouter, an analytics engine, and published northbound APIs for cloud and NFV. OpenContrail is also available for free from Juniper. Contrail promotes and use MPLS in datacenter. NEC Programmable Flow No Commercial NEC provides its own SDN controller and switches. NEC SDN platform is one of choices of enterprises and has lots of traction and some case studies.   Avaya SDN Fx controller Yes Commercial Based on OpenDaylight, bundled as a solution package.   Big Cloud Fabric No Commercial BigSwitch networks solution is based on Floodlight opensource project. BigCloud Fabric is a robust, clean SDN controller and works with bare metal whitebox switches. BigCloud Fabric includes SwitchLightOS which is a switch operating system can be loaded on whitebox switches with Broadcom Trident 2 or Tomahawk silicons. The benefit of BigCloud Fabric is that you are not bound to any hardware and you can use baremetal switches from any vendor.   Ciena's Agility Yes Commercial Ciena's Agility multilayer WAN controller is built atop the open-source baseline of the OpenDaylight Project—an open, modular framework created by a vendor-neutral ecosystem (rather than a vendor-centric ego-system) that will enable network operators to source network services and applications from both Ciena's Agility and others. HP VAN (Virtual Application Network) No Commercial The building block of the HP open SDN ecosystem, the controller allows third-party developers to deliver innovative SDN solutions. Huawei Agile controller Yes and No (based on editions) Commercial Huawei's SDN controller which integrates as a solution with Huawei enterprise switches Nuage No Commercial Nuage Networks VSP provides SDN capabilities for clouds of all sizes. It is implemented as a non-disruptive overlay for all existing virtualized and non-virtualized server and network resources. Pluribus Netvisor No Commercial Netvisor Premium and Open Netvisor Linux are distributed network operating systems. Open Netvisor integrates atraditional, interoperable networking stack (L2/L3/VXLAN) with an SDN distributed controller that runs in everyswitch of the fabric. VMware NSX No Commercial VMware NSX is an Overlay type of SDN, which currently works with VMware vSphere. The plan is to support OpenStack in future. VMware NSX also has built-in firewall, router and L4 load balancers allowing micro segmentation. OpenDaylight as an SDN controller Previously, we went through the role of SDN controller, and a brief history of ODL.ODL is a modular open SDN platform which allows developers to build any network or business application on top of it to drive the network in the way they want. Currently OpenDaylight has reached to its fifth release (Boron, which is the fifth element in periodic table). ODL releases are named based on periodic table elements, started from first release the Hydrogen. ODL has a 6 month release period, with many developers working on expanding the ODL, 2 releases per year is expected from community. For technical readers to understand it more clearly, the following diagram will help: ODL platform has a broad set of use cases for multivendor, brown field, green fields, service providers and enterprises. ODL is a foundation for networks of the future. Service providers are using ODL to migrate their services to a software enabled level with automatic service delivery and coming out of circuit-based mindset of service delivery. Also they work on providing a virtualized CPE with NFV support in order to provide flexible offerings. Enterprises use ODL for many use cases, from datacenter networking, Cloud and NFS, network automation and resource optimization, visibility, control to deploying a fully SDN campus network. ODL uses a MD-SAL which makes it very scalable and lets it incorporate new applications and protocols faster. ODL supports multiple standard and proprietary southbound protocols, for example with full support of OpenFlow and OVSDB, ODL can communicate with any standard hardware (or even the virtual switches such as Open vSwitch(OVS) supporting such protocols). With such support, ODL can be deployed and used in multivendor environments and control hardware from different vendors from a single console no matter what vendor and what device it is, as long as they support standard southbound protocols. ODL uses a micro service architecture model which allows users to control applications, protocols and plugins while deploying SDN applications. Also ODL is able to manage the connection between external consumers and providers. The followingdiagram explains the ODL footprint and different components and projects within the ODL: Micro servicesarchitecture ODL stores its YANG data structure in a common data store and uses messaging infrastructure between different components to enable a model-driven approach to describe the network and functions. In ODL MD-SAL, any SDN application can be integrated as a service and then loaded into the SDN controller. These services (apps) can be chained together in any number and ways to match the application needs. This concept allows users to only install and enable the protocols and services they need which makes the system light and efficient. Also services and applications created by users can be shared among others in the ecosystem since the SDN application deployment for ODL follows a modular design. ODL supports multiple southbound protocols. OpenFlow and OpenFlow extension such as Table Type Patterns (TTP), as well as other protocols including NETCONF, BGP/PCEP, CAPWAP and OVSDB. Also ODL supports Cisco OpFlex protocol: ODL platform provides a framework for authentication, authorization and accounting (AAA), as well as automatic discovery and securing of network devices and controllers. Another key area in security is to use encrypted and authenticated communication trough southbound protocols with switches and routers within the SDN domain. Most of southbound protocols support security encryption mechanisms. Summary In this article we learned about SDN, and why it is important. We reviewed the SDN controller products, the ODL history as well as core features of SDN controllers and market leader controllers. We managed to dive in some details about SDN . Resources for Article: Further resources on this subject: Managing Network Devices [article] Setting Up a Network Backup Server with Bacula [article] Point-to-Point Networks [article]

Unlike Clojure, Haskell, F#, and the likes, Kotlin is not a pure functional programming language, where immutability is forced; rather, we may refer to Kotlin as a perfect blend of functional programming and OOP languages. It contains the major benefits of both worlds. So, instead of forcing immutability like pure functional programming languages, Kotlin encourages immutability, giving it automatic preference wherever possible. In this article, we'll understand the various methods of implementing immutability in Kotlin. This article has been taken from the book, Functional Kotlin, by Mario Arias and Rivu Chakraborty. In other words, Kotlin has immutable variables (val), but no language mechanisms that would guarantee true deep immutability of the state. If a val variable references a mutable object, its contents can still be modified. We will have a more elaborate discussion and a deeper dive on this topic, but first let us have a look at how we can get referential immutability in Kotlin and the differences between var, val, and const val. By true deep immutability of the state, we mean a property will always return the same value whenever it is called and that the property never changes its value; we can easily avoid this if we have a val  property that has a custom getter. You can find more details at the following link: https://artemzin.com/blog/kotlin-val-does-not-mean-immutable-it-just-means-readonly-yeah/ The difference between var and val So, in order to encourage immutability but still let the developers have the choice, Kotlin introduced two types of variables. The first one is var, which is just a simple variable, just like in any imperative language. On the other hand, val brings us a bit closer to immutability; again, it doesn't guarantee immutability. So, what exactly does the val variable provide us? It enforces read-only, you cannot write into a val variable after initialization. So, if you use a val variable without a custom getter, you can achieve referential immutability. Let's have a look; the following program will not compile: fun main(args: Array<String>) { val x:String = "Kotlin" x+="Immutable"//(1) } As I mentioned earlier, the preceding program will not compile; it will give an error on comment (1). As we've declared variable x as val, x will be read-only and once we initialize x; we cannot modify it afterward. So, now you're probably asking why we cannot guarantee immutability with val ? Let's inspect this with the following example: object MutableVal { var count = 0 val myString:String = "Mutable" get() {//(1) return "$field ${++count}"//(2) } } fun main(args: Array<String>) { println("Calling 1st time ${MutableVal.myString}") println("Calling 2nd time ${MutableVal.myString}") println("Calling 3rd time ${MutableVal.myString}")//(3) } In this program, we declared myString as a val property, but implemented a custom get function, where we tweaked the value of myString before returning it. Have a look at the output first, then we will further look into the program: As you can see, the myString property, despite being val, returned different values every time we accessed it. So, now, let us look into the code to understand such behavior. On comment (1), we declared a custom getter for the val property myString. On comment (2), we pre-incremented the value of count and added it after the value of the field value, myString, and returned the same from the getter. So, whenever we requested the myString property, count got incremented and, on the next request, we got a different value. As a result, we broke the immutable behavior of a val property. Compile time constants So, how can we overcome this? How can we enforce immutability? The const val properties are here to help us. Just modify val myString with const val myString and you cannot implement the custom getter. While val properties are read-only variables, const val, on the other hand, are compile time constants. You cannot assign the outcome (result) of a function to const val. Let's discuss some of the differences between val and const val: The val properties are read-only variables, while const val are compile time constants The val properties can have custom getters, but const val cannot We can have val properties anywhere in our Kotlin code, inside functions, as a class member, anywhere, but const val has to be a top-level member of a class/object You cannot write delegates for the const val properties We can have the val property of any type, be it our custom class or any primitive data type, but only primitive data types and String are allowed with a const val property We cannot have nullable data types with the const val properties; as a result, we cannot have null values for the const val properties either As a result, the const val properties guarantee immutability of value but have lesser flexibility and you are bound to use only primitive data types with const val, which cannot always serve our purposes. Now, that I've used the word referential immutability quite a few times, let us now inspect what it means and how many types of immutability there are. Types of immutability There are basically the following two types of immutability: Referential immutability Immutable values Immutable reference  (referential immutability) Referential immutability enforces that, once a reference is assigned, it can't be assigned to something else. Think of having it as a val property of a custom class, or even MutableList or MutableMap; after you initialize the property, you cannot reference something else from that property, except the underlying value from the object. For example, take the following program: class MutableObj { var value = "" override fun toString(): String { return "MutableObj(value='$value')" } } fun main(args: Array<String>) { val mutableObj:MutableObj = MutableObj()//(1) println("MutableObj $mutableObj") mutableObj.value = "Changed"//(2) println("MutableObj $mutableObj") val list = mutableListOf("a","b","c","d","e")//(3) println(list) list.add("f")//(4) println(list) } Have a look at the output before we proceed with explaining the program: So, in this program we've two val properties—list and mutableObj. We initialized mutableObj with the default constructor of MutableObj, since it's a val property it'll always refer to that specific object; but, if you concentrate on comment (2), we changed the value property of mutableObj, as the value property of the MutableObj class is mutable (var). It's the same with the list property, we can add items to the list after initialization, changing its underlying value. Both list and mutableObj are perfect examples of immutable reference; once initialized, the properties can't be assigned to something else, but their underlying values can be changed (you can refer the output). The reason behind that is the data type we used to assign to those properties. Both the MutableObj class and the MutableList<String> data structures are mutable themselves, so we cannot restrict value changes for their instances. Immutable values The immutable values, on the other hand, enforce no change on values as well; it is really complex to maintain. In Kotlin, the const val properties enforce immutability of value, but they lack flexibility (we already discussed them) and you're bound to use only primitive types, which can be troublesome in real-life scenarios. Immutable collections Kotlin gives preference to immutability wherever possible, but leaves the choice to the developer whether or when to use it. This power of choice makes the language even more powerful. Unlike most languages, where they have either only mutable (like Java, C#, and so on) or only immutable collections (like F#, Haskell, Clojure, and so on), Kotlin has both and distinguishes between them, leaving the developer with the freedom to choose whether to use an immutable or mutable one. Kotlin has two interfaces for collection objects—Collection<out E> and MutableCollection<out E>; all the collection classes (for example, List, Set, or Map) implement either of them. As the name suggests, the two interfaces are designed to serve immutable and mutable collections respectively. Let us have an example: fun main(args: Array<String>) { val immutableList = listOf(1,2,3,4,5,6,7)//(1) println("Immutable List $immutableList") val mutableList:MutableList<Int> = immutableList.toMutableList()//(2) println("Mutable List $mutableList") mutableList.add(8)//(3) println("Mutable List after add $mutableList") println("Mutable List after add $immutableList") } The output is as follows: So, in this program, we created an immutable list with the help of the listOf method of Kotlin, on comment (1). The listOf method creates an immutable list with the elements (varargs) passed to it. This method also has a generic type parameter, which can be skipped if the elements array is not empty. The listOf method also has a mutable version—mutableListOf() which is identical except that it returns MutableList instead. We can convert an immutable list to a mutable one with the help of the toMutableList() extension function, we did the same in comment (2), to add an element to it on comment (3). However, if you check the output, the original Immutable List remains the same without any changes, the item is, however, added to the newly created MutableList instead. So now you know how to implement immutability in Kotlin. If you found this tutorial helpful, and would like to learn more, head on over to purchase the full book, Functional Kotlin, by Mario Arias and Rivu Chakraborty. Extension functions in Kotlin: everything you need to know Building RESTful web services with Kotlin Building chat application with Kotlin using Node.js, the powerful Server-side JavaScript platform

In this article by Martin Abbott, Ashish Bhambhani, James Corbould, Gautam Gyanendra, Abhishek Kumar, and Mahindra Morar, authors of the book Robust Cloud Integration with Azure, we learn that it is important to know how to control and manage API assets that exist or are built as part of any enterprise development. (For more resources related to this topic, see here.) Typically, modern APIs are used to achieve one of the following two outcomes: First, to expose the on-premises line of business applications, such as Customer Relationship Management (CRM) or Enterprise Resource Planning (ERP) solutions to other applications that need to consume and interact with these enterprise assets both on-premises and in the cloud Second, to provide access to the API for commercial purposes to monetize access to the assets exposed by the API The latter use case is important as it allows organizations to extend the use of their API investment, and it has led to what has become known as the API economy. The API economy provides a mechanism to gain additional value from data contained within the organizational boundary whether that data exists in the cloud or on-premises. When providing access to information via an API, two considerations are important: Compliance: This ensures that an access to the API and the use of the API meets requirements around internal or legal policies and procedures, and it provides reporting and auditing information Governance: This ensures the API is accessed and used only by those authorized to do so, and in a way, that is controlled and if necessary metered, and provides reporting and auditing information, which can be used, for example, to provide usage information for billing In order to achieve this at scale in an organization, a tool is required that can be used to apply both compliance and governance structures to an exposed endpoint. This is required to ensure that the usage of the information behind that endpoint is limited only to those who should be allowed access and only in a way that meets the requirements and policies of the organization. This is where API Management plays a significant role. There are two main types of tools that fit within the landscape that broadly fall under the banner of API Management: API Management: These tools provide the compliance and governance control required to ensure that the exposed API is used appropriately and data presented in the correct format. For example, a message may be received in the XML format, but the consuming service may need the data in the JSON format. They can also provide monitoring tools and access control that allows organizations to gain insight into the use of the API, perhaps with the view to charge a fee for access. API Gateway: These tools provide the same or similar level of management as normal API Management tools, but often include other functionality that allows some message mediation and message orchestration thereby allowing more complex interactions and business processes to be modeled, exposed, and governed. Microsoft Azure API Management falls under the first category above whilst Logic Apps, provide the capabilities (and more) that API Gateways offer. Another important aspect of providing management of APIs is creating documentation that can be used by consumers, so they know how to interact with and get the best out of the API. For APIs, generally, it is not a case of build it and they will come, so some form of documentation that includes endpoint and operation information, along with sample code, can lead to greater uptake of usage of the API. Azure API Management is currently offered in three tiers: Developer, Standard, and Premium. The details associated with these tiers at the time of writing are shown in the following table:   Developer Standard Premium API Calls (per unit) 32 K / day ( ~1 M / month ) 7 M / day ( ~217 M / month ) 32 M / day ( ~1 B / month ) Data Transfer (per unit) 161 MB / day ( ~5 GB / month ) 32 GB / day ( ~1 TB / month ) 161 GB / day ( ~5 TB / month ) Cache 10 MB 1 GB 5 GB Scale-out N/A 4 units Unlimited SLA N/A 99.9% 99.95% Multi-Region Deployment N/A N/A Yes Azure Active Directory Integration Unlimited user accounts N/A Unlimited user accounts VPN Yes N/A Yes Key items of note in the table are Scale-out, multiregion deployment, and Azure Active Directory Integration. Scale-out: This defines how many instances, or units, of the API instance are possible; this is configured through the Azure Classic Portal Multi-region deployment: When using Premium tier, it is possible to deploy the API Management instance to many locations to provided geographically distributed load Azure Active Directory Integration: If an organization synchronizes an on-premises Active Directory domain to Azure, access to the API endpoints can be configured to use Azure Active Directory to provide same sign-on capabilities The main use case for Premium tier is if an organization has many hundreds or even thousands of APIs they want to expose to developers, or in cases where scale and integration with line of business APIs is critical. The anatomy of Azure API Management To understand how to get the best out of an API, it is important to understand some terms that are used for APIs and within Azure API Management, and these are described here. API and operations An API provides an abstraction layer through an endpoint that allows interaction with entities or processes that would otherwise be difficult to consume. Most API developers favor using a RESTful approach to API applications since this allows us easy understanding on how to work with the operations that the API exposes and provides scalability, modifiability, reliability, and performance. Representational State Transfer (REST) is an architectural style that was introduced by Roy Fielding in his doctoral thesis in 2000 (http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm). Typically, modern APIs are exposed using HTTP since this makes it easier for different types of clients to interact with it, and this increased interoperability provides the greatest opportunity to offer additional value and greater adoption across different technology stacks. When building an API, a set of methods or operations is exposed that a user can interact with in a predictable way. While RESTful services do not have to use HTTP as a transfer method, nearly all modern APIs do, since the HTTP standard is well known to most developers, and it is simple and straightforward to use. Since the operations are called via HTTP, a distinct endpoint or Unified Resource Identifier (URI) is required to ensure sufficient modularity of the API service. When calling an endpoint, which may for example represent, an entity in a line of business system, HTTP verbs (GET, POST, PUT, and DELETE, for example) are used to provide a standard way of interacting with the object. An example of how these verbs are used by a developer to interact with an entity is given in the following table: TYPE GET POST PUT DELETE Collection Retrieve a list of entities and their URIs Create a new entity in the collection Replace (update) a collection Delete the entire collection Entity Retrieve a specific entity and its information usually in a particular data format Create a new entity in the collection, not generally used Replace (update) an entity in the collection, or if it does not exist, create it Delete a specific entity from a collection When passing data to and receiving data from an API operation, the data needs to be encapsulated in a specific format. When services and entities were exposed through SOAP-based services, this data format was typically XML. For modern APIs, JavaScript Object Notation (JSON) has become the norm. JSON has become the format of choice since it has a smaller payload than XML and a smaller processing overhead, which suits the limited needs of mobile devices (often running on battery power). JavaScript (as the acronym JSON implies) also has good support for processing and generating JSON, and this suits developers, who can leverage existing toolsets and knowledge. API operations should abstract small amounts of work to be efficient, and in order to provide scalability, they should be stateless, and they can be scaled independently. Furthermore, PUT and DELETE operations must be created that ensure consistent state regardless of how many times the specific operation is performed, this leads to the need of those operations being idempotent. Idempotency describes an operation that when performed multiple times produces the same result on the object that is being operated on. This is an important concept in computing, particularly, where you cannot guarantee that an operation will only be performed once, such as with interactions over the Internet. Another outcome of using a URI to expose entities is that the operation is easily modified and versioned because any new version can simply be made available on a different URI, and because HTTP is used as a transport mechanism, endpoint calls can be cached to provide better performance and HTTP Headers can be used to provide additional information, for example security. By default, when an instance of API Management is provisioned, it has a single API already available named Echo API. This has the following operations: Creating resource Modifying resource Removing resource Retrieving header only Retrieving resource Retrieving resource (cached) In order to get some understanding of how objects are connected, this API can be used, and some information is given in the next section. Objects within API Management Within Azure API Management, there are a number of key objects that help define a structure and provide the governance, compliance, and security artifacts required to get the best out of a deployed API, as shown in the following diagram: As can be seen, the most important object is a PRODUCT. A product has a title and description and is used to define a set of APIs that are exposed to developers for consumption. They can be Open or Protected, with an Open product being publicly available and a Protected product requiring a subscription once published. Groups provide a mechanism to organize the visibility of and access to the APIs within a product to the development community wishing to consume the exposed APIs. By default, a product has three standard groups that cannot be deleted: Administrators: Subscription administrators are included by default, and the members of this group manage API services instances, API creation, API policies, operations, and products Developers: The members of this group have authenticated access to the Developer Portal; they are the developers who have chosen to build applications that consume APIs exposed as a specific product Guests: Guests are able to browse products through the Developer Portal and examine documentation, and they have read-only access to information about the products In addition to these built-in groups, it is possible to create new groups as required, including the use of groups within an Azure Active Directory tenant. When a new instance of API Management is provisioned, it has the following two products already configured: Starter: This product limits subscribers to a maximum of five calls per minute up to a maximum of 100 calls per week Unlimited: This product has no limits on use, but subscribers can only use it with the administrator approval Both of these products are protected, meaning that they need to be subscribed to and published. They can be used to help gain some understanding of how the objects within API Management interact. These products are configured with a number of sample policies that can be used to provide a starting point. Azure API Management policies API Management policies are the mechanism used to provide governance structures around the API. They can define, for instance, the number of call requests allowed within a period, cross-origin resource sharing (CORS), or certificate authentication to a service backend. Policies are defined using XML and can be stored in source control to provide active management. Policies are discussed in greater detail later in the article. Working with Azure API Management Azure API Management is the outcome of the acquisition by Microsoft of Apiphany, and as such it has its own management interfaces. Therefore, it has a slightly different look and feel to the standard Azure Portal content. The Developer and Publisher Portals are described in detail in this section, but first a new instance of API Management is required. Once created and the provisioning in the Azure infrastructure can take some time, most interactions take place through the Developer and Publisher Portals. Policies in Azure API Management In order to provide control over interactions with Products or APIs in Azure API Management, policies are used. Policies make it possible to change the default behavior of an API in the Product, for example, to meet the governance needs of your company or Product, and are a series of statements executed sequentially on each request or response of an API. Three demo scenarios will provide a "taster" of this powerful feature of Azure API Management. How to use Policies in Azure API Management Policies are created and managed through the Publisher Portal. The first step in policy creation is to determine at what scope the policy should be applied. Policies can be assigned to all Products, individual Products, the individual APIs associated with a Product, and finally the individual operations associated with an API. Secure your API in Azure API Management We have previously discussed how it is possible to organize APIs in Products with those products further refined through the use of Policies. Access to and visibility of products is controlled through the use of Groups and developer subscriptions for those APIs requiring subscriptions. In most enterprise scenarios where you are providing access to some line of business system on-premises, it is necessary to provide sufficient security on the API endpoint to ensure that the solution remains compliant. There are a number of ways to achieve this level of security using Azure API Management, such as using certificates, Azure Active Directory, or extending the corporate network into Microsoft Azure using a Virtual Private Network (VPN), and creating a hybrid cloud solution. Securing your API backend with Mutual Certificates Certificate exchange allows Azure API Management and an API to create a trust boundary based on encryption that is well understood and easy to use. In this scenario, because Azure API Management is communicating with an API that has been provided, a self-signed certificate is allowed as the key exchange for the certificate is via a trusted party. For an in-depth discussion on how to configure Mutual Certificate authentication to secure your API, please refer to the Azure API Management documentation (https://azure.microsoft.com/en-us/documentation/articles/api-management-howto-mutual-certificates/). Securing your API backend with Azure Active Directory If an enterprise already uses Azure Active Directory to provide single or same sign-on to cloud-based services, for instance, on-premises Active Directory synchronization via ADConnect or DirSync, then this provides a good opportunity to leverage Azure Active Directory to provide a security and trust boundary to on-premises API solutions. For an in-depth discussion on how to add Azure Active Directory to an API Management instance, please see the Azure API Management documentation (https://azure.microsoft.com/en-us/documentation/articles/api-management-howto-protect-backend-with-aad/). VPN connection in Azure API Management Another way of providing a security boundary between Azure API Management and the API is managing the creation of a virtual private network. A VPN creates a tunnel between the corporate network edge and Azure, essentially creating a hybrid cloud solution. Azure API Management supports site-to-site VPNs, and these are created using the Classic Portal. If an organization already has an ExpressRoute circuit provisioned, this can also be used to provide connectivity via private peering. Because a VPN needs to communicate to on-premises assets, a number of firewall port exclusions need to be created to ensure the traffic can flow between the Azure API Management instance and the API endpoint. Monitoring your API Any application tool is only as good as the insight you can gain from the operation of the tool. Azure API Management is no exception and provides a number of ways of getting information about how the APIs are being used and are performing. Summary API Management can be used to provide developer access to key information in your organization, information that could be sensitive, or that needs to be limited in use. Through the use of Products, Policies, and Security, it is possible to ensure that firm control is maintained over the API estate. The developer experience can be tailored to provide a virtual storefront to any APIs along with information and blogs to help drive deeper developer engagement. Although not discussed in this article, it is also possible for developers to publish their own applications to the API Management instance for other developers to use. Resources for Article: Further resources on this subject: Creating Multitenant Applications in Azure [article] Building A Recommendation System with Azure [article] Putting Your Database at the Heart of Azure Solutions [article]

[box type="note" align="" class="" width=""]The following extract is taken from the book Deep Learning with Keras, co-authored by Antonio Gulli and Sujit Pal. [/box] Keras has a lot of built-in functionality for you to build all your deep learning models without much need for customization. In this article, the authors explain how your Keras models can be customized for better and more efficient deep learning. As you will recall, Keras is a high level API that delegates to either a TensorFlow or Theano backend for the computational heavy lifting. Any code you build for your customization will call out to one of these backends. In order to keep your code portable across the two backends, your custom code should use the Keras backend API (https://keras.io/backend/), which provides a set of functions that act like a facade over your chosen backend. Depending on the backend selected, the call to the backend facade will translate to the appropriate TensorFlow or Theano call. The full list of functions available and their detailed descriptions can be found on the Keras backend page. In addition to portability, using the backend API also results in more maintainable code, since Keras code is generally more high-level and compact compared to equivalent TensorFlow or Theano code. In the unlikely case that you do need to switch to using the backend directly, your Keras components can be used directly inside TensorFlow (not Theano though) code as described in this Keras blog (https://blog.keras.io/keras-as-a-simplified-interface-to-tensorflow-tutorial.html) Customizing Keras typically means writing your own custom layer or custom distance function. In this section, we will demonstrate how to build some simple Keras layers. You will see more examples of using the backend functions to build other custom Keras components, such as objectives (loss functions), in subsequent sections. Keras example — using the lambda layer Keras provides a lambda layer; it can wrap a function of your choosing. For example, if you wanted to build a layer that squares its input tensor element-wise, you can say simply: model.add(lambda(lambda x: x ** 2)) You can also wrap functions within a lambda layer. For example, if you want to build a custom layer that computes the element-wise euclidean distance between two input tensors, you would define the function to compute the value itself, as well as one that returns the output shape from this function, like so: def euclidean_distance(vecs): x, y = vecs return K.sqrt(K.sum(K.square(x - y), axis=1, keepdims=True)) def euclidean_distance_output_shape(shapes): shape1, shape2 = shapes return (shape1[0], 1) You can then call these functions using the lambda layer shown as follows: lhs_input = Input(shape=(VECTOR_SIZE,)) lhs = dense(1024, kernel_initializer="glorot_uniform", activation="relu")(lhs_input) rhs_input = Input(shape=(VECTOR_SIZE,)) rhs = dense(1024, kernel_initializer="glorot_uniform", activation="relu")(rhs_input) sim = lambda(euclidean_distance, output_shape=euclidean_distance_output_shape)([lhs, rhs]) Keras example - building a custom normalization layer While the lambda layer can be very useful, sometimes you need more control. As an example, we will look at the code for a normalization layer that implements a technique called local response normalization. This technique normalizes the input over local input regions, but has since fallen out of favor because it turned out not to be as effective as other regularization methods such as dropout and batch normalization, as well as better initialization methods. Building custom layers typically involves working with the backend functions, so it involves thinking about the code in terms of tensors. As you will recall, working with tensors is a two step process. First, you define the tensors and arrange them in a computation graph, and then you run the graph with actual data. So working at this level is harder than working in the rest of Keras. The Keras documentation has some guidelines for building custom layers (https://keras.io/layers/writing-your-own-keras-layers/), which you should definitely read. One of the ways to make it easier to develop code in the backend API is to have a small test harness that you can run to verify that your code is doing what you want it to do. Here is a small harness I adapted from the Keras source to run your layer against some input and return a result: from keras.models import Sequential from keras.layers.core import Dropout, Reshape def test_layer(layer, x): layer_config = layer.get_config() layer_config["input_shape"] = x.shape layer = layer.__class__.from_config(layer_config) model = Sequential() model.add(layer) model.compile("rmsprop", "mse") x_ = np.expand_dims(x, axis=0) return model.predict(x_)[0] And here are some tests with layer objects provided by Keras to make sure that the harness runs okay: from keras.layers.core import Dropout, Reshape from keras.layers.convolutional import ZeroPadding2D import numpy as np x = np.random.randn(10, 10) layer = Dropout(0.5) y = test_layer(layer, x) assert(x.shape == y.shape) x = np.random.randn(10, 10, 3) layer = ZeroPadding2D(padding=(1,1)) y = test_layer(layer, x) assert(x.shape[0] + 2 == y.shape[0]) assert(x.shape[1] + 2 == y.shape[1]) x = np.random.randn(10, 10) layer = Reshape((5, 20)) y = test_layer(layer, x) assert(y.shape == (5, 20)) Before we begin building our local response normalization layer, we need to take a moment to understand what it really does. This technique was originally used with Caffe, and the Caffe documentation (http://caffe.berkeleyvision.org/tutorial/layers/lrn.html), describes it as a kind of lateral inhibition that works by normalizing over local input regions. In ACROSS_CHANNEL mode, the local regions extend across nearby channels but have no spatial extent. In WITHIN_CHANNEL mode, the local regions extend spatially, but are in separate channels. We will implement the WITHIN_CHANNEL model as follows. The formula for local response normalization in the WITHIN_CHANNEL model is given by: The code for the custom layer follows the standard structure. The __init__ method is used to set the application specific parameters, that is, the hyperparameters associated with the layer. Since our layer only does a forward computation and doesn't have any learnable weights, all we do in the build method is to set the input shape and delegate to the superclass's build method, which takes care of any necessary book-keeping. In layers where learnable weights are involved, this method is where you would set the initial values. The call method does the actual computation. Notice that we need to account for dimension ordering. Another thing to note is that the batch size is usually unknown at design times, so you need to write your operations so that the batch size is not explicitly invoked. The computation itself is fairly straightforward and follows the formula closely. The sum in the denominator can also be thought of as average pooling over the row and column dimension with a padding size of (n, n) and a stride of (1, 1). Because the pooled data is averaged already, we no longer need to divide the sum by n. The last part of the class is the get_output_shape_for method. Since the layer normalizes each element of the input tensor, the output size is identical to the input size: from keras import backend as K from keras.engine.topology import Layer, InputSpec class LocalResponseNormalization(Layer): def __init__(self, n=5, alpha=0.0005, beta=0.75, k=2, **kwargs): self.n = n self.alpha = alpha self.beta = beta self.k = k super(LocalResponseNormalization, self).__init__(**kwargs) def build(self, input_shape): self.shape = input_shape super(LocalResponseNormalization, self).build(input_shape) def call(self, x, mask=None): if K.image_dim_ordering == "th": _, f, r, c = self.shape Else: _, r, c, f = self.shape squared = K.square(x) pooled = K.pool2d(squared, (n, n), strides=(1, 1), padding="same", pool_mode="avg") if K.image_dim_ordering == "th": summed = K.sum(pooled, axis=1, keepdims=True) averaged = self.alpha * K.repeat_elements(summed, f, axis=1) Else: summed = K.sum(pooled, axis=3, keepdims=True) averaged = self.alpha * K.repeat_elements(summed, f, axis=3) denom = K.pow(self.k + averaged, self.beta) return x / denom def get_output_shape_for(self, input_shape): return input_shape You can test this layer during development using the test harness we described here. It is easier to run this instead of trying to build a whole network to put this into, or worse, waiting till you have fully specified the layer before running it: x = np.random.randn(225, 225, 3) layer = LocalResponseNormalization() y = test_layer(layer, x) assert(x.shape == y.shape) Now that you have a good idea of how to build a custom Keras layer, you might find it instructive to look at Keunwoo Choi's melspectogram (https://keunwoochoi.wordpress.com/2016/11/18/for-beginners-writing-a-custom-keras-layer/) Though building custom Keras layers seems to be fairly commonplace for experienced Keras developers, but they may not be widely useful in a general context. Custom layers are usually built to serve a specific narrow purpose, depending on the use-case in question, and Keras gives you enough flexibility to do so with ease. If you found our post useful, make sure to check out our best selling title Deep Learning with Keras, for other intriguing deep learning concepts and their implementation using Keras.    

The Godot Engine has built-in functionalities that makes it easy to create navigation in the game world. This post will cover how to make an object follow a fixed path and how to go between two points avoiding the obstacles in the way. Following a fixed path Godot has a couple of nodes that help you create a path that can be followed by another node. One use of this is to make an NPC follow a fixed path in the map. Assuming you have a new project, create a Path2D node. You can then use the controls on the toolbar to make a curve representing the path you will need to follow. Curve buttons After adding the points and adjusting the curves, you will have something like the following: Path curve Now you need to add a PathFollow2D node as a child of Path2D. This will do the actual movement based on the Offset property. Then add an AnimationPlayer node as child of the PathFollow2D. Create a new animation in the player. Set the length to five seconds. Add a keyframe on the start with value 0 for the Unit Offset property of PathFollow2D. You can do that by clicking on the key icon next to the property in the Inspector dock. Then go to the end of the animation and add a frame with the value of 1. This will make Unit Offset go from 0 to 1 in the period of the animation (five seconds in this case). Set the animation to loop and autoplay. To see the effect in practice, add a Sprite node as child of PathFollow2D. You can use the default Godot icon as the texture for it. Enable the Visible Navigation under the Debug Options menu (last button in the top center bar) to make it easier to see. Save the scene and play it to see the Godot robot run around the screen: Sprite following path That's it! Making an object follow a fixed path is quite easy with the built-in resources of Godot. Not even scripting is needed for this example. Navigation and Avoiding Obstacles Sometimes you don't have a fixed path to follow. It might change dynamically or your AI must determine the path and avoid the obstacles and walls that might be in the way. Don't worry because Godot will also help you in this regard. Create a new scene and add a Node2D as the root. Then add a Navigation2D as its child. This will be responsible for creating the paths for you. You now need to add a NavigationPolygonInstance node as child of the Navigation2D. This will hold the polygon used for navigation, to determine what the passable areas are. To create the polygon itself, click on the pencil button on the toolbar (it will appear only if the NavigationPolygonInstance node is selected). The first time you try to add a point, the editor will warn you that there's no NavigationPolygon resource and will offer you to create one. Click on the Create button and all will be set. Navitation resource warning First you need to create the outer boundaries of the navigable area. The polygon can have as many points as you need, but it does need to be a closed polygon. Note that you can right-click on points to remove them and hold Ctrl while clicking on lines to add points. Once you finish the boundaries, click the pencil button again and create polygons inside it to make the impassable areas, such as the walls. You will end up with something like the following: Navigation polygon Add a Sprite node as child of the root Node2D and set the texture of it (you can use the default Godot icon). This will be the object navigating through the space. Now add the following script to the root node. The most important detail here is the get_simple_path function, which returns a list of points to travel from start to end without passing through the walls. extends Node2D # Global variables var start = Vector2() var end = Vector2() var path = [] var speed = 1 var transition = 0 var path_points = 0 func _ready(): # Enable the processing of user input set_process_input(true) # Enable the general process callback set_process(true) func _input(event): # If the user press a mouse button if event.type == InputEvent.MOUSE_BUTTON and event.pressed: if event.button_index == BUTTON_LEFT: # If it's the left button, set the starting point start = event.global_pos elif event.button_index == BUTTON_RIGHT: # If it's the right button, set the ending point end = event.global_pos # Reset the sprite position get_node("Sprite").set_global_pos(start) transition = 0 func _process(delta): # Get the list of points that compose the path path = get_node("Navigation2D").get_simple_path(start, end) # If the path has less points than it did before, reset the transition point if path.size() < path_points: transition = 0 # Update the current amount of points path_points = path.size() # If there's less than 2 points, nothing can be done if path_points < 2: return var sprite = get_node("Sprite") # This uses the linear interpolation function from Vector2 to move the sprite in a constant # rate through the points of the path. Transition is a value from 0 to 1 to be used as a ratio. sprite.set_global_pos(sprite.get_global_pos().linear_interpolate(path[1], transition)) start = sprite.get_global_pos() transition += speed * delta # Reset the transition when it gets to the point. if transition > 1: transition = 0 # Update the node so the _draw() function is called update() func _draw(): # This draw a white circle with radius of 10px for each point in the path for p in path: draw_circle(p, 10, Color(1, 1, 1)) Enable the Visible Navigation in the Debug Options button to help you visualize the effect. Save and run the scene. You can then left-click somewhere to define a starting point, and right-click to define the ending point. The points will be marked as white circles and the Sprite will follow the path, clearing the intermediate points as it travels along. Navigating Godot bot Conclusion The Godot Engine has many features to ease the development of all kinds of games. The navigation functions have many utilities in top-down games, be it an RPG or an RTS. Tilesets also embed navigation polygons that can be used in a similar fashion. About the Author: George Marques is a Brazilian software developer who has been playing with programming in a variety of environments since he was a kid. He works as a freelancer programmer for web technologies based on open source solutions such as WordPress and Open Journal Systems. He's also one of the regular contributors of the Godot Engine, helping solving bugs and adding new features to the software, while also giving solutions to the community for the questions they have.

At the DefCon 27, Eyal Itkin, a vulnerability researcher at Check Point Software Technologies, demonstrated how vulnerabilities in the Picture Transfer Protocol (PTP) allowed him to infect a Canon EOS 80D DSLR with ransomware over a rogue WiFi connection. The PTP along with image transfer also contains dozens of different commands that support anything from taking a live picture to upgrading the camera’s firmware. The researcher chose Canon’s EOS 80D DSLR camera for three major reasons: Canon is the largest DSLR maker, controlling more than 50% of the market. The EOS 80D supports both USB and WiFi. Canon has an extensive “modding” community, called Magic Lantern, an open-source free software add-on that adds new features to the Canon EOS cameras. Eyal Itkin highlighted six vulnerabilities in the PTP that can easily allow a hacker to infiltrate the DSLRs and inject ransomware and lock the device. Next, the users might have to pay ransom to free up their camera and picture files. CVE-2019-5994 – Buffer Overflow in SendObjectInfo  (opcode 0x100C) CVE-2019-5998 – Buffer Overflow in NotifyBtStatus (opcode 0x91F9) CVE-2019-5999– Buffer Overflow in BLERequest (opcode 0x914C) CVE-2019-6000– Buffer Overflow in SendHostInfo (opcode0x91E4) CVE-2019-6001– Buffer Overflow in SetAdapterBatteryReport (opcode 0x91FD) CVE-2019-5995 – Silent malicious firmware update Itkin’s team informed Canon about the vulnerabilities in their DSLR on March 31, 2019. Recently, on August 6, Canon published a security advisory informing users that, “at this point, there have been no confirmed cases of these vulnerabilities being exploited to cause harm” and asking them to take advised measures to ensure safety. Itkin told The Verge, “due to the complexity of the protocol, we do believe that other vendors might be vulnerable as well, however, it depends on their respective implementation”. Though Itkin said he worked only with the Canon model, he also said DSLRs of other companies may also be at high risk. Vulnerability discovery by Itkin’s team in Canon’s DSLR After Itkin’s team was successful in dumping the camera’s firmware and loading it into their disassembler (IDA Pro), they say finding the PTP layer was an easy task. This is because, The PTP layer is command-based, and every command has a unique numeric opcode. The firmware contains many indicative strings, which eases the task of reverse-engineering it. Next, the team traversed back from the PTP OpenSession handler and found the main function that registers all of the PTP handlers according to their opcodes. “When looking on the registration function, we realized that the PTP layer is a promising attack surface. The function registers 148 different handlers, pointing to the fact that the vendor supports many proprietary commands. With almost 150 different commands implemented, the odds of finding a critical vulnerability in one of them is very high,” Itkin wrote in the research report. Each PTP command handler implements the same code API. The API makes use of the ptp_context object, an object that is partially documented thanks to ML, Itkin said. The team realized that most of the commands were relatively simple. “They receive only a few numeric arguments, as the protocol supports up to 5 such arguments for every command. After scanning all of the supported commands, the list of 148 commands was quickly narrowed down to 38 commands that receive an input buffer,” Itkin writes. “From an attacker’s viewpoint, we have full control of this input buffer, and therefore, we can start looking for vulnerabilities in this much smaller set of commands. Luckily for us, the parsing code for each command uses plain C code and is quite straight-forward to analyze,” he further added. Following this, they were able to find their first vulnerabilities and then the rest. Check Point and Canon have advised users to ensure that their cameras are using the latest firmware and install patches whenever they become available. Also, if the device is not in use camera owners should keep the device’s Wi-Fi turned off. A user on HackerNews points out, “It could get even worse if the perpetrator instead of bricking the device decides to install a backdoor that silently uploads photos to a server whenever a wifi connection is established.” Another user on Petapixel explained what quick measures they should take,  “A custom firmware can close the vulnerability also if they put in the work. Just turn off wifi and don't use random computers in grungy cafes to connect to your USB port and you should be fine. It may or may not happen but it leaves the door open for awesome custom firmware to show up. Easy ones are real CLOG for 1dx2. For the 5D4, I would imagine 24fps HDR, higher res 120fps, and free Canon Log for starters. For non tech savvy people that just leave wifi on all the time, that visit high traffic touristy photo landmarks they should update. Especially if they have no interest in custom firmware.” Another user on Petapixel highlighted the fact, “this hack relies on a serious number of things to be in play before it works, there is no mention of how to get the camera working again, is it just a case of flashing the firmware and accepting you may have lost a few images ?... there’s a lot more things to worry about than this.” Check Point has demonstrated the entire attack in the following YouTube video. https://youtu.be/75fVog7MKgg To know more about this news in detail, read Eyal Itkin’s complete research on Check Point. Researchers reveal vulnerability that can bypass payment limits in contactless Visa card Apple patched vulnerability in Mac’s Zoom Client; plans to address ‘video on by default’ VLC media player affected by a major vulnerability in a 3rd library, libebml

This article, written by Roberto Freato and Marco Parenzan, is from the book Mastering Cloud Development using Microsoft Azure by Packt Publishing, and it teaches us how to create multitenant applications in Azure. This book guides you through the many efficient ways of mastering the cloud services and using Microsoft Azure and its services to its maximum capacity. (For more resources related to this topic, see here.) A tenant is a private space for a user or a group of users in an application. A typical way to identify a tenant is by its domain name. If multiple users share a domain name, we say that these users live inside the same tenant. If a group of users use a different reserved domain name, they live in a reserved tenant. From this, we can infer that different names are used to identify different tenants. Different domain names can imply different app instances, but we cannot say the same about deployed resources. Multitenancy is one of the funding principles of Cloud Computing. Developers need to reach economy of scale, which allows every cloud user to scale as needed without paying for overprovisioned resources or suffering for underprovisioned resources. To do this, cloud infrastructure needs to be oversized for a single user and sized for a pool of potential users that share the same group of resources during a certain period of time. Multitenancy is a pattern. Legacy on-premise applications usually tend to be a single-tenant app, shared between users because of the lack of specific DevOps tasks. Provisioning an app for every user can be a costly operation. Cloud environments invite reserving a single tenant for each user (or group of users) to enforce better security policies and to customize tenants for specific users because all DevOps tasks can be automated via management APIs. The cloud invites reserving resource instances for a tenant and deploying a group of tenants on the same resources. In general, this is a new way of handling app deployment. We will now take a look at how to develop an app in this way. Scenario CloudMakers.xyz, a cloud-based development company, decided to develop a personal accountant web application—MyAccountant. Professionals or small companies can register themselves on this app as a single customer and record all of their invoices on it. A single customer represents the tenant; different companies use different tenants. Every tenant needs its own private data to enforce data security, so we will reserve a dedicated database for a single tenant. Access to a single database is not an intensive task because invoice registration will generally occur once daily. Every tenant will have its own domain name to enforce company identity. A new tenant can be created from the company portal application, where new customers register themselves, specifying the tenant name. For sample purposes, without the objective of creating production-quality styling, we use the default ASP.NET MVC templates to style and build up apps and focus on tenant topics. Creating the tenant app A tenant app is an invoice recording application. To brand the tenant, we record tenant name in the app settings inside the web.config file: <add key="TenantName" value="{put_your_tenant_name}" /> To simplify this, we "brand" the application that stores the tenant name in the main layout file where the application name is displayed. The application content is represented by an Invoices page where we record data with a CRUD process. The entry for the Invoices page is in the Navigation bar: <ul class="nav navbar-nav"> <li>@Html.ActionLink("Home", "Index", "Home")</li> <li>@Html.ActionLink("Invoices", "Index", "Invoices")</li> <!-- other code omitted --> First, we need to define a model for the application in the models folder. As we need to store data in an Azure SQL database, we can use entity framework to create the model from an empty code. However, first we use the following code: public class InvoicesModel : DbContext { public InvoicesModel() : base("name=InvoicesModel") { } public virtual DbSet<Invoice> Invoices { get; set; } } As we can see, data will be accessed by a SQL database that is referenced by a connectionString in the web.config file: <add name="InvoicesModel" connectionString="data source=(LocalDb)MSSQLLocalDB;initial catalog=Tenant.Web.Models.InvoicesModel;integrated security=True;MultipleActiveResultSets=True; App=EntityFramework" providerName="System.Data.SqlClient" /></connectionStrings> This model class is just for demo purposes: public class Invoice { public int InvoiceId { get; set; } public int Number { get; set; } public DateTime Date { get; set; } public string Customer { get; set; } public decimal Amount { get; set; } public DateTime DueDate { get; set; } } After this, we try to compile the project to check whether we have not made any mistake. We can now scaffold this model into an MVC controller so that we can have a simple but working app skeleton. Creating the portal app We now need to create the portal app starting from the MVC default template. Its registration workflow is useful for the creation of our tenant registration. In particular, we utilize user registration as the tenant registration. The main information acquires the tenant name and triggers tenant deployment. We need to make two changes on the UI. First, in the RegisterViewModel defined under the Models folder, we add a TenantName property to the AccountViewModels.cs file: public class RegisterViewModel { [Required] [Display(Name = "Tenant Name")] public string TenantName { get; set; } [Required] [EmailAddress] [Display(Name = "Email")] public string Email { get; set; } // other code omitted } In the Register.cshtml view page under ViewsAccount folder, we add an input box: @using (Html.BeginForm("Register", "Account", FormMethod.Post, new { @class = "form-horizontal", role = "form" })) { @Html.AntiForgeryToken() <h4>Create a new account.</h4> <hr /> @Html.ValidationSummary("", new { @class = "text-danger" }) <div class="form-group"> @Html.LabelFor(m => m.TenantName, new { @class = "col-md-2 control-label" }) <div class="col-md-10"> @Html.TextBoxFor(m => m.TenantName, new { @class = "form-control" }) </div> </div> <div class="form-group"> @Html.LabelFor(m => m.Email, new { @class = "col-md-2 control-label" }) <div class="col-md-10"> @Html.TextBoxFor(m => m.Email, new { @class = "form- control" }) </div> </div> <!-- other code omitted --> } Portal application can be great to allow the tenant owner to manage its own tenant, configuring or handling subscription-related tasks to the supplier company. Deploying the portal application Before tenant deployment, we need to deploy the portal itself. MyAccountant is a complex solution made up of multiple Azure services, which needs to be deployed together. First, we need to create an Azure Resource Group to collect all the services: As we already discussed earlier, all data from different tenants, including the portal itself, need to be contained inside distinct Azure SQL databases. Every user will have their own DB as a personal service, which they don't use frequently. It can be a waste of money assigning a reserved quantity of Database Transaction Units (DTUs) to a single database. We can invest on a pool of DTUs that should be shared among all SQL database instances. We begin by creating an SQL Server service from the portal: We need to create a pool of DTUs, which are shared among databases, and configure the pricing tier, which defines the maximum resources allocation per DB: The first database that we need to manually deploy is the portal database, where users will register as tenants. From the MyAccountantPool blade, we can create a new database that will be immediately associated to the pool: From the database blade, we read the connection: We use this connection string to configure the portal app in web.config: <connectionStrings> <add name="DefaultConnection" connectionString="Server=tcp: {portal_db}.database.windows.net,1433;Data Source={portal_db}; .database.windows.net;Initial Catalog=Portal;Persist Security Info=False;User ID={your_username};Password={your_password}; Pooling=False;MultipleActiveResultSets=False;Encrypt=True; TrustServerCertificate=False;Connection Timeout=30;" providerName="System.Data.SqlClient" /> </connectionStrings> We need to create a shared resource for the Web. In this case, we need to create an App Service Plan where we'll host portal and tenants apps. The initial size is not a problem because we can decide to scale up or scale out the solution at any time (in this case, only when application is able to scale out—we don't handle this scenario here). Then, we need to create portal web app that will be associated with the service plan that we just created: The portal can be deployed from Visual Studio to the Azure subscription by right-clicking on the project root in Solution Explorer and selecting Microsoft Azure Web App from Publish. After deployment, the portal is up and running: Deploy the tenant app After tenant registration from the portal, we need to deploy tenant itself, which is made up of the following: The app itself that is considered as the artifact that has to be deployed A web app that runs the app, hosted on the already defined web app plan The Azure SQL database that contains data inside the elastic pool The connection string that connect database to the web app in the web.config file It's a complex activity because it involves many different resources and different kinds of tasks from deployment to configuration. For this purpose, we have the Azure Resource Group project in Visual Studio, where we can configure web app deployment and configuration via Azure Resource Manager templates. This project will be called Tenant.Deploy, and we choose a blank template to do this. In the azuredeploy.json file, we can type a template such as https://github.com/marcoparenzan/CreateMultitenantAppsInAzure/blob/master/Tenant.Deploy/Templates/azuredeploy.json. This template is quite complex. Remember that in the SQL connection string, the username and password should be provided inside the template. We need to reference the Tenant.Web project from the deployment project because we need to deploy tenant artifacts (the project bits). To support deployment, we need to create an Azure Storage Account back to the Azure portal: To understand how it works, we can manually run a deployment directly from Visual Studio by right-clicking on Deployment project from Solution Explorer and selecting Deploy. When we deploy a "sample" tenant, the first dialog will appear. You can connect to the Azure subscription, selecting an existing resource group or creating a new one and the template that describes the deployment composition. The template requires the following parameters from Edit Parameters window: The tenant name The artifact location and SAS token that are automatically added having selected the Azure Storage account from the previous dialog Now, via the included Deploy-AzureResourceGroup.ps1 PowerShell file, Azure resources are deployed. The artifact is copied with AzCopy.exe command to the Azure storage in the Tenant.Web container as a package.zip file and the resource manager starts allocating resources. We can see that tenant is deployed in the following screenshot: Automating the tenant deployment process Now, in order to complete our solution, we need to invoke this deployment process from the portal application during a registration process call in ASP.NET MVC controls. For the purpose of this article, we will just invoke the execution without defining a production-quality deployment process. We can use the following checklist before proceeding: We already have an Azure Resource Manager template that deploys the tenant app customized for the user Deployment is made with a PowerShell script in the Visual Studio deployment project A new registered user for our application does not have an Azure account; we, as service publisher, need to offer a dedicated Azure account with our credentials to deploy the new tenants Azure offers many different ways to interact with an Azure subscription: The classic portal (https://manage.windowsazure.com) The new portal (https://portal.azure.com) The resource portal (https://resources.azure.com) The Azure REST API (https://msdn.microsoft.com/en-us/library/azure/mt420159.aspx) The Azure .NET SDK (https://github.com/Azure/azure-sdk-for-net) and other platforms The Azure CLI open source CLI (https://github.com/Azure/azure-xplat-cli) PowerShell (https://github.com/Azure/azure-powershell) For our needs, this means integrating in our application. We can make these considerations: We need to reuse the same ARM template that we defined We can reuse PowerShell experience, but we can also use our experience as .NET, REST, or other platform developers Authentication is the real discriminator in our solution: the user is not an Azure subscription user and we don't want to make a constraint on this Interacting with Azure REST API, which is the API on which every other solution depends, requires that all invocations need to be authenticated to the Azure Active Directory of the subscription tenant. We already mentioned that the user is not a subscription-authenticated user. Therefore, we need an unattended authentication to our Azure API subscription using a dedicated user for this purpose, encapsulated into a component that is executed by the ASP.NET MVC application in a secure manner to make the tenant deployment. The only environment that offers an out-of-the box solution for our needs (so that we need to write less code) is the Azure Automation Service. Before proceeding, we create a dedicated user for this purpose. Therefore, for security reasons, we can disable a specific user at any time. You should take note of two things: Never use the credentials that you used to register Azure subscription in a production environment! For automation implementation, you need a Azure AD tenant user, so you cannot use Microsoft accounts (Live or Hotmail). To create the user, we need to go to the classic portal, as Azure Active Directory has no equivalent management UI in the new portal. We need to select the tenant directory, that is, the one in the new portal that is visible in the upper right corner. From the classic portal, go to to Azure Active Directory and select the tenant. Click on Add User and type in a new username: Then, go to Administrator Management in the Setting tab of the portal because we need to define the user as a co-administrator in the subscription that we need to use for deployment. Now, with the temporary password, we need to log in manually to https://portal.azure.com/ (open the browser in private mode) with these credentials because we need to change the password, as it is generated as "expired". We are now ready to proceed. Back in the new portal, we select a new Azure Automation account: The first thing that we need to do inside the account is create a credential asset to store the newly-created AAD credentials and use the inside PowerShell scripts to log on in Azure: We can now create a runbook, which is an automation task that can be expressed in different ways: Graphical PowerShell We choose the second one: As we can edit it directly from portal, we can write a PowerShell script for our purposes. This is an adaptation from the one that we used in a standard way in the deployment project inside Visual Studio. The difference is that it is runable inside a runbook and Azure, and it uses already deployed artifacts that are already in the Azure Storage account that we created before. Before proceeding, we need of two IDs from our subscription: The subscription ID The tenant ID These two parameters can be discovered with PowerShell because we can perform Login-AzureRmAccount. Run it through the command line and copy them from the output: The following code is not production quality (needs some optimization) but for demo purposes: param ( $WebhookData, $TenantName ) # If runbook was called from Webhook, WebhookData will not be null. if ($WebhookData -ne $null) { $Body = ConvertFrom-Json -InputObject $WebhookData.RequestBody $TenantName = $Body.TenantName } # Authenticate to Azure resources retrieving the credential asset $Credentials = Get-AutomationPSCredential -Name "myaccountant" $subscriptionId = '{your subscriptionId}' $tenantId = '{your tenantId}' Login-AzureRmAccount -Credential $Credentials -SubscriptionId $subscriptionId -TenantId $tenantId $artifactsLocation = 'https://myaccountant.blob.core.windows.net/ myaccountant-stageartifacts' $ResourceGroupName = 'MyAccountant' # generate a temporary StorageSasToken (in a SecureString form) to give ARM template the access to the templatea artifacts$StorageAccountName = 'myaccountant' $StorageContainer = 'myaccountant-stageartifacts' $StorageAccountKey = (Get-AzureRmStorageAccountKey - ResourceGroupName $ResourceGroupName -Name $StorageAccountName).Key1 $StorageAccountContext = (Get-AzureRmStorageAccount - ResourceGroupName $ResourceGroupName -Name $StorageAccountName).Context $StorageSasToken = New-AzureStorageContainerSASToken -Container $StorageContainer -Context $StorageAccountContext -Permission r -ExpiryTime (Get-Date).AddHours(4) $SecureStorageSasToken = ConvertTo-SecureString $StorageSasToken -AsPlainText -Force #prepare parameters for the template $ParameterObject = New-Object -TypeName Hashtable $ParameterObject['TenantName'] = $TenantName $ParameterObject['_artifactsLocation'] = $artifactsLocation $ParameterObject['_artifactsLocationSasToken'] = $SecureStorageSasToken $deploymentName = 'MyAccountant' + '-' + $TenantName + '-'+ ((Get-Date).ToUniversalTime()).ToString('MMdd-HHmm') $templateLocation = $artifactsLocation + '/Tenant.Deploy/Templates/azuredeploy.json' + $StorageSasToken # execute New-AzureRmResourceGroupDeployment -Name $deploymentName ` -ResourceGroupName $ResourceGroupName ` -TemplateFile $templateLocation ` @ParameterObject ` -Force -Verbose The script is executable in the Test pane, but for production purposes, it needs to be deployed with the Publish button. Now, we need to execute this runbook from outside ASP.NET MVC portal that we already created. We can use Webhooks for this purpose. Webhooks are user-defined HTTP callbacks that are usually triggered by some event. In our case, this is new tenant registration. As they use HTTP, they can be integrated into web services without adding new infrastructure. Runbooks can directly be exposed as a Webhooks that provides HTTP endpoint natively without the need to provide one by ourself. We need to remember some things: Webhooks are public with a shared secret in the URL, so it is "secure" if we don't share it As a shared secret, it expires, so we need to handle Webhook update in the service lifecycle As a shared secret if more users are needed, more Webhooks are needed, as the URL is the only way to recognize who invoked it (again, don't share Webhooks) Copy the URL at this stage as it is not possible to recover it but it needs to be deleted and generate a new one Write it directly in portal web.config app settings: <add key="DeplyNewTenantWebHook" value="https://s2events.azure- automation.net/webhooks?token={your_token}"/> We can set some default parameters if needed, then we can create it. To invoke the Webhook, we use System.Net.HttpClient to create a POST request, placing a JSON object containing TenantName in the body: var requestBody = new { TenantName = model.TenantName }; var httpClient = new HttpClient(); var responseMessage = await httpClient.PostAsync( ConfigurationManager.AppSettings ["DeplyNewTenantWebHook"], new StringContent(JsonConvert.SerializeObject (requestBody)) ); This code is used to customize the registration process in AccountController: public async Task<ActionResult> Register(RegisterViewModel model) { if (ModelState.IsValid) { var user = new ApplicationUser { UserName = model.Email, Email = model.Email }; var result = await UserManager.CreateAsync(user, model.Password); if (result.Succeeded) { await SignInManager.SignInAsync(user, isPersistent:false, rememberBrowser:false); // handle webhook invocation here return RedirectToAction("Index", "Home"); } AddErrors(result); } The responseMessage is again a JSON object that contains JobId that we can use to programmatically access the executed job. Conclusion There are a lot of things that can be done with the set of topics that we covered in this article. These are a few of them: We can write better .NET code for multitenant apps We can authenticate users on with the Azure Active Directory service We can leverage deployment tasks with Azure Service Bus messaging We can create more interaction and feedback during tenant deployment We can learn how to customize ARM templates to deploy other Azure Storage services, such as DocumentDB, Azure Storage, and Azure Search We can handle more PowerShell for the Azure Management tasks Summary Azure can change the way we write our solutions, giving us a set of new patterns and powerful services to develop with. In particular, we learned how to think about multitenant apps to ensure confidentiality to the users. We looked at deploying ASP.NET web apps in app services and providing computing resources with App Services Plans. We looked at how to deploy SQL in Azure SQL databases and computing resources with elastic pool. We declared a deployment script with Azure Resource Manager, Azure Resource Template with Visual Studio cloud deployment projects, and automated ARM PowerShell script execution with Azure Automation and runbooks. The content we looked at in the earlier section will be content for future articles. Code can be found on GitHub at https://github.com/marcoparenzan/CreateMultitenantAppsInAzure. Have fun! Resources for Article: Further resources on this subject: Introduction to Microsoft Azure Cloud Services [article] Microsoft Azure – Developing Web API for Mobile Apps [article] Security in Microsoft Azure [article]

In this article by Kamil Przeorski, the author of the book Mastering Full Stack React Web Development introduces Universal JavaScript or isomorphic JavaScript features that we are going to implement in thisarticle. To be more exact: we will develop our app the way that we will render the app's pages both on server and client side. It's different to Angular1 or Backbone single-page apps which are mainly rendered on the client side. Our approach is more complicated in technological terms as you need to deploy your full-stack skills which working on a server side rendering, but on the other-side having this experience will make you more desirable programmer so you can advance your career to the next level—you will be able to charge more for your skills on the market. (For more resources related to this topic, see here.) When the server-side is worth implementing h1 The server-side rendering is very useful feature in text's content (like news portals) related startups/companies because it helps to be better indexed by different search engines. It's an essential feature for any news and content heavy websites, because it helps grow them organic traffic. In this article, we will also run our app with server-side rendering. Second segment of companies where server-side rendering may be very useful are entertainment one where users have less patience and they can close the www's browser if a webpage is loading slowly. In general, all B2C (consumer facing) apps shall use server-side rendering to improve its experience with the masses of people who are visiting their websites. Our focus for article will include the following: Making whole server-side code rearrangement to prepare for the server-side rendering Start using react-dom/server and it's renderToString method Are you ready? Our first step is to mock the database's response on the backend (we will create a real DB query after whole server-side rendering will work correctly on the mocked data). Mocking the database response h2 First of all, we will mock our database response on the backend in order to get prepared to go into server-side rendering directly. $ [[you are in the server directory of your project]] $ touch fetchServerSide.js The fetchServerSide.js file will consist of all functions that will fetch data from our database in order to make the server side works.As was mentioned earlier we will mock it for the meanwhile with following code in fetchServerSide.js: export default () => { return { 'article': { '0': { 'articleTitle': 'SERVER-SIDE Lorem ipsum - article one', 'articleContent':'SERVER-SIDE Here goes the content of the article' }, '1': { 'articleTitle':'SERVER-SIDE Lorem ipsum - article two', 'articleContent':'SERVER-SIDE Sky is the limit, the content goes here.' } } } } The goal of making this mocked object once again, is that we will be able to see if our server-side rendering works correctly after implementation because as you probably have already spotted that we have added this SERVER-SIDE in the beginning of each title and content—so it will help us to learn that our app is getting the data from server-side rendering. Later this function will be replaced with a query to MongoDB. Next thing that will help us implement the server-side rendering is to make a handleServerSideRender function that will be triggered each time a request hits the server. In order to make the handleServerSideRender trigger every time the frontend calls our backend we need to use the Express middleware using app.use. So far we were using some external libraries like: app.use(cors()) app.use(bodyParser.json({extended: false})) Now, we will write our own small's middleware function that behaves similar way to the cors or bodyParser (the external libs that are also middlewares). Before doing so, let's import our dependencies that are required in React's server-side rendering (server/server.js): import React from 'react'; import {createStore} from 'redux'; import {Provider} from 'react-redux'; import {renderToStaticMarkup} from 'react-dom/server'; import ReactRouter from 'react-router'; import {RoutingContext, match} from 'react-router'; import * as hist from 'history'; import rootReducer from '../src/reducers'; import reactRoutes from '../src/routes'; import fetchServerSide from './fetchServerSide'; After adding all those imports of the server/server.js, the file will be looking as following: import http from 'http'; import express from 'express'; import cors from 'cors'; import bodyParser from 'body-parser'; import falcor from 'falcor'; import falcorExpress from 'falcor-express'; import falcorRouter from 'falcor-router'; import routes from './routes.js'; import React from 'react' import { createStore } from 'redux' import { Provider } from 'react-redux' import { renderToStaticMarkup } from 'react-dom/server' import ReactRouter from 'react-router'; import { RoutingContext, match } from 'react-router'; import * as hist from 'history'; import rootReducer from '../src/reducers'; import reactRoutes from '../src/routes'; import fetchServerSide from './fetchServerSide'; Important is to import history in the given way as in the example import * as hist from 'history'. The RoutingContext, match is the way of using React-Router on the server side. The renderToStaticMarkup function is going to generate for us a HTML markup on serverside. After we have added those new imports then under falcor's middleware setup: // this already exists in your codebase app.use('/model.json', falcorExpress.dataSourceRoute((req, res) => { return new falcorRouter(routes); // this alrady exsits in your codebase })); Under themodel.jsonfile's code, please add the following: let handleServerSideRender = (req, res) => { return; }; let renderFullHtml = (html, initialState) => { return; };app.use(handleServerSideRender); The app.use(handleServerSideRender) is fired each time the server side receives a request from a client's application. Then we have prepared empty functions that we will use: handleServerSideRender:It will use renderToString in order to create a valid server-side's HTML's markup renderFullHtml:The helper's function will embed our new React's HTML markup into a whole HTML's document as you can later in a moment down below. Summary We have done the basic server-side rendering in this article. Resources for Article: Further resources on this subject: Basic Website using Node.js and MySQL database [article] How to integrate social media with your WordPress website [article] Laravel 5.0 Essentials [article]