How-To Tutorials

Web developers work hard to build a working application that they can be proud of. But how can we ensure a painless maintainability in the future? A comprehensive automated testing layer will become our lifeline once our application begins to scale up and we have to mitigate the impact of bugs caused by new functionalities colliding with the already existing ones. In this article, we will see why we need to perform unit testing and the process of setting up Jasmine for our tests. This tutorial is an excerpt taken from the book 'Learning Angular - second edition' written by Christoffer Noring, Pablo Deeleman.  Why do we need unit tests? What is a unit test? Unit tests are part of an engineering philosophy that takes a stand for efficient and agile development processes, by adding an additional layer of automated testing to the code, before it is developed. The core concept is that each piece of code is delivered with its own test, and both pieces of code are built by the developer who is working on that code. First, we design the test against the module we want to deliver, checking the accuracy of its output and behavior. Since the module is still not implemented, the test will fail. Hence, our job is to build the module in such a way that it passes its own test. Unit testing is quite controversial. While there is a common agreement about how beneficial test-driven development for ensuring code quality and maintenance over time is, not everybody undertakes unit testing in their daily practice. Why is that? Well, building tests while we develop our code can feel like a burden sometimes, particularly when the test winds up being bigger in size than the piece of functionality it aims to test. However, the arguments favoring testing outnumber the arguments against it: Building tests contribute to better code design. Our code must conform to the test requirements and not the other way around. In that sense, if we try to test an existing piece of code and we find ourselves blocked at some point, chances are that the piece of code we aim to test is not well designed and shows off a convoluted interface that requires some rethinking. On the other hand, building testable modules can help with early detection of side effects on other modules. Refactoring tested code is the lifeline against introducing bugs in later stages. Any development is meant to evolve with time, and on every refactor the risk of introducing a bug, that will only pop up in another part of our application, is high. Unit tests are a good way to ensure that we catch bugs at an early stage, either when introducing new features or when updating existing ones. Building tests is a good way to document our code APIs and functionalities. And this becomes a priceless resource when someone not acquainted with the code base takes over the development endeavor. These are only a few arguments, but you can find countless resources on the web about the benefits of testing your code. If you do not feel convinced yet, give it a try. Otherwise, let's continue with our journey and see the overall form of a test. Unit testing in Jasmine There are many different ways to test a piece of code, but in this article, we will look at the anatomy of a test, what it is made up of. The first thing we need, for testing any code, is a test framework. The test framework should provide utility functions for building test suites, containing one or several test specs each. So what are these concepts? Test suite: A suite creates a logical grouping for a bunch of tests. A suite can, for example, be all the tests for a product page. Test spec: This is another name for a unit test. The following shows what a test file can look like where we are using a test suite and placing a number of related tests inside. The chosen framework for this is Jasmine. In Jasmine, the describe() function helps us to define a test suite. The describe() method takes a name as the first parameter and a function as the second parameter. Inside of the describe() function are a number of invocations to the it() method. The it() function is our unit test; it takes the name of the test as the first parameter and a function as the second parameter: // Test suite describe('A math library', () => { // Test spec it('add(1,1,) should return 2', () => { // Test spec implementation goes here }); }); Each test spec checks out a specific functionality of the feature described in the suite description argument and declares one or several expectations in its body. Each expectation takes a value, which we call the expected value, and is compared against an actual value by means of a matcher function, which checks whether expected and actual values match accordingly. This is what we call an assertion, and the test framework will pass or fail the spec depending on the result of such assertions. The code is as follows: // Test suite describe('A math library', () => { // Test spec it('add(1,1) should return 2', () => { // Test assertion expect(add(1,1,)).toBe(2); }); it('subtract(2,1)', () =>{ //Test assertion expect(subtract(2,1)).toBe(1); }) }); In the previous example, add(1,1) will return the actual value that is supposed to match the expected value declared in the toBe() matcher function. Worth noting from the previous example is the addition of a second test that tests our subtract() function. We can clearly see that this test deals with yet another mathematical operation, thus it makes sense to group both these tests under one suite. So far, we have learned about test suites and how to group tests according to their function. Furthermore, we have learned about invoking the code you want to test and asserting that it does what you think it does. There are, however, more concepts to a unit test worth knowing about, namely setup and teardown functionality. A setup functionality is something that sets up your code before the test is run usually. It's a way to keep your code cleaner so you can focus on just invoking the code and asserting. A tear-down functionality is the opposite of a setup functionality and is dedicated to tearing down what you set up initially; essentially it's a way to clean up after the test. Let's see how this can look in practice with a code example, using the Jasmine framework. In Jasmine, the beforeEach() method is used for setup functionality; it runs before every unit test. The afterEach() method is used to run tear-down logic. The code is as follows: describe('a Product service', () => { let productService; beforeEach(() => { productService = new ProductService(); }); it('should return data', () => { let actual = productService.getData(); assert(actual.length).toBe(1); }); afterEach(() => { productService = null; }); }); We can see in the preceding code how the beforeEach() function is responsible for instantiating the productService, which means the test only has to care about invoking production code and asserting the outcome. This makes the test look cleaner. It should be said, though, in reality, tests tend to have a lot of setup going on and having a beforeEach() function can really make the tests look cleaner; above all, it tends to make it easier to add new tests, which is great. What you want at the end of the day is well-tested code; the easier it is to write and maintain such code, the better for your software. Testing web applications in general and Angular applications, in particular, poses a myriad of scenarios that usually need a specific approach. Remember that if a specific test requires a cumbersome and convoluted solution, we are probably facing a good case for a module redesign instead. There are several paths to compound our knowledge of web application testing in Angular and enable us to become great testing ninjas.  In this we saw the importance of unit testing in our Angular applications, the basic shape of a unit test, and the process of setting up Jasmine for our tests. If you found this post useful, do check out the book 'Learning Angular - Second Edition'  to learn more about unit testing and how to implement it for routes, inputs, outputs, directives, etc. Everything new in Angular 6: Angular Elements, CLI commands and more Angular 6 is here packed with exciting new features! Getting started with Angular CLI and build your first Angular Component

Last week, the Digital, Culture, Media and Sport Sub-Committee held its hearing on Disinformation with Aaron Greenspan as the witness. Aaron is the founder, president, and CEO of Think Computer Corporation, an IT consulting service and the author of the book ‘Authoritas: One Student's Harvard Admissions and the Founding of the Facebook Era’. Aaron who claims to have had the original idea for Facebook, has been a long standing critic of the social network. In January this year, Aaron published a 75-page report, in which he states that fake accounts made up more than half of Facebook's 2.2 billion users. He has also testified the same, in front of the UK Parliamentary Committee. He minced no words when he said Mark Zuckerberg is a liar, fraudster and unfit to be the CEO of Facebook. Fake accounts Facebook differentiates between duplicated accounts and fake accounts. But Aaron considers “any account which is not in your name and you have a second account for whatever purpose, that account is ultimately a fake account”. He noticed the issue of fake accounts on Facebook last year, after which he did his own enquiry and found some “alarming” numbers. At the end of 2017, Facebook claimed that “only 1% of their accounts are fake”. By their own definition, two weeks ago they have said that the number of “fake accounts have increased to 5%.” This means that in a span of two years, their estimate has increased fivefold. The second issue, Aaron highlights is that it's “extremely unclear” how they have arrived at that estimate. Two years ago, Facebook came up with a ‘Transparency portal’, due to public pressure. The transparency portal of Facebook aims at publishing “regular reports to give our community visibility into how we enforce policies, respond to data requests and protect intellectual property, while monitoring dynamics that limit access to Facebook products.” Aaron states that he found “very difficult to reconcile the SEC filings of Facebook with the numbers from the transparency portal”. He says that the number of fake accounts published on the transparency portal of Facebook do not match with the SEC filings. This is the reason why he decided to do his own report and find out if the numbers from the “two sources of Facebook aligned” and has come to the conclusion that they “don’t”. Instead Aaron has arrived at a conditional conclusion that “around 30% of accounts on Facebook are fake”. He claims that Facebook always minimizes their numbers to make the problem look smaller than it is. Also he adds that “Based on my total use of the platform, the historical trend starting in 2006, when it was made public, up until the present, it seemed like it was safe that 50% are fake and I think this could actually be higher.” Some weeks ago, Facebook “finally updated their transparency portal and announced that in the fourth quarter of 2018, that they have disabled 1.2 billion fake accounts. And in the first quarter of 2019, the numbers counted to 2.2 billion fake accounts, which is an exponential growth curve in fake accounts, according to their own numbers, which has not been audited by any respective body”. If all the numbers are added, “by a conservative guess, it can be said that there are 10billion fake accounts, for a platform which has 2.2 billion active users”. While “Facebook says that fake accounts don’t matter very much and within their undisclosed methodology, they are doing a good job.” Aaron believes that transparency around fake accounts is the number one problem for Facebook to look after. FB is more ‘unwilling’ rather than ‘unable’ to tackle Fake accounts When asked, if he thinks that “Facebook wants to tackle these problems of fake accounts and others”, Aaron answered “No”. He believes very strongly that Mark has no clear intention of complying with the law, as he’s not appearing in front of the parliament, in the UK or in Canada or anywhere where serious questions will be aimed at him. This is because, “he has no genuine answers in many cases. So i have no faith in Facebook, I don't think it can be trusted and don't think it should be trusted and i would suggest independent analysis.” The illusion of Behavioural advertising effectiveness Aaron claims that “Behavioural advertising produces 4% benefit of revenue to advertisers”. So while adexchangers like “Google or Facebook might charge 59% more on average for targeted ads, or upto 4900% more” in some cases, they are going to get customers as they get benefit from them. He also described Facebook as a "black box", and claimed advertisers were "in the dark" about how effective their campaigns actually are on Facebook and if it’s actually reaching real users. Aaron adds that “I have come to the conclusion that Facebook is not in anybody’s control. The company has lost its capability to control its own platform. And i don’t think they can truly regain that ability”, he thus likened the social network to the ‘Chernobyl disaster’, the largest catastrophic nuclear disaster to hit the world in 1986. In this particular situation, there was a technology which was hyped and was “expected to be transformative and make some other problems go away”. Aaron says that in 2004, Mark had described the Facebook system as “something that would involve the problem of reaching critical mass, a nuclear power reference”. Aaron believes that by designing Facebook, the way it is now, “Mark has effectively removed the control from the reactor core”, and the result is an “enormous uninhabitable zone in the internet, which is polluted with disinformation and falsity, much like radiation, that transpires nearly impossible reverse.” History of Facebook Aaron who has always claimed that he is the original creator of Facebook, says that “My need to create ‘Universal Facebook’, was based on Harvard’s structure as an organization, while Mark wanted to build something cool.” He also states that “certainly neither of us had thought of this to be a global encompassing system”. Aaron says that if he would have known that Mark was planning on “something like a huge global system”, he would have made it clear to Mark that “this could end up being a privacy nightmare”. He adds that “In the early days, Facebook lost control of the platform and it will never get it back”, and “unfortunately, the Media has been a significant member in propelling Facebook since the last 10-12 years.” Facebook is not growing Aaron believes that Mark has been lying to investors and the global community about Facebook’s growth, which is not true. He says that from indicators, as an outside observer, everything suggests that the usage of Facebook is falling drastically as users are having concerns about their privacy and yet it is being publicised that Facebook is growing. In reality, Facebook is growing in countries like India, Philippines, Vietnam, and Indonesia, which are the same countries where Facebook claims in their disclaimer that “we have more fake accounts coming from these countries than anyone else.” This means that Facebook is growing in countries where there is a known problem of ‘fake accounts’ and this is more problematic than the rest of the world. Aaron alleges that this misrepresentation of facts to shareholders amounts to “fraud”. On respecting Privacy and Personal Data Aaron states that Facebook used to lie initially that “Openness is a universal good”, while the new lie Facebook says is that “Encryption is the same as privacy,” which is not true, “and that privacy is the universal good.” So Aaron believes that Mark is going completely in contrast to his earlier beliefs, as the “previous model was not working for him anymore so he made a new model, and this is going to increase” further for every next level of Facebook. Aaron also adds that “Encryption comes with a lot of pitfalls.” On the question of whether Zuckerberg respects personal data, Aaron has claimed that Mark does not believe in the concept of Personal Data as he has been performing security fraud on a number of occasions, in an incredibly blatant manner. He states that “the SEC has done nothing about it because they are afraid of targeting a billionaire”. He also pointed out that Mark is not the only executive who lies to stockholders, and claimed that even other tech giants get away with this. For example, “Elon Musk does it.” When asked “if there were any warning signs of the Cambridge Analytica scandal”, prior to 2016. Aaron says that “This wasn’t so much a breach as it was a designed behaviour, and that design was made so on Mark’s orders”. Aaron also recalled that, in 2007, when he was working with Ed Baker, Mark Zuckerberg’s current colleague, he was planning to break the law with a technique, allegedly designed to steal customer data. He added that “When I worked with Ed, he suggested for the software that we were building that, we should ask users for access to their address book. And regardless of whether the answer was yes or no, we should take that data anyway, and use that to send emails to other potential users”. Aaron claims that “At that point, I quit”. Eventually, Ed joined Facebook and is part of its growth team now. Antitrust Law Debate If an antitrust action is taken against Facebook, it would result in Whatsapp and Instagram being separated from its mother platform. However, Mark will still be responsible for 12 billion of users data of Facebook. According to Aaron, this is a huge problem. And this is the reason he believes that “antitrust action against Facebook, is not going to be effective in the long run”. He also adds that some other major methods needs to be undertaken to make Facebook behave responsibly. How do we solve a problem like Facebook? Aaron has some ideas Aaron believes that regulating an entity as large and complex like Facebook requires technical knowledge, and “by large US regulators lack the technical knowledge, to effectively enforce the laws that are already on the book.” Aaron proposes a number of solutions for ways to regulate Facebook, out of which, the most important step he believed is “to remove Mark from the C.E.O. position.” He considers Mark incapable in being a responsible CEO to Facebook. Aaron’s recommendation comes at a time when this week, around 68% of independent investors wanted the company to have an independent chairman. Despite the revolt, the proposal was not passed as Mark owns upto 75% of Class B stock, i.e., he has almost 60% of the voting power at Facebook. Obviously, he and his colleagues voted down the independent Chairman proposal very smoothly. Next, Aaron suggested to regulate Facebook, along the lines of a government regulated bank. He proposes to have KYC requirements for all social media at this point of time. Also he adds that “I think anonymous speech should not be banned, as it plays an important role, but if there’s a problem in any case, then the anonymous person should be held to account.” He says that as “Transparency around fake accounts, is the number one problem faced by Facebook”, a ‘Social Media Tax’ should be levied on all its users. He believes that it will provide some revenue to fund investigative journalism, and due to payment involved, it will require some kind of authentication. This can play a major role in identification of fake accounts. He says that this will make “the entire process manageable for governments.” Check out the full hearing on the Parliament TV website. US regulators plan to probe Google on anti-trust issues; Facebook, Amazon & Apple also under legal scrutiny Facebook argues it didn’t violate users’ privacy rights and thinks there’s no expectation of privacy because there is no privacy on social media Experts present most pressing issues facing global lawmakers on citizens’ privacy, democracy and rights to freedom of speech

In this tutorial, we will understand how to leverage chatbots to assist in network operations. As we move toward intelligent operations, another area to focus on is mobility. It's good to have a script to perform configurations, remediations, or even troubleshooting, but it still requires a presence to monitor, initiate, or even execute those programs or scripts. Nokia's MIKA is a good example of a chatbot that operations personnel can use for network troubleshooting and repair. According to Nokia's blog,  MIKA responds with an alarm prioritization information based on the realities for this individual network and also compare's the current situation to a whole service history of past events from this network and others, in order to identify the best solution for the current problem. Let's create a chatbot to assist in network operations. For this use case, we will use a widely-used chat application, Slack. Referring to the intelligent data analysis capabilities of Splunk, we would see some user chat interaction with the chatbot, to get some insight into the environment. This tutorial is an excerpt from a book written by Abhishek Ratan titled Practical Network Automation - Second Edition. This book will acquaint you with the fundamental concepts of network automation and help you improve your data center's robustness and security. The code for this tutorial can be found on GitHub. As we have our web framework deployed, we'll leverage the same framework to interact with the Slack chatbot, which in turn will interact with Splunk. It can also interact directly with network devices so we can initiate some complex chats, such as rebooting a router from Slack if need be. This eventually gives mobility to an engineer who can work on tasks from anywhere (even from a cellphone) without being tied to a certain location or office. To create a chatbot, here are the basic steps: Create a workspace (or account) on Slack: Create an application in your workspace (in our case, we have created an app called mybot): Here is the basic information about the application (App ID and Client ID can be used along with other information that uniquely identifies this application): Add a bot capability to this application: Add the event subscriptions and mapping to the external API that the messages will be posted to. An event subscription is when someone types the reference to the chatbot on the chat, then which API will be called with the data that is being typed in the chat with this chatbot: Here, a crucial step is once we type in the URL that accepts chat messages, that particular URL needs to be verified from Slack. A verification involves the API endpoint sending the same response back as a string or JSON that is being sent to that endpoint from Slack. If we receive the same response, Slack confirms that the endpoint is authentic and marks it as verified. This is a one-time process and any changes in the API URL will result in repeating this step. Here is the Python code in the Ops API framework that responds to this specific query: import falcon import json def on_get(self,req,resp): # Handles GET request resp.status=falcon.HTTP_200 # Default status resp.body=json.dumps({"Server is Up!"}) def on_post(self,req,resp): # Handles POST Request print("In post") data=req.bounded_stream.read() try: # Authenticating end point to Slack data=json.loads(data)["challenge"] # Default status resp.status=falcon.HTTP_200 # Send challenge string back as response resp.body=data except: # URL already verified resp.status=falcon.HTTP_200 resp.body="" This would validate, and if a challenge is sent from Slack, it would respond back with the same challenge value that confirms it to be the right endpoint for the Slack channel to send chat data to. Install this application (or chatbot) into any channels (this is similar to adding a user in a group chat): The core API framework code that responds to specific chat messages, performs the following actions: Acknowledges any post sent to Slack with a response of 200 in three seconds. If this is not done, Slack reports back: endpoint not reachable. Ensures any message sent from chatbot (not from any real user) is again not sent back as a reply. This can create a loop, since a message sent from a chatbot, would be treated as a new message in Slack chat and it would be sent again to URL. This would eventually make the chat unusable, causing repetitive messages on the chat. Authenticates the response with a token that would be sent back to Slack to ensure the response coming to Slack is from an authenticated source. The code is as follows: import falcon import json import requests import base64 from splunkquery import run from splunk_alexa import alexa from channel import channel_connect,set_data class Bot_BECJ82A3V(): def on_get(self,req,resp): # Handles GET request resp.status=falcon.HTTP_200 # Default status resp.body=json.dumps({"Server is Up!"}) def on_post(self,req,resp): # Handles POST Request print("In post") data=req.bounded_stream.read() try: bot_id=json.loads(data)["event"]["bot_id"] if bot_id=="BECJ82A3V": print("Ignore message from same bot") resp.status=falcon.HTTP_200 resp.body="" return except: print("Life goes on. . .") try: # Authenticating end point to Slack data=json.loads(data)["challenge"] # Default status resp.status=falcon.HTTP_200 # Send challenge string back as response resp.body=data except: # URL already verified resp.status=falcon.HTTP_200 resp.body="" print(data) data=json.loads(data) #Get the channel and data information channel=data["event"]["channel"] text=data["event"]["text"] # Authenticate Agent to access Slack endpoint token="xoxp-xxxxxx" # Set parameters print(type(data)) print(text) set_data(channel,token,resp) # Process request and connect to slack channel channel_connect(text) return # falcon.API instance , callable from gunicorn app= falcon.API() # instantiate helloWorld class Bot3V=Bot_BECJ82A3V() # map URL to helloWorld class app.add_route("/slack",Bot3V) Performing a channel interaction response: This code takes care of interpreting specific chats that are performed with chat-bot, in the chat channel. Additionally, this would respond with the reply, to the specific user or channel ID and with authentication token to the Slack API https://slack.com/api/chat.postMessage. This ensures the message or reply back to the Slack chat is shown on the specific channel, from where it originated. As a sample, we would use the chat to encrypt or decrypt a specific value. For example, if we write encrypt username[:]password, it would return an encrypted string with a base64 value. Similarly, if we write decrypt <encoded string>, the chatbot would return a <username/password> after decrypting the encoded string. The code is as follows: import json import requests import base64 from splunk_alexa import alexa channl="" token="" resp="" def set_data(Channel,Token,Response): global channl,token,resp channl=Channel token=Token resp=Response def send_data(text): global channl,token,res print(channl) resp = requests.post("https://slack.com/api/chat.postMessage",data='{"channel":"'+channl+'","text":"'+text+'"}',headers={"Content-type": "application/json","Authorization": "Bearer "+token},verify=False) def channel_connect(text): global channl,token,resp try: print(text) arg=text.split(' ') print(str(arg)) path=arg[0].lower() print(path in ["decode","encode"]) if path in ["decode","encode"]: print("deecode api") else: result=alexa(arg,resp) text="" try: for i in result: print(i) print(str(i.values())) for j in i.values(): print(j) text=text+' '+j #print(j) if text=="" or text==None: text="None" send_data(text) return except: text="None" send_data(text) return decode=arg[1] except: print("Please enter a string to decode") text="<decode> argument cannot be empty" send_data(text) return deencode(arg,text) def deencode(arg,text): global channl,token,resp decode=arg[1] if arg[1]=='--help': #print("Sinput") text="encode/decode <encoded_string>" send_data(text) return if arg[0].lower()=="encode": encoded=base64.b64encode(str.encode(decode)) if '[:]' in decode: text="Encoded string: "+encoded.decode('utf-8') send_data(text) return else: text="sample string format username[:]password" send_data(text) return try: creds=base64.b64decode(decode) creds=creds.decode("utf-8") except: print("problem while decoding String") text="Error decoding the string. Check your encoded string." send_data(text) return if '[:]' in str(creds): print("[:] substring exists in the decoded base64 credentials") # split based on the first match of "[:]" credentials = str(creds).split('[:]',1) username = str(credentials[0]) password = str(credentials[1]) status = 'success' else: text="encoded string is not in standard format, use username[:]password" send_data(text) print("the encoded base64 is not in standard format username[:]password") username = "Invalid" password = "Invalid" status = 'failed' temp_dict = {} temp_dict['output'] = {'username':username,'password':password} temp_dict['status'] = status temp_dict['identifier'] = "" temp_dict['type'] = "" #result.append(temp_dict) print(temp_dict) text="<username> "+username+" <password> "+password send_data(text) print(resp.text) print(resp.status_code) return This code queries the Splunk instance for a particular chat with the chatbot. The chat would ask for any management interface (Loopback45) that is currently down. Additionally, in the chat, a user can ask for all routers on which the management interface is up. This English response is converted into a Splunk query and, based upon the response from Splunk, it returns the status to the Slack chat. Let us see the code that performs the action to respond the result, to Slack chat: from splunkquery import run def alexa(data,resp): try: string=data.split(' ') except: string=data search=' '.join(string[0:-1]) param=string[-1] print("param"+param) match_dict={0:"routers management interface",1:"routers management loopback"} for no in range(2): print(match_dict[no].split(' ')) print(search.split(' ')) test=list(map(lambda x:x in search.split(' '),match_dict[no].split(' '))) print(test) print(no) if False in test: pass else: if no in [0,1]: if param.lower()=="up": query="search%20index%3D%22main%22%20earliest%3D0%20%7C%20dedup%20interface_name%2Crouter_name%20%7C%20where%20interface_name%3D%22Loopback45%22%20%20and%20interface_status%3D%22up%22%20%7C%20table%20router_name" elif param.lower()=="down": query="search%20index%3D%22main%22%20earliest%3D0%20%7C%20dedup%20interface_name%2Crouter_name%20%7C%20where%20interface_name%3D%22Loopback45%22%20%20and%20interface_status%21%3D%22up%22%20%7C%20table%20router_name" else: return "None" result=run(query,resp) return result The following Splunk query fetches the status: For UP interface: The query would be as follows: index="main" earliest=0 | dedup interface_name,router_name | where interface_name="Loopback45" and interface_status="up" | table router_name For DOWN interface (any status except ): The query would be as follows: index="main" earliest=0 | dedup interface_name,router_name | where interface_name="Loopback45" and interface_status!="up" | table router_name Let's see the end result of chatting with the chatbot and the responses being sent back based on the chats. The encoding/decoding example is as follows: As we can see here, we sent a chat with the encode abhishek[:]password123 message. This chat was sent as a POST request to the API, which in turn encrypted it to base64 and responded back with the added words as Encoded string: <encoded string>. In the next chat, we passed the same string with the decode option. This responds back with decoding the information from API function, and responds back to Slack chat, with username abhishek and password password123. Let's see the example of the Splunk query chat: In this query, we have shut down the Loopback45 interface on rtr1. During our scheduled discovery of those interfaces through the Python script, the data is now in Splunk. When queried on which management interface (Loopback45) is down, it would respond back with rtr1. The slack chat, On which routers the management interface is down, would pass this to the API, which, upon receiving this payload, will run the Splunk query to get the stats. The return value (which, in this case, is rtr1) will be given back as a response in the chat. Similarly, a reverse query of, On which routers the management interface is up, will query Splunk and eventually share back the response as rtr2, rtr3, and rtr4 (as interfaces on all these routers are UP). This chat use case can be extended to ensure that full end-to-end troubleshooting can occur using a simple chat. Extensive cases can be built using various backend functions, starting from a basic identification of problems to complex tasks, such as remediation based upon identified situations. Summary In this tutorial, we implemented some real-life use cases and looked at techniques to perform troubleshooting using chatbot. The use cases gave us insight into performing intelligent remediation as well as performing audits at scale, which are key challenges in the current environment. To learn how to automate your own network without any hassle while leveraging the power of Python, check out our book Practical Network Automation - Second Edition.  Preparing and automating a task in Python [Tutorial] PyPy 7.0 released for Python 2.7, 3.5, and 3.6 alpha 5 blog posts that could make you a better Python programmer

In this article, David Mark Clements, the author of the book, Node.js Cookbook, we will be covering the following points to introduce you to using Node.js  for exploratory data analysis: Node's module system Initializing a module Writing a module Tooling around modules Publishing modules Setting up a private module repository Best practices (For more resources related to this topic, see here.) In idiomatic Node, the module is the fundamental unit of logic. Any typical application or system consists of generic code and application code. As a best practice, generic shareable code should be held in discrete modules, which can be composed together at the application level with minimal amounts of domain-specific logic. In this article, we'll learn how Node's module system works, how to create modules for various scenarios, and how we can reuse and share our code. Scaffolding a module Let's begin our exploration by setting up a typical file and directory structure for a Node module. At the same time, we'll be learning how to automatically generate a package.json file (we refer to this throughout as initializing a folder as a package) and to configure npm (Node's package managing tool) with some defaults, which can then be used as part of the package generation process. In this recipe, we'll create the initial scaffolding for a full Node module. Getting ready Installing Node If we don't already have Node installed, we can go to https://nodejs.org to pick up the latest version for our operating system. If Node is on our system, then so is the npm executable; npm is the default package manager for Node. It's useful for creating, managing, installing, and publishing modules. Before we run any commands, let's tweak the npm configuration a little: npm config set init.author.name "<name here>" This will speed up module creation and ensure that each package we create has a consistent author name, thus avoiding typos and variations of our name. npm stands for... Contrary to popular belief, npm is not an acronym for Node Package Manager; in fact, it stands for npm is Not An Acronym, which is why it's not called NINAA. How to do it… Let's say we want to create a module that converts HSL (hue, saturation, luminosity) values into a hex-based RGB representation, such as will be used in CSS (for example,  #fb4a45 ). The name hsl-to-hex seems good, so let's make a new folder for our module and cd into it: mkdir hsl-to-hex cd hsl-to-hex Every Node module must have a package.json file, which holds metadata about the module. Instead of manually creating a package.json file, we can simply execute the following command in our newly created module folder: npm init This will ask a series of questions. We can hit enter for every question without supplying an answer. Note how the default module name corresponds to the current working directory, and the default author is the init.author.name value we set earlier. An npm init should look like this: Upon completion, we should have a package.json file that looks something like the following: { "name": "hsl-to-hex", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "echo "Error: no test specified" && exit 1" }, "author": "David Mark Clements", "license": "MIT" } How it works… When Node is installed on our system, npm comes bundled with it. The npm executable is written in JavaScript and runs on Node. The npm config command can be used to permanently alter settings. In our case, we changed the init.author.name setting so that npm init would reference it for the default during a module's initialization. We can list all the current configuration settings with npm config ls . Config Docs Refer to https://docs.npmjs.com/misc/config for all possible npm configuration settings. When we run npm init, the answers to prompts are stored in an object, serialized as JSON and then saved to a newly created package.json file in the current directory. There's more… Let's find out some more ways to automatically manage the content of the package.json file via the npm command. Reinitializing Sometimes additional metadata can be available after we've created a module. A typical scenario can arise when we initialize our module as a git repository and add a remote endpoint after creating the module. Git and GitHub If we've not used the git tool and GitHub before, we can refer to http://help.github.com to get started. If we don't have a GitHub account, we can head to http://github.com to get a free account. To demonstrate, let's create a GitHub repository for our module. Head to GitHub and click on the plus symbol in the top-right, then select New repository: Select New repository. Specify the name as hsl-to-hex and click on Create Repository. Back in the Terminal, inside our module folder, we can now run this: echo -e "node_modulesn*.log" > .gitignore git init git add . git commit -m '1st' git remote add origin http://github.com/<username>/hsl-to-hex git push -u origin master Now here comes the magic part; let's initialize again (simply press enter for every question): npm init This time the Git remote we just added was detected and became the default answer for the git repository question. Accepting this default answer meant that the repository, bugs, and homepage fields were added to package.json . A repository field in package.json is an important addition when it comes to publishing open source modules since it will be rendered as a link on the modules information page at http://npmjs.com. A repository link enables potential users to peruse the code prior to installation. Modules that can't be viewed before use are far less likely to be considered viable. Versioning The npm tool supplies other functionalities to help with module creation and management workflow. For instance, the npm version command can allow us to manage our module's version number according to SemVer semantics. SemVer SemVer is a versioning standard. A version consists of three numbers separated by a dot, for example, 2.4.16. The position of a number denotes specific information about the version in comparison to the other versions. The three positions are known as MAJOR.MINOR.PATCH. The PATCH number is increased when changes have been made that don't break the existing functionality or add any new functionality. For instance, a bug fix will be considered a patch. The MINOR number should be increased when new backward compatible functionality is added. For instance, the adding of a method. The MAJOR number increases when backwards-incompatible changes are made. Refer to http://semver.org/ for more information. If we were to a fix a bug, we would want to increase the PATCH number. We can either manually edit the version field in package.json , setting it to 1.0.1, or we can execute the following: npm version patch This will increase the version field in one command. Additionally, if our module is a Git repository, it will add a commit based on the version (in our case, v1.0.1), which we can then immediately push. When we ran the command, npm output the new version number. However, we can double-check the version number of our module without opening package.json: npm version This will output something similar to the following: { 'hsl-to-hex': '1.0.1', npm: '2.14.17', ares: '1.10.1-DEV', http_parser: '2.6.2', icu: '56.1', modules: '47', node: '5.7.0', openssl: '1.0.2f', uv: '1.8.0', v8: '4.6.85.31', zlib: '1.2.8' } The first field is our module along with its version number. If we added a new backwards-compatible functionality, we can run this: npm version minor Now our version is 1.1.0. Finally, we can run the following for a major version bump: npm version major This sets our modules version to 2.0.0. Since we're just experimenting and didn't make any changes, we should set our version back to 1.0.0. We can do this via the npm command as well: npm version 1.0.0 See also Refer to the following recipes: Writing module code Publishing a module Installing dependencies In most cases, it's most wise to compose a module out of other modules. In this recipe, we will install a dependency. Getting ready For this recipe, all we need is Command Prompt open in the hsl-to-hex folder from the Scaffolding a module recipe. How to do it… Our hsl-to-hex module can be implemented in two steps: Convert the hue degrees, saturation percentage, and luminosity percentage to corresponding red, green, and blue numbers between 0 and 255. Convert the RGB values to HEX. Before we tear into writing an HSL to the RGB algorithm, we should check whether this problem has already been solved. The easiest way to check is to head to http://npmjs.com and perform a search: Oh, look! Somebody already solved this. After some research, we decide that the hsl-to-rgb-for-reals module is the best fit. Ensuring that we are in the hsl-to-hex folder, we can now install our dependency with the following: npm install --save hsl-to-rgb-for-reals Now let's take a look at the bottom of package.json: tail package.json #linux/osx type package.json #windows Tail output should give us this: "bugs": { "url": "https://github.com/davidmarkclements/hsl-to-hex/issues" }, "homepage": "https://github.com/davidmarkclements/hsl-to-hex#readme", "description": "", "dependencies": { "hsl-to-rgb-for-reals": "^1.1.0" } } We can see that the dependency we installed has been added to a dependencies object in the package.json file. How it works… The top two results of the npm search are hsl-to-rgb and hsl-to-rgb-for-reals . The first result is unusable because the author of the package forgot to export it and is unresponsive to fixing it. The hsl-to-rgb-for-reals module is a fixed version of hsl-to-rgb . This situation serves to illustrate the nature of the npm ecosystem. On the one hand, there are over 200,000 modules and counting, and on the other many of these modules are of low value. Nevertheless, the system is also self-healing in that if a module is broken and not fixed by the original maintainer, a second developer often assumes responsibility and publishes a fixed version of the module. When we run npm install in a folder with a package.json file, a node_modules folder is created (if it doesn't already exist). Then, the package is downloaded from the npm registry and saved into a subdirectory of node_modules (for example, node_modules/hsl-to-rgb-for-reals ). npm 2 vs npm 3 Our installed module doesn't have any dependencies of its own. However, if it did, the sub-dependencies would be installed differently depending on whether we're using version 2 or version 3 of npm. Essentially, npm 2 installs dependencies in a tree structure, for instance, node_modules/dep/node_modules/sub-dep-of-dep/node_modules/sub-dep-of-sub-dep. Conversely, npm 3 follows a maximally flat strategy where sub-dependencies are installed in the top level node_modules folder when possible, for example, node_modules/dep, node_modules/sub-dep-of-dep, and node_modules/sub-dep-of-sub-dep. This results in fewer downloads and less disk space usage; npm 3 resorts to a tree structure in cases where there are two versions of a sub-dependency, which is why it's called a maximally flat strategy. Typically, if we've installed Node 4 or above, we'll be using npm version 3. There's more… Let's explore development dependencies, creating module management scripts and installing global modules without requiring root access. Installing development dependencies We usually need some tooling to assist with development and maintenance of a module or application. The ecosystem is full of programming support modules, from linting to testing to browser bundling to transpilation. In general, we don't want consumers of our module to download dependencies they don't need. Similarly, if we're deploying a system built-in node, we don't want to burden the continuous integration and deployment processes with superfluous, pointless work. So, we separate our dependencies into production and development categories. When we use npm --save install <dep>, we're installing a production module. To install a development dependency, we use --save-dev. Let's go ahead and install a linter. JavaScript Standard Style A standard is a JavaScript linter that enforces an unconfigurable ruleset. The premise of this approach is that we should stop using precious time up on bikeshedding about syntax. All the code in this article uses the standard linter, so we'll install that: npm install --save-dev standard semistandard If the absence of semicolons is abhorrent, we can choose to install semistandard instead of standard at this point. The lint rules match those of standard, with the obvious exception of requiring semicolons. Further, any code written using standard can be reformatted to semistandard using the semistandard-format command tool. Simply, run npm -g i semistandard-format to get started with it. Now, let's take a look at the package.json file: { "name": "hsl-to-hex", "version": "1.0.0", "main": "index.js", "scripts": { "test": "echo "Error: no test specified" && exit 1" }, "author": "David Mark Clements", "license": "MIT", "repository": { "type": "git", "url": "git+ssh://git@github.com/davidmarkclements/hsl-to-hex.git" }, "bugs": { "url": "https://github.com/davidmarkclements/hsl-to-hex/issues" }, "homepage": "https://github.com/davidmarkclements/hsl-to- hex#readme", "description": "", "dependencies": { "hsl-to-rgb-for-reals": "^1.1.0" }, "devDependencies": { "standard": "^6.0.8" } } We now have a devDependencies field alongside the dependencies field. When our module is installed as a sub-dependency of another package, only the hsl-to-rgb-for-reals module will be installed while the standard module will be ignored since it's irrelevant to our module's actual implementation. If this package.json file represented a production system, we could run the install step with the --production flag, as shown: npm install --production Alternatively, this can be set in the production environment with the following command: npm config set production true Currently, we can run our linter using the executable installed in the node_modules/.bin folder. Consider this example: ./node_modules/.bin/standard This is ugly and not at all ideal. Refer to Using npm run scripts for a more elegant approach. Using npm run scripts Our package.json file currently has a scripts property that looks like this: "scripts": { "test": "echo "Error: no test specified" && exit 1" }, Let's edit the package.json file and add another field, called lint, as follows: "scripts": { "test": "echo "Error: no test specified" && exit 1", "lint": "standard" }, Now, as long as we have standard installed as a development dependency of our module (refer to Installing Development Dependencies), we can run the following command to run a lint check on our code: npm run-script lint This can be shortened to the following: npm run lint When we run an npm script, the current directory's node_modules/.bin folder is appended to the execution context's PATH environment variable. This means even if we don't have the standard executable in our usual system PATH, we can reference it in an npm script as if it was in our PATH. Some consider lint checks to be a precursor to tests. Let's alter the scripts.test field, as illustrated: "scripts": { "test": "npm run lint", "lint": "standard" }, Chaining commands Later, we can append other commands to the test script using the double ampersand (&&) to run a chain of checks. For instance, "test": "npm run lint && tap test". Now, let's run the test script: npm run test Since the test script is special, we can simply run this: npm test Eliminating the need for sudo The npm executable can install both the local and global modules. Global modules are mostly installed so to allow command line utilities to be used system wide. On OS X and Linux, the default npm setup requires sudo access to install a module. For example, the following will fail on a typical OS X or Linux system with the default npm setup: npm -g install cute-stack # <-- oh oh needs sudo This is unsuitable for several reasons. Forgetting to use sudo becomes frustrating; we're trusting npm with root access and accidentally using sudo for a local install causes permission problems (particularly with the npm local cache). The prefix setting stores the location for globally installed modules; we can view this with the following: npm config get prefix Usually, the output will be /usr/local . To avoid the use of sudo, all we have to do is set ownership permissions on any subfolders in /usr/local used by npm: sudo chown -R $(whoami) $(npm config get prefix)/{lib/node_modules,bin,share} Now we can install global modules without root access: npm -g install cute-stack # <-- now works without sudo If changing ownership of system folders isn't feasible, we can use a second approach, which involves changing the prefix setting to a folder in our home path: mkdir ~/npm-global npm config set prefix ~/npm-global We'll also need to set our PATH: export PATH=$PATH:~/npm-global/bin source ~/.profile The source essentially refreshes the Terminal environment to reflect the changes we've made. See also Scaffolding a module Writing module code Publishing a module Resources for Article: Further resources on this subject: Understanding and Developing Node Modules [article] Working with Pluginlib, Nodelets, and Gazebo Plugins [article] Basic Website using Node.js and MySQL database [article]

The World Wide Web turned 30 years old yesterday. As a part of the celebration, its creator, Tim Berners-Lee published an open letter on Monday, sharing his vision for the future of the web. In this year’s letter, he also expressed his concerns about the direction in which the web is heading and how we can make it as the one he envisioned. To celebrate #Web30, Tim Berners-Lee is on a 30-hour trip and his first stop was the birthplace of WWW, the European Organization for Nuclear Research, CERN. https://twitter.com/timberners_lee/status/1105400740112203777 Back in 1989, Tim Berners-Lee, as a research fellow at the CERN researching lab, wrote a proposal to his boss titled Information Management: A Proposal. This proposal was for building an information system that would allow researchers to share general information about accelerators and experiments. Initially, he named the project “The Mesh”, which combined hypertext with internet TCP and domain name system. The project did not go that well, but Berners-Lee’s boss, Mike Sendall did remark that the idea is “vague but exciting”. Later on, in 1990, he actually started coding for the project and this time he named the project, what we know today as, the World Wide Web. Fast forward to now, the simple innocent system that he built has become so large, connecting millions and millions of people across the globe. If you are curious to know how WWW looked back then, check out its revived version by a CERN team: https://twitter.com/CERN/status/1105457772626358273 The three dysfunctions the Web is now facing World Wide Web has come a long way. It has opened various opportunities, given voice to marginalized groups, and has made our daily lives much convenient and easier. At the same time, it has also given opportunities to scammers, provided a platform for hate speech, and made it extremely easy for committing crimes while sitting behind a computer screen. Berners-Lee listed down three sources of problems that are affecting today’s web and also suggested a few ways we can minimize or prevent them: “Deliberate, malicious intent, such as state-sponsored hacking and attacks, criminal behavior, and online harassment.” Though it is really not possible to completely eliminate this dysfunction, policymakers can come up with laws and developers can take the responsibility to write code that will help minimize this behavior. “System design that creates perverse incentives where user value is sacrificed, such as ad-based revenue models that commercially reward clickbait and the viral spread of misinformation.” These type of systems introduces the wrong ways of rewarding that encourage others to sacrifice the user’s interests. To prevent this problem developers need to rethink the incentives and accordingly redesign the systems so that they are not promoting these wrong behaviors. “Unintended negative consequences of benevolent design, such as the outraged and polarised tone and quality of online discourse.” These are the systems that are created thoughtfully and with good intent but still result in negative outcomes. Actually, the problem is that it is really difficult to tell what are all the outcomes of the system you are building. Berners-Lee in an interview with The Guardian said, “Given there are more web pages than there are neurons in your brain, it’s a complicated thing. You build Reddit, and people on it behave in a particular way. For a while, they all behave in a very positive, constructive way. And then you find a subreddit in which they behave in a nasty way.” This problem could be eliminated by researching and understanding of existing systems. Based on this research, we can then model possible new systems or enhance those we already have. Contract for the Web Berners Lee further explained that we can’t just really put the blame on the government or a social network for all the loopholes and dysfunctions that are affecting the Web. He said, “You can’t generalise. You can’t say, you know, social networks tend to be bad, tend to be nasty.” We need to find the root causes and to do exactly that we all need to come together as a global web community. “As the web reshapes, we are responsible for making sure that it is seen as a human right and is built for the public good”, he wrote in the open letter. To address these problems, Berners-Lee has a radical solution. Back in November last year at the Web Summit, he, with The Web Foundation, introduced Contract for the Web. The contract aims to bring together governments, companies, and citizens who believe that there is a need for setting clear norms, laws, and standards that underpin the web. “Governments, companies, and citizens are all contributing, and we aim to have a result later this year,” he shared. In theory, the contract defines people’s online rights and lists the key principles and duties government, companies, and citizens should follow. In Berners-Lee’s mind, it will restore some degree of equilibrium and transparency to the digital realm. The contract is part of a broader project that Berners-Lee believes is essential if we are to ‘save’ the web from its current problems. First, we need to create an open web for the users who are already connected to the web and give them the power of fixing issues that we have with the existing web. Secondly, we need to bring the other half of the world, which is not yet connected to the web. Many people are agreeing on the points Berners-Lee discussed in the open letter. Here is what some of the Twitter users are saying: https://twitter.com/girlygeekdom/status/1105375206829256704 https://twitter.com/solutionpoint/status/1105366111678279681 Contract for the Web, as Berners-Lee says, is about “going back to the values”. His idea of bringing together governments, companies, and citizens to make the Web safer and accessible to everyone looks pretty solid. Read the full open letter by Tim Berners-Lee on the Web Foundation’s website. Web Summit 2018: day 2 highlights Tim Berners-Lee is on a mission to save the web he invented UN on Web Summit 2018: How we can create a safe and beneficial digital future for all  

In order to demonstrate Django's rapid development potential, we will begin by constructing a simple, but fully-featured, e-commerce store. The goal is to be up and running with a product catalog and products for sale, including a simple payment processing interface, in about half-an-hour. If this seems ambitious, remember that Django offers a lot of built-in shortcuts for the most common web-related development tasks. We will be taking full advantage of these and there will be side discussions of their general use. In addition to building our starter storefront, this article aims to demonstrate some other Django tools and techniques. In this article by Jesse Legg, author of Django 1.2 e-commerce, we will: Create our Django Product model to take advantage of the automatic admin tool Build a flexible but easy to use categorization system, to better organize our catalog of products Utilize Django's generic view framework to expose a quick set of views on our catalog data Finally, create a simple template for selling products through the Google Checkout API (Read more interesting articles on Django 1.2 e-commerce here.) Before we begin, let's take a moment to check our project setup. Our project layout includes two directories: one for files specific to our personal project (settings, URLs, and so on), and the other for our collection of e-commerce Python modules (coleman). This latter location is where the bulk of the code will live. If you have downloaded the source code from the Packt website, the contents of the archive download represents everything in this second location. Designing a product catalog The starting point of our e-commerce application is the product catalog. In the real world, businesses may produce multiple catalogs for mutually exclusive or overlapping subsets of their products. Some examples are: fall and spring catalogs, catalogs based on a genre or sub-category of product such as catalogs for differing kinds of music (for example, rock versus classical), and many other possibilities. In some cases a single catalog may suffice, but allowing for multiple catalogs is a simple enhancement that will add flexibility and robustness to our application. As an example, we will imagine a fictitious food and beverage company, CranStore.com, that specializes in cranberry products: cranberry drinks, food, and desserts. In addition, to promote tourism at their cranberry bog, they sell numerous gift items, including t-shirts, hats, mouse pads, and the like. We will consider this business to illustrate examples as they relate to the online store we are building. We will begin by defining a catalog model called Catalog. The basic model structure will look like this: class Catalog(models.Model): name = models.CharField(max_length=255 slug = models.SlugField(max_length=150) publisher = models.CharField(max_length=300) description = models.TextField() pub_date = models.DateTimeField(default=datetime.now) This is potentially the simplest model we will create. It contains only five, very simple fields. But it is a good starting point for a short discussion about Django model design. Notice that we have not included any relationships to other models here. For example, there is no products ManyToManyField. New Django developers tend to overlook simple design decisions such as the one shown previously, but the ramifications are quite important. The first reason for this design is a purely practical one. Using Django's built-in admin tool can be a pleasure or a burden, depending on the design of your models. If we were to include a products field in the Catalog design, it would be a ManyToManyField represented in the admin as an enormous multiple-select HTML widget. This is practically useless in cases where there could be thousands of possible selections. If, instead, we attach a ForeignKey to Catalog on a Product model (which we will build shortly), we instantly increase the usability of Django's automatic admin tool. Instead of a select-box where we must shift-click to choose multiple products, we have a much simpler HTML drop-down interface with significantly fewer choices. This should ultimately increase the usability of the admin for our users. For example, CranStore.com sells lots of t-shirts during the fall when cranberries are ready to harvest and tourism spikes. They may wish to run a special catalog of touristy products on their website during this time. For the rest of the year, they sell a smaller selection of items online. The developers at CranStore create two catalogs: one is named Fall Collection and the other is called Standard Collection. When creating product information, the marketing team can decide which catalog an individual product belongs to by simply selecting them from the product editing page. This is more intuitive than selecting individual products out of a giant list of all products from the catalog admin page. Secondly, designing the Catalog model this way prevents potential "bloat" from creeping into our models. Imagine that CranStore decides to start printing paper versions of their catalogs and mailing them to a subscriber list. This would be a second potential ManyToManyField on our Catalog model, a field called subscribers. As you can see, this pattern could repeat with each new feature CranStore decides to implement. By keeping models as simple as possible, we prevent all kinds of needless complexity. In addition we also adhere to a basic Django design principle, that of "loose coupling". At the database level, the tables Django generates will be very similar regardless of where our ManyToManyField lives. Usually the only difference will be in the table name. Thus it generally makes more sense to focus on the practical aspects of Django model design. Django's excellent reverse relationship feature also allows a great deal of flexibility when it comes time to using the ORM to access our data. Model design is difficult and planning up-front can pay great dividends later. Ideally, we want to take advantage of the automatic, built-in features that make Django so great. The admin tool is a huge part of this. Anyone who has had to build a CRUD interface by hand so that non-developers can manage content should recognize the power of this feature. In many ways it is Django's "killer app". Creating the product model Finally, let's implement our product model. We will start with a very basic set of fields that represent common and shared properties amongst all the products we're likely to sell. Things like a picture of the item, its name, a short description, and pricing information. class Product(models.Model): name = models.CharField(max_length=300) slug = models.SlugField(max_length=150) description = models.TextField() photo = models.ImageField(upload_to='product_photo', blank=True) manufacturer = models.CharField(max_length=300, blank=True) price_in_dollars = models.DecimalField(max_digits=6, decimal_places=2) Most e-commerce applications will need to capture many additional details about their products. We will add the ability to create arbitrary sets of attributes and add them as details to our products later in this article. For now, let's assume that these six fields are sufficient. A few notes about this model: first, we have used a DecimalField to represent the product's price. Django makes it relatively simple to implement a custom field and such a field may be appropriate here. But for now we'll keep it simple and use a plain and built-in DecimalField to represent currency values. Notice, too, the way we're storing the manufacturer information as a plain CharField. Depending on your application, it may be beneficial to build a Manufacturer model and convert this field to a ForeignKey. Lastly, you may have realized by now that there is no connection to a Catalog model, either by a ForeignKey or ManyToManyField. Earlier we discussed the placement of this field in terms of whether it belonged to the Catalog or in the Product model and decided, for several reasons, that the Product was the better place. We will be adding a ForeignKey to our Product model, but not directly to the Catalog. In order to support categorization of products within a catalog, we will be creating a new model in the next section and using that as the connection point for our products.

In this article by Helder Vasconcelos, the author of Asynchronous Android Programming book - Second Edition, will present the most common asynchronous techniques techniques used on Android to develop an application, that using the multicore CPUs available on the most recent devices, is able to deliver up to date results quickly, respond to user interactions immediately and produce smooth transitions between the different UI states without draining the device battery. (For more resources related to this topic, see here.) Several reports have shown that an efficient application that provides a great user experience have better review ratings, higher user engagement and are able to achieve higher revenues. Why do we need Asynchronous Programming on Android? The Android system, by default, executes the UI rendering pipeline, base components (Activity, Fragment, Service, BroadcastReceiver, ...) lifecycle callback handling and UI interaction processing on a single thread, sometimes known as UI thread or main thread. The main thread handles his work sequentially collecting its work from a queue of tasks (Message Queue) that are queued to be processed by a particular application component. When any unit of work, such as a I/O operation, takes a significant period of time to complete, it will block and prevent the main thread from handling the next tasks waiting on the main thread queue to processed. Besides that, most Android devices refresh the screen 60 times per second, so every 16 milliseconds (1s/60 frames) a UI rendering task has to be processed by the main thread in order to draw and update the device screen. The next figure shows up a typical main thread timeline that runs the UI rendering task every 16ms. When a long lasting operation, prevents the main thread from executing frame rendering in time, the current frame drawing is deferred or some frame drawings are missed generating a UI glitch noticeable to the application user. A typical long lasting operation could be: Network data communication HTTP REST Request SOAP Service Access File Upload or Backup Reading or writing of files to the filesystem Shared Preferences Files File Cache Access Internal Database reading or writing Camera, Image, Video, Binary file processing. A user Interface glitch produced by dropped frames on Android is known on Android as jank. The Android SDK command systrace (https://developer.android.com/studio/profile/systrace.html) comes with the ability to measure the performance of UI rendering and then diagnose and identify problems that may arrive from various threads that are running on the application process. In the next image we illustrate a typical main thread timeline when a blocking operation dominates the main thread for more than 2 frame rendering windows: As you can perceive, 2 frames are dropped and the 1 UI Rendering frame was deferred because our blocking operation took approximately 35ms, to finish: When the long running operation that runs on the main thread does not complete within 5 seconds, the Android System displays an “Application not Responding” (ANR) dialog to the user giving him the option to close the application. Hence, in order to execute compute-intensive or blocking I/O operations without dropping a UI frame, generate UI glitches or degrade the application responsiveness, we have to offload the task execution to a background thread, with less priority, that runs concurrently and asynchronously in an independent line of execution, like shown on the following picture. Although, the use of asynchronous and multithreaded techniques always introduces complexity to the application, Android SDK and some well known open source libraries provide high level asynchronous constructs that allow us to perform reliable asynchronous work that relieve the main thread from the hard work. Each asynchronous construct has advantages and disadvantages and by choosing the right construct for your requirements can make your code more reliable, simpler, easier to maintain and less error prone. Let’s enumerate the most common techniques that are covered in detail in the “Asynchronous Android Programming” book. AsyncTask AsyncTask is simple construct available on the Android platform since Android Cupcake (API Level 3) and is the most widely used asynchronous construct. The AsyncTask was designed to run short-background operations that once finished update the UI. The AsyncTask construct performs the time consuming operation, defined on the doInBackground function, on a global static thread pool of background threads. Once doInBackground terminates with success, the AsyncTask construct switches back the processing to the main thread (onPostExecute) delivering the operation result for further processing. This technique if it is not used properly can lead to memory leaks or inconsistent results. HandlerThread The HandlerThread is a Threat that incorporates a message queue and an Android Looper that runs continuously waiting for incoming operations to execute. To submit new work to the Thread we have to instantiate a Handler that is attached to HandlerThread Looper. public class DownloadImageTask extends AsyncTask<URL, Integer, Bitmap> { protected Long doInBackground(URL... urls) {} protected void onProgressUpdate(Integer... progress) {} protected void onPostExecute(Bitmap image) {} } HandlerThread handlerThread = new HandlerThread("MyHandlerThread"); handlerThread.start(); Looper looper = handlerThread.getLooper(); Handler handler = new Handler(looper); handler.post(new Runnable(){ @Override public void run() {} }); The Handler interface allow us to submit a Message or a Runnable subclass object that could aggregate data and a chunk of work to be executed. Loader The Loader construct allow us to run asynchronous operations that load content from a content provider or a data source, such as an Internal Database or a HTTP service. The API can load data asynchronously, detect data changes, cache data and is aware of the Fragment and Activity lifecycle. The Loader API was introduced to the Android platform at API level 11, but are available for backwards compatibility through the Android Support libraries. public static class TextLoader extends AsyncTaskLoader<String> { @Override public String loadInBackground() { // Background work } @Override public void deliverResult(String text) {} @Override protected void onStartLoading() {} @Override protected void onStopLoading() {} @Override public void onCanceled(String text) {} @Override protected void onReset() {} } IntentService The IntentService class is a specialized subclass of Service that implements a background work queue using a single HandlerThread. When work is submitted to an IntentService, it is queued for processing by a HandlerThread, and processed in order of submission. public class BackupService extends IntentService { @Override protected void onHandleIntent(Intent workIntent) { // Background Work } } JobScheduler The JobScheduler API allow us to execute jobs in background when several prerequisites are fulfilled and taking into the account the energy and network context of the device. This technique allows us to defer and batch job executions until the device is charging or an unmetered network is available. JobScheduler scheduler = (JobScheduler) getSystemService( Context.JOB_SCHEDULER_SERVICE ); JobInfo.Builder builder = new JobInfo.Builder(JOB_ID, serviceComponent); builder.setRequiredNetworkType(JobInfo.NETWORK_TYPE_UNMETERED); builder.setRequiresCharging(true); scheduler.schedule(builder.build()); RxJava RxJava is an implementation of the Reactive Extensions (ReactiveX) on the JVM, that was developed by Netflix, used to compose asynchronous event processing that react to an observable source of events. The framework extends the Observer pattern by allowing us to create a stream of events, that could be intercepted by operators (input/output) that modify the original stream of events and deliver the result or an error to a final Observer. The RxJava Schedulers allow us to control in which thread our Observable will begin operating on and in which thread the event is delivered to the final Observer or Subscriber. Subscription subscription = getPostsFromNetwork() .map(new Func1<Post, Post>() { @Override public Post call(Post Post) { ... return result; } }) .subscribeOn(Schedulers.io()) .observeOn(AndroidSchedulers.mainThread()) .subscribe(new Observer<Post>() { @Override public void onCompleted() {} @Override public void onError() {} @Override public void onNext(Post post) { // Process the result } }); Summary As you have seen, there are several high level asynchronous constructs available to offload the long computing or blocking tasks from the UI thread and it is up to developer to choose right construct for each situation because there is not an ideal choice that could be applied everywhere. Asynchronous multithreaded programming, that produces reliable results, is difficult and error prone task so using a high level technique tends to simplify the application source code and the multithreading processing logic required to scale the application over the available CPU cores. Remember that keeping your application responsive and smooth is essential to delight your users and increase your chances to create a notorious mobile application. The techniques and asynchronous constructs summarized on the previous paragraphs are covered in detail in the Asynchronous Android Programming book published by Packt Publishing. Resources for Article: Further resources on this subject: Getting started with Android Development [article] Practical How-To Recipes for Android [article] Speeding up Gradle builds for Android [article]

The Google Walkout for Real Change group tweeted out their protest against the news of ‘Google confirming that it paid $135 million as exit packages to the two top execs accused of sexual assault, on Twitter, earlier this week. The group castigated the ‘multi-million dollar payouts’ and asked people to use the hashtag #GooglePayoutsForAll to demonstrate different and better ways this obscenely large amount of ‘hush money’ could have been used. https://twitter.com/GoogleWalkout/status/1105556617662214145 The news of Google paying its senior execs, namely, Amit Singhal (former Senior VP of Google search) and Andy Rubin (creator of Android) high exit packages was first highlighted in a report by the New York Times, last October. As per the report, Google paid $90 million to Rubin and $15 million to Singhal. A lawsuit filed by James Martin, an Alphabet shareholder, on Monday this week, further confirmed this news. The lawsuit states that this decision taken by directors of Alphabet caused significant financial harm to the company apart from deteriorating its reputation, goodwill, and market capitalization. Meredith Whittaker, one of the early organizers of the Google Walkout in November last month tweeted, “$135 million could fix Flint's water crisis and still have $80 million left.” Vicki Tardif, another Googler summed up the sentiments in her tweet, “$135M is 1.35 times what Google.org  gave out in grants in 2016.” An ACLU researcher pointed out that $135M could have in addition to feeding the hungry, housing the homeless and pay off some student loans, It could also support local journalism killed by online ads. The public support to the call for protest using the hashtag #GooglePayoutsForAll has been awe-inspiring. Some shared their stories of injustice in cases of sexual assault, some condemned Google for its handling of sexual misconduct, while others put the amount of money Google wasted on these execs into a larger perspective. Better ways Google could have used $135 million it wasted on execs payouts, according to Twitter Invest in people to reduce structural inequities in the company $135M could have been paid to the actual victims who faced harassment and sexual assault. https://twitter.com/xzzzxxzx/status/1105681517584572416 Google could have used the money to fix the wage and level gap for women of color within the company. https://twitter.com/sparker2/status/1105511306465992705 $135 million could be used to adjust the 16% median pay gap of the 1240 women working in Google’s UK offices https://twitter.com/crschmidt/status/1105645484104998913 $135M could have been used by Google for TVC benefits. It could also be used to provide rigorous training to the Google employees on what impact misinformation within the company can have on women and other marginalized groups.   https://twitter.com/EricaAmerica/status/1105546835526107136 For $135M, Google could have paid the 114 creators featured in its annual "YouTube Rewind" who are otherwise unpaid for their time and participation. https://twitter.com/crschmidt/status/1105641872033230848 Improve communities by supporting social causes Google could have paid $135M to RAINN, a largest American nonprofit anti-sexual assault organization, covering its expenses for the next 18 years. https://twitter.com/GoogleWalkout/status/1105450565193121792 For funding 1800 school psychologists for 1 year in public schools https://twitter.com/markfickett/status/1105640930936324097 To build real, affordable housing solutions in collaboration with London Breed, SFGOV, and other Bay Area officials https://twitter.com/jillianpuente/status/1105922474930245636 $135M could provide insulin for nearly 10,000 people with Type 1 diabetes in the US https://twitter.com/GoogleWalkout/status/1105585078590210051 To pay for the first year for 1,000 people with stage IV breast cancer https://twitter.com/GoogleWalkout/status/1105845951938347008 Be a responsible corporate citizen To fund approximately 5300 low-cost electric vehicles for Google staff, and saving around 25300 metric tons of carbon dioxide from vehicle emissions per year. https://twitter.com/crschmidt/status/1105698893361233926 Providing free Google Fiber internet to 225,000 homes for a year https://twitter.com/markfickett/status/1105641215389773825 To give $5/hr raise to 12,980 service workers at Silicon Valley tech campuses https://twitter.com/LAuerhahn/status/1105487572069801985 $135M could have been used for the construction of affordable homes, protecting 1,100 low-income families in San Jose from coming rent hikes of Google’s planned mega-campus. https://twitter.com/JRBinSV/status/1105478979543154688 #GooglePayoutsForAll: Another initiative to promote awareness of structural inequities in tech   The core idea behind launching #GooglePayoutsForAll on Twitter by the Google walkout group was to promote awareness among people regarding the real issues within the company. It urged people to discuss how Google is failing at maintaining the ‘open culture’ that it promises to the outside world. It also highlights how mottos such as “Don’t be Evil” and “Do the right thing” that Google stood by only make for pretty wall decor and there’s still a long way to go to see those ideals in action. The group gained its name when more than 20,000 Google employees along with vendors, and contractors, temps, organized Google “walkout for real change” and walked out of their offices in November 2018. The walkout was a protest against the hushed and unfair handling of sexual misconduct within Google. Ever since then, Googlers have been consistently taking initiatives to bring more transparency, accountability, and fairness within the company. For instance, the team launched an industry-wide awareness campaign to fight against forced arbitration in January, where they shared information about arbitration on their Twitter and Instagram accounts throughout the day. The campaign was a success as Google finally ended its forced arbitration policy which goes into effect this month for all the employees (including contractors, temps, vendors) and for all kinds of discrimination. Also, House and Senate members in the US have proposed a bipartisan bill to prohibit companies from using forced arbitration clauses, last month.    Although many found the #GooglePayoutsForAll idea praiseworthy, some believe this initiative doesn’t put any real pressure on Google to bring about a real change within the company. https://twitter.com/Jeffanie16/status/1105541489722081290 https://twitter.com/Jeffanie16/status/1105546783063752709 https://twitter.com/Jeffanie16/status/1105547341862457344 Now, we don’t necessarily disagree with this opinion, however, the initiative can't be completely disregarded as it managed to make people who’d otherwise hesitate to open up talk extensively regarding the real issues within the company. As Liz Fong-Jones puts it, “Strikes and walkouts are more sustainable long-term than letting Google drive each organizer out one by one. But yes, people *are* taking action in addition to speaking up. And speaking up is a bold step in companies where workers haven't spoken up before”. The Google Walkout group have not yet announced what they intend to do next following this digital protest. However, the group has been organizing meetups such as the one earlier this month on March 6th where it invited the tech contract workers for discussion about building solidarity to make work better for everyone. We are only seeing the beginning of a powerful worker movement take shape in Silicon Valley. Recode Decode #GoogleWalkout interview shows why data and evidence don’t always lead to right decisions in even the world’s most data-driven company Liz Fong Jones, prominent ex-Googler shares her experience at Google and ‘grave concerns’ for the company Google’s pay equity analysis finds men, not women, are underpaid; critics call out design flaws in the analysis

In this article by Paul Asjes, the author of the book, Building Slack Bots, we'll have a look at webhooks in Slack. (For more resources related to this topic, see here.) Slack is a great way of communicating at your work environment—it's easy to use, intuitive, and highly extensible. Did you know that you can make Slack do even more for you and your team by developing your own bots? This article will teach you how to implement incoming and outgoing webhooks for Slack, supercharging your Slack team into even greater levels of productivity. The programming language we'll use here is JavaScript; however, webhooks can be programmed with any language capable of HTTP requests. Webhooks First let's talk basics: a webhook is a way of altering or augmenting a web application through HTTP methods. Webhooks allow us to post messages to and from Slack using regular HTTP requests with a JSON payloads. What makes a webhook a bot is its ability to post messages to Slack as if it were a bot user. These webhooks can be divided into incoming and outgoing webhooks, each with their own purposes and uses. Incoming webhooks An example of an incoming webhook is a service that relays information from an external source to a Slack channel without being explicitly requested, such as GitHub Slack integration: The GitHub integration posts messages about repositories we are interested in In the preceding screenshot, we see how a message was sent to Slack after a new branch was made on a repository this team was watching. This data wasn't explicitly requested by a team member but automatically sent to the channel as a result of the incoming webhook. Other popular examples include Jenkins integration, where infrastructure changes can be monitored in Slack (for example, if a server watched by Jenkins goes down, a warning message can be posted immediately to a relevant Slack channel). Let's start with setting up an incoming webhook that sends a simple "Hello world" message: First, navigate to the Custom Integration Slack team page, as shown in the following screenshot (https://my.slack.com/apps/build/custom-integration): The various flavors of custom integrations Select Incoming WebHooks from the list and then select which channel you'd like your webhook app to post messages to: Webhook apps will post to a channel of your choosing Once you've clicked on the Add Incoming WebHooks integration button, you will be presented with this options page, which allows you to customize your integration a little further: Names, descriptions, and icons can be set from this menu Set a customized icon for your integration (for this example, the wave emoji was used) and copy down the webhook URL, which has the following format:https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX This generated URL is unique to your team, meaning that any JSON payloads sent via this URL will only appear in your team's Slack channels. Now, let's throw together a quick test of our incoming webhook in Node. Start a new Node project (remember: you can use npm init to create your package.json file) and install the superagent AJAX library by running the following command in your terminal: npm install superagent –save Create a file named index.js and paste the following JavaScript code within it: const WEBHOOK_URL = [YOUR_WEBHOOK_URL]; const request = require('superagent'); request .post(WEBHOOK_URL) .send({ text: 'Hello! I am an incoming Webhook bot!' }) .end((err, res) => { console.log(res); }); Remember to replace [YOUR_WEBHOOK_URL] with your newly generated URL, and then run the program by executing the following command: nodemon index.js Two things should happen now: firstly, a long response should be logged in your terminal, and secondly, you should see a message like the following in the Slack client: The incoming webhook equivalent of "hello world" The res object we logged in our terminal is the response from the AJAX request. Taking the form of a large JavaScript object, it displays information about the HTTP POST request we made to our webhook URL. Looking at the message received in the Slack client, notice how the name and icon are the same ones we set in our integration setup on the team admin site. Remember that the default icon, name, and channel are used if none are provided, so let's see what happens when we change that. Replace your request AJAX call in index.js with the following: request .post(WEBHOOK_URL) .send({ username: "Incoming bot", channel: "#general", icon_emoji: ":+1:", text: 'Hello! I am different from the previous bot!' }) .end((err, res) => { console.log(res); }); Save the file, and nodemon will automatically restart the program. Switch over to the Slack client and you should see a message like the following pop up in your #general channel: New name, icon, and message In place of icon_emoji, you could also use icon_url to link to a specific image of your choosing. If you wish your message to be sent only to one user, you can supply a username as the value for the channel property: channel: "@paul" This will cause the message to be sent from within the Slackbot direct message. The message's icon and username will match either what you configured in the setup or set in the body of the POST request. Finally, let's look at sending links in our integration. Replace the text property with the following and save index.js: text: 'Hello! Here is a fun link: <http://www.github.com|Github is great!>' Slack will automatically parse any links it finds, whether it's in the http://www.example.com or www.example.com formats. By enclosing the URL in angled brackets and using the | character, we can specify what we would like the URL to be shown as: Formatted links are easier to read than long URLs For more information on message formatting, visit https://api.slack.com/docs/formatting. Note that as this is a custom webhook integration, we can change the name, icon, and channel of the integration. If we were to package the integration as a Slack app (an app installable by other teams), then it is not possible to override the default channel, username, and icon set. Incoming webhooks are triggered by external sources; an example would be when a new user signs up to your service or a product is sold. The goal of the incoming webhook is to provide information to your team that is easy to reach and comprehend. The opposite of this would be if you wanted users to get data out of Slack, which can be done via the medium of outgoing webhooks. Outgoing webhooks Outgoing webhooks differ from the incoming variety in that they send data out of Slack and to a service of your choosing, which in turn can respond with a message to the Slack channel. To set up an outgoing webhook, visit the custom integration page of your Slack team's admin page again—https://my.slack.com/apps/build/custom-integration—and this time, select the Outgoing WebHooks option. On the next screen, be sure to select a channel, name, and icon. Notice how there is a target URL field to be filled in; we will fill this out shortly. When an outgoing webhook is triggered in Slack, an HTTP POST request is made to the URL (or URLs, as you can specify multiple ones) you provide. So first, we need to build a server that can accept our webhook. In index.js, paste the following code: 'use strict'; const http = require('http'); // create a simple server with node's built in http module http.createServer((req, res) => { res.writeHead(200, {'Content-Type': 'text/plain'}); // get the data embedded in the POST request req.on('data', (chunk) => { // chunk is a buffer, so first convert it to // a string and split it to make it more legible as an array console.log('Body:', chunk.toString().split('&')); }); // create a response let response = JSON.stringify({ text: 'Outgoing webhook received!' }); // send the response to Slack as a message res.end(response); }).listen(8080, '0.0.0.0'); console.log('Server running at http://0.0.0.0:8080/'); Notice how we require the http module despite not installing it with NPM. This is because the http module is a core Node dependency and is automatically included with your installation of Node. In this block of code, we start a simple server on port 8080 and listen for incoming requests. In this example, we set our server to run at 0.0.0.0 rather than localhost. This is important as Slack is sending a request to our server, so it needs to be accessible from the Internet. Setting the IP of your server to 0.0.0.0 tells Node to use your computer's network-assigned IP address. Therefore, if you set the IP of your server to 0.0.0.0, Slack can reach your server by hitting your IP on port 8080 (for example, http://123.456.78.90:8080). If you are having trouble with Slack reaching your server, it is most likely because you are behind a router or firewall. To circumvent this issue, you can use a service such as ngrok (https://ngrok.com/). Alternatively, look at port forwarding settings for your router or firewall. Let's update our outgoing webhook settings accordingly: The outgoing webhook settings, with a destination URL Save your settings and run your Node app; test whether the outgoing webhook works by typing a message into the channel you specified in the webhook's settings. You should then see something like this in Slack: We built a spam bot Well, the good news is that our server is receiving requests and returning a message to send to Slack each time. The issue here is that we skipped over the Trigger Word(s) field in the webhook settings page. Without a trigger word, any message sent to the specified channel will trigger the outgoing webhook. This causes our webhook to be triggered by a message sent by the outgoing webhook in the first place, creating an infinite loop. To fix this, we could do one of two things: Refrain from returning a message to the channel when listening to all the channel's messages. Specify one or more trigger words to ensure we don't spam the channel. Returning a message is optional yet encouraged to ensure a better user experience. Even a confirmation message such as Message received! is better than no message as it confirms to the user that their message was received and is being processed. Let's therefore presume we prefer the second option, and add a trigger word: Trigger words keep our webhooks organized Let's try that again, this time sending a message with the trigger word at the beginning of the message. Restart your Node app and send a new message: Our outgoing webhook app now functions a lot like our bots from earlier Great, now switch over to your terminal and see what that message logged: Body: [ 'token=KJcfN8xakBegb5RReelRKJng', 'team_id=T000001', 'team_domain=buildingbots', 'service_id=34210109492', 'channel_id=C0J4E5SG6', 'channel_name=bot-test', 'timestamp=1460684994.000598', 'user_id=U0HKKH1TR', 'user_name=paul', 'text=webhook+hi+bot%21', 'trigger_word=webhook' ] This array contains the body of the HTTP POST request sent by Slack; in it, we have some useful data, such as the user's name, the message sent, and the team ID. We can use this data to customize the response or to perform some validation to make sure the user is authorized to use this webhook. In our response, we simply sent back a Message received string; however, like with incoming webhooks, we can set our own username and icon. The channel cannot be different from the channel specified in the webhook's settings, however. The same restrictions apply when the webhook is not a custom integration. This means that if the webhook was installed as a Slack app for another team, it can only post messages as the username and icon specified in the setup screen. An important thing to note is that webhooks, either incoming or outgoing, can only be set up in public channels. This is predominantly to discourage abuse and uphold privacy, as we've seen that it's simple to set up a webhook that can record all the activity on a channel. Summary In this article, you learned what webhooks are and how you can use them to get data in and out of Slack. You learned how to send messages as a bot user and how to interact with your users in the native Slack client. Resources for Article: Further resources on this subject: Keystone – OpenStack Identity Service[article] A Sample LEMP Stack[article] Implementing Stacks using JavaScript[article]

(For more resources related to this topic, see here.) Smartphones, by now, have entered our lives not only as users and consumers but also as producers of our own content. Though this kind of device has been on the market since 1992 (the first was the Simon model by IBM), the big diffusion was driven by Apple's iPhone, when it was produced in 2007 (last year, the fifth generation of this device was released). Meanwhile, another big giant, Google, developed an open source product to be used as the internal operating system in mobile devices; in a different manner from the leader of the market, this company doesn't constraint itself to a unique hardware-specific device, but allows third-party companies to use it on their cell phones, which have different characteristics. The big advantage was also to be able to sell this device to consumers that don't want to (or can't have) spend as much money as the Apple phone costs. This allowed Android to win the battle of diffusion. But there is another side to the coin. A variety of devices by different producers means more fragmentation of the underlying system and a non-uniform user experience that can be really disappointing. As programmers, we have to take into account these problems and this article strives to be a useful guideline to solve that problem. The Android platform was born in 2003, as the product of a company which at first was known as Android Inc. and which was acquired by Google in 2005. Its direct competitors were and are still today the iOS platform by Apple and the RIM, know as Blackberry. Technically speaking, its core is an operating system using a Linux Kernel, aimed to be installed on devices with very different hardware (mainly mobile devices, but today it is also used in general embedded systems, for example, the game console OUYA that features a modified version of Android 4.0). Like any software that has been around for a while, many changes happened to the functionality and many versions came out, each with a name of a dessert: Apple Pie (API level 1) Banana Bread (API level 2) 1.5 Cupcake (API level 3) 1.6 – Donut (API level 4) 2.0-2.1x – Eclair (API level 5 to 7) 2.2 – Froyo (API level 8) 2.3 – Gingerbread (API level 9 and 10) 3.0-3.2 – Honeycomb (API level 11 to 13) 4.0 – Ice Cream Sandwich (API level 14 and 15) 4.1 – Jelly Bean (API level 16) Like in many other software projects, the names are in alphabetical order (another project that follows this approach is the Ubuntu distribution). The API level written in the parenthesis is the main point about the fragmentation. Each version of software introduces or removes features and bugs. In its lifetime, an operating system such as Android aims to add more fantastic innovations while avoiding breaking pre-installed applications in older versions, but also aims to make available to these older versions the same features with a process technically called backporting. For more information about the API levels, carefully read the official documentation available at http://developer.android.com/guide/topics/manifest/uses-sdk- element.html#ApiLevels. If you look at the diffusion of these versions as given by the following pie chart you can see there are more than 50 percent of devices have installed versions that we consider outdated with the latest; all that you will read in the article is thought to address these problems, mainly using backporting; in particular, to specifically address the backward compatibility issues with version 3.0 of the Android operating system—the version named Honeycomb. Version 3.0 was first intended to be installed on tablets, and in general, on devices with large screens. Android is a platform that from the beginning was intended to be used on devices with very different characteristics (think of a system where an application must be usable on VGA screens, with or without physical keyboards, with a camera, and so on); with the release of 3.0, all this was improved with specific APIs thought to extend and make developing applications easier, and also to create new patterns with the graphical user interfaces. The more important innovation was the introduction of the Fragment class. Earlier, the only main class in developing the Android applications was Activity, a class that provides the user with a screen in order to accomplish a specific task, but that was too coarse grain and not re-usable enough to be used in the applications with large screens such as a tablet. With the introduction of the Fragment class to be used as the basic block, it is now possible to create responsive mobile design; that is, producing content adapting to the context and optimizing the block's placement, using reflowing or a combination of each Fragment inside the main Activity. These are concepts inspired by the so called responsive web design, where developers build web pages that adapt to the viewport's size; the preeminent article about this argument is Responsive Web Design, Ethan Marcotte. For sake of completeness, let me list other new capabilities introduced with Honeycomb (look into the official documentation for a better understanding of them): Copy and Paste: A clipboard-based framework Loaders: Load data asynchronously Drag and Drop: Permits the moving of data between views Property animation framework: Supersedes the old Animation package, allowing the animation of almost everything into an application Hardware acceleration: From API level 11, the graphic pipeline uses dedicated hardware when it is present Support for encrypted storage In particular for address these changes and new features, Google make available a particular library called "Support library" that backports Fragment and Loader. Although the main characteristics of this classes are maintained, in the article is explained in detail how to use the low level API related with the threading stuff. Indeed an Android application is not a unique block of instructions executed one after the other, but is composed of multiple pipelines of execution. The main concepts here are the process and thread. When an application is started, the operating system creates a process (technically a Linux process) and each component is associated to this process. Together with the process, a thread of execution named main is also created. This is a very important thread because it is in charge of dispatching events to the appropriate user interface elements and receiving events from them. This thread is also called UI Thread. It's important to note that the system does not create a separate thread for each element, but instead uses the same UI thread for all of them. This can be dangerous for the responsiveness of your application, since if you perform an intensive or time expensive operation, this will block the entire UI. All Android developers fight against the ANR (Application Not Responding) message that is presented when the UI is not responsive for more than 5 seconds. Following Android's documentation, there are only two rules to follow to avoid the ANR: Do not block the UI thread Do not access the Android UI toolkit from outside the UI thread These two rules can seem simple, but there are some particulars that have to be clear. In the article some examples are shown using not only the Thread class (and the Runnable interface) but also the (very) low-level classes named Looper and Handler. Also the interaction between GUI elements and these classes are investigated to avoid nasty exceptions. Another important element introduced in Google's platform is the UI pattern named ActionBar—a piece of interface at the top of an application where the more important menu's buttons are visualized in order to be easily accessible. Also a new contextual menu is available in the action bar. When, for example, one or more items in a list are selected (such as, the Gmail application), the appearance of the bar changes and shows new buttons related to the actions available for the selected items. One thing not addressed by the compatibility package is ActionBar. Since this is a very important element for integration with the Android ecosystem, some alternatives are born, the first one from Google itself, as a simple code sample named ActionBar Compatibility that you can find in the sample/directory of the Android SDK. In the article, we will follow a different approach, using a famous open source project, ActionBarSherlock. The code for this library is not available from SDK, so we need to download it from its website (http://actionbarsherlock.com/). This library allows us to use the most part of the functionality of the original ActionBar implementation such as UP button (that permits a hierarchical navigation), ActionView and contextual action menu. Summary Thus in this article we learned about Android Fragmentation Management. Resources for Article : Further resources on this subject: Android Native Application API [Article] New Connectivity APIs – Android Beam [Article] So, what is Spring for Android? [Article]

In this article from the book Game Programming using Qt by authors Witold Wysota and Lorenz Haas, you will be taught how to communicate with the Internet servers and with sockets in general. First, we will have a look at QNetworkAccessManager, which makes sending network requests and receiving replies really easy. Building on this basic knowledge, we will then use Google's Distance API to get information about the distance between two locations and the time it would take to get from one location to the other. (For more resources related to this topic, see here.) QNetworkAccessManager The easiest way to access files on the Internet is to use Qt's Network Access API. This API is centered on QNetworkAccessManager, which handles the complete communication between your game and the Internet. When we develop and test a network-enabled application, it is recommended that you use a private, local network if feasible. This way, it is possible to debug both ends of the connection and the errors will not expose sensitive data. If you are not familiar with setting up a web server locally on your machine, there are luckily a number of all-in-one installers that are freely available. These will automatically configure Apache2, MySQL, PHP, and much more on your system. On Windows, for example, you could use XAMPP (http://www.apachefriends.org/en) or the Uniform Server (http://www.uniformserver.com); on Apple computers there is MAMP (http://www.mamp.info/en); and on Linux, you normally don't have to do anything since there is already localhost. If not, open your preferred package manager, search for a package called apache2 or similar, and install it. Alternatively, have a look at your distribution's documentation. Before you go and install Apache on your machine, think about using a virtual machine like VirtualBox (http://www.virtualbox.org) for this task. This way, you keep your machine clean and you can easily try different settings of your test server. With multiple virtual machines, you can even test the interaction between different instances of your game. If you are on UNIX, Docker (http://www.docker.com) might be worth to have a look at too. Downloading files over HTTP For downloading files over HTTP, first set up a local server and create a file called version.txt in the root directory of the installed server. The file should contain a small text like "I am a file on localhost" or something similar. To test whether the server and the file are correctly set up, start a web browser and open http://localhost/version.txt. You then should see the file's content. Of course, if you have access to a domain, you can also use that. Just alter the URL used in the example correspondingly. If you fail, it may be the case that your server does not allow to display text files. Instead of getting lost in the server's configuration, just rename the file to version .html. This should do the trick! Result of requesting http://localhost/version.txt on a browser As you might have guessed, because of the filename, the real-life scenario could be to check whether there is an updated version of your game or application on the server. To get the content of a file, only five lines of code are needed. Time for action – downloading a file First, create an instance of QNetworkAccessManager: QNetworkAccessManager *m_nam = new QNetworkAccessManager(this); Since QNetworkAccessManager inherits QObject, it takes a pointer to QObject, which is used as a parent. Thus, you do not have to take care of deleting the manager later on. Furthermore, one single instance of QNetworkAccessManager is enough for an entire application. So, either pass a pointer to the network access manager in your game around or, for ease of use, create a singleton pattern and access the manager through that. A singleton pattern ensures that a class is instantiated exactly once. The pattern is useful for accessing application-wide configurations or—in our case—an instance of QNetworkAccessManager. On the wiki pages for qtcentre.org and qt-project.org, you will find examples for different singleton patterns. A simple template-based approach would look like this (as a header file): template <class T> class Singleton { public: static T& Instance() { static T _instance; return _instance; } private: Singleton(); ~Singleton(); Singleton(const Singleton &); Singleton& operator=(const Singleton &); }; In the source code, you would include this header file and acquire a singleton of a class called MyClass with: MyClass *singleton = &Singleton<MyClass>::Instance(); If you are using Qt Quick, you can directly use the view instance of QNetworkAccessManager: QQuickView *view = new QQuickView; QNetworkAccessManager *m_nam = view->engine()->networkAccessManager(); Secondly, we connect the manager's finished() signal to a slot of our choice. For example, in our class, we have a slot called downloadFinished(): connect(m_nam, SIGNAL(finished(QNetworkReply*)), this, SLOT(downloadFinished(QNetworkReply*))); Then, it actually request's the version.txt file from localhost: m_nam->get(QNetworkRequest(QUrl("http://localhost/version.txt"))); With get(), a request to get the contents of the file, specified by the URL, is posted. The function expects QNetworkRequest, which defines all the information needed to send a request over the network. The main information of such a request is naturally the URL of the file. This is the reason why QNetworkRequest takes a QUrl as an argument in its constructor. You can also set the URL with setUrl() to a request. If you like to define some additional headers, you can either use setHeader() for the most common header or use setRawHeader() to be fully flexible. If you want to set, for example, a custom user agent to the request, the call would look like: QNetworkRequest request; request.setUrl(QUrl("http://localhost/version.txt")); request.setHeader(QNetworkRequest::UserAgentHeader, "MyGame"); m_nam->get(request); The setHeader() function takes two arguments, the first is a value of the enumeration QNetworkRequest::KnownHeaders, which holds the most common—self-explanatory—headers such as LastModifiedHeader or ContentTypeHeader, and the second is the actual value. You could also have written the header by using of setRawHeader(): request.setRawHeader("User-Agent", "MyGame"); When you use setRawHeader(), you have to write the header field names yourself. Beside that, it behaves like setHeader(). A list of all available headers for the HTTP protocol Version 1.1 can be found in section 14 at http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14. With the get() function we requested the version.txt file from localhost. All we have to do from now on is to wait for the server to reply. As soon as the server's reply is finished, the slot downloadFinished() will be called. That was defined by the previous connection statement. As an argument the reply of type QNetworkReply is transferred to the slot and we can read the reply's data and set it to m_edit, an instance of QPlainTextEdit, using the following code: void FileDownload::downloadFinished(QNetworkReply *reply) { const QByteArray content = reply->readAll(); m_edit->setPlainText(content); reply->deleteLater(); } Since QNetworkReply inherits QIODevice, there are also other possibilities to read the contents of the reply including QDataStream or QTextStream to either read and interpret binary data or textual data. Here, as fourth command, QIODevice::readAll() is used to get the complete content of the requested file in a QByteArray. The responsibility for the transferred pointer to the corresponding QNetworkReply lies with us, so we need to delete it at the end of the slot. This would be the fifth line of code needed to download a file with Qt. However, be careful and do not call delete on the reply directly. Always use deleteLater() as the documentation suggests! Have a go hero – extending the basic file downloader If you haven't set up a localhost, just alter the URL in the source code to download another file. Of course, having to alter the source code in order to download another file is far from an ideal approach. So try to extend the dialog, by adding a line edit where you can specify the URL you want to download. Also, you can offer a file dialog to choose the location to where the downloaded file should be saved. Error handling If you do not see the content of the file, something went wrong. Just as in real life, this can always happen so we better make sure, that there is good error handling in such cases to inform the user what is going on. Time for action – displaying a proper error message Fortunately QNetworkReply offers several possibilities to do this. In the slot called downloadFinished() we first want to check if an error occurred: if (reply->error() != QNetworkReply::NoError) {/* error occurred */} The function QNetworkReply::error() returns the error that occurred while handling the request. The error is encoded as a value of type QNetworkReply::NetworkError. The two most common errors are probably these: Error code Meaning ContentNotFoundError This error indicates that the URL of the request could not be found. It is similar to the HTTP error code 404. ContentAccessDenied This error indicates that you do not have the permission to access the requested file. It is similar to the HTTP error 401. You can look up the other 23 error codes in the documentation. But normally you do not need to know exactly what went wrong. You only need to know if everything worked out—QNetworkReply::NoError would be the return value in this case—or if something went wrong. Since QNetworkReply::NoError has the value 0, you can shorten the test phrase to check if an error occurred to: if (reply->error()) { // an error occurred } To provide the user with a meaningful error description you can use QIODevice::errorString(). The text is already set up with the corresponding error message and we only have to display it: if (reply->error()) { const QString error = reply->errorString(); m_edit->setPlainText(error); return; } In our example, assuming we had an error in the URL and wrote versions.txt by mistake, the application would look like this: If the request was a HTTP request and the status code is of interest, it could be retrieved by QNetworkReply::attribute(): reply->attribute(QNetworkRequest::HttpStatusCodeAttribute) Since it returns QVariant, you can either use QVariant::toInt() to get the code as an integer or QVariant::toString() to get the number as a QString. Beside the HTTP status code you can query through attribute() a lot of other information. Have a look at the description of the enumeration QNetworkRequest::Attribute in the documentation. There you also will find QNetworkRequest::HttpReasonPhraseAttribute which holds a human readable reason phrase of the HTTP status code. For example "Not Found" if an HTTP error 404 occurred. The value of this attribute is used to set the error text for QIODevice::errorString(). So you can either use the default error description provided by errorString() or compose your own by interpreting the reply's attributes. If a download failed and you want to resume it or if you only want to download a specific part of a file, you can use the range header: QNetworkRequest req(QUrl("...")); req.setRawHeader("Range", "bytes=300-500"); QNetworkReply *reply = m_nam->get(req); In this example only the bytes 300 to 500 would be downloaded. However, the server must support this. Downloading files over FTP As simple as it is to download files over HTTP, as simple it is to download a file over FTP. If it is an anonymous FTP server for which you do not need an authentication, just use the URL like we did earlier. Assuming there is again a file called version.txt on the FTP server on localhost, type: m_nam->get(QNetworkRequest(QUrl("ftp://localhost/version.txt"))); That is all, everything else stays the same. If the FTP server requires an authentication you'll get an error, for example: Setting the user name and the user password to access an FTP server is likewise easy. Either write it in the URL or use QUrl functions setUserName() and setPassword(). If the server does not use a standard port, you can set the port explicitly with QUrl::setPort(). To upload a file to a FTP server use QNetworkAccessManager::put() which takes as first argument a QNetworkRequest, calling a URL that defines the name of the new file on the server, and as second argument the actual data, that should be uploaded. For small uploads, you can pass the content as a QByteArray. For larger contents, better use a pointer to a QIODevice. Make sure the device is open and stays available until the upload is done. Downloading files in parallel A very important note on QNetworkAccessManager: it works asynchronously. This means you can post a network request without blocking the main event loop and this is what keeps the GUI responsive. If you post more than one request, they are put on the manager's queue. Depending on the protocol used they get processed in parallel. If you are sending HTTP requests, normally up to six requests will be handled at a time. This will not block the application. Therefore, there is really no need to encapsulate QNetworkAccessManager in a thread, unfortunately, this unnecessary approach is frequently recommended all over the Internet. QNetworkAccessManager already threads internally. Really, don't move QNetworkAccessManager to a thread—unless you know exactly what you are doing. If you send multiple requests, the slot connected to the manager's finished() signal is called in an arbitrary order depending on how quickly a request gets a reply from the server. This is why you need to know to which request a reply belongs. This is one reason why every QNetworkReply carries its related QNetworkRequest. It can be accessed through QNetworkReply::request(). Even if the determination of the replies and their purpose may work for a small application in a single slot, it will quickly get large and confusing if you send a lot of requests. This problem is aggravated by the fact that all replies are delivered to only one slot. Since most probably there are different types of replies that need different treatments, it would be better to bundle them to specific slots, specialized for a special task. Fortunately this can be achieved very easily. QNetworkAccessManager::get() returns a pointer to the QNetworkReply which will get all information about the request you post with get(). By using this pointer, you can then connect specific slots to the reply's signals. For example if you have several URLs and you want to save all linked images from these sites to the hard drive, then you would request all web pages via QNetworkAccessManager::get() and connect their replies to a slot specialized for parsing the received HTML. If links to images are found, this slot would request them again with get(). However, this time the replies to these requests would be connected to a second slot, which is designed for saving the images to the disk. Thus you can separate the two tasks, parsing HTML and saving data to a local drive. The most important signals of QNetworkReply are. The finished signal The finished() signal is equivalent with the QNetworkAccessManager::finished() signal we used earlier. It is triggered as soon as a reply has been returned—successfully or not. After this signal has been emitted, neither the reply's data nor its metadata will be altered anymore. With this signal you are now able to connect a reply to a specific slot. This way you can realize the scenario outlined previously. However, one problem remains: if you post simultaneous requests, you do not know which one has finished and thus called the connected slot. Unlike QNetworkAccessManager::finished(), QNetworkReply::finished() does not pass a pointer to QNetworkReply; this would actually be a pointer to itself in this case. A quick solution to solve this problem is to use sender(). It returns a pointer to the QObject instance that has called the slot. Since we know that it was a QNetworkReply, we can write: QNetworkReply *reply = qobject_cast<QNetworkReply*>(sender()); if (!reply) return; This was done by casting sender() to a pointer of type QNetworkReply. Whenever casting classes that inherit QObject, use qobject_cast. Unlike dynamic_cast it does not use RTTI and works across dynamic library boundaries. Although we can be pretty confident the cast will work, do not forget to check if the pointer is valid. If it is a null pointer, exit the slot. Time for action – writing OOP conform code by using QSignalMapper A more elegant way that does not rely on sender(), would be to use QSignalMapper and a local hash, in which all replies that are connected to that slot are stored. So whenever you call QNetworkAccessManager::get() store the returned pointer in a member variable of type QHash<int, QNetworkReply*> and set up the mapper. Let's assume that we have following member variables and that they are set up properly: QNetworkAccessManager *m_nam; QSignalMapper *m_mapper; QHash<int, QNetworkReply*> m_replies; Then you would connect the finished() signal of a reply this way: QNetworkReply *reply = m_nam->get(QNetworkRequest(QUrl(/*...*/))); connect(reply, SIGNAL(finished()), m_mapper, SLOT(map())); int id = /* unique id, not already used in m_replies*/; m_replies.insert(id, reply); m_mapper->setMapping(reply, id); What just happened? First we post the request and fetch the pointer to the QNetworkReply with reply. Then we connect the reply's finished signal to the mapper's slot map(). Next we have to find a unique ID which must not already be in use in the m_replies variable. One could use random numbers generated with qrand() and fetch numbers as long as they are not unique. To determine if a key is already in use, call QHash::contains(). It takes the key as an argument against which it should be checked. Or even simpler: count up another private member variable. Once we have a unique ID we insert the pointer to QNetworkReply in the hash using the ID as a key. Last, with setMapping(), we set up the mapper's mapping: the ID's value corresponds to the actual reply. At a prominent place, most likely the constructor of the class, we already have connected the mappers map() signal to a custom slot. For example: connect(m_mapper, SIGNAL(mapped(int)), this, SLOT(downloadFinished(int))); When the slot downloadFinished() is called, we can get the corresponding reply with: void SomeClass::downloadFinished(int id) { QNetworkReply *reply = m_replies.take(id); // do some stuff with reply here reply->deleteLater(); } QSignalMapper also allows to map with QString as an identifier instead of an integer as used above. So you could rewrite the example and use the URL to identify the corresponding QNetworkReply; at least as long as the URLs are unique. The error signal If you download files sequentially, you can swap the error handling out. Instead of dealing with errors in the slot connected to the finished() signal, you can use the reply's signal error() which passes the error of type QNetworkReply::NetworkError to the slot. After the error() signal has been emitted, the finished() signal will most likely also be emitted shortly. The readyRead signal Until now, we used the slot connected to the finished() signal to get the reply's content. That works perfectly if you deal with small files. However, this approach is unsuitable when dealing with large files since they would unnecessarily bind too many resources. For larger files it is better to read and save transferred data as soon as it is available. We get informed by QIODevice::readyRead() whenever new data is available to be read. So for large files you should type in the following: connect(reply, SIGNAL(readyRead()), this, SLOT(readContent())); file.open(QIODevice::WriteOnly); This will help you connect the reply's signal readyRead() to a slot, set up QFile and open it. In the connected slot, type in the following snippet: const QByteArray ba = reply->readAll(); file.write(ba); file.flush(); Now you can fetch the content, which was transferred so far, and save it to the (already opened) file. This way the needed resources are minimized. Don't forget to close the file after the finished() signal was emitted. In this context it would be helpful if one could know upfront the size of the file one wants to download. Therefore, we can use QNetworkAccessManager::head(). It behaves like the get() function, but does not transfer the content of the file. Only the headers are transferred. And if we are lucky, the server sends the "Content-Length" header, which holds the file size in bytes. To get that information we type: reply->head(QNetworkRequest::ContentLengthHeader).toInt(); With this information, we could also check upfront if there is enough space left on the disk. The downloadProgress method Especially when a big file is being downloaded, the user usually wants to know how much data has already been downloaded and how long it will approximately take for the download to finish. Time for action – showing the download progress In order to achieve this we can use the reply's downloadProgress() signal. As a first argument it passes the information on how many bytes have already been received and as a second argument how many there are in total. This gives us the possibility to indicate the progress of the download with QProgressBar. As the passed arguments are of type qint64 we can't use them directly with QProgressBar since it only accepts int. So in the connected slot we first calculate the percentage of the download progress: void SomeClass::downloadProgress(qint64 bytesReceived, qint64 bytesTotal) { qreal progress = (bytesTotal < 1) ? 1.0 : bytesReceived * 100.0 / bytesTotal; progressBar->setValue(progress * progressBar->maximum()); } What just happened? With the percentage we set the new value for the progress bar where progressBar is the pointer to this bar. However, what value will progressBar->maximum() have and where do we set the range for the progress bar? What is nice is that you do not have to set it for every new download. It is only done once, for example in the constructor of the class containing the bar. As range values I would recommend: progressBar->setRange(0, 2048); The reason is that if you take for example a range of 0 to 100 and the progress bar is 500 pixels wide, the bar would jump 5 pixels forward for every value change. This will look ugly. To get a smooth progression where the bar expands by 1 pixel at a time, a range of 0 to 99.999.999 would surely work but would be highly inefficient. This is because the current value of the bar would change a lot without any graphical depiction. So the best value for the range would be 0 to the actual bar's width in pixel. Unfortunately, the width of the bar can change depending on the actual widget width and frequently querying the actual size of the bar every time the value change is also not a good solution. Why 2048, then? The idea behind this value is the resolution of the screen. Full HD monitors normally have a width of 1920 pixels, thus taking 2^11, aka 2048, ensure that a progress bar runs smoothly, even if it is fully expanded. So 2048 isn't the perfect number but a fairly good compromise. If you are targeting smaller devices, choose a smaller, more appropriate number. To be able to calculate the remaining time for the download to finish you have to start a timer. In this case use QElapsedTimer. After posting the request with QNetworkAccessManager::get() start the timer by calling QElapsedTimer::start(). Assuming the timer is called m_timer, the calculation would be: qint64 total = m_timer.elapsed() / progress; qint64 remaining = (total – m_timer.elapsed()) / 1000; QElapsedTimer::elapsed() returns the milliseconds counting from the moment when the timer was started. This value divided by the progress equals the estimated total download time. If you subtract the elapsed time and divide the result by 1000, you'll get the remaining time in seconds. Using a proxy If you like to use a proxy you first have to set up a QNetworkProxy. You have to define the type of the proxy with setType(). As arguments you most likely want to pass QNetworkProxy::Socks5Proxy or QNetworkProxy::HttpProxy. Then set up the host name with setHostName(), the user name with setUserName() and the password with setPassword(). The last two properties are, of course, only needed if the proxy requires an authentication. Once the proxy is set up you can set it to the access manager via QNetworkAccessManager::setProxy(). Now, all new requests will use that proxy. Summary In this article you familiarized yourself with QNetworkAccessManager. This class is at the heart of your code whenever you want to download or upload files to the Internet. After having gone through the different signals that you can use to fetch errors, to get notified about new data or to show the progress, you should now know everything you need on that topic. Resources for Article: Further resources on this subject: GUI Components in Qt 5[article] Code interlude – signals and slots [article] Configuring Your Operating System [article]

In this article by Chetan Giridhar, author of the book, Learning Python Design Patterns - Second Edition, we will get introduced to the Façade design pattern and how it is used in software application development. We will work with a sample use case and implement it in Python v3.5. In brief, we will cover the following topics in this article: An understanding of the Façade design pattern with a UML diagram A real-world use case with the Python v3.5 code implementation The Façade pattern and principle of least knowledge (For more resources related to this topic, see here.) Understanding the Façade design pattern Façade is generally referred to as the face of the building, especially an attractive one. It can be also referred to as a behavior or appearance that gives a false idea of someone's true feelings or situation. When people walk past a façade, they can appreciate the exterior face but aren't aware of the complexities of the structure within. This is how a façade pattern is used. Façade hides the complexities of the internal system and provides an interface to the client that can access the system in a very simplified way. Consider an example of a storekeeper. Now, when you, as a customer, visit a store to buy certain items, you're not aware of the layout of the store. You typically approach the storekeeper who is well aware of the store system. Based on your requirements, the storekeeper picks up items and hands them over to you. Isn't this easy? The customer need not know how the store looks and s/he gets the stuff done through a simple interface, the storekeeper. The Façade design pattern essentially does the following: It provides a unified interface to a set of interfaces in a subsystem and defines a high-level interface that helps the client use the subsystem in an easy way. Façade discusses representing a complex subsystem with a single interface object. It doesn't encapsulate the subsystem but actually combines the underlying subsystems. It promotes the decoupling of the implementation with multiple clients. A UML class diagram We will now discuss the Façade pattern with the help of the following UML diagram: As we observe the UML diagram, you'll realize that there are three main participants in this pattern: Façade: The main responsibility of a façade is to wrap up a complex group of subsystems so that it can provide a pleasing look to the outside world. System: This represents a set of varied subsystems that make the whole system compound and difficult to view or work with. Client: The client interacts with the Façade so that it can easily communicate with the subsystem and get the work completed. It doesn't have to bother about the complex nature of the system. You will now learn a little more about the three main participants from the data structure's perspective. Façade The following points will give us a better idea of Façade: It is an interface that knows which subsystems are responsible for a request It delegates the client's requests to the appropriate subsystem objects using composition For example, if the client is looking for some work to be accomplished, it need not have to go to individual subsystems but can simply contact the interface (Façade) that gets the work done System In the Façade world, System is an entity that performs the following: It implements subsystem functionality and is represented by a class. Ideally, a System is represented by a group of classes that are responsible for different operations. It handles the work assigned by the Façade object but has no knowledge of the façade and keeps no reference to it. For instance, when the client requests the Façade for a certain service, Façade chooses the right subsystem that delivers the service based on the type of service Client Here's how we can describe the client: The client is a class that instantiates the Façade It makes requests to the Façade to get the work done from the subsystems Implementing the Façade pattern in the real world To demonstrate the applications of the Façade pattern, let's take an example that we'd have experienced in our lifetime. Consider that you have a marriage in your family and you are in charge of all the arrangements. Whoa! That's a tough job on your hands. You have to book a hotel or place for marriage, talk to a caterer for food arrangements, organize a florist for all the decorations, and finally handle the musical arrangements expected for the event. In yesteryears, you'd have done all this by yourself, such as talking to the relevant folks, coordinating with them, negotiating on the pricing, but now life is simpler. You go and talk to an event manager who handles this for you. S/he will make sure that they talk to the individual service providers and get the best deal for you. From the Façade pattern perspective we will have the following three main participants: Client: It's you who need all the marriage preparations to be completed in time before the wedding. They should be top class and guests should love the celebrations. Façade: The event manager who's responsible for talking to all the folks that need to work on specific arrangements such as food, flower decorations, among others Subsystems: They represent the systems that provide services such as catering, hotel management, and flower decorations Let's develop an application in Python v3.5 and implement this use case. We start with the client first. It's you! Remember, you're the one who has been given the responsibility to make sure that the marriage preparations are done and the event goes fine! However, you're being clever here and passing on the responsibility to the event manager, isn't it? Let's now look at the You class. In this example, you create an object of the EventManager class so that the manager can work with the relevant folks on marriage preparations while you relax. class You(object):     def __init__(self):         print("You:: Whoa! Marriage Arrangements??!!!")     def askEventManager(self):         print("You:: Let's Contact the Event Manager\n\n")         em = EventManager()         em.arrange()     def __del__(self):         print("You:: Thanks to Event Manager, all preparations done! Phew!") Let's now move ahead and talk about the Façade class. As discussed earlier, the Façade class simplifies the interface for the client. In this case, EventManager acts as a façade and simplifies the work for You. Façade talks to the subsystems and does all the booking and preparations for the marriage on your behalf. Here is the Python code for the EventManager class: class EventManager(object):         def __init__(self):         print("Event Manager:: Let me talk to the folks\n")         def arrange(self):         self.hotelier = Hotelier()         self.hotelier.bookHotel()                 self.florist = Florist()         self.florist.setFlowerRequirements()                  self.caterer = Caterer()         self.caterer.setCuisine()                 self.musician = Musician()         self.musician.setMusicType() Now that we're done with the Façade and client, let's dive into the subsystems. We have developed the following classes for this scenario: Hotelier is for the hotel bookings. It has a method to check whether the hotel is free on that day (__isAvailable) and if it is free for booking the Hotel (bookHotel). The Florist class is responsible for flower decorations. Florist has the setFlowerRequirements() method to be used to set the expectations on the kind of flowers needed for the marriage decoration. The Caterer class is used to deal with the caterer and is responsible for the food arrangements. Caterer exposes the setCuisine() method to accept the type of cuisine to be served at the marriage. The Musician class is designed for musical arrangements at the marriage. It uses the setMusicType() method to understand the music requirements for the event. class Hotelier(object):     def __init__(self):         print("Arranging the Hotel for Marriage? --")         def __isAvailable(self):         print("Is the Hotel free for the event on given day?")         return True       def bookHotel(self):         if self.__isAvailable():             print("Registered the Booking\n\n")     class Florist(object):     def __init__(self):         print("Flower Decorations for the Event? --")         def setFlowerRequirements(self):         print("Carnations, Roses and Lilies would be used for Decorations\n\n")     class Caterer(object):     def __init__(self):         print("Food Arrangements for the Event --")         def setCuisine(self):         print("Chinese & Continental Cuisine to be served\n\n")     class Musician(object):     def __init__(self):         print("Musical Arrangements for the Marriage --")         def setMusicType(self):         print("Jazz and Classical will be played\n\n")   you = You() you.askEventManager() The output of the preceding code is given here: In the preceding code example: The EventManager class is the Façade that simplifies the interface for You EventManager uses composition to create objects of the subsystems such as Hotelier, Caterer, and others The principle of least knowledge As you have learned in the initial parts of this article, the Façade provides a unified system that makes subsystems easy to use. It also decouples the client from the subsystem of components. The design principle that is employed behind the Façade pattern is the principle of least knowledge. The principle of least knowledge guides us to reduce the interactions between objects to just a few friends that are close enough to you. In real terms, it means the following:: When designing a system, for every object created, one should look at the number of classes that it interacts with and the way in which the interaction happens. Following the principle, make sure that we avoid situations where there are many classes created tightly coupled to each other. If there are a lot of dependencies between classes, the system becomes hard to maintain. Any changes in one part of the system can lead to unintentional changes to other parts of the system, which means that the system is exposed to regressions and this should be avoided. Summary We began the article by first understanding the Façade design pattern and the context in which it's used. We understood the basis of Façade and how it is effectively used in software architecture. We looked at how Façade design patterns create a simplified interface for clients to use. It simplifies the complexity of subsystems so that the client benefits. The Façade doesn't encapsulate the subsystem and the client is free to access the subsystems even without going through the Façade. You also learned the pattern with a UML diagram and sample code implementation in Python v3.5. We understood the principle of least knowledge and how its philosophy governs the Façade design patterns. Further resources on this subject: Asynchronous Programming with Python [article] Optimization in Python [article] The Essentials of Working with Python Collections [article]

Hello there! If you are reading this article by Adith Jagadish Boloor, the author of the book Arduino By Example, it means that you've taken your first step to make fascinating projects using Arduinos. This article will teach you how to set up an Arduino and write your first Arduino code. You'll be in good hands whilst you learn some of the basics aspects of coding using the Arduino platform, which will allow you to build almost anything from robots, home automation systems, touch interfaces, sensory systems, and so on. Firstly, you will learn how to install the powerful Arduino software, then set that up, followed by hooking up your Arduino board and, after making sure that everything is fine and well, you will write your first code! Once you are comfortable with that, we will modify the code to make it do something more, which is often what Arduino coders do. We do not just create completely new programs but often we build on what has been done before, to make it better and more suited to our objectives. The contents of this article are divided into the following topics: Prerequisites Setting up Hello World Summary (For more resources related to this topic, see here.) Prerequisites Well, you can't jump onto a horse without putting on a saddle first, can you? This section will cover what components you need to start coding on an Arduino. These can be purchased from your favorite electrical hobby store or simply ordered online. Materials needed 1x Arduino compatible board such as an Arduino UNO 1x USB cable A to B 2x LEDs 2x 330Ω resistors A mini breadboard 5x male-to-male jumper wires Note The UNO can be substituted for any other Arduino board (Mega, Leonardo and so on) for most of the projects. These boards have their own extra features. For example, the Mega has almost double the number of I/O (input/output) pins for added functionality. The Leonardo has a feature which enables it to control the keyboard and mouse of your computer. Setting up This topic involves downloading the Arduino software, installing the drivers, hooking up the Arduino, and understanding the IDE menus. Downloading and installing the software Arduino is open source-oriented. This means all the software is free to use non-commercially. Go to http://arduino.cc/en/Main/Software and download the latest version for your specific operating system. If you are using a Mac, make sure you choose the right Java version, and similarly on Linux, download the 32 or 64 bit version according to your computer. Arduino download page Windows Once you have downloaded the setup file, run it. If it asks for administrator privileges, allow it. Install it in its default location (C:Program FilesArduino or C:Program Files (x86)Arduino). Create a new folder in this location and rename it My Codes or something where you can conveniently store all your programs. Mac OS X Once the ZIP file has finished downloading, double-click to expand it. Copy the Arduino application to the Applications folder. You won't have to install additional drivers to make the Arduino work since we will be using only the Arduino UNO and MEGA. You're all set. If you didn't get anything to work, go to https://www.arduino.cc/en/guide/macOSX. Linux (Ubuntu 12.04 and above) Once you have downloaded the latest version of Arduino from the above link, install the compiler and the library packages using the following command: sudo apt-get update && sudo apt-get install arduino arduino-core If you are using a different version of Linux, this official Arduino walkthrough at http://playground.arduino.cc/Learning/Linux will help you out. Connecting the Arduino It is time to hook up the Arduino board. Plug in the respective USB terminals to the USB cable and the tiny LEDs on the Arduino should begin to flash. Arduino UNO plugged in If the LEDs didn't turn on, ensure that the USB port on your computer is functioning and make sure the cable isn't faulty. If it still does not light up, there is something wrong with your board and you should get it checked. Windows The computer will begin to install the drivers for the Arduino by itself. If it does not succeed, do the following: Open Device Manager Click on Ports (COM & LPT) Right-click on Unknown Device and select Properties Click install driver and choose browse files on the computer Choose the drivers folder in the previously installed Arduino folder The computer should say that your Arduino UNO (USB) has been successfully installed on COM port (xx). Here xx refers to a single or double digit number. If this message didn't pop up, go back to the Device Manager and check if it has been installed under COM ports. Arduino UNO COM port Remember the (COMxx) port that the Arduino UNO was installed on. Mac OS X If you are using Mac OS, a dialog box will tell you that a new network interface has been detected. Click Network Preferences and select Apply. Even though the Arduino board may show up as Not Configured, it should be working perfectly. Linux You are ready to go. The serial ports for Mac OS and Linux will be obtained once the Arduino software has been launched. The Arduino IDE The Arduino software, commonly referred to as the Arduino IDE (integrated development environment). The IDE for Windows, Mac OS and Linux is almost identical. Now let's look at some of the highlights of this software. Arduino IDE This is the window that you will see when you first start up the IDE. The tick/check mark verifies that your code's syntax is correct. The arrow pointing right is the button that uploads the code to the board and checks if the code has been changed since the last upload or verification. The magnifying glass is the Serial Monitor. This is used to input text or output debugging statements or sensor values. Examples of Arduino Every Arduino programmer starts by using one of these examples. Even after mastering Arduino, one would still return here to find examples to use. Arduino tools The screenshot shows the tools that are available in the Arduino IDE. The Board option opens up all the different boards that the software supports. Hello World The easiest way to start working with Arduinos begins here. You'll learn how to output print statements. The Arduino uses a Serial Monitor for displaying information such as print statements, sensor data and the like. This is a very powerful tool for debugging long codes. Now for your first code! Writing a simple print statement Open up the Arduino IDE and copy the following code into a new sketch. void setup() { Serial.begin(9600); Serial.println("Hello World!"); } void loop() { } Open Tools | Board and choose Arduino UNO, as shown in the following screenshot: Open Tools | Port and choose the appropriate port (remember the previous COM xx number? select that), as shown in the following screenshot. For Mac and Linux users, once you have connected the Arduino board, going to Tools | Serial Port will give you a list of ports. The Arduino is typically something like /dev/tty.usbmodem12345 where 12345 will be different. Selecting port Finally, hit the upload button. If everything is fine, the LEDs on the Arduino should start flickering as the code is uploaded to the Arduino. The code will then have uploaded to the Arduino. To see what you have accomplished, click on the Serial Monitor button on the right side and switch the baud rate on the Serial Monitor window to 9600. You should see your message Hello World! waiting for you there. LED blink That wasn't too bad but it isn't cool enough. This article will enlighten you, literally. Open up a new sketch. Go to File | Examples | Basics | Blink. Blink example Before we upload the code, we need to make sure of one more thing. Remember the LED that we spoke about in the prerequisites? Let’s learn a bit about it before plugging it in. LED basics We will make use of it now. Plug in the LED such that the longer leg goes into pin 13 and the shorter leg goes into the GND pin as in the following images: LED blink setup (Fritzing) This diagram is made using software called Fritzing. This software will be used in future projects to make it cleaner to see and easier to understand as compared to a photograph with all the wires running around. Fritzing is opensource software which you can learn more about at www.fritzing.org. Arduino LED setup Upload the code. Your LED will start blinking, as shown in the following image. Lit up LED Isn't it just fascinating? You just programmed your first hardware. There's no stopping you now. Let's see what the code does and what happens when you change it. This is the blink example code that you just used. /* Blink Turns on an LED on for one second, then off for one second, repeatedly. This example code is in the public domain. */ //Pin 13 has an LED connected on most Arduino boards. //give it a name: int led = 13; //the setup routine runs once when you press reset: void setup() { // initialize the digital pin as an output. pinMode(led, OUTPUT); } //the loop routine runs over and over again forever: void loop() { digitalWrite(led, HIGH); // turn the LED on (HIGH is the voltage level) delay(1000); // wait for a second digitalWrite(led, LOW); // turn the LED off by making the voltage LOW delay(1000); // wait for a second } We have three major sections in this code. int led = 13; This line simply stores the numerical PIN value onto a variable called led. void setup() { // initialize the digital pin as an output. pinMode(led, OUTPUT); } This is the setup function. Here is where you tell the Arduino what is connected on each used pin. In this case, we tell the Arduino that there is an output device (LED) on pin 13. void loop() { digitalWrite(led, HIGH); // turn the LED on (HIGH is the voltage level) delay(1000); // wait for a second digitalWrite(led, LOW); // turn the LED off by making the voltage LOW delay(1000); // wait for a second } This is the loop function. It tells the Arduino to keep repeating whatever is inside it in a sequence. The digitalWrite command is like a switch that can be turned ON (HIGH) or OFF (LOW). The delay(1000) function simply makes the Arduino wait for a second before heading to the next line. If you wanted to add another LED, you'd need some additional tools and some changes to the code. This is the setup that you want to create. Connecting two LEDs to an Arduino If this is your first time using a breadboard, take some time to make sure all the connections are in the right place. The colors of the wires don't matter. However, GND is denoted using a black wire and VCC/5V/PWR is denoted with a red wire. The two resistors, each connected in series (acting like a connecting wire itself) with the LEDs limit the current flowing to the LEDs, making sure they don't blow up. As before, create a new sketch and paste in the following code: /* Double Blink Turns on and off two LEDs alternatively for one second each repeatedly. This example code is in the public domain. */ int led1 = 12; int led2 = 13; void setup() { // initialize the digital pins as an output. pinMode(led1, OUTPUT); pinMode(led2, OUTPUT); // turn off LEDs before loop begins digitalWrite(led1, LOW); // turn the LED off (LOW is the voltage level) digitalWrite(led2, LOW); // turn the LED off (LOW is the voltage level) } //the loop routine runs over and over again forever: void loop() { digitalWrite(led1, HIGH); // turn the LED on (HIGH is the voltage level) digitalWrite(led2, LOW); // turn the LED off (LOW is the voltage level) delay(1000); // wait for a second digitalWrite(led1, LOW); // turn the LED off (LOW is the voltage level) digitalWrite(led2, HIGH); // turn the LED on (HIGH is the voltage level) delay(1000); // wait for a second } Once again, make sure the connections are made properly, especially the positive LEDs (longer to OUTPUT PIN) and the negative (shorter to GND) terminals. Save the code into DoubleBlink.ino. Now, if you make any changes to it, you can always retrieve it. Upload the code. 3… 2… 1… And there you have it, an alternating LED blink cycle created purely with the Arduino. You can try changing the delay to see its effects. For the sake of completeness, I would like to mention that you could take this mini-project further by using a battery to power the system and decorate your desk/room/house. Summary You have now completed the basic introduction to the world of Arduino. In short, you have successfully set up your Arduino and have written your first code. You also learned how to modify the existing code to create something new, making it more suitable for your specific needs. This methodology will be applied repeatedly while programming, because almost all the code available is open source and it saves time and energy. Resources for Article: Further resources on this subject: Central Air and Heating Thermostat [article] Christmas Light Sequencer [article] Webcam and Video Wizardry [article]

Yael Niv is an Associate Professor of Psychology at the Princeton Neuroscience Institute since 2007. Her preferred areas of research include human and animal reinforcement learning and decision making. At her Niv lab, she studies day-to-day processes that animals and humans use to learn by trial and error, without explicit instructions given. In order to predict future events and to act upon the current environment so as to maximize reward and minimize the damage. Our article aims to deliver key points from Yael Niv’s keynote presentation at NIPS 2017. She talks about the ability of Artificial Intelligence systems to perform simple human-like tasks effectively using State representations in the human brain. The talk also deconstructs the complex human decision-making process. Further, we explore how a human brain breaks down complex procedures into simple states and how these states determine our decision-making capabilities.This, in turn, gives valuable insights into the design and architecture of smart AI systems with decision-making capabilities. Staying Simple is Complex What do you think happens when a human being crosses a road, especially when it’s a busy street and you constantly need to keep an eye on multiple checkpoints in order to be safe and sound? The answer is quite ironical. The human brain breaks down the complex process into multiple simple blocks. The blocks can be termed as states - and these states then determine decisions such as when to cross the road or at what speed to cross the road. In other words, the states can be anything - from determining the incoming traffic density to maintaining the calculation of your walking speed. These states help the brain to ignore other spurious or latent tasks in order to complete the priority task at hand. Hence, the computational power of the brain is optimized. The human brain possesses the capability to focus on the most important task at hand and then breaks it down into multiple simple tasks. The process of making smarter AI systems with complex decision-making capabilities can take inspiration from this process. The Practical Human Experiment To observe how the human brain behaves when urged to draw complex decisions, a few experiments were performed. The primary objective of these experiments was to verify the hypothesis that the decision making information in the human brain is stored in a part of the frontal brain called as Orbitofrontal cortex. The two experiments performed are described in brief below: Experiment 1 The participants were given sets of circles at random and they were asked to guess the number of circles in the cluster within 2 minutes. After they guessed the first time, the experimenter disclosed the correct number of circles. Then the subjects were further given a cluster of circles in two different colors (red and yellow) to repeat the guessing activity for each cluster. However, the experimenter never disclosed the fact that they will be given different colored clusters next. Observation: The most important observation derived from the experiment was that after the subject knew the correct count, their guesses revolved around that number irrespective of whether that count mattered for the next set of circle clusters given. That is, the count had actually changed for the two color specimens given to them. The important factor here is that the participants were not told that color would be a parameter to determine the number of circles in each set and still it played a huge part in guessing the number of circles in each set. This way it acted as a latent factor, which was present in the subconscious of the participants and was not a direct parameter. And, this being a latent factor was not in the list of parameters which played an important in determining the number of circles. But still, it played an important part in changing the overall count which was significantly higher for the red color than for the yellow color cluster. Hence, the experiment proved the hypothesis that latent factors are an integral part of intelligent decision-making capabilities in human beings. Experiment 2 The second experiment was performed to ascertain the hypothesis that the Orbitofrontal cortex contains all the data to help the human brain make complex decisions. For this, human brains were monitored using MRI to track the brain activity during the decision making process. In this experiment, the subjects were given a straight line and a dot. They were then asked to predict the next line from the dot - both in terms of line direction and its length. After completing this process for a given number of times, the participants were asked to remember the length and direction of the first line. There was a minor change among the sets of lines and dots. One group had a gradual change in line length and direction and another group had a drastic change in the middle. Observation: The results showed that the group with a gradual change of line length and direction were more helpful in preserving the first data and the one with drastic change was less accurate. The MRI reports showed signs that the classification information was primarily stored in the Orbitofrontal cortex. Hence it is considered as one of the most important parts of the human decision-making process. Shallow Learning with Deep Representations The decision-making capabilities and the effect of latent factors involved in it form the basis of dormant memory in humans. An experiment on rats was performed to explain this phenomenon. In the experiment, 4 rats were given electric shock accompanied by a particular type of sound for a day or two. On the third day, they reacted to the sound even without being given electric shocks. Ivan Pavlov has coined this term as Classical Conditioning theory wherein a relatively permanent change in behavior can be seen as a result of experience or continuous practice. Such instances of conditioning can be deeply damaging, for example in case of PTSD (Post Traumatic Stress Disorder) patients and other trauma victims. In order to understand the process of State representations being stored in memory, the reversal mechanism, i.e how to reverse the process also needs to be understood. For that, three techniques were tested on these rats: The rats were not given any shock but were subjected to the sound The rats were given shocks accompanied by sound at regular intervals and sounds without shock The shocks were slowly reduced in numbers but the sound continued The best results in reversing the memory were observed in case of the third technique, which is known as gradual extinction. In this way, a simple reinforcement learning mechanism is shown to be very effective because it helps in creating simple states which are manageable efficiently and trainable easily. Along with this, if we could extract information from brain imaging data derived from the Orbitofrontal cortex, these simple representational states can shed a lot of light into making complex computational processes simpler and enable us to make smarter AI systems for a better future.

Last Friday’s uncontrolled spread of horrific videos on the Christchurch mosque attack and a propaganda coup for espousing hateful ideologies raised questions about social media. The tech companies scrambled to take action on time due to the speed and volume of content which was uploaded, reuploaded and shared by the users worldwide. In Washington and Silicon Valley, the incident crystallized growing concerns about the extent to which government and market forces have failed to check the power of social media. The failure highlighted the social media companies struggle to police content that are massively lucrative and persistently vulnerable to outside manipulation despite years of promises to do better. After the white supremacist live-streamed the attack and uploaded the video to Facebook, Twitter, YouTube, and other platforms across the internet. These tech companies faced back lashes from the media and internet users worldwide, to an extent where they were regarded as complicit in promoting white supremacism too. In response to the backlash, Google and Facebook provides status report on what they went through when the video was reported, the kind of challenges they faced and what are the next steps to combat such incidents in future. Google’s report so far... Google in an email to Motherboard says it employs 10,000 people across to moderate the company’s platforms and products. They also described a process they would follow when a user reports a piece of potentially violating content—such as the attack video; which is The user flagged report will go to a human moderator to assess. The moderator is instructed to flag all pieces of content related to the attack as “Terrorist Content,” including full-length or sections of the manifesto. Because of the document’s length the email tells moderators not to spend an extensive amount of time trying to confirm whether a piece of content does contain part of the manifesto. Instead, if the moderator is unsure, they should err on the side of caution and still label the content as “Terrorist Content,” which will then be reviewed by a second moderator. The second moderator is told to take time to verify that it is a piece of the manifesto, and appropriately mark the content as terrorism no matter how long or short the section may be. Moderators are told to mark the manifesto or video as terrorism content unless there is an Educational, Documentary, Scientific, or Artistic (EDSA) context to it. Further Google adds that they want to preserve journalistic or educational coverage of the event, but does not want to allow the video or manifesto itself to spread throughout the company’s services without additional context. Google at some point had taken the unusual step of automatically rejecting any footage of violence from the attack video, cutting out the process of a human determining the context of the clip. If, say, a news organization was impacted by this change, the outlet could appeal the decision, Google commented. “We made the call to basically err on the side of machine intelligence, as opposed to waiting for human review,” YouTube’s Product Officer Neal Mohan told the Washington Post in an article published Monday. Google also tweaked the search function to show results from authoritative news sources. It suspended the ability to search for clips by upload date, making it harder for people to find copies of the attack footage. "Since Friday’s horrific tragedy, we’ve removed tens of thousands of videos and terminated hundreds of accounts created to promote or glorify the shooter," a YouTube spokesperson said. “Our teams are continuing to work around the clock to prevent violent and graphic content from spreading, we know there is much more work to do,” the statement added. Facebook’s update so far... Facebook on Wednesday also shared an update on how they have been working with the New Zealand Police to support their investigation. It provided additional information on how their products were used to circulate videos and how they plan to improve them. So far Facebook has provided the following information: The video was viewed fewer than 200 times during the live broadcast. No users reported the video during the live broadcast. Including the views during the live broadcast, the video was viewed about 4,000 times in total before being removed from Facebook. Before Facebook was alerted to the video, a user on 8chan posted a link to a copy of the video on a file-sharing site. The first user report on the original video came in 29 minutes after the video started, and 12 minutes after the live broadcast ended. In the first 24 hours, Facebook removed more than 1.2 million videos of the attack at upload, which were therefore prevented from being seen on our services. Approximately 300,000 additional copies were removed after they were posted. As there were questions asked to Facebook about why artificial intelligence (AI) didn’t detect the video automatically. Facebook says AI has made massive progress over the years to proactively detect the vast majority of the content it can remove. But it’s not perfect. “To achieve that we will need to provide our systems with large volumes of data of this specific kind of content, something which is difficult as these events are thankfully rare.” says Guy Rosen VP Product Management at Facebook. Guy further adds, “AI is an incredibly important part of our fight against terrorist content on our platforms, and while its effectiveness continues to improve, it is never going to be perfect. People will continue to be part of the equation, whether it’s the people on our team who review content, or people who use our services and report content to us. That’s why last year Facebook more than doubled the number of people working on safety and security to over 30,000 people, including about 15,000 content reviewers to report content that they find disturbing.” Facebook further plans to: Improve the image and video matching technology so that they can stop the spread of viral videos of such nature, regardless of how they were originally produced. React faster to this kind of content on a live streamed video. Continue to combat hate speech of all kinds on their platform. Expand industry collaboration through the Global Internet Forum to Counter Terrorism (GIFCT). Challenges Google and Facebook faced to report the video content According to Motherboard, Google saw an unprecedented number of attempts to post footage from the attack, sometimes as fast as a piece of content per second. But the challenge they faced was to block access to the killer’s so-called manifesto, a 74-page document that spouted racist views and explicit calls for violence. Google described the difficulties of moderating the manifesto, pointing to its length and the issue of users sharing the snippets of the manifesto that Google’s content moderators may not immediately recognise. “The manifesto will be particularly challenging to enforce against given the length of the document and that you may see various segments of various lengths within the content you are reviewing,” says Google. A source with knowledge of Google’s strategy for moderating the New Zealand attack material said this can complicate moderation efforts because some outlets did use parts of the video and manifesto. UK newspaper The Daily Mail let readers download the terrorist’s manifesto directly from the paper’s own website, and Sky News Australia aired parts of the attack footage, BuzzFeed News reported. On the other hand Facebook faces a challenge to automatically discern such content from visually similar, innocuous content. For example if thousands of videos from live-streamed video games are flagged by the systems, reviewers could miss the important real-world videos where they could alert first responders to get help on the ground. Another challenge for Facebook is similar to what Google faces, which is the proliferation of many different variants of videos makes it difficult for the image and video matching technology to prevent spreading further. Facebook found that a core community of bad actors working together to continually re-upload edited versions of the video in ways designed to defeat their detection. Second, a broader set of people distributed the video and unintentionally made it harder to match copies. Websites and pages, eager to get attention from people seeking out the video, re-cut and re-recorded the video into various formats. In total, Facebook found and blocked over 800 visually-distinct variants of the video that were circulating. Both companies seem to be working hard to improve their products and gain user’s trust and confidence back. How social media enabled and amplified the Christchurch terrorist attack Google to be the founding member of CDF (Continuous Delivery Foundation) Google announces the stable release of Android Jetpack Navigation