Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

7010 Articles
article-image-performing-sentiment-analysis-on-social-media-platforms-using-corenlp
Packt Editorial Staff
27 Mar 2019
9 min read
Save for later

Dr Joshua Eckroth on performing Sentiment Analysis on social media platforms using CoreNLP

Packt Editorial Staff
27 Mar 2019
9 min read
Sentiment analysis is achieved by labeling individual words as positive or negative, among other possible sentiments such as happy, worried, and so on. The sentiment of the sentence or phrase as a whole is determined by a procedure that aggregates the sentiment of individual words. In this article we'll demonstrate how to perform sentiment analysis on social media platforms using the CoreNLP library. This article is taken from the book AI Blueprints written by Dr Joshua Eckroth.The book covers several paradigms of AI, including deep learning, natural language processing, planning, and logic programming. Consider the sentence, I didn't like a single minute of this film. A simplistic sentiment analysis system would probably label the word like as positive and the other words as neutral, yielding an overall positive sentiment. More advanced systems analyze the "dependency tree" of the sentence to identify which words are modifiers for other words. In this case, didn't is a modifier for like, so the sentiment of like is reversed due to this modifier. Likewise, a phrase such as It's definitely not dull, exhibits a similar property, and ...not only good but amazing exhibits a further nuance of the English language. It is clear a simple dictionary of positive and negative words is insufficient for accurate sentiment analysis. The presence of modifiers can change the polarity of a word. Wilson and others' work on sentiment analysis (Recognizing contextual polarity in phrase-level sentiment analysis, Wilson, Theresa, Janyce Wiebe, and Paul Hoffmann, published in Proceedings of the conference on human language technology and empirical methods in natural language processing, pp. 347-354, 2005) is foundational in the dependency tree approach. They start with a lexicon (that is, collection) of 8,000 words that serve as "subjectivity clues" and are tagged with polarity (positive or negative). Using just this dictionary, they achieved 48% accuracy in identifying the sentiment of about 3,700 phrases. To improve on this, they adopted a two-step approach. First, they used a statistical model to determine whether a subjectivity clue is used in a neutral or polar context. When used in a neutral context, the word can be ignored as it does not contribute to the overall sentiment. The statistical model for determining whether a word is used in a neutral or polar context uses 28 features, including the nearby words, binary features such as whether the word not appears immediately before, and part-of-speech information such as whether the word is a noun, verb, adjective, and so on. Next, words that have polarity, that is, those that have not been filtered out by the neutral/polar context identifier, are fed into another statistical model that determines their polarity: positive, negative, both, or neutral. Ten features are used for polarity classification, including the word itself and its polarity from the lexicon, whether or not the word is being negated, and the presence of certain nearby modifiers such as little, lack, and abate. These modifiers themselves have polarity: neutral, negative, and positive, respectively. Their final procedure achieves 65.7% percent accuracy for detecting sentiment. Their approach is implemented in the open source OpinionFinder. Sentiment analysis using Natural Language Processing A more modern approach may be found in Stanford's open source CoreNLP project . CoreNLP supports a wide range of NLP processing such as sentence detection, word detection, part-of-speech tagging, named-entity recognition (finding names of people, places, dates, and so on), and sentiment analysis. Several NLP features, such as sentiment analysis, depend on prior processing including sentence detection, word detection, and part-of-speech tagging. As described in the following text, a sentence's dependency tree, which shows the subject, object, verbs, adjectives, and prepositions of a sentence, is critical for sentiment analysis. CoreNLP's sentiment analysis technique has been shown to achieve 85.4% accuracy for detecting positive/negative sentiment of sentences. Their technique is state-of-the-art and has been specifically designed to better handle negation in various places in a sentence, a limitation of simpler sentiment analysis techniques as previously described. CoreNLP's sentiment analysis uses a technique known as recursive neural tensor networks (RNTN) (Here, a sentence or phrase is parsed into a binary tree, as seen in Figure 1. Every node is labeled with its part-of-speech: NP (noun phrase), VP (verb phrase), NN (noun), JJ (adjective), and so on. Each leaf node, that is, each word node, has a corresponding word vector. A word vector is an array of about 30 numbers (the actual size depends on a parameter that is determined experimentally). The values of the word vector for each word are learned during training, as is the sentiment of each individual word. Just having word vectors will not be enough since we have already seen how sentiment cannot be accurately determined by looking at words independently of their context. The next step in the RNTN procedure collapses the tree, one node at a time, by calculating a vector for each node based on its children. The bottom-right node of the figure, the NP node with children own and crashes, will have a vector that is the same size of the word vectors but is computed based on those child word vectors. The computation multiplies each child word vector and sums the results. The exact multipliers to use are learned during training. The RNTN approach, unlike prior but similar tree collapsing techniques, uses a single combiner function for all nodes. Ultimately, the combiner function and the word vectors are learned simultaneously using thousands of example sentences with the known sentiment. Figure 1: CoreNLP's dependency tree parse of the sentence, "Self-driving car companies should not be allowed to investigate their own crashes" The dependency tree from the preceding figure has twelve leaf nodes and twelve combiner nodes. Each leaf node has an associated word vector learned during training. The sentiment of each leaf node is also learned during training. Thus, the word crashes, for example, has a neutral sentiment with 0.631 confidence, while the word not has negative sentiment with 0.974 confidence. The parent node of allowed and the phrase to investigate their own crashes has a negative sentiment, confidence 0.614, even though no word or combiner node among its descendants have anything but neutral sentiment. This demonstrates that the RNTN learned a complex combiner function that operates on the word vectors of its children and not just a simple rule such as, If both children are neutral, then this node is neutral, or if one child is neutral, but one is positive, this node is positive, .... The sentiment values and confidence of each node in the tree is shown in the output of CoreNLP shown in the following code block. Note that sentiment values are coded: 0 = very negative 1 = negative 2 = neutral 3 = positive 4 = very positive (ROOT|sentiment=1|prob=0.606  (NP|sentiment=2|prob=0.484    (JJ|sentiment=2|prob=0.631 Self-driving) (NP|sentiment=2|prob=0.511   (NN|sentiment=2|prob=0.994 car)   (NNS|sentiment=2|prob=0.631 companies)))  (S|sentiment=1|prob=0.577 (VP|sentiment=2|prob=0.457      (VP|sentiment=2|prob=0.587     (MD|sentiment=2|prob=0.998 should)     (RB|sentiment=1|prob=0.974 not))      (VP|sentiment=1|prob=0.703        (VB|sentiment=2|prob=0.994 be)        (VP|sentiment=1|prob=0.614      (VBN|sentiment=2|prob=0.969 allowed)          (S|sentiment=2|prob=0.724            (TO|sentiment=2|prob=0.990 to)            (VP|sentiment=2|prob=0.557              (VB|sentiment=2|prob=0.887 investigate)              (NP|sentiment=2|prob=0.823                (PRP|sentiment=2|prob=0.997 their)                (NP|sentiment=2|prob=0.873                  (JJ|sentiment=2|prob=0.996 own)                  (NNS|sentiment=2|prob=0.631 crashes))))))))    (.|sentiment=2|prob=0.997 .))) We see from these sentiment values that allowed to investigate their own crashes is labeled with negative sentiment. We can investigate how CoreNLP handles words such as allowed and not by running through a few variations. These are shown in the following table: Sentence Sentiment Confidence They investigate their own crashes. Neutral 0.506 They are allowed to investigate their own crashes. Negative 0.697 They are not allowed to investigate their own crashes. Negative 0.672 They are happy to investigate their own crashes. Positive 0.717 They are not happy to investigate their own crashes. Negative 0.586 They are willing to investigate their own crashes. Neutral 0.507 They are not willing to investigate their own crashes. Negative 0.599 They are unwilling to investigate their own crashes. Negative 0.486 They are not unwilling to investigate their own crashes. Negative 0.625 Table 1: Variations of a sentence with CoreNLP's sentiment analysis It is clear from Table 1 that the phrase investigates their own crashes is not contributing strongly to the sentiment of the whole sentence. The verb, allowed, happy, or willing can dramatically change the sentiment. The modifier not can flip the sentiment, though curiously not unwilling is still considered negative. We should be particularly careful to study CoreNLP's sentiment analysis with sentence fragments and other kinds of invalid English that is commonly seen on Twitter. For example, the Twitter API will deliver phrases such as, Ford's self-driving car network will launch 'at scale' in 2021 - Ford hasn't been shy about... with the ... in the actual tweet. CoreNLP labels this sentence as negative with confidence 0.597. CoreNLP was trained on movie reviews, so news articles, tweets, and Reddit comments may not match the same kind of words and grammar found in movie reviews. We might have a domain mismatch between the training domain and the actual domain. CoreNLP can be trained on a different dataset but doing so requires that thousands (or 10's or 100's of thousands) of examples with known sentiment are available. Every node in the dependency tree of every sentence must be labeled with a known sentiment. This is very time-consuming. The authors of CoreNLP used Amazon Mechanical Turk to recruit humans to perform this labeling task. To summarize, in this article, we have demonstrated how to identify the sentiment, or general mood, of the feedback from customers and the general public on social media platforms (e.g. Twitter). We also performed sentiment analysis using a method based on machine learning using CoreNLP. Master essential AI blueprints to program real-world business applications from the book AI Blueprints by Dr Joshua Eckroth. How to perform sentiment analysis using Python [Tutorial] Understanding Sentiment Analysis and other key NLP concepts Sentiment Analysis of the 2017 US elections on Twitter
Read more
  • 0
  • 0
  • 27321

article-image-apples-march-event-changes-gears-to-services
Sugandha Lahoti
26 Mar 2019
7 min read
Save for later

Apple’s March Event: Apple changes gears to services, is now your bank, news source, gaming zone, and TV

Sugandha Lahoti
26 Mar 2019
7 min read
Apple’s main business model has always been hardware-centric - to sell phones and computers. However, in light of the recent news of Apple’s iPhone sales dwindling, the company is now shifting its focus to other means of revenue growth to keep its consumers occupied in the world of Apple. That is exactly what happened yesterday when Apple unveiled a set of new services at the Apple March Event. Gadi Schwartz, NBC News correspondent rightly sums up Apple’s latest plan. https://twitter.com/GadiNBC/status/1110270953001410560 Here’s the detailed report. Video subscription service: Apple TV+ The Apple TV+ is a new television subscription service (yes in the likes of Netflix and Amazon Prime) which will give subscribers access to the many shows the company has been developing. Apple TV+, the company says, “will become the new home for the world’s most creative storytellers featuring exclusive original shows, movies, and documentaries.”  Apple plans to launch Apple TV+ in over 100 countries sometime this fall, though it did not disclose the pricing. The subscription service will be ad-free, available on demand, and viewable both online and offline. https://youtu.be/Bt5k5Ix_wS8 Apple also announced Apple TV Channels as a part of the Apple TV app, which will let customers pay and watch HBO, Showtime, Starz, CBS All Access, and other services directly through the TV app. Apple TV + puts the company in direct competition to Netflix, Hulu, and Disney who also offer their own video subscription services. The company is trying to find new ways to market the Apple experience to consumers. With iPhone’s sales slowly receding, Apple’s foray into video subscription services is a new initiative to bring everyone into the walled orchids of Apple. https://twitter.com/DylanByers/status/1110144534132908037 Media subscription service: Apple news+ Next in line, is the Apple News Plus service, which adds newspapers and magazines to the Apple News app. The service costing $9.99 per month will feature almost 300 magazines and newspapers including People, Vogue, National Geographic Magazine, ELLE, Glamour, The Wall Street Journal, Los Angeles Times, and more. Surprisingly, The New York Times and The Washington Post, have opted out of joining the subscription service. Although the publishers were not authorized to speak publicly about the plans, it is speculated that opting out from this subscription service is because of two major reasons. First, Apple is asking for a cut of roughly half of the subscription revenue involved in the service. Second, Apple has also asked publishers to give unlimited access to all their content which is concerning. Combined, the subscriptions provided through Apple News+ would cost more than $8,000 per year. Apple News Plus will also come with “Live Covers,” which shows animated images instead of static photos for a magazine’s cover. https://youtu.be/Im5c5WR9vMQ Apple has been quite vocal about maintaining privacy. A striking feature of Apple news+ is the heavy emphasis on private recommendations inside the news app, including magazines. The app downloads a set of articles and manages recommendations on-device. It also does not give any data to advertisers. The company noted in the live stream, "Apple doesn't know what you read." Apple News+ is available in the U.S. and Canada. The first month is free. In Canada, the service will be offered at $12.99 per month. Later this year, Apple News+ will arrive in Europe and Australia. Game subscription service: Apple Arcade Apple is now your new gaming zone with a new game subscription service, the Apple Arcade. Presented as the “world’s first game subscription service for mobile, desktop, and living room”, it will feature over 100 new and exclusive games. These games will be from acclaimed indie developers, major studios as well as renowned creators. Apple will also be contributing to the development costs for such games. With the subscription service, players can try any game in the service without risk. Every game includes access to the full experience, including all game features, content and future updates with no additional purchases required. Apple says Arcade games don’t track usage, gameplay or the titles a user plays more. Apple Arcade will launch in fall 2019 in more than 150 countries. Arcade as a single subscription package may also possibly bring premium games the traction they may have been lacking otherwise. People also pointed out that Apple’s primary target for Arcade may be parents. A comment on Hacker News reads, “I think a lot of folks looking at this from the point of view of an adult gamer are missing the point: the audience for this is parents. For 10 bucks (or whatever) a month you can load the iPad up with games and not worry about microtransactions or scummy ads targeting your kids. "Curated" is a signal that can trust age recommendations and not worry about inappropriate content.” Netizens also believe that gaming subscription services will payout more than traditional models. “The difference between this and music is that most people do not want to hear the same songs over and over again. The $10 is spread across so many artists. Video games will capture the attention of a person for hours in a month. I can see that a big chunk of the monthly fee going to a couple of titles.”, reads a comment on Hacker News. Payment subscription service: Apple Card Probably the most important service, Apple is now venturing into the banking sector, with a new digital credit card with simpler applications, no fees, lower interest rates, and daily rewards. The Apple Card is created in partnership with Goldman Sachs and Mastercard. It is available as two options. First, as a digital card which users will be able to access by signing up on their iPhone in the Apple Wallet app.  Second, as a physical titanium card with no credit card number, CVV, expiration date, or signature. All of the authorization information is stored directly in the Apple Wallet app. The card makes use of machine learning and Apple Maps to label stores and categorize them based on color. Users can easily track purchases across categories like “food and drink” or “shopping.” It also has a rewards program, “Daily Cash,” which adds 2 percent of the daily purchase amount in cash to your Apple Cash account, also within the Wallet app. Though, purchases made through the physical card will get just 1 percent cash back. Again, privacy is the most important feature here. Apple will store the spending, tracking and other information directly on the device. Jennifer Bailey, VP of Apple Pay said, “Apple doesn’t know what you bought, where you bought it, and how much you paid for it. Goldman Sachs will never sell your data to third parties for marketing and advertising.” This is probably the service that has got people the most excited. https://twitter.com/byjacobward/status/1110237925889851393 https://twitter.com/DylanByers/status/1110248441152561158 Apple Card will be available in the US this summer. Why is Apple changing focus to services? With iPhone sales growth slowing, Apple needs new measures to bring in a large number of users to its world. What better than to foray into subscription streaming services and premium original content. With the new announcements made, Apple is indeed playing the perfect middleman between its users and TV, gaming, news, and other services, bolstering privacy as their major selling point, while also earning huge revenues. As perfectly summed by a report from NBC News, “The better Apple's suite of services — movies and shows, but also music, news, fitness tracking, mobile payments, etc. — the more revenue Apple will see from subscribers.” Spotify files an EU antitrust complaint against Apple; Apple says Spotify’s aim is to make more money off other’s work. Donald Trump called Apple CEO Tim Cook ‘Tim Apple’ Apple to merge the iPhone, iPad, and Mac apps by 2021
Read more
  • 0
  • 0
  • 28249

article-image-evade-intrusion-detection-systems-using-proxy-cannon-tutorial
Packt Editorial Staff
26 Mar 2019
10 min read
Save for later

Adrian Pruteanu shows how to evade Intrusion Detection Systems using Proxy Cannon [Tutorial]

Packt Editorial Staff
26 Mar 2019
10 min read
These days, it is fairly common for mature companies to implement Intrusion detection system (IDS), intrusion prevention systems (IPS), and security information and event management (SIEM) when they detect abuse against a particular application. When an unknown IP is performing too many operations in a short time on a protected application, IDS or IPS may take action against the source. If we are conducting a password spraying attack, we may avoid lockouts but we're still hammering the server from one source: our machine. A good way to evade these types of detection systems is to distribute the connection requests from the attacker machine over many IPs, which is commonly done by malicious actors through networks of compromised hosts. With the advent of cloud computing and computing time becoming increasingly cheap, even free in some cases, we don't have to stray outside of the law and build a botnet. In this article we'll see how to use Proxy cannon to evade intrusion detection systems (IDS). This article is taken from the book Becoming the Hacker written by Adrian Pruteanu. This book will teach you how to approach web penetration testing with an attacker's mindset. While testing web applications for performance is common, the ever-changing threat landscape makes security testing much more difficult for the defender. The Tor Project was started to provide a way for users to browse the internet anonymously. It is by far the best way to anonymize traffic and best of all, it's free. It is an effective way to change the public IP during an attack. The Tor network Tor is a network of independently operated nodes interconnected to form a network through which packets can be routed. The following graphic shows how a user, Alice, can connect to Bob through a randomly generated path or circuit, through the Tor network: Figure 1: The Tor network traffic flow (source: https://www.torproject.org/) Instead of connecting directly to the destination, the client connection from Alice to Bob will be routed through a randomly chosen set of nodes in the Tor network. Each packet is encrypted and every node can only decrypt enough information to route it to the next hop along the path. The exit node is the final node in the chain, which will make the connection to the intended destination on behalf of the client. When the packet arrives at Bob's machine, the request will look like it's coming from the exit node and not Alice's public IP. Note: More information on Tor can be found on the official site: https://www.torproject.org. While Tor is important for anonymity, we're not really concerned with staying completely anonymous. We can, however, leverage the randomly chosen exit nodes to mask the public IP when attacking an application. There are a couple of issues with conducting attacks through the Tor network. The routing protocol is inherently slower than a more direct connection. This is because Tor adds several layers of encryption to each transmission, and each transmission is forwarded through three Tor nodes on top of the normal routing that internet communication requires. This process improves anonymity but also increases communication delay significantly. The lag is noticeable for normal web browsing, but this is a tolerable trade-off. For large volume scans, it may not be the ideal transport. Warning: It should also be noted that Tor is used heavily in regions of the world where privacy is of utmost importance. Conducting large volume attacks through Tor is discouraged, as it can lead to unnecessary network slowdowns and can impact legitimate users. Low and slow attacks shouldn't cause any problems. Some red-team engagements may even require testing from the Tor network to verify related IDS/IPS rules are working as intended, but caution should be taken when launching attacks through a limited-resource public medium. Proxy cannon An alternative to using Tor for diversifying our attack IPs is to simply use the cloud. There are countless Infrastructure as a Service (IaaS) providers, each with a large IP space available for free to VM instances. VMs are cheap and sometimes free as well, so routing our traffic through them should be fairly cost effective. Amazon, Microsoft, and Google all have an easy-to-use API for automating the management of VM instances. If we can spawn a new VM with a new external IP periodically, we can route our traffic to the target application through it and mask our true origin. This should make it much more difficult for automated systems to detect and alert on our activities. Cue ProxyCannon, a great tool that does all the heavy lifting of talking to Amazon's AWS API, creating and destroying VM instances, rotating external IPs, and routing our traffic through them. Note: ProxyCannon was developed by Shellntel and is available on GitHub: https://github.com/Shellntel/scripts/blob/master/proxyCannon.py. ProxyCannon requires boto, a Python library that provides API access to Amazon's AWS. We can use Python's pip command to install the required dependency: root@kali:~/tools# pip install -U boto Collecting boto  Downloading boto-2.48.0-py2.py3-none-any.whl (1.4MB) [...] Installing collected packages: boto Successfully installed boto-2.48.0 The ProxyCannon tool should now be ready to use with the -h option showing all of the available options: root@kali:~/tools# python proxyCannon.py -h usage: proxyCannon.py [-h] [-id [IMAGE_ID]] [-t [IMAGE_TYPE]]             [--region [REGION]] [-r] [-v] [--name [NAME]]             [-i [INTERFACE]] [-l]             num_of_instances positional arguments:  num_of_instances   The number of amazon instances you'd like to launch. optional arguments:  -h, --help         show this help message and exit  -id [IMAGE_ID], --image-id [IMAGE_ID]             Amazon ami image ID. Example: ami-d05e75b8. If not             set, ami-d05e75b8.  -t [IMAGE_TYPE], --image-type [IMAGE_TYPE]             Amazon ami image type Example: t2.nano. If not             set, defaults to t2.nano.  --region [REGION] Select the region: Example: us-east-1. If             not set, defaults to us-east-1. positional arguments:  num_of_instances   The number of amazon instances you'd like to launch. optional arguments:  -h, --help         show this help message and exit  -id [IMAGE_ID], --image-id [IMAGE_ID]             Amazon ami image ID. Example: ami-d05e75b8. If not             set, ami-d05e75b8.  -t [IMAGE_TYPE], --image-type [IMAGE_TYPE]             Amazon ami image type Example: t2.nano. If not             set, defaults to t2.nano.  --region [REGION] Select the region: Example: us-east-1. If             not set, defaults to us-east-1. Output is to /tmp/ By default, ProxyCannon creates t2.nano virtual instances in AWS, which should be free for a limited time with new accounts. They have very little resources but are typically enough for most attacks. To change the type of instance, we can supply the -t switch. The default region is us-east-1 and can be adjusted using the --region switch. ProxyCannon will create as many instances as specified in the num_of_instances and using the -r switch, it will rotate them regularly. The -l switch is also useful to keep track of what public IPs ProxyCannon is using over the course of the execution. This is useful for reporting purposes: the blue team may need a list of all the IPs used in the attack. In order for the tool to be able to communicate with our AWS account and to manage instances automatically, we have to create API access keys in the AWS console. The interface is fairly straightforward and can be accessed in the account Security Credentials page. The access key ID and the secret keys are randomly generated and should be stored securely. Once the engagement is over, you should delete the keys in the AWS console. Figure 2: Generating a new AWS API access key We can start ProxyCannon using the -r and -l switches, and specify that we want 3 instances running at the same time. root@kali:~/tools# python proxyCannon.py -r -l 3 What is the AWS Access Key Id: d2hhdCBhcmUgeW91IGRvaW5n What is the AWS Secret Access Key: dW5mb3J0dW5hdGVseSB0aGlzIGlzIG5vdCB0aGUgcmVhbCBrZXku [...] Upon first run, ProxyCannon will ask you for these values and store them in the ~/.boto file. root@kali:~/tools# cat ~/.boto [default] aws_access_key_id = d2hhdCBhcmUgeW91IGRvaW5n aws_secret_access_key = dW5mb3J0dW5hdGVseSB0aGlzIGlzIG5vdCB0aGUgcmVhbCBrZXku As you can see, these are stored in plaintext, so make sure this file is properly protected. Amazon recommends that these keys are rotated frequently. It's probably a good idea to create new ones for each engagement and delete them from AWS as soon as they're not required anymore. ProxyCannon will connect to Amazon EC2, setup the SSH keys, adjust the security groups, and start the VM instances. This process may take a couple of minutes to complete. [*] Connecting to Amazon's EC2... [*] Generating ssh keypairs... [*] Generating Amazon Security Group... [~] Starting 3 instances, please give about 4 minutes for them to fully boot [====================] 100% ProxyCannon will overwrite the current system iptables configuration to properly route all traffic through whatever instance is chosen: [*] Provisioning Hosts..... [*] Saving existing iptables state [*] Building new iptables... [*] Done! +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + Leave this terminal open and start another to run your commands.+ +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ [~] Press ctrl + c to terminate the script gracefully. [...] As promised, ProxyCannon will periodically rotate our effective external IP using SSH tunnels and by modifying the routing table. All of this is done automatically, in the background, while Burp Suite or ZAP runs the password spraying attack. The following is the periodic output from ProxyCannon showing the IPs being rotated: [*] Rotating IPs. [*] Replaced 107.21.177.36 with 34.207.187.254 on tun0 [*] Replaced 34.234.91.233 with 52.91.91.157 on tun1 [*] Replaced 34.202.237.230 with 34.228.167.195 on tun2 [*] Replaced 34.207.187.254 with 34.228.158.208 on tun0 [*] Replaced 52.91.91.157 with 54.198.223.114 on tun1 On the AWS console, we can see the started t2.nano instances and their public IPs: Figure 3: AWS instances created to route our traffic through We can test ProxyCannon by repeating a curl request to our target application using the watch command. We don't need to drop in a shell similar to torsocks because ProxyCannon modifies the local system routing to help us change our external IP. root@kali:~# watch -n30 curl http://c2.spider.ml On the target application side, c2.spider.ml, the server log, shows connection attempts from various IPs belonging to the Amazon address space: 52.91.91.157 - - [13:01:16] "GET / HTTP/1.1" 200 - 52.91.91.157 - - [13:01:22] "GET / HTTP/1.1" 200 - 34.228.158.208 - - [13:01:43] "GET / HTTP/1.1" 200 - 34.228.158.208 - - [13:01:48] "GET / HTTP/1.1" 200 - 54.198.223.114 - - [13:06:34] "GET / HTTP/1.1" 200 - 54.198.223.114 - - [13:06:39] "GET / HTTP/1.1" 200 - It should be noted that there is a lower limit to how often we can rotate the IPs on Amazon or any cloud provider for that matter. It takes a while for instances to boot and IP addresses to be reserved, associated, and become active. ProxyCannon has a hardcoded value of about 90 seconds to ensure the effective IP actually changes. In this article, we looked at Proxy cannon for staying under the radar while conducting brute-force attacks during an engagement. Becoming the Hacker is a playbook to help you become an ethical hacker and protect the web. Learn about the tricks of a web attacker. 6 common use cases of Reverse Proxy scenarios MarioNet: A browser-based attack that allows hackers to run malicious code even if users’ exit a web page Black Hat hackers used IPMI cards to launch JungleSec Ransomware, affects most of the Linux servers
Read more
  • 0
  • 0
  • 25152

article-image-how-ai-is-transforming-the-smart-cities-iot-tutorial
Natasha Mathur
23 Mar 2019
11 min read
Save for later

How AI is transforming the Smart Cities IoT? [Tutorial]

Natasha Mathur
23 Mar 2019
11 min read
According to techopedia, a smart city is a city that utilizes information and communication technologies so that it enhances the quality and performance of urban services (such as energy and transportation) so that there's a reduction in resource consumption, wastage, and overall costs. In this article, we will look at components of a smart city and its AI-powered-IoT use cases, how AI helps with the adaption of IoT in Smart cities, and an example of AI-powered-IoT solution. Deakin and AI Waer list four factors that contribute to the definition of a smart city: Using a wide range of electronic and digital technologies in the city infrastructure Employing Information and Communication Technology (ICT) to transform living and working environment Embedding ICT in government systems Implementing practices and policies that bring people and ICT together to promote innovation and enhance the knowledge that they offer Hence, a smart city would be a city that not only possesses ICT but also employs technology in a way that positively impacts the inhabitants. This article is an excerpt taken from the book 'Hands-On Artificial Intelligence for IoT' written by  Amita Kapoor.  The book explores building smarter systems by combining artificial intelligence and the Internet of Things—two of the most talked about topics today. Artificial Intelligence (AI), together with IoT, has the potential to address the key challenges posed by excessive urban population; they can help with traffic management, healthcare, energy crisis, and many other issues. IoT data and AI technology can improve the lives of the citizens and businesses that inhabit a smart city.  Let's see how. Smart city and its AI-powered-IoT use cases A smart city has lots of use cases for AI-powered IoT-enabled technology, from maintaining a healthier environment to enhancing public transport and safety. In the following diagram, you can see some the of use cases for a smart city: Smart city components Let's have a look at some of the most popular use cases that have already been implemented in smart cities across the world. Smart traffic management AI and IoT can implement smart traffic solutions to ensure that inhabitants of a smart city get from one point to another in the city as safely and efficiently as possible. Los Angeles, one of the most congested cities in the world, has implemented a smart traffic solution to control the flow of traffic. It has installed road-surface sensors and closed-circuit television cameras that send real-time updates about the traffic flow to a central traffic management system. The data feed from the sensors and cameras is analyzed, and it notifies the users of congestion and traffic signal malfunctions. In July 2018, the city further installed Advanced Transportation Controller (ATC) cabinets at each intersection. Enabled with vehicle-to-infrastructure (V2I) communications and 5G connectivity, this allows them to communicate with cars that have the traffic light information feature, such as Audi A4 or Q7. You can learn more about the Los Angeles smart transportation system from their website. The launch of automated vehicles embedded with sensors can provide both the location and speed of the vehicle; they can directly communicate with the smart traffic lights and prevent congestion. Additionally, using historical data, future traffic could be predicted and used to prevent any possible congestion. Smart parking Anyone living in a city must have felt the struggle of finding a parking spot, especially during the holiday time. Smart parking can ease the struggle. With road surface sensors embedded in the ground on parking spots, smart parking solutions can determine whether the parking spots are free or occupied and create a real-time parking map. The city of Adelaide installed a smart parking system in February 2018, they are also launching a mobile app: Park Adelaide, which will provide the user with accurate and real-time parking information. The app can provide users with the ability to locate, pay for, and even extend the parking session remotely. The smart parking system of the city of Adelaide aims to also improve traffic flow, reduce traffic congestion, and decrease carbon emissions. The details of the smart parking system are available in the city of Adelaide website. The San Francisco Municipal Transportation Agency (SAFTA) implemented SFpark a smart parking system. They use wireless sensors to detect real-time parking-space occupancy in metered spaces. Launched in the year 2013, SFpark has reduced weekday greenhouse gas emissions by 25%, the traffic volume has gone down, and drivers' search time has reduced by 50%. In London, the city of Westminster also established a smart parking system in the year 2014 in association with Machina Research. Earlier, drivers had to wait an average of 12 minutes, resulting in congestion and pollution, but since the installation of the smart parking system, there's no need to wait; drivers can find an available parking spot using the mobile. These are some of the use-cases mentioned. Other use-cases include smart waste management, smart policing, smart lighting, and smart governance. What can AI do for IoT adaption in smart cities? Building a smart city is not a one-day business, neither is it the work of one person or organization. It requires the collaboration of many strategic partners, leaders, and even citizens. Let's explore what the AI community can do, what are the areas that provide us with a career or entrepreneurship opportunity. Any IoT platform will necessarily require the following: A network of smart things (sensors, cameras, actuators, and so on) for gathering data Field (cloud) gateways that can gather the data from low power IoT devices, store it, and forward it securely to the cloud Streaming data processor for aggregating numerous data streams and distributing them to a data lake and control applications A data lake for storing all the raw data, even the ones that seem of no value yet A data warehouse that can clean and structure the collected data Tools for analyzing and visualizing the data collected by sensors AI algorithms and techniques for automating city services based on long-term data analysis and finding ways to improve the performance of control applications Control applications for sending commands to the IoT actuators User applications for connecting smart things and citizens Besides this, there will be issues regarding security and privacy, and the service provider will have to ensure that these smart services do not pose any threat to citizens' wellbeing. The services themselves should be easy to use and employ so that citizens can adopt them. As you can see, this offers a range of job opportunities, specifically for AI engineers. The IoT-generated data needs to be processed, and to benefit from it truly, we will need to go beyond monitoring and basic analysis. The AI tools will be required to identify patterns and hidden correlations in the sensor data. Analysis of historical sensor data using ML/AI tools can help in identifying trends and create predictive models based on them. These models can then be used by control applications that send commands to IoT devices' actuators. The process of building a smart city will be an iterative process, with more processing and analysis added at each iteration. Let's now have a look at an example of AI-powered-IoT solution. Detecting crime using San Francisco crime data The San Francisco city also has an open data portal providing data from different departments online. In this section, we take the dataset providing about 12 years (from January 2003 to May 2015) of crime reports from across all of San Francisco's neighborhoods and train a model to predict the category of crime that occurred. There are 39 discreet crime categories, thus it's a multi-class classification problem. We will use make use of Apache's PySpark and use its easy to use text processing features for this dataset. So the first step will be to create a Spark session: The first step is to import the necessary modules and create a Spark session: from pyspark.ml.classification import LogisticRegression as LR from pyspark.ml.feature import RegexTokenizer as RT from pyspark.ml.feature import StopWordsRemover as SWR from pyspark.ml.feature import CountVectorizer from pyspark.ml.feature import OneHotEncoder, StringIndexer, VectorAssembler from pyspark.ml import Pipeline from pyspark.sql.functions import col from pyspark.sql import SparkSession spark = SparkSession.builder \ .appName("Crime Category Prediction") \ .config("spark.executor.memory", "70g") \ .config("spark.driver.memory", "50g") \ .config("spark.memory.offHeap.enabled",True) \ .config("spark.memory.offHeap.size","16g") \ .getOrCreate() We load the dataset available in a csv file: data = spark.read.format("csv"). \ options(header="true", inferschema="true"). \ load("sf_crime_dataset.csv") data.columns The data contains nine columns: [Dates, Category, Descript, DayOfWeek, PdDistrict, Resolution, Address, X, Y], we will need only Category and Descript fields for training and testing dataset: drop_data = ['Dates', 'DayOfWeek', 'PdDistrict', 'Resolution', 'Address', 'X', 'Y'] data = data.select([column for column in data.columns if column not in drop_data]) data.show(5) Now the dataset we have has textual data, so we will need to perform text processing. The three important text processing steps are: tokenizing the data, remove the stop words and vectorize the words into vectors. We will use RegexTokenizer which will uses regex to tokenize the sentence into a list of words, since punctuation or special characters do not add anything to the meaning, we retain only the words containing alphanumeric content. There are some words like the, which will be very commonly present in the text, but not add any meaning to context. We can remove these words (also called stop words) using the inbuilt StopWordsRemover class. We use standard stop words ["http","https","amp","rt","t","c","the"]. And finally using the CountVectorizer, we convert the words to numeric vector (features). It's these numeric features that will be used as input to train the model. The output for our data is the Category column, but it's also textual with 36 distinct categories, and so, we need to convert it to one hot encoded vector; the PySpark's StringIndexer can be easily used for it. We add all these transformations into our data Pipeline: # regular expression tokenizer re_Tokenizer = RT(inputCol="Descript", outputCol="words", pattern="\\W") # stop words stop_words = ["http","https","amp","rt","t","c","the"] stop_words_remover = SWR(inputCol="words", outputCol="filtered").setStopWords(stop_words) # bag of words count count_vectors = CountVectorizer(inputCol="filtered", outputCol="features", vocabSize=10000, minDF=5) #One hot encoding the label label_string_Idx = StringIndexer(inputCol = "Category", outputCol = "label") # Create the pipeline pipeline = Pipeline(stages=[re_Tokenizer, stop_words_remover, count_vectors, label_string_Idx]) # Fit the pipeline to data. pipeline_fit = pipeline.fit(data) dataset = pipeline_fit.transform(data) dataset.show(5) Now, the data is ready, we split it into training and test dataset: # Split the data randomly into training and test data sets. (trainingData, testData) = dataset.randomSplit([0.7, 0.3], seed = 100) print("Training Dataset Size: " + str(trainingData.count())) print("Test Dataset Size: " + str(testData.count())) Let's fit a simple logistic regression model for it. On the test dataset, it provides a 97% accuracy. Yahoo!: # Build the model logistic_regrssor = LR(maxIter=20, regParam=0.3, elasticNetParam=0) # Train model with Training Data model = logistic_regrssor.fit(trainingData) # Make predictions on Test Data predictions = model.transform(testData) # evaluate the model on test data set evaluator = MulticlassClassificationEvaluator(predictionCol="prediction") evaluator.evaluate(predictions) AI is changing the way cities operate, deliver, and maintain public amenities, from lighting and transportation to connectivity and health services. However, the adoption can be obstructed by the selection of technology that doesn't efficiently work together or integrate with other city services. For cities to truly benefit from the potential that smart cities offer, a change in mindset is required. The authorities should plan longer and across multiple departments. The city of Barcelona is a prime example where the implementation of IoT systems created an estimated 47,000 jobs, saved €42.5 million on water, and generated an extra €36.5 million a year through smart parking. We can easily see that cities can benefit tremendously from the technological advances that utilize AI-powered IoT solutions. AI-powered IoT solutions can help connect cities and manage multiple infrastructure, and public services. In this article, we looked at use-cases of smart-cities from smart lighting and road traffic to connected public transport, and waste management. We also learned to use tools that can help categorize the data from the San Francisco crime reports done in a period of 12 years. If you want to explore more topics in the book, be sure to check out the book 'Hands-On Artificial Intelligence for IoT'. IBM Watson announces pre-trained AI tools to accelerate IoT operations Implementing cost-effective IoT analytics for predictive maintenance [Tutorial] AI and the Raspberry Pi: Machine Learning and IoT, What’s the Impact?
Read more
  • 0
  • 0
  • 34760

article-image-four-versions-of-wikipedia-goes-offline-in-a-protest-against-eu-copyright-directive-which-will-affect-free-speech-online
Savia Lobo
22 Mar 2019
5 min read
Save for later

Four versions of Wikipedia goes offline in a protest against EU copyright Directive which will affect free speech online

Savia Lobo
22 Mar 2019
5 min read
Yesterday, March 21, four versions of Wikipedia, German, Danish, Czech, and Slovak were blacked off as a move to oppose the recent EU Copyright Directive, which will be up for voting on Tuesday, March 26. These long-awaited updates to the copyright law include “important wins for the open community in the current text”, the Wikimedia foundation reports. However, “the inclusion of Articles 11 and 13 will harm the way people find and share information online”, Wikimedia further states. However, the major opposition is towards the controversial Article 13. Article 11 states that if a text contains more than a snippet from an article, it must be licensed and paid for by whoever quotes the text. “While each country can define "snippet" however it wants, the Directive does not stop countries from making laws that pass using as little as three words from a news story”, the Electronic Frontier Foundation mentions. Article 13 is, however, the most controversial and is all set to restructure how copyright works on the web. As of now, in order to take down content that is subject to copyright infringement, the rights holder just have to send a ‘takedown notice’. However, with Article 13 in place, there will be no protection for online services and also “relieves rights-holders of the need to check the Internet for infringement and send out notices. Instead, it says that online platforms have a duty to ensure that none of their users infringe copyright.” According to The Next Web, “To make people understand how serious the effects of the Copyright Reform will be if it’s passed, Reddit and Wikipedia will hinder access to their sites in the EU to mimic the effects of the directive.” Both Article 11 and 13 were reintroduced under the leadership of German Member of the European Parliament (MEP) Axel Voss. However, these had already been discarded as unworkable after expert advice. “Voss's insistence that Articles 11 and 13 be included in the final Directive has been a flashpoint for public anger, drawing criticism from the world's top technical, copyright, journalistic, and human rights experts and organizations”, the Electronic Frontier Foundation reports. “Critics say the politicians behind the legislation do not understand the breadth of the laws they are proposing, and that the directive, if implemented, will harm free expression online”, The Verge reports. Platforms such as Tumblr, YouTube, and many others, that host user-generated content will be under the radar if Article 13 is passed and will be legally responsible if the users upload copyrighted content. According to The Verge, “The only way to stop these uploads, say critics, will be to scan content before its uploaded, leading to the creation of filters that will likely be error-prone and abused by copyright trolls.” Many have protested against Article 13 in recent weeks. In Germany, about 3,500 people took out a rally in Berlin as a protest against the new copyright plans. Also, a petition ‘Save the Internet’ has already gathered more than five million signatures. Reddit has also taken an action against the Copyright Directive by flashing a simulated error message citing failure when Reddit desktop users in EU countries attempt to make a top-level post on Reddit. According to Reddit, “This experience, meant to mimic the automated filters that users would encounter should the Directive pass, will last through March 23rd, when IRL demonstrations are planned across Europe.” Julia Reda, a member of the European Parliament from Germany, in her blog post mentions, “For two years we’ve debated different drafts and versions of the controversial Articles 11 and 13. Now, there is no more ambiguity: This law will fundamentally change the internet as we know it – if it is adopted in the upcoming final vote. But we can still prevent that!” United Nations’ free-speech rapporteur, David Kaye, said, “Europe has a responsibility to modernize its copyright law to address the challenges of the digital age. But this should not be done at the expense of the freedom of expression that Europeans enjoy today… Article 13 of the proposed Directive appears destined to drive internet platforms toward monitoring and restriction of user-generated content even at the point of upload. Such sweeping pressure for pre-publication filtering is neither a necessary nor proportionate response to copyright infringement online.” A user on HackerNews writes, “I hope they win and that Article 11 and 13 will be removed. I think this is an important moment in the birth of EU democracy because it feels to me that one of the first times, there is a big public discussion about an issue and the people at the center aren't national politicians like Merkel or Macron but EU MEPs, namely Voss vs Reda. The EU has rightfully been criticized of not being democratic enough, and this discussion feels like it's very much democratic.” https://twitter.com/Wikipedia/status/1108595296068501504 Five EU countries oppose the EU copyright directive Reddit’s 2018 Transparency report includes copyright removals, restorations, and more! Drafts of Article 13 and the EU Copyright Directive have been finalized
Read more
  • 0
  • 0
  • 19267

article-image-google-facebook-working-hard-to-clean-image-after-media-backlash-from-attack
Fatema Patrawala
22 Mar 2019
8 min read
Save for later

Google and Facebook working hard to clean image after the media backlash from the Christchurch terrorist attack

Fatema Patrawala
22 Mar 2019
8 min read
Last Friday’s uncontrolled spread of horrific videos on the Christchurch mosque attack and a propaganda coup for espousing hateful ideologies raised questions about social media. The tech companies scrambled to take action on time due to the speed and volume of content which was uploaded, reuploaded and shared by the users worldwide. In Washington and Silicon Valley, the incident crystallized growing concerns about the extent to which government and market forces have failed to check the power of social media. The failure highlighted the social media companies struggle to police content that are massively lucrative and persistently vulnerable to outside manipulation despite years of promises to do better. After the white supremacist live-streamed the attack and uploaded the video to Facebook, Twitter, YouTube, and other platforms across the internet. These tech companies faced back lashes from the media and internet users worldwide, to an extent where they were regarded as complicit in promoting white supremacism too. In response to the backlash, Google and Facebook provides status report on what they went through when the video was reported, the kind of challenges they faced and what are the next steps to combat such incidents in future. Google’s report so far... Google in an email to Motherboard says it employs 10,000 people across to moderate the company’s platforms and products. They also described a process they would follow when a user reports a piece of potentially violating content—such as the attack video; which is The user flagged report will go to a human moderator to assess. The moderator is instructed to flag all pieces of content related to the attack as “Terrorist Content,” including full-length or sections of the manifesto. Because of the document’s length the email tells moderators not to spend an extensive amount of time trying to confirm whether a piece of content does contain part of the manifesto. Instead, if the moderator is unsure, they should err on the side of caution and still label the content as “Terrorist Content,” which will then be reviewed by a second moderator. The second moderator is told to take time to verify that it is a piece of the manifesto, and appropriately mark the content as terrorism no matter how long or short the section may be. Moderators are told to mark the manifesto or video as terrorism content unless there is an Educational, Documentary, Scientific, or Artistic (EDSA) context to it. Further Google adds that they want to preserve journalistic or educational coverage of the event, but does not want to allow the video or manifesto itself to spread throughout the company’s services without additional context. Google at some point had taken the unusual step of automatically rejecting any footage of violence from the attack video, cutting out the process of a human determining the context of the clip. If, say, a news organization was impacted by this change, the outlet could appeal the decision, Google commented. “We made the call to basically err on the side of machine intelligence, as opposed to waiting for human review,” YouTube’s Product Officer Neal Mohan told the Washington Post in an article published Monday. Google also tweaked the search function to show results from authoritative news sources. It suspended the ability to search for clips by upload date, making it harder for people to find copies of the attack footage. "Since Friday’s horrific tragedy, we’ve removed tens of thousands of videos and terminated hundreds of accounts created to promote or glorify the shooter," a YouTube spokesperson said. “Our teams are continuing to work around the clock to prevent violent and graphic content from spreading, we know there is much more work to do,” the statement added. Facebook’s update so far... Facebook on Wednesday also shared an update on how they have been working with the New Zealand Police to support their investigation. It provided additional information on how their products were used to circulate videos and how they plan to improve them. So far Facebook has provided the following information: The video was viewed fewer than 200 times during the live broadcast. No users reported the video during the live broadcast. Including the views during the live broadcast, the video was viewed about 4,000 times in total before being removed from Facebook. Before Facebook was alerted to the video, a user on 8chan posted a link to a copy of the video on a file-sharing site. The first user report on the original video came in 29 minutes after the video started, and 12 minutes after the live broadcast ended. In the first 24 hours, Facebook removed more than 1.2 million videos of the attack at upload, which were therefore prevented from being seen on our services. Approximately 300,000 additional copies were removed after they were posted. As there were questions asked to Facebook about why artificial intelligence (AI) didn’t detect the video automatically. Facebook says AI has made massive progress over the years to proactively detect the vast majority of the content it can remove. But it’s not perfect. “To achieve that we will need to provide our systems with large volumes of data of this specific kind of content, something which is difficult as these events are thankfully rare.” says Guy Rosen VP Product Management at Facebook. Guy further adds, “AI is an incredibly important part of our fight against terrorist content on our platforms, and while its effectiveness continues to improve, it is never going to be perfect. People will continue to be part of the equation, whether it’s the people on our team who review content, or people who use our services and report content to us. That’s why last year Facebook more than doubled the number of people working on safety and security to over 30,000 people, including about 15,000 content reviewers to report content that they find disturbing.” Facebook further plans to: Improve the image and video matching technology so that they can stop the spread of viral videos of such nature, regardless of how they were originally produced. React faster to this kind of content on a live streamed video. Continue to combat hate speech of all kinds on their platform. Expand industry collaboration through the Global Internet Forum to Counter Terrorism (GIFCT). Challenges Google and Facebook faced to report the video content According to Motherboard, Google saw an unprecedented number of attempts to post footage from the attack, sometimes as fast as a piece of content per second. But the challenge they faced was to block access to the killer’s so-called manifesto, a 74-page document that spouted racist views and explicit calls for violence. Google described the difficulties of moderating the manifesto, pointing to its length and the issue of users sharing the snippets of the manifesto that Google’s content moderators may not immediately recognise. “The manifesto will be particularly challenging to enforce against given the length of the document and that you may see various segments of various lengths within the content you are reviewing,” says Google. A source with knowledge of Google’s strategy for moderating the New Zealand attack material said this can complicate moderation efforts because some outlets did use parts of the video and manifesto. UK newspaper The Daily Mail let readers download the terrorist’s manifesto directly from the paper’s own website, and Sky News Australia aired parts of the attack footage, BuzzFeed News reported. On the other hand Facebook faces a challenge to automatically discern such content from visually similar, innocuous content. For example if thousands of videos from live-streamed video games are flagged by the systems, reviewers could miss the important real-world videos where they could alert first responders to get help on the ground. Another challenge for Facebook is similar to what Google faces, which is the proliferation of many different variants of videos makes it difficult for the image and video matching technology to prevent spreading further. Facebook found that a core community of bad actors working together to continually re-upload edited versions of the video in ways designed to defeat their detection. Second, a broader set of people distributed the video and unintentionally made it harder to match copies. Websites and pages, eager to get attention from people seeking out the video, re-cut and re-recorded the video into various formats. In total, Facebook found and blocked over 800 visually-distinct variants of the video that were circulating. Both companies seem to be working hard to improve their products and gain user’s trust and confidence back. How social media enabled and amplified the Christchurch terrorist attack Google to be the founding member of CDF (Continuous Delivery Foundation) Google announces the stable release of Android Jetpack Navigation
Read more
  • 0
  • 0
  • 14346
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €18.99/month. Cancel anytime
article-image-qt-installation-on-different-platforms-tutorial
Amrata Joshi
22 Mar 2019
10 min read
Save for later

Qt installation on different platforms [Tutorial]

Amrata Joshi
22 Mar 2019
10 min read
Qt provides a different look for mobile and embedded devices where users expect a different style of presentation. This is controlled within the framework so the developer can concentrate on developing a single application. The Qt framework is released in two separate distributions, one commercial and one open source (known as dual licensing). In this manner, they can support open source-compliant applications for free, while providing unrestricted usage for closed source commercial projects. Before the year 2000 (with the release of 2.2), the source code for the free distribution had been under various licenses that some groups considered incompatible with common open source initiatives. For the 2.2 release, it was changed to GPL licensing, which settled any concerns about the group's commitment to true open source freedoms. In 2007, Qt 4.5 was released and they added LGPL as an option for developers who prefer the more permissive license. This article is an excerpt taken from the book Hands-On GUI Application Development in Go. This book covers the benefits and complexities of building native graphical applications, the procedure for building platform and developing graphical Windows applications using Walk.  This article covers the basics of therecipe/qt, multiple platforms, installation of qt (the bindings), and much more. Getting started with therecipe/qt To begin our exploration of Qt and the binding to Go, we'll build a simple hello world application. To be able to do so, we first need to install therecipe/qt, which depends on various prerequisites that we must first set up. As with Go-GTK, we'll be relying on a native library that requires that we both set up the CGo functionality and install the Qt library appropriate for the current platform. Preparing CGo The Qt Go bindings, like many of the other toolkits featured in this book, require the presence of CGo to utilize native libraries. On a full development system, it's likely that this is already set up. Installing Qt The Qt website offers various methods of installation, including a customized online installer available to anyone with a Qt account (which is free to sign up for). Typically, a Qt installation comes with Qt Creator (the project IDE), the GUI designer, additional tools, and examples. Visiting the preceding site will automatically detect your system and suggest the most appropriate download (this is normally the best option). Be aware that the Qt installation can be quite large. If you don't have at least 40 GB of space on your hard drive, you need to make a little space before installing. Some operating systems offer Qt libraries and tools as part of their package manager, which often provides a more lightweight installation that'll automatically stay up to date. Installing Qt on multiple platforms macOS On Apple macOS, the best approach to installation is to use the installer application available at the Qt download site. Visit www.qt.io/download and download the macOS installer. Once it has downloaded, open the package and run the program inside; this will install the selected compilers, tools, and supporting applications. If you encounter any errors during installation, the first step would be to check that your Xcode installation is complete and up to date. Windows Installing on Windows is more straightforward than some of the other toolkits we've looked at, as the Qt installer has a mingw package bundled to set up most of the compiling requirements (though it's still recommended to have your own compiler set up for the binding phase next). To install it, go to the download page listed previously and access the Windows installer. Run the downloaded executable and follow the onscreen instructions. It's recommended to install to the default location. Once that's done, you're ready to set up the bindings. Linux Using the online installer from Qt's website is the easiest approach, though it may be possible to install through your system's package manager (if you want to try the package manager approach, then first read the Qt Linux documentation. On most Linux platforms, the Qt downloads website will correctly detect the platform and offer a simple run installer. After downloading the file, you should make it executable and then run it: On Linux, you need to make the install file executable and run it This will start the installer just as on macOS; from here, follow the onscreen instructions and complete the installation. License / Qt account When it comes to the login screen, then you should enter your Qt account details if you have them. If you qualify for their open source license (GPL or LGPL), you can skip this step—to do so; make sure the email and password fields are empty. Installing qt (the bindings) To use qt (the Go Qt bindings), we need to download the project and its dependencies and then run a setup script to configure and compile the library. If using Windows, it's recommended to use the MSYS2 Terminal. If you installed the Qt download to anything other than the default location, then make sure to set up the QT_DIR environment variable to the location you chose. First, the library and its dependencies should be installed using the go tools, by running go get github.com/sirupsen/logrus and go get github.com/therecipe/qt. Once the download has completed, we need to run the qtsetup tool, which is included in the qt project; so, within the cmd/qtsetup folder, execute go run main.go. Using a Linux Terminal, it should look something like this: Executing the qtsetup script for therecipe/qt bindings Once this process completes, the bindings should be ready to use. If you encounter errors, then it's probably because the Qt tools aren't correctly installed or the location was customized and you forgot to set the QT_DIR environment variable. Build To build our first qt application with Go, let's make another Hello World application. As with previous examples, we'll make use of a simple vertical box layout within a single application window. The following code should be sufficient to load your first application: package main import ( "os" "github.com/therecipe/qt/widgets" ) func main() { app := widgets.NewQApplication(len(os.Args), os.Args) window := widgets.NewQMainWindow(nil, 0) window.SetWindowTitle("Hello World") widget := widgets.NewQWidget(window, 0) widget.SetLayout(widgets.NewQVBoxLayout()) window.SetCentralWidget(widget) label := widgets.NewQLabel2("Hello World!", window, 0) widget.Layout().AddWidget(label) button := widgets.NewQPushButton2("Quit", window) button.ConnectClicked(func(bool) { app.QuitDefault() }) widget.Layout().AddWidget(button) window.Show() widgets.QApplication_Exec() } Let's note a few details from this code snippet. You'll see that each of the widget constructor functions takes (typically) two parameters, each is the parent widget and a flags parameter. Additional types passed in will usually be added before these values with a note in the function name that there are additional parameters. For example, widgets.NewQLabel2(title, parent, flags) is equivalent to widgets.NewQLabel(parent, flags).SetTitle(title). Additionally, you'll see that the layout is applied to a new widgets.QWidget through SetLayout(layout), and that's set to the window content through window.SetCentralWidget(widget). To load the display and run the application, we call window.Show() and then widgets.QApplication_Exec(). This file is built in the usual way with go build hello.go: Building is simple though the output file is rather large The file built is quite large due to the size of the Qt framework. This will be reduced significantly when packaging for a specific distribution. Run The output of the build phase is a binary that can be executed on the current computer, either on the command line or by double-clicking in a file manager. Additionally, you could execute it directly with go run hello.go—either way, you should see a simple window, as shown here: qt Hello on Linux Running on macOS At this stage, the binaries can be executed on a computer with the same architecture that also has Qt installed. Object model and event handling The Qt framework is written using the C++ language, and so much of its architecture will be familiar to those who've coded in C++ before. It's important to note that Go isn't a complete object-oriented language and, as such, doesn't match these capabilities directly. In particular, we should look at inheritance as it's important to the Qt object model. Inheritance The Qt API is a fully object-oriented model that makes heavy use of the inheritance model. While Go doesn't truly support object-oriented inheritance in the traditional manner, its composition approach is very powerful and works well in its place. The result means that you probably won't notice the difference! Memory management As you'll have noticed in the preceding example, each widget expects the parent to be passed to the constructing function. This enables the Qt framework to handle the tidying up and freeing of memory when a tree of widgets is removed. QObject (which is the base object for all of the Qt API) keeps track of its child objects and so, when being removed, can remove its children too. This makes the creation and deletion of complex widget hierarchies easier to handle correctly. To make use of this feature, you should always remember to pass the parent object to a widget's constructor (the Go functions starting with New...), despite the fact that passing nil may look like it's working. Signals and slots Qt is similar to GTK+, an event-driven framework and uses signals extensively to handle event management and data communications. In Qt, this concept is split into signals and slots; a signal is what will be generated when an event occurs and a slot is what can receive a signal. The action of setting a slot to receive a signal is called connecting and this causes a slot function to be called when its connected signal is invoked. In Qt, these are typed events meaning that each signal has a list of type parameters associated with it. When the signal is defined, this type is set and any slot wishing to connect to the signal will need to have the same type. s.ConnectMySignal( func(msg string) { log.Println("Signalled message", msg) } ) Signals and slots are what power user interfaces generated with Qt Designer and are the recommended way of handling multi-threaded applications. A signal may fire from a background thread and the user interface code can connect this signal to its own slot—in essence, listening for the signal. When the signal fires, any associated data (parameters to the signal) will be passed from one thread to another so it can be used safely within the GUI updates. As Qt is a lightweight binding to the Qt API, the Go-specific documentation is minimal but you can find out a lot more about the Qt design and all of the classes available in the official documentation available at Qt's blog post. In this article, we have learned about the Qt framework and the multiple platforms, therecipe/qt, installation of qt (the bindings), and much more. To know more about Go-GTK and platforms with GTK, check out the book Hands-On GUI Application Development in Go. Qt Creator 4.9 Beta released with QML support, programming language support and more! Qt Design Studio 1.1 released with Qt Photoshop bridge, updated timeline and more Qt for Python 5.12 released with PySide2, Qt GUI and more
Read more
  • 0
  • 0
  • 23592

article-image-debugging-and-profiling-python-scripts-tutorial
Melisha Dsouza
21 Mar 2019
12 min read
Save for later

Debugging and Profiling Python Scripts [Tutorial]

Melisha Dsouza
21 Mar 2019
12 min read
Debugging and profiling play an important role in Python development. The debugger helps programmers to analyze the complete code. The debugger sets the breakpoints whereas the profilers run our code and give us the details of the execution time. The profilers will identify the bottlenecks in your programs. In this tutorial, we'll learn about the pdb Python debugger, cProfile module, and timeit module to time the execution of Python code. This tutorial is an excerpt from a book written by Ganesh Sanjiv Naik titled Mastering Python Scripting for System Administrators. This book will show you how to leverage Python for tasks ranging from text processing, network administration, building GUI, web-scraping as well as database administration including data analytics & reporting. Python debugging techniques Debugging is a process that resolves the issues that occur in your code and prevent your software from running properly. In Python, debugging is very easy. The Python debugger sets conditional breakpoints and debugs the source code one line at a time. We'll debug our Python scripts using a pdb module that's present in the Python standard library. To better debug a Python program, various techniques are available. We're going to look at four techniques for Python debugging: print() statement: This is the simplest way of knowing what's exactly happening so you can check what has been executed. logging: This is like a print statement but with more contextual information so you can understand it fully. pdb debugger: This is a commonly used debugging technique. The advantage of using pdb is that you can use pdb from the command line, within an interpreter, and within a program. IDE debugger: IDE has an integrated debugger. It allows developers to execute their code and then the developer can inspect while the program executes. Error handling (exception handling) In this section, we're going to learn how Python handles exceptions. An exception is an error that occurs during program execution. Whenever any error occurs, Python generates an exception that will be handled using a try…except block. Some exceptions can't be handled by programs so they result in error messages. Now, we are going to see some exception examples. In your Terminal, start the python3 interactive console and we will see some exception examples: student@ubuntu:~$ python3 Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> >>> 50 / 0 Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: division by zero >>> >>> 6 + abc*5 Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'abc' is not defined >>> >>> 'abc' + 2 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: Can't convert 'int' object to str implicitly >>> >>> import abcd Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named 'abcd' >>> These are some examples of exceptions. Now, we will see how we can handle the exceptions. Whenever errors occur in your Python program, exceptions are raised. We can also forcefully raise an exception using raise keyword. Now we are going to see a try…except block that handles an exception. In the try block, we will write a code that may generate an exception. In the except block, we will write a solution for that exception. The syntax for try…except is as follows: try: statement(s) except: statement(s) A try block can have multiple except statements. We can handle specific exceptions also by entering the exception name after the except keyword. The syntax for handling a specific exception is as follows: try: statement(s) except exception_name: statement(s) We are going to create an exception_example.py script to catch ZeroDivisionError. Write the following code in your script: a = 35 b = 57 try: c = a + b print("The value of c is: ", c) d = b / 0 print("The value of d is: ", d) except: print("Division by zero is not possible") print("Out of try...except block") Run the script as follows and you will get the following output: student@ubuntu:~$ python3 exception_example.py The value of c is: 92 Division by zero is not possible Out of try...except block Debuggers tools There are many debugging tools supported in Python: winpdb pydev pydb pdb gdb pyDebug In this section, we are going to learn about pdb Python debugger. pdb module is a part of Python's standard library and is always available to use. The pdb debugger The pdb module is used to debug Python programs. Python programs use pdb interactive source code debugger to debug the programs. pdb sets breakpoints and inspects the stack frames, and lists the source code. Now we will learn about how we can use the pdb debugger. There are three ways to use this debugger: Within an interpreter From a command line Within a Python script We are going to create a pdb_example.py script and add the following content in that script: class Student: def __init__(self, std): self.count = std def print_std(self): for i in range(self.count): print(i) return if __name__ == '__main__': Student(5).print_std() Using this script as an example to learn Python debugging, we will see how we can start the debugger in detail. Within an interpreter To start the debugger from the Python interactive console, we are using run() or runeval(). Start your python3 interactive console. Run the following command to start the console: $ python3 Import our pdb_example script name and the pdb module. Now, we are going to use run() and we are passing a string expression as an argument to run() that will be evaluated by the Python interpreter itself: student@ubuntu:~$ python3 Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> >>> import pdb_example >>> import pdb >>> pdb.run('pdb_example.Student(5).print_std()') > <string>(1)<module>() (Pdb) To continue debugging, enter continue after the (Pdb) prompt and press Enter. If you want to know the options we can use in this, then after the (Pdb) prompt press the Tab key twice. Now, after entering continue, we will get the output as follows: student@ubuntu:~$ python3 Python 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609] on linux Type "help", "copyright", "credits" or "license" for more information. >>> >>> import pdb_example >>> import pdb >>> pdb.run('pdb_example.Student(5).print_std()') > <string>(1)<module>() (Pdb) continue 0 1 2 3 4 >>> From a command line The simplest and most straightforward way to run a debugger is from a command line. Our program will act as input to the debugger. You can use the debugger from command line as follows: $ python3 -m pdb pdb_example.py When you run the debugger from the command line, source code will be loaded and it will stop the execution on the first line it finds. Enter continue to continue the debugging. Here's the output: student@ubuntu:~$ python3 -m pdb pdb_example.py > /home/student/pdb_example.py(1)<module>() -> class Student: (Pdb) continue 0 1 2 3 4 The program finished and will be restarted > /home/student/pdb_example.py(1)<module>() -> class Student: (Pdb) Within a Python script The previous two techniques will start the debugger at the beginning of a Python program. But this third technique is best for long-running processes. To start the debugger within a script, use set_trace(). Now, modify your pdb_example.py file as follows: import pdb class Student: def __init__(self, std): self.count = std def print_std(self): for i in range(self.count): pdb.set_trace() print(i) return if __name__ == '__main__': Student(5).print_std() Now, run the program as follows: student@ubuntu:~$ python3 pdb_example.py > /home/student/pdb_example.py(10)print_std() -> print(i) (Pdb) continue 0 > /home/student/pdb_example.py(9)print_std() -> pdb.set_trace() (Pdb) set_trace() is a Python function, therefore you can call it at any point in your program. So, these are the three ways by which you can start a debugger. Debugging basic program crashes In this section, we are going to see the trace module. The trace module helps in tracing the program execution. So, whenever your Python program crashes, we can understand where it crashes. We can use trace module by importing it into your script as well as from the command line. Now, we will create a script named trace_example.py and write the following content in the script: class Student: def __init__(self, std): self.count = std def go(self): for i in range(self.count): print(i) return if __name__ == '__main__': Student(5).go() The output will be as follows: student@ubuntu:~$ python3 -m trace --trace trace_example.py --- modulename: trace_example, funcname: <module> trace_example.py(1): class Student: --- modulename: trace_example, funcname: Student trace_example.py(1): class Student: trace_example.py(2): def __init__(self, std): trace_example.py(5): def go(self): trace_example.py(10): if __name__ == '__main__': trace_example.py(11): Student(5).go() --- modulename: trace_example, funcname: init trace_example.py(3): self.count = std --- modulename: trace_example, funcname: go trace_example.py(6): for i in range(self.count): trace_example.py(7): print(i) 0 trace_example.py(6): for i in range(self.count): trace_example.py(7): print(i) 1 trace_example.py(6): for i in range(self.count): trace_example.py(7): print(i) 2 trace_example.py(6): for i in range(self.count): trace_example.py(7): print(i) 3 trace_example.py(6): for i in range(self.count): trace_example.py(7): print(i) 4 So, by using trace --trace at the command line, the developer can trace the program line-by-line. So, whenever the program crashes, the developer will know the instance where it crashes. Profiling and timing programs Profiling a Python program means measuring an execution time of a program. It measures the time spent in each function. Python's cProfile module is used for profiling a Python program. The cProfile module As discussed previously, profiling means measuring the execution time of a program. We are going to use the cProfile Python module for profiling a program. Now, we will write a cprof_example.py script and write the following code in it: mul_value = 0 def mul_numbers( num1, num2 ): mul_value = num1 * num2; print ("Local Value: ", mul_value) return mul_value mul_numbers( 58, 77 ) print ("Global Value: ", mul_value) Run the program and you will see the output as follows: student@ubuntu:~$ python3 -m cProfile cprof_example.py Local Value: 4466 Global Value: 0 6 function calls in 0.000 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.000 0.000 cprof_example.py:1(<module>) 1 0.000 0.000 0.000 0.000 cprof_example.py:2(mul_numbers) 1 0.000 0.000 0.000 0.000 {built-in method builtins.exec} 2 0.000 0.000 0.000 0.000 {built-in method builtins.print} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} So, using cProfile, all functions that are called will get printed with the time spent on each function. Now, we will see what these column headings mean: ncalls: Number of calls tottime: Total time spent in the given function percall: Quotient of tottime divided by ncalls cumtime: Cumulative time spent in this and all subfunctions percall: Quotient of cumtime divided by primitive calls filename:lineno(function): Provides the respective data of each function timeit timeit is a Python module used to time small parts of your Python script. You can call timeit from the command line as well as import the timeit module into your script. We are going to write a script to time a piece of code. Create a timeit_example.py script and write the following content into it: import timeit prg_setup = "from math import sqrt" prg_code = ''' def timeit_example(): list1 = [] for x in range(50): list1.append(sqrt(x)) ''' # timeit statement print(timeit.timeit(setup = prg_setup, stmt = prg_code, number = 10000)) Using timeit, we can decide what piece of code we want to measure the performance of. So, we can easily define the setup code as well as the code snippet on which we want to perform the test separately. The main code runs 1 million times, which is the default time, whereas the setup code runs only once. Making programs run faster There are various ways to make your Python programs run faster, such as the following: Profile your code so you can identify the bottlenecks Use built-in functions and libraries so the interpreter doesn't need to execute loops Avoid using globals as Python is very slow in accessing global variables Use existing packages Summary In this tutorial, we learned about the importance of debugging and profiling programs. We learned what the different techniques available for debugging are. We learned about the pdb Python debugger and how to handle exceptions and how to use the cProfile and timeit modules of Python while profiling and timing our scripts. We also learned how to make your scripts run faster. To learn how to to use the latest features of Python and be able to build powerful tools that will solve challenging, real-world tasks, check out our book Mastering Python Scripting for System Administrators. 5 blog posts that could make you a better Python programmer Using Python Automation to interact with network devices [Tutorial] 4 tips for learning Data Visualization with Python
Read more
  • 0
  • 0
  • 68721

article-image-how-to-remotely-monitor-hosts-over-telnet-and-ssh-tutorial
Melisha Dsouza
20 Mar 2019
14 min read
Save for later

How to remotely monitor hosts over Telnet and SSH [Tutorial]

Melisha Dsouza
20 Mar 2019
14 min read
In this tutorial, you will learn how to carry out basic configurations on a server with Telnet and SSH configured. We will begin by using the Telnet module, after which we will implement the same configurations using the preferred method: SSH using different modules in Python. You will also learn about how telnetlib, subprocess, fabric, Netmiko, and paramiko modules work.  This tutorial is an excerpt from a book written by Ganesh Sanjiv Naik titled Mastering Python Scripting for System Administrators. This book will take you through a set of specific software patterns and  you will learn, in detail, how to apply these patterns and build working software on top of a serverless system. The telnetlib() module In this section, we are going to learn about the Telnet protocol and then we will do Telnet operations using the telnetlib module over a remote server. Telnet is a network protocol that allows a user to communicate with remote servers. It is mostly used by network administrators to remotely access and manage devices. To access the device, run the Telnet command with the IP address or hostname of a remote server in your Terminal. Telnet uses TCP on the default port number 23. To use Telnet, make sure it is installed on your system. If not, run the following command to install it: $ sudo apt-get install telnetd To run Telnet using the simple Terminal, you just have to enter the following command: $ telnet ip_address_of_your_remote_server Python has the telnetlib module to perform Telnet functions through Python scripts. Before telnetting your remote device or router, make sure they are configured properly and, if not, you can do basic configuration by using the following command in the router's Terminal: configure terminal enable password 'set_Your_password_to_access_router' username 'set_username' password 'set_password_for_remote_access' line vty 0 4 login local transport input all interface f0/0 ip add 'set_ip_address_to_the_router' 'put_subnet_mask' no shut end show ip interface brief Now, let's see the example of Telnetting a remote device. For that, create a telnet_example.py script and write following content in it: import telnetlib import getpass import sys HOST_IP = "your host ip address" host_user = input("Enter your telnet username: ") password = getpass.getpass() t = telnetlib.Telnet(HOST_IP) t.read_until(b"Username:") t.write(host_user.encode("ascii") + b"\n") if password: t.read_until(b"Password:") t.write(password.encode("ascii") + b"\n") t.write(b"enable\n") t.write(b"enter_remote_device_password\n") #password of your remote device t.write(b"conf t\n") t.write(b"int loop 1\n") t.write(b"ip add 10.1.1.1 255.255.255.255\n") t.write(b"int loop 2\n") t.write(b"ip add 20.2.2.2 255.255.255.255\n") t.write(b"end\n") t.write(b"exit\n") print(t.read_all().decode("ascii") ) Run the script and you will get the output as follows: student@ubuntu:~$ python3 telnet_example.py Output: Enter your telnet username: student Password: server>enable Password: server#conf t Enter configuration commands, one per line. End with CNTL/Z. server(config)#int loop 1 server(config-if)#ip add 10.1.1.1 255.255.255.255 server(config-if)#int loop 23 server(config-if)#ip add 20.2.2.2 255.255.255.255 server(config-if)#end server#exit In the preceding example, we accessed and configured a Cisco router using the telnetlib module. In this script, first, we took the username and password from the user to initialize the Telnet connection with a remote device. When the connection was established, we did further configuration on the remote device. After telnetting, we will be able to access a remote server or device. But there is one very important disadvantage of this Telnet protocol, and that is all the data, including usernames and passwords, is sent over a network in a text manner, which may cause a security risk. Because of that, nowadays Telnet is rarely used and has been replaced by a very secure protocol named Secure Shell, known as SSH. Install SSH by running the following command in your Terminal: $ sudo apt install ssh Also, on a remote server where the user wants to communicate, an SSH server must be installed and running. SSH uses the TCP protocol and works on port number 22 by default. You can run the ssh command through the Terminal as follows: $ ssh host_name@host_ip_address Now, you will learn to do SSH by using different modules in Python, such as subprocess, fabric, Netmiko, and Paramiko. Now, we will see those modules one by one. The subprocess.Popen() module The Popen class handles the process creation and management. By using this module, developers can handle less common cases. The child program execution will be done in a new process. To execute a child program on Unix/Linux, the class will use the os.execvp() function. To execute a child program in Windows, the class will use the CreateProcess() function. Now, let's see some useful arguments of subprocess.Popen(): class subprocess.Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0) Let's look at each argument: args: It can be a sequence of program arguments or a single string. If args is a sequence, the first item in args is executed. If args is a string, it recommends to pass args as a sequence. shell: The shell argument is by default set to False and it specifies whether to use shell for execution of the program. If shell is True, it recommends to pass args as a string. In Linux, if shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. bufsize: If bufsize is 0 (by default, it is 0), it means unbuffered and if bufsize is 1, it means line buffered. If bufsize is any other positive value, use a buffer of the given size. If bufsize is any other negative value, it means fully buffered. executable: It specifies that the replacement program to be executed. stdin, stdout, and stderr: These arguments define the standard input, standard output, and standard error respectively. preexec_fn: This is set to a callable object and will be called just before the child is executed in the child process. close_fds: In Linux, if close_fds is true, all file descriptors except 0, 1, and 2 will be closed before the child process is executed. In Windows, if close_fds is true then the child process will inherit no handles. env: If the value is not None, then mapping will define environment variables for new process. universal_newlines: If the value is True then stdout and stderr will be opened as text files in newlines mode. Now, we are going to see an example of subprocess.Popen(). For that, create a ssh_using_sub.py  script and write the following content in it: import subprocess import sys HOST="your host username@host ip" COMMAND= "ls" ssh_obj = subprocess.Popen(["ssh", "%s" % HOST, COMMAND], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE) result = ssh_obj.stdout.readlines() if result == []: err = ssh_obj.stderr.readlines() print(sys.stderr, "ERROR: %s" % err) else: print(result) Run the script and you will get the output as follows: student@ubuntu:~$ python3 ssh_using_sub.py Output : student@192.168.0.106's password: [b'Desktop\n', b'Documents\n', b'Downloads\n', b'examples.desktop\n', b'Music\n', b'Pictures\n', b'Public\n', b'sample.py\n', b'spark\n', b'spark-2.3.1-bin-hadoop2.7\n', b'spark-2.3.1-bin-hadoop2.7.tgz\n', b'ssh\n', b'Templates\n', b'test_folder\n', b'test.txt\n', b'Untitled1.ipynb\n', b'Untitled.ipynb\n', b'Videos\n', b'work\n'] In the preceding example, first, we imported the subprocess module, then we defined the host address where you want to establish the SSH connection. After that, we gave one simple command that executed over the remote device. After all this was set up, we put this information in the subprocess.Popen() function. This function executed the arguments defined inside that function to create a connection with the remote device. After the SSH connection was established, our defined command was executed and provided the result. Then we printed the result of SSH on the Terminal, as shown in the output. SSH using fabric module Fabric is a Python library as well as a command-line tool for the use of SSH. It is used for system administration and application deployment over the network. We can also execute shell commands over SSH. To use fabric module, first you have to install it using the following command: $ pip3 install fabric3 Now, we will see an example. Create a fabfile.py script and write the following content in it: from fabric.api import * env.hosts=["host_name@host_ip"] env.password='your password' def dir(): run('mkdir fabric') print('Directory named fabric has been created on your host network') def diskspace(): run('df') Run the script and you will get the output as follows: student@ubuntu:~$ fab dir Output: [student@192.168.0.106] Executing task 'dir' [student@192.168.0.106] run: mkdir fabric Done. Disconnecting from 192.168.0.106... done. In the preceding example, first, we imported the fabric.api module, then we set the hostname and password to get connected with the host network. After that, we set a different task to perform over SSH. Therefore, to execute our program instead of the Python3 fabfile.py, we used the fab utility (fab dir), and after that we stated that the required tasks should be performed from our fabfile.py.  In our case, we performed the dir task, which creates a directory with the name 'fabric' on your remote network. You can add your specific task in your Python file. It can be executed using the fab utility of the fabric module. SSH using the Paramiko library Paramiko is a library that implements the SSHv2 protocol for secure connections to remote devices. Paramiko is a pure Python interface around SSH. Before using Paramiko, make sure you have installed it properly on your system. If it is not installed, you can install it by running the following command in your Terminal: $ sudo pip3 install paramiko Now, we will see an example of using paramiko. For this paramiko connection, we are using a Cisco device. Paramiko supports both password-based and  key-pair based authentication for a secure connection with the server. In our script, we are using password-based authentication, which means we check for a password and, if available, authentication is attempted using plain username/password authentication. Before we are going to do SSH to your remote device or multi-layer router, make sure they are configured properly and, if not, you can do basic configuration by using the following command in a multi-layer router Terminal: configure t ip domain-name cciepython.com crypto key generate rsa How many bits in the modulus [512]: 1024 interface range f0/0 - 1 switchport mode access switchport access vlan 1 no shut int vlan 1 ip add 'set_ip_address_to_the_router' 'put_subnet_mask' no shut exit enable password 'set_Your_password_to_access_router' username 'set_username' password 'set_password_for_remote_access' username 'username' privilege 15 line vty 0 4 login local transport input all end Now, create a pmiko.py script and write the following content in it: import paramiko import time ip_address = "host_ip_address" usr = "host_username" pwd = "host_password" c = paramiko.SSHClient() c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) c.connect(hostname=ip_address,username=usr,password=pwd) print("SSH connection is successfully established with ", ip_address) rc = c.invoke_shell() for n in range (2,6): print("Creating VLAN " + str(n)) rc.send("vlan database\n") rc.send("vlan " + str(n) + "\n") rc.send("exit\n") time.sleep(0.5) time.sleep(1) output = rc.recv(65535) print(output) c.close Run the script and you will get the output as follows: student@ubuntu:~$ python3 pmiko.py Output: SSH connection is successfuly established with 192.168.0.70 Creating VLAN 2 Creating VLAN 3 Creating VLAN 4 Creating VLAN 5 In the preceding example, first, we imported the paramiko module, then we defined the SSH credentials required to connect the remote device. After providing credentials, we created an instance 'c' of paramiko.SSHclient(), which is the primary client used to establish connections with the remote device and execute commands or operations. The creation of an SSHClient object allows us to establish remote connections using the .connect() function. Then, we set the policy paramiko connection because, by default, paramiko.SSHclient sets the SSH policy in reject policy state. That causes the policy to reject any SSH connection without any validation. In our script, we are neglecting this possibility of SSH connection drop by using the AutoAddPolicy() function that automatically adds the server's host key without prompting it. We can use this policy for testing purposes only, but this is not a good option in a production environment because of security purpose. When an SSH connection is established, you can do any configuration or operation that you want on your device. Here, we created a few virtual LANs on a remote device. After creating VLANs, we just closed the connection. SSH using the Netmiko library In this section, we will learn about Netmiko. The Netmiko library is an advanced version of Paramiko. It is a multi_vendor library that is based on Paramiko. Netmiko simplifies SSH connection to a network device and takes particular operation on the device. Before going doing SSH to your remote device or multi-layer router, make sure they are configured properly and, if not, you can do basic configuration by command mentioned in the Paramiko section. Now, let's see an example. Create a nmiko.py script and write the following code in it: from netmiko import ConnectHandler remote_device={ 'device_type': 'cisco_ios', 'ip': 'your remote_device ip address', 'username': 'username', 'password': 'password', } remote_connection = ConnectHandler(**remote_device) #net_connect.find_prompt() for n in range (2,6): print("Creating VLAN " + str(n)) commands = ['exit','vlan database','vlan ' + str(n), 'exit'] output = remote_connection.send_config_set(commands) print(output) command = remote_connection.send_command('show vlan-switch brief') print(command) Run the script and you will get the output as follows: student@ubuntu:~$ python3 nmiko.py Output: Creating VLAN 2 config term Enter configuration commands, one per line. End with CNTL/Z. server(config)#exit server #vlan database server (vlan)#vlan 2 VLAN 2 modified: server (vlan)#exit APPLY completed. Exiting.... server # .. .. .. .. switch# Creating VLAN 5 config term Enter configuration commands, one per line. End with CNTL/Z. server (config)#exit server #vlan database server (vlan)#vlan 5 VLAN 5 modified: server (vlan)#exit APPLY completed. Exiting.... VLAN Name Status Ports ---- -------------------------------- --------- ------------------------------- 1 default active Fa0/0, Fa0/1, Fa0/2, Fa0/3, Fa0/4, Fa0/5, Fa0/6, Fa0/7, Fa0/8, Fa0/9, Fa0/10, Fa0/11, Fa0/12, Fa0/13, Fa0/14, Fa0/15 2 VLAN0002 active 3 VLAN0003 active 4 VLAN0004 active 5 VLAN0005 active 1002 fddi-default active 1003 token-ring-default active 1004 fddinet-default active 1005 trnet-default active In the preceding example, we use Netmiko library to do SSH, instead of Paramiko. In this script, first, we imported ConnectHandler from the Netmiko library, which we used to establish an SSH connection to the remote network devices by passing in the device dictionary. In our case, that dictionary is remote_device. After the connection is established, we executed configuration commands to create a number of virtual LANs using the send_config_set() function. When we use this type (.send_config_set()) of function to pass commands on a remote device, it automatically sets our device in configuration mode. After sending configuration commands, we also passed a simple command to get the information about the configured device. Summary In this tutorial, you learned about Telnet and SSH and different Python modules such as telnetlib, subprocess, fabric, Netmiko, and Paramiko, using which we perform Telnet and SSH. SSH uses the public key encryption for security purposes and is more secure than Telnet. To learn how to leverage the features and libraries of Python to administrate your environment efficiently, check out our book Mastering Python Scripting for System Administrators. 5 blog posts that could make you a better Python programmer “With Python, you can create self-explanatory, concise, and engaging data visuals, and present insights that impact your business” – Tim Großmann and Mario Döbler [Interview] Using Python Automation to interact with network devices [Tutorial]
Read more
  • 0
  • 0
  • 59858

article-image-women-win-all-open-board-director-seats-in-open-source-initiative-2019-board-elections
Savia Lobo
19 Mar 2019
3 min read
Save for later

Women win all open board director seats in Open Source Initiative 2019 board elections

Savia Lobo
19 Mar 2019
3 min read
The recently held Open Source Initiative’s 2019 Board elections elected six Board of Directors to its eleven-person Board. Two were elected from the affiliate membership, and four from the individual membership. If it wasn’t incredible enough that many women ran for the seats,  they have won all the seats! The six seats include two returning directors, Carol Smith and Molly de Blanc; and three new directors Pamela Chestek, Elana Hashman, and Hong Phuc Dang. Pamela Chestek (nominated by The Document Foundation) and Molly de Blanc (nominated by the Debian Project) captured the most votes from OSI Affiliate Members. The last seat is a tie between Christine Hall and Mariatta Wijaya and hence a runoff election will be required to identify the final OSI Board Director. The run off election started yesterday, March 18th (opening at 12:00 a.m. / 00:00) and will end on Monday, March 25th (closing at 12:00 a.m. / 00:00). Mariatta Wijaya, a core Python developer and a platform engineer at Zapier, told in a statement to Business Insider that she found not all open source projects were as welcoming, especially to women. That's one reason why she's running for the board of the Open Source Initiative, an influential organization that promotes and protects open source software communities. Wijaya also said, "I really want to see better diversity across the people who contribute to open source. Not just the users, the creators of open source. I would love to see that diversity improve. I would like to see a better representation. I did find it a barrier initially, not seeing more people who look like me in this space, and I felt like an outsider." A person discussed six female candidates in misogynistic language on Slashdot, which is a tech-focussed social news website. The post also then labeled each woman with how much of a "threat" they were. Slashdot immediately took down this post “shortly afterward the OSI started seeing inappropriate comments posted on its website”. https://twitter.com/alicegoldfuss/status/1102609189342371840 Molly de Blanc and Patrick Masson said this was the first time they saw such type of harassment of female OSI board candidates. They also said that such harassments in open source are not uncommon. Joshua R. Simmons, an Open source advocate, and web developer tweeted, “women winning 100% of the open seats in an election that drew attention from a cadre of horrible misogynists” https://twitter.com/joshsimmons/status/1107303020293832704 OSI President, Simon Phipps said that the OSI committee is “thrilled the electorate has picked an all-female cohort to the new Board” https://twitter.com/webmink/status/1107367907825274886 To know more about these elections in detail, head over to the OSI official blog post. UPDATED: In the previous draft, Pamela Chestek who was listed as returning board member, is a new board member; and Carol Smith who was listed as a new board member, is a returning member. #GoogleWalkout demanded a ‘truly equitable culture for everyone’; Pichai shares a “comprehensive” plan for employees to safely report sexual harassment MongoDB withdraws controversial Server Side Public License from the Open Source Initiative’s approval process Google’s pay equity analysis finds men, not women, are underpaid; critics call out design flaws in the analysis  
Read more
  • 0
  • 0
  • 14912
article-image-social-media-enabled-and-amplified-christchurch-terrorist-attack
Fatema Patrawala
19 Mar 2019
11 min read
Save for later

How social media enabled and amplified the Christchurch terrorist attack

Fatema Patrawala
19 Mar 2019
11 min read
The recent horrifying terrorist attack in New Zealand has cast new blame on how technology platforms police content. There are now questions about whether global internet services are designed to work this way? And if online viral hate is uncontainable? Fifty one people so far have been reported to be dead and 50 more injured after the terrorist attacks on two New Zealand mosques on Friday. The victims included children as young as 3 and 4 years old, and elderly men and women. The alleged shooter is identified as a 28 year old Australian man named Brenton Tarrant. Brenton announced the attack on the anonymous-troll message board 8chan. There, he posted images of the weapons days before the attack, and made an announcement an hour before the shooting. On 8chan, Facebook and Twitter, he also posted links to a 74-page manifesto, titled “The Great Replacement,” blaming immigration for the displacement of whites in Oceania and elsewhere. The manifesto cites “white genocide” as a motive for the attack, and calls for “a future for white children” as its goal. Further he live-streamed the attacks on Facebook, YouTube; and posted a link to the stream on 8chan. It’s terrifying and disgusting, especially when 8chan is one of the sites where disaffected internet misfits create memes and other messages to provoke dismay and sow chaos among people. “8chan became the new digital home for some of the most offensive people on the internet, people who really believe in white supremacy and the inferiority of women,” Ethan Chiel wrote. “It’s time to stop shitposting,” the alleged shooter’s 8chan post reads, “and time to make a real-life effort post.” Many of the responses, anonymous by 8chan’s nature, celebrate the attack, with some posting congratulatory Nazi memes. A few seem to decry it, just for logistical quibbles. And others lament that the whole affair might destroy the site, a concern that betrays its users’ priorities. Social media encourages performance crime The use of social media technology and livestreaming marks the attack as different from many other terrorist incidents. It is a form of violent “performance crime”. That is, the video streaming is a central component of the violence itself, it’s not somehow incidental to the crime, or a trophy for the perpetrator to re-watch later. In the past, terrorism functioned according to what has been called the “theatre of terror”, which required the media to report on the spectacle of violence created by the group. Nowadays with social media in our hands it's much easier for someone to both create the spectacle of horrific violence and distribute it widely by themselves. There is a tragic and recent history of performance crime videos that use live streaming and social media video services as part of their tactics. In 2017, for example, the sickening murder video of an elderly man in Ohio was uploaded to Facebook, and the torture of a man with disabilities in Chicago was live streamed. In 2015, the murder of two journalists was simultaneously broadcast on-air, and live streamed. Tech companies on the radar Social-media companies scrambled to take action as the news—and the video—of the attack spread. Facebook finally managed to pull down Tarrant’s profiles and the video, but only after New Zealand police brought the live-stream to the company’s attention. It has been working "around the clock" to remove videos of the incident shared on its platform. In a statement posted to Twitter on Sunday, the tech company said that within 24 hours of Friday’s shooting it had removed 1.5 million videos of the attack from its platform globally. YouTube said it had also removed an “unprecedented volume” of videos of the shooting. Twitter also suspended Tarrant’s account, where he had posted links to the manifesto from several file-sharing sites. The chaotic aftermath mostly took place while many North Americans slept unaware, waking up to the news and its associated confusion. By morning on the East Coast, news outlets had already weighed in on whether technology companies might be partly to blame for catastrophes such as the New Zealand massacre because they have failed to catch offensive content before it spreads. One of the tweets say Google, Twitter and Facebook made a choice to not use tools available to them to stop white supremacist terrorism. https://twitter.com/samswey/status/1107055372949286912 Countries like Germany and France already have a law in place that demands social media sites move quickly to remove hate speech, fake news and illegal material. Sites that do not remove "obviously illegal" posts could face fines of up to 50m euro (£44.3m). In the wake of the attack, a consortium of New Zealand’s major companies has pledged to pull their advertising from Facebook. In a joint statement, the Association of New Zealand Advertisers (ANZA) and the Commercial Communications Council asked domestic companies to think about where “their advertising dollars are spent, and carefully consider, with their agency partners, where their ads appear.” They added, “We challenge Facebook and other platform owners to immediately take steps to effectively moderate hate content before another tragedy can be streamed online.” Additionally internet service providers like Vodafone, Spark and Vocus in New Zealand are blocking access to websites that do not respond or refuse to comply to requests to remove reuploads of the shooter’s original live stream. The free speech vs safety debate puts social media platforms in the crosshairs Tech Companies are facing new questions on content moderation following the New Zealand attack. The shooter posted a link to the live stream, and soon after he was apprehended, reuploads were found on other platforms like YouTube and Twitter. “Tech companies basically don’t see this as a priority,” the counter-extremism policy adviser Lucinda Creighton commented. “They say this is terrible, but what they’re not doing is preventing this from reappearing.” Others affirmed the importance of quelling the spread of the manifesto, video, and related materials, for fear of producing copycats, or of at least furthering radicalization among those who would be receptive to the message. The circulation of ideas might have motivated the shooter as much as, or even more than, ethnic violence. As Charlie Warzel wrote at The New York Times, the New Zealand massacre seems to have been made to go viral. Tarrant teased his intentions and preparations on 8chan. When the time came to carry out the act, he provided a trove of resources for his anonymous members, scattered to the winds of mirror sites and repositories. Once the live-stream started, one of the 8chan user posted “capped for posterity” on Tarrant’s thread, meaning that he had downloaded the stream’s video for archival and, presumably, future upload to other services, such as Reddit or 4chan, where other like-minded trolls or radicals would ensure the images spread even further. As Warzel put it, “Platforms like Facebook, Twitter, and YouTube … were no match for the speed of their users.” The internet is a Pandora’s box that never had a lid. Camouflaging stories is easy but companies trying hard in building AI to catch it Last year, Mark Zuckerberg defended himself and Facebook before Congress against myriad failures, which included Russian operatives disrupting American elections and permitting illegal housing ads that discriminate by race. Mark Zuckerberg repeatedly invoked artificial intelligence as a solution for the problems his and other global internet companies have created. There’s just too much content for human moderators to process, even when pressed hard to do so under poor working conditions. The answer, Zuckerberg has argued, is to train AI to do the work for them. But that technique has proved insufficient. That’s because detecting and scrubbing undesirable content automatically is extremely difficult. False positives enrage earnest users or foment conspiracy theories among paranoid ones, thanks to the black-box nature of computer systems. Worse, given a pool of billions of users, the clever ones will always find ways to trick any computer system, for example, by slightly modifying images or videos in order to make them appear different to the computer but identical to human eyes. 8chan, as it happens, is largely populated by computer-savvy people who have self-organized to perpetrate exactly those kinds of tricks. The primary sources of content are only part of the problem. Long after the deed, YouTube users have bolstered conspiracy theories about murders, successfully replacing truth with lies among broad populations of users who might not even know they are being deceived. Even stock-photo providers are licensing stills from the New Zealand shooter’s video; a Reuters image that shows the perpetrator wielding his rifle as he enters the mosque is simply credited, “Social media.” Interpreting real motives is difficult on social The video is just the tip of the iceberg. Many smaller and less obviously inflamed messages have no hope of being found, isolated, and removed by technology services. The shooter praised Donald Trump as a “symbol of renewed white identity” and incited the conservative commentator Candace Owens, who took the bait on Twitter in a post that got retweeted thousands of times by the morning after the attack. The shooter’s forum posts and video are littered with memes and inside references that bear special meaning within certain communities on 8chan, 4chan, Reddit, and other corners of the internet, offering tempting receptors for consumption and further spread. Perhaps worst of all, the forum posts, the manifesto, and even the shooting itself might not have been carried out with the purpose that a literal read of their contents suggests. At the first glance, it seems impossible to deny that this terrorist act was motivated by white-extremist hatred, an animosity that authorities like the FBI expert and the Facebook officials would want to snuff out before it spreads. But 8chan is notorious for users with an ironic and rude behaviour under the shades of anonymity.They use humor, memes and urban slang to promote chaos and divisive rhetoric. As the internet separates images from context and action from intention, and then spreads those messages quickly among billions of people scattered all around the globe. That structure makes it impossible to even know what individuals like Tarrant “really mean” by their words and actions. As it spreads, social-media content neuters earnest purpose entirely, putting it on the same level as anarchic randomness. What a message means collapses into how it gets used and interpreted. For 8chan trolls, any ideology might be as good as any other, so long as it produces chaos. We all have a role to play It’s easy to say that technology companies can do better. They can, and they should. But ultimately, content moderation is not the solution by itself. The problem is the media ecosystem they have created. The only surprise is that anyone would still be surprised that social media produce this tragic abyss, for this is what social media are supposed to do, what they were designed to do: spread the images and messages that accelerate interest and invoke raw emotions, without check, and absent concern for their consequences. We hope that social media companies get better at filtering out violent content and explore alternative business models, and governments think critically about cyber laws that protect both people and speech. But until they do we should reflect on our own behavior too. As news outlets, we shape the narrative through our informed perspectives which makes it imperative to publish legitimate & authentic content. Let’s as users too make a choice of liking and sharing content on social platforms. Let’s consider how our activities could contribute to an overall spectacle society that might inspire future perpetrator-produced videos of such gruesome crime – and act accordingly. In this era of social spectacle, we all have a role to play in ensuring that terrorists aren’t rewarded for their crimes with our clicks and shares. The Indian government proposes to censor social media content and monitor WhatsApp messages Virality of fake news on social media: Are weaponized AI bots to blame, questions Destin Sandlin Mastodon 2.7, a decentralized alternative to social media silos, is now out!
Read more
  • 0
  • 0
  • 17401

article-image-application-server-clustering-using-various-cloud-providers-tutorial
Melisha Dsouza
19 Mar 2019
11 min read
Save for later

Application server clustering using various cloud providers [Tutorial]

Melisha Dsouza
19 Mar 2019
11 min read
In this tutorial, we will illustrate how applications are clustered, using different cloud providers and frameworks. You can set up a lot of applications in a highly available way, and spread their workloads between the on-premises and cloud environments. You can also set them up in different cloud environments if the technical requirements are set. We will describe solutions to do the same and learn how to implement them to be independent of one cloud vendor and avoid a vendor login. This tutorial is an excerpt from a book written by Florian Klaffenbach, Markus Klein, Suresh Sundaresan titled Multi-Cloud for Architects. This book will be your go-to guide to find perfect solutions at completely adapting to any Cloud and its services, no matter the size of your infrastructure. Technical requirements for cross-cloud application servers To design a cross-cloud application server environment, the requirements needed are: Network connectivity between the different clouds Single identity management solutions for all servers Supported applications for georedundancy Networking connectivity between different clouds No matter what cloud you need to connect to, networking and networking security are always the key. This means that you will need reliable and secured networking connectivity, as there is a possibility that not all of the traffic is encrypted, depending on the software architecture for high availability. As the Windows cluster environment basically has all of the nodes in one physical network location, each newer release can work with the nodes in physically different locations. So, for example, one part of the server nodes could be in cloud A, and the other one in cloud B. The software requirements are set for applications using different cloud vendors. As each cloud vendor is running different connection gateways, the most common solution is to have the same network virtual appliance as a VM instance (single, or even double redundant) in each environment, and to design each cloud as an independent data center location. There is no requirement to have direct connectivity (using MPLS) or remote connectivity, but you will need to make sure that the network package round-trips are as quick as possible. Single identity management solutions for all servers The second pillar of cross-platform application designs is the IDM solution. As not every cloud vendor offers managed IDM solutions, the valid options are setting them up, for example, Active Directory as VMs and joining all servers in the Cloud into these domain or using a managed IDM solution like Azure AD which is not only support being used in Azure, but also works in other public cloud environments to have all servers join the one Microsoft Azure AD. Supported applications for georedundancy Finally, the application itself needs to support georedundancy in its node placement design. The application needs to be designed to work in low latency environments (for example, Microsoft SQL Server or Microsoft Exchange Server); it can also be designed quite easily, by placing a georedundant load balancer (for example, a web server). A sample design of a cross-cloud virtual machine running clustered application nodes is as follows: The preceding diagram shows Azure on the left and AWS on the right, both connected to a single-network environment and using Azure AD as a single identity management solution. Whether you choose the left one or the right one for your preferred cloud vendor, the result will be the same, and the service will be reachable from both cloud environments. Examples of clustered application servers Let's take a look at some sample application servers (based on virtual machines) in a multi-cloud design. Microsoft SQL Server Microsoft Exchange Server Web servers Microsoft SQL Server Microsoft SQL Server is a robust relational database application server that can run either based on Microsoft Windows Server or in Linux, as an operating system (starting with the 2017 version). Surely, you could set up a single SQL server as a virtual machine in your preferred cloud, we need to look at the high availability features of the application. With Microsoft SQL Server, the basic keyword for HA is availability groups. Using this feature, you can set up your application server to have one or more replicas of the database itself, organized in an availability group. You can design the availability groups based on your needs, and split them between servers, no matter where the virtual machine really lives; then, you can configure database replication. One availability group supports one primary, and up to eight secondary, databases. Keep in mind that a secondary database is not equal to a database backup, as the replica contains the same information that the primary database does. Since the 2017 release, there have been two options for availability groups, as follows: Always on for redundancy: This means that if a database replica fails, one or more are still available for the application requesting the data. Always on for read optimization: This means that if an application needs to read from data, it uses its nearest database server. For write operations, the primary database replica needs to be available. The replica operation itself can be synchronous or asynchronous, depending on the requirements and design of the application working with the database(s). The following chart illustrates the availability group design of Microsoft SQL Servers: As you can see in the preceding chart, there are different nodes available. Each of them can either reside in the same cloud, or a different one. Regarding risk management, a single cloud vendor can go offline without affecting the availability of your database servers themselves. When you take a look at the SQL Server Management Studio, you will see the availability groups and their health statuses, as follows: Microsoft Exchange Server Microsoft Exchange Server is a groupware and email solution that is, in Microsoft cloud technology, the technical basis of the Office 365 SaaS email functionality. If you decide to run Exchange Server on your own, the most recent release is Exchange Server 2019. And, in case it is needed, there is full support to run Exchange Server as virtual machines in your cloud environment. Of course, it is possible to run a simple VM with all Exchange Services on it, but, like with almost every company groupware solution that requires high availability, there is the possibility to set up Exchange Server environments in a multi-server environment. As Exchange Server has to support low latency network environments by default, it supports running an Exchange Server environment in different networking regions. This means that you can set up some nodes of Exchange Server in cloud A, and the others in cloud B. This feature is called availability groups, too. A typical design would look as follows: As you can see, the preceding diagram shows two Azure AD sites with redundancy, using Database Availability Groups (DAGs). The best practice for running Exchange Server on AWS is illustrated as follows: There is also a best practice design for Azure in hybrid cloud environments; it looks as follows: A primary database residing on one Exchange mailbox server could have up to 16 database copies. The placement of these copies is dependent on customer requirements. Configuring high availability within Exchange Server is the default, and it is quite easy to handle, too, from either the Exchange Server Management Console or from PowerShell. Supporting cross-cloud implementations using geo load balancers If an application that needs to be redesigned for a multi-cloud environment works based on port communications, the redesign will be quite easy, as you will just need to set up a georedundant load balancer to support the different cloud targets, and route the traffic correspondingly. A georedundant load balancer is a more complex solution; it's like a default load balancer that just routes the traffic between different servers in one region or cloud environment. It generally works with the same technology and uses DNS name resolutions for redundancy and traffic routing, but, in comparison to DNS round robin technologies, a load balancer knows the available targets for resolving requests, and can work with technologies such as IP range mapping. Azure Traffic Manager Azure Traffic Manager is the Microsoft solution for georedundant traffic routing. It is available in each Azure region, and it provides transparent load balancing for services that coexist in different Azure regions, non-Microsoft clouds, or on premises. It provides the following features: Flexible traffic routing Reduced application downtime Improve performance and content delivery Distributed traffic over multiple locations Support for all available cloud solutions (private and public clouds) As you can see in the following diagram, Azure Traffic Manager is a flexible solution for traffic routing, and can point to each target that you need in your application design: Incoming traffic is routed to the appropriate site using Traffic Manager metrics, and if a site is down or degraded, Traffic Manager routes the traffic to another available site. You can compare the Traffic Manager to an intelligent router that knows the origin of the traffic and reroutes the requests to the nearest available service. AWS Route 53 In AWS, the Route 53 service provides an easy solution for routing traffic based on load and availability. It is a PaaS service, like Azure Traffic Manager, and works based on DNS name resolutions, too. The technical design works as follows; it is fully integrated into the DNS service: As you can see, the Route 53 design is quite comparable to Azure Traffic Manager. If you need to decide which service to use in your design, it is not a technical decision at all, as the technology is nearly the same. Rather, the choice is based on other requirements, involving technology and pricing. Managing multi-cloud virtual machines for clustered application servers If you decide to design your applications in a multi-cloud environment, it does not facilitate designing automation and replaying configurations. Azure works with ARM templates and AWS with AWS CloudFormation. Both languages are JSON based, but they are different. If you plan to use cloud solutions to transform your on-premises solutions, you should think about automation and ways to replay configurations. If you need to deal with two (or even more) different dialects, you will need to set up a process to create and update the corresponding templates. Therefore, implementing another layer of templating will be required, if you do not want to rely on manual processes. There is a very small number of vendors that provide the technology to avoid relying on different dialects. A common one is Terraform, but Ansible and Puppet are other options. Terraform works based on a language called Hashicorp Configuration Language (HCL). It is designed for human consumption, so users can quickly interpret and understand the infrastructure configurations. HCL includes a full JSON parser for machine-generated configurations. If you compare HCL to the JSON language, it looks as follows: # An AMI variable "ami" { description = "the AMI to use" } /* A multi line comment. */ resource "aws_instance" "web" { ami = "${var.ami}" count = 2 source_dest_check = false connection { user = "root" } } Terraform gives us providers to translate the deployments into the corresponding cloud vendor languages. There are a lot of providers available, as you can see in the following screenshot: A provider works as follows: If you decide to work with Terraform to make your cloud automation processes smooth and independent of your cloud vendor, you can install it from most cloud vendor marketplaces as a virtual machine. Troubleshooting cross-cloud application servers If you have decided on a multi-cloud design for your applications, you will need to have a plan for troubleshooting; network connectivity and having identities between the different cloud environments could be reasons for unavailability issues. Otherwise, the troubleshooting mechanisms are the same ones that you're already familiar with, and they are included in the application servers themselves, in general. Summary In this tutorial, we learned that it is quite easy to design a multi-cloud environment. In case there is a need to change the components in this solution, you can even decide to change services as a part of your solution from one cloud vendor to another. To learn how to architect a multi-cloud solution for your organization, check out our book  Multi-Cloud for Architects. Modern Cloud Native architectures: Microservices, Containers, and Serverless – Part 1 The 10 best cloud and infrastructure conferences happening in 2019 VMware Essential PKS: Use upstream Kubernetes to build a flexible, cost-effective cloud-native platform
Read more
  • 0
  • 0
  • 4540

article-image-the-u-s-dod-wants-to-dominate-russia-and-china-in-artificial-intelligence-last-week-gave-us-a-glimpse-into-that-vision
Savia Lobo
18 Mar 2019
9 min read
Save for later

The U.S. DoD wants to dominate Russia and China in Artificial Intelligence. Last week gave us a glimpse into that vision.

Savia Lobo
18 Mar 2019
9 min read
In a hearing on March 12, the sub-committee on emerging threats and capabilities received testimonies on Artificial Intelligence Initiatives within the Department of Defense(DoD). The panel included Peter Highnam, Deputy Director of the Defense Advanced Research Projects Agency; Michael Brown, DoD Defense Innovation Unit Director; and Lieutenant General John Shanahan, director of the Joint Artificial Intelligence Center (JAIC). The panel broadly testified to senators that AI will significantly transform DoD’s capabilities and that it is critical the U.S. remain competitive with China and Russia in developing AI applications. Dr. Peter T. Highnam on DARPA’s achievements and future goals Dr. Peter T. Highnam, Deputy Director, Defense Advanced Research Projects Agency talked about DARPA’s significant role in the development of AI technologies that have produced game-changing capabilities for the Department of Defense and beyond. In his testimony, he mentions, “DARPA’s AI Next effort is simply a continuing part of its 166 historic investment in the exploration and advancement of AI technologies.” Dr. Highnam highlighted different waves of AI technologies. The first wave, which was nearly 70 years ago, emphasized handcrafted knowledge, and computer scientists constructed so-called expert systems that captured the rules that the system could then apply to situations of interest. However, handcrafting rules was costly and time-consuming. The second wave that brought in machine learning that applies statistical and probabilistic methods to large data sets to create generalized representations that can be applied to future samples. However, this required training deep learning (artificial) neural networks with a variety of classification and prediction tasks when adequate historical data. Therein lies the rub, however, as the task of collecting, labelling, and vetting data on which to train. Such a process is prohibitively costly and time-consuming too. He says, “DARPA envisions a future in which machines are more than just tools that execute human programmed rules or generalize from human-curated data sets. Rather, the machines DARPA envisions will function more as colleagues than as tools.” Towards this end, DARPA is focusing its investments on a “third wave” of AI technologies that brings forth machines that can reason in context. Incorporating these technologies in military systems that collaborate with warfighters will facilitate better decisions in complex, time-critical, battlefield environments; enable a shared understanding of massive, incomplete, and contradictory information; and empower unmanned systems to perform critical missions safely and with high degrees of autonomy. DARPA’s more than $2 billion “AI Next” campaign, announced in September 2018, includes providing robust foundations for second wave technologies, aggressively applying the second wave AI technologies into appropriate systems, and exploring and creating third wave AI science and technologies. DARPA’s third wave research efforts will forge new theories and methods that will make it possible for machines to adapt contextually to changing situations, advancing computers from tools to true collaborative partners. Furthermore, the agency will be fearless about exploring these new technologies and their capabilities – DARPA’s core function – pushing critical frontiers ahead of our nation’s adversaries. To know more about this in detail, read Dr. Peter T. Highnam’s complete statement. Michael Brown on (Defense Innovation Unit) DIU’s efforts in Artificial Intelligence Michael Brown, Director of the Defense Innovation Unit, started the talk by highlighting on the fact how China and Russia are investing heavily to become dominant in AI.  “By 2025, China will aim to achieve major breakthroughs in AI and increase its domestic market to reach $59.6 billion (RMB 400 billion) To achieve these targets, China’s National Development and Reform Commission (China’s industrial policy-making agency) funded the creation of a national AI laboratory, and Chinese local governments have pledged more than $7 billion in AI funding”, Brown said in his statement. He said that these Chinese firms are in a way leveraging U.S. talent by setting up research institutes in the state, investing in U.S. AI-related startups and firms, recruiting U.S.-based talent, and commercial and academic partnerships. Brown said that DIU will engage with DARPA and JAIC(Joint Artificial Intelligence Center) and also make its commercial knowledge and relationships with potential vendors available to any of the Services and Service Labs. DIU also anticipates that with its close partnership with the JAIC, DIU will be at the leading edge of the Department’s National Mission Initiatives (NMIs), proving that commercial technology can be applied to critical national security challenges via accelerated prototypes that lay the groundwork for future scaling through JAIC. “DIU looks to bring in key elements of AI development pursued by the commercial sector, which relies heavily on continuous feedback loops, vigorous experimentation using data, and iterative development, all to achieve the measurable outcome, mission impact”, Brown mentions. DIU’s AI portfolio team combines depth of commercial AI, machine learning, and data science experience from the commercial sector with military operators. However, they have specifically prioritized projects that address three major impact areas or use cases which employ AI technology, including: Computer vision The DIU is prototyping computer vision algorithms in humanitarian assistance and disaster recovery scenarios. “This use of AI holds the potential to automate post-disaster assessments and accelerate search and rescue efforts on a global scale”, Brown said in his statement. Large dataset analytics and predictions DIU is prototyping predictive maintenance applications for Air Force and Army platforms. For this DIU plans to partner with JAIC to scale this solution across multiple aircraft platforms, as well as ground vehicles beginning with DIU’s complementary predictive maintenance project focusing on the Army’s Bradley Fighting Vehicle. Brown says this is one of DIU’s highest priority projects for FY19 given its enormous potential for impact on readiness and reducing costs. Strategic reasoning DIU is prototyping an application from Project VOLTRON that leverages AI to reason about high-level strategic questions, map probabilistic chains of events, and develop alternative strategies. This will make DoD owned systems more resilient to cyber attacks and inform program offices of configuration errors faster and with fewer errors than humans. Know more about what more DIU plans in partnership with DARPA and JAIC, in detail, in Michael Brown’s complete testimony. Lieutenant General Jack Shanahan on making JAIC “AI-Ready” Lieutenant General Jack Shanahan, Director, Joint Artificial Intelligence Center, touches upon  how the JAIC is partnering with the Under Secretary of Defense (USD) Research & Engineering (R&E), the role of the Military Services, the Department’s initial focus areas for AI delivery, and how JAIC is supporting whole-of-government efforts in AI. “To derive maximum value from AI application throughout the Department, JAIC will operate across an end-to-end lifecycle of problem identification, prototyping, integration, scaling, transition, and sustainment. Emphasizing commerciality to the maximum extent practicable, JAIC will partner with the Services and other components across the Joint Force to systematically identify, prioritize, and select new AI mission initiatives”, Shanahan mentions in his testimony. The AI capability delivery efforts that will go through this lifecycle will fall into two categories including National Mission Initiatives (NMI) and Component Mission Initiatives (CMI). NMI is an operational or business reform joint challenge, typically identified from the National Defense Strategy’s key operational problems and requiring multi-service innovation, coordination, and the parallel introduction of new technology and new operating concepts. On the other hand, Component Mission Initiatives (CMI) is a component-level challenge that can be solved through AI. JAIC will work closely with individual components on CMIs to help identify, shape, and accelerate their Component-specific AI deployments through: funding support; usage of common foundational tools, libraries, cloud infrastructure; application of best practices; partnerships with industry and academia; and so on. The Component will be responsible for identifying and implementing the organizational structure required to accomplish its project in coordination and partnership with the JAIC. Following are some examples of early NMI’s by JAIC to deliver mission impact at speed, demonstrate the proof of concept for the JAIC operational model, enable rapid learning and iterative process refinement, and build their library of reusable tools while validating JAIC’s enterprise cloud architecture. Perception Improve the speed, completeness, and accuracy of Intelligence, Surveillance, Reconnaissance (ISR) Processing, Exploitation, and Dissemination (PED). Shanahan says Project Maven’s efforts are included here. Predictive Maintenance (PMx) Provide computational tools to decision-makers to help them better forecast, diagnose, and manage maintenance issues to increase availability, improve operational effectiveness, and ensure safety, at a reduced cost. Humanitarian Assistance/Disaster Relief (HA/DR) Reduce the time associated with search and discovery, resource allocation decisions, and executing rescue and relief operations to save lives and livelihood during disaster operations. Here, JAIC plans to apply lessons learned and reusable tools from Project Maven to field AI capabilities in support of federal responses to events such as wildfires and hurricanes—where DoD plays a supporting role. Cyber Sensemaking Detect and deter advanced adversarial cyber actors who infiltrate and operate within the DoD Information Network (DoDIN) to increase DoDIN security, safeguard sensitive information, and allow warfighters and engineers to focus on strategic analysis and response. Shanahan states, “Under the DoD CIO’s authorities and as delineated in the JAIC establishment memo, JAIC will coordinate all DoD AI-related projects above $15 million annually.” “It does mean that we will start to ensure, for example, that they begin to leverage common tools and libraries, manage data using best practices, reflect a common governance framework, adhere to rigorous testing and evaluation methodologies, share lessons learned, and comply with architectural principles and standards that enable scale”, he further added. To know more about this in detail, read Lieutenant General Jack Shanahan’s complete testimony. To know more about this news in detail, watch the entire hearing on 'Artificial Intelligence Initiatives within the Department of Defense' So, you want to learn artificial intelligence. Here’s how you do it. What can happen when artificial intelligence decides on your loan request Mozilla partners with Ubisoft to Clever-Commit its code, an artificial intelligence assisted assistant
Read more
  • 0
  • 0
  • 6681
article-image-applying-modern-css-to-create-react-app-projects-tutorial
Bhagyashree R
18 Mar 2019
13 min read
Save for later

Applying Modern CSS to Create React App Projects [Tutorial]

Bhagyashree R
18 Mar 2019
13 min read
Previously with Create React App, you actually didn't have a lot of options to be able to clean things up visually. You were frequently at the whims and mercy of random Cascading Style Sheets (CSS) project maintainers, and trying to get other libraries, frameworks, or preprocessors involved in the project compilation process was frequently a nightmare. A preprocessor in the context of Create React App is basically one of the steps in the build process. In this case, we're talking about something that takes some of the style code (CSS or another format), compiles it down to basic CSS, and adds it to the output of the build process. This article is taken from the book  Create React App 2 Quick Start Guide by Brandon Richey. This book is intended for those who want to get intimately familiar with the Create React App tool. It covers all the commands in Create React App and all of the new additions in version 2.  To follow along with the examples implemented in this article, you can download the code from the book’s GitHub repository. Over the span of this article, we'll be covering materials that span the gamut of style-related functionality and highlighting what is, in my mind, one of the best new features in Create React App: support for CSS Modules and SASS. Introducing CSS Modules CSS Modules give you the ability to modularize any CSS code that you import in a way that prevents introducing global, overlapping namespaces, despite the fact that the end result is still just one giant CSS file. Better project organization Let's start off by cleaning up our directory structure in our project a little bit better. What we're going to do is just separate out each component that has CSS and JavaScript code into their own folders. Let's first create NewTodo, Todo, App, TodoList, and Divider folders and place all of their related code in each of those. We'll also need to create a new file in each of these directories called index.js, which will be responsible for only importing and exporting the appropriate component. For example, the App index file (src/App/index.js) will look like this: import App from "./App"; export default App; The new index file of Todo (src/Todo/index.js) will look like this: import Todo from "./Todo"; export default Todo; You can probably guess what the index files NewTodo, TodoList, and Divider will look like as well, based on this pattern. Next, we'll need to change each place that these files are referenced to make it easier to import all of them. This will unfortunately be a little bit of grunt work, but we'll need to do it all the same to make sure we don't break anything in the process. First, in src/App/App.js, change the TodoList import component to the following: import TodoList from "../TodoList"; There's nothing we need to do for Divider since it is a component with no imports. NewTodo and Todo are of a similar type, so we can skip them as well. src/TodoList/TodoList.js, on the other hand, has a lot we need to deal with, since it's one of our highest-level components and imports a lot: import Todo from "../Todo"; import NewTodo from "../NewTodo"; import Divider from "../Divider"; But that's not all. Our test file, src/TodoList/TodoList.test.js, also needs to be modified to include these new paths for our files or else our tests will fail! We'll need nearly the same list of imports as earlier: import TodoList from "./TodoList"; import NewTodo from "../NewTodo"; import Todo from "../Todo"; Now, when you reload your application, your code should still be working just fine, your tests should all pass, and everything should be cleanly separated out! Our full project structure should now look like this: src/ App/ App.css App.js App.test.js index.js Divider/ Divider.css Divider.js index.js NewTodo/ NewTodo.css NewTodo.js NewTodo.test.js index.js Todo/ Todo.css Todo.js Todo.test.js index.js TodoList/ TodoList.css TodoList.js TodoList.test.js index.js index.css index.js setupTests.js ... etc ... Introducing CSS Modules to our application If we want to use CSS Modules, there are a few simple guidelines that we need to follow. The first is that we need to name our files [whatever].module.css, instead of [whatever].css. The next thing we need to do is to make sure that our styles are named simply and are easy to reference. Let's start off by following these conventions and by renaming our CSS file for Todo as src/Todo/Todo.module.css, and then we'll change the contents a tiny bit: .todo { border: 2px solid black; text-align: center; background: #f5f5f5; color: #333; margin: 20px; padding: 20px; } .done { background: #f5a5a5; } Next, we'll open up src/Todo/Todo.js to take advantage of CSS Modules instead. We created a helper function in our Todo component called cssClasses(), which returns the styles we should be using in our component, and there's not much we need to change to make this all work exactly the same as earlier. We'll need to change our import statement at the top as well, since we renamed the file and are changing how our CSS is getting loaded into our code! Take a look at the following: import styles from "./Todo.module.css"; This enables our code to take advantage of any class names defined in Todo.module.css by referencing them as styles.[className]. For example, in the previous file, we defined two CSS class names: todo and done, so we can now reference them in our component via styles.Todo and styles.done. We'll need to change the cssClasses() function to use this, so let's make those exact changes now. In src/Todo/Todo.js, our cssClasses() function should now read as follows: cssClasses() { let classes = [styles.todo]; if (this.state.done) { classes = [...classes, styles.done]; } return classes.join(' '); } Save and reload, and our application should be back to normal! Next, let's change the hr tags inside of the todo components to have their own styles and effects. Head back into src/Todo/Todo.module.css and add the following block for our hr tag, which we'll give a new class of redDivider: .redDivider { border: 2px solid red; } And finally, return back to our render() function in src/Todo/Todo.js, and change our render() function's hr tag inclusion to the following: <hr className={styles.redDivider} /> Save and reload, and now we should have fully compartmentalized CSS code without worrying about collisions and global namespaces! Here's how the output looks like: Composability with CSS Modules That's not all that CSS Modules give us, although it's certainly one of the great parts of CSS Modules that we get immediately and with no fuss. We also get CSS composability, which is the ability to inherit CSS classes off of other classes, whether they're in the main file or not. This can be incredibly useful when you're setting up more complicated nested components that all need to handle slightly different style sheets, but are not wildly different from each other. Let's say we want to have the ability to mark some components as critical instead of just regular Todos. We don't want to change too much about the component; we want it to inherit the same basic rules as all of the other Todos. We'll need to set up some code to make this happen. Back in src/Todo/Todo.js, we're going to make some modifications to allow a new state property named critical. We'll start off in the constructor component, where we'll add our new state property and a bind tag for a function: constructor(props) { super(props); this.state = { done: false, critical: false }; this.markAsDone = this.markAsDone.bind(this); this.removeTodo = this.removeTodo.bind(this); this.markCritical = this.markCritical.bind(this); } We add a new critical property in our state property and set it to a default value of false. Then we also reference a function (which we haven't written yet) called markCritical, and we bind this, since we'll be using it in an event handler later. Next, we'll tackle the markCritical() function: markCritical() { this.setState({ critical: true }); } We'll also need to modify our cssClasses() function so that it can react to this new state property. To demonstrate the composability function of CSS Modules, we'll set it so that classes is originally an empty array, and then the first item either becomes critical or todo, depending on whether or not the item is marked as critical: cssClasses() { let classes = []; if (this.state.critical) { classes = [styles.critical]; } else { classes = [styles.todo]; } if (this.state.done) { classes = [...classes, styles.done]; } return classes.join(' '); } And finally, in our render function, we'll create the button tag to mark items as critical: render() { return ( <div className={this.cssClasses()}> {this.props.description} <br /> <hr className={styles.hr} /> <button className="MarkDone" onClick={this.markAsDone}> Mark as Done </button> <button className="RemoveTodo" onClick={this.removeTodo}> Remove Me </button> <button className="MarkCritical" onClick={this.markCritical}> Mark as Critical </button> </div> ); } We're not quite done yet, although we're at least 90% of the way there. We'll also want to go back to src/Todo/Todo.module.css and add a new block for the critical class name, and we'll use our composable property as well: .critical { composes: todo; border: 4px dashed red; } To use composition, all you need to do is add a new CSS property called composes and give it a class name (or multiple class names) that you want it to compose. Compose, in this case, is a fancy way of saying that it inherits the behavior of the other class names and allows you to override others. In the previous case, we're saying critical is a CSS module class that is composed of a todo model as the base, and adds a border component of a big red dashed line since, well, we'll just say that this means it is critical. Save and reload, as always, and you should be able to mark items as Mark as Done, Mark as Critical, or both, or remove them by clicking Remove Me, as in the following screenshot: And that about covers it for our brief introduction to CSS Modules! Before you move on, you'll also want to quickly update your snapshots for your tests by hitting U in the yarn test screen. Introducing SASS to our project SASS is essentially CSS with extended feature support. When I say extended feature support here, though, I mean it! SASS supports the following feature set, which is missing in CSS: Variables Nesting Partial CSS files Import support Mixins Extensions and inheritance Operators and calculations Installing and configuring SASS The good news is that getting SASS support working in a Create React App project is incredibly simple. We first need to install it via yarn, or npm. $ yarn add node-sass We'll see a ton of output from it, but assuming there are no errors and everything goes well, we should be able to restart our development server and get started with some SASS. Let's create a more general utility SASS file that will be responsible for storing standardized colors that we'll want to use throughout our application, and something to store that neat gradient hr pattern in case we want to use it elsewhere. We'll also change some of the colors that we're using so that there is some red, green, and blue, depending on whether the item is critical, done, or neither, respectively. In addition, we'll need to change up our project a little bit and add a new file to have a concept of some shared styles and colors. So, let's begin: Create a new file, src/shared.scss, in our project and give it the following body: $todo-critical: #f5a5a5; $todo-normal: #a5a5f5; $todo-complete: #a5f5a5; $fancy-gradient: linear-gradient( to right, rgba(0, 0, 0, 0), rgba(0, 0, 0, 0.8), rgba(0, 0, 0, 0) ); Next, hop over to src/Divider/Divider.css and rename the file to src/Divider/Divider.scss. Next, we'll change the reference to Divider.css in src/Divider/Divider.js, as follows: import "./Divider.scss"; Now we'll need to change up the code in Divider.scss to import in our shared variables file and use a variable as part of it: @import "../shared"; hr { border: 0; height: 1px; background-image: $fancy-gradient; } So, we import in our new shared SASS file in src/, and then the background-image value just references the $fancy-gradient variable that we created, which means we can now recreate that fancy gradient whenever we need to without having to rewrite it over and over. Save and reload, and you should see that nothing major has changed. Mixing SASS and CSS Modules The good news is that it's basically no more complicated to introduce SASS to CSS Modules in Create React App. In fact, the steps are borderline identical! So, if we want to start mixing the two, all we need to do is rename some files and change how our imports are handled. Let's see this in action: First, head back to our src/Todo/Todo.module.css file and make a very minor modification. Specifically, let's rename it src/Todo/Todo.module.scss. Next, we need to change our import statement in src/Todo/Todo.js, otherwise the whole thing will fall apart: import styles from "./Todo.module.scss"; Now, we should have our SASS working for CSS Modules with the Todo component, so let's start taking advantage of it. Again, we'll need to import our shared file into this SASS file as well. Note the following back in src/Todo/Todo.module.scss: @import '../shared'; Next, we'll need to start changing the references to our various background colors. We'll change the background for regular Todos to $todo-normal. Then, we'll change the finished Todo background to $todo-complete. Finally, we'll want to change the critical items to $todo-critical: .todo { border: 2px solid black; text-align: center; background: $todo-normal; color: #333; margin: 20px; padding: 20px; } .done { background: $todo-complete; } .hr { border: 2px solid red; } .critical { composes: todo; background: $todo-critical; } Save and reload our project, and let's make sure the new color scheme is being respected: Now, we have CSS Modules and SASS  integrated nicely in our Create React App project without having to install a single new dependency. We have them playing nicely together even, which is an even greater achievement! If you found this post useful, do check out the book, Create React App 2 Quick Start Guide. In addition to getting familiar with Create React App 2, you will also build modern, React projects with, SASS, and progressive web applications. React Native 0.59 is now out with React Hooks, updated JavaScriptCore, and more! React Native Vs Ionic: Which one is the better mobile app development framework? How to create a native mobile app with React Native [Tutorial]
Read more
  • 0
  • 0
  • 25435

article-image-designing-a-multi-cloud-environment-with-iaas-paas-and-saas-tutorial
Melisha Dsouza
17 Mar 2019
15 min read
Save for later

Designing a Multi-Cloud Environment with IaaS, PaaS, and SaaS [Tutorial] 

Melisha Dsouza
17 Mar 2019
15 min read
In this tutorial, you will understand a scenario that describes how to use solutions from different cloud providers and frameworks. You will learn how to interact with and create a design to fit into the requirements that will be as transparent as possible to the end customer. We will conclude the tutorial by designing a real-world scenario with Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS), in multi-cloud environments (private, public, and hybrid). This tutorial is an excerpt from a book written by Florian Klaffenbach, Markus Klein, Suresh Sundaresan titled Multi-Cloud for Architects. This book is a practical step-by-step guide that will teach you to architect effective Cloud computing solutions and services efficiently. Design guidelines for multi-cloud solutions To design a multi-cloud environment you will need: Network connectivity between the different clouds A single identity management solution for all servers Supported application for georedundancy Containerization As virtual machines are complex and not easy to manage, there is often a requirement to bring in flexibility to custom IT services. This is where containerization comes into play. The concept of containers is defined as follows: a container is an infrastructure independent sandbox running in a container environment, without an operating system in the container. You can easily move containers between environments, and you can scale by adding another container to an existing environment. Items that are usually bundled into a container include the following: Applications Dependencies Libraries Binaries Configuration files Container services have been an approach of different solutions in the past and even came from the community. In the meantime, the following container solutions survived. Cloud Foundry Cloud Foundry was mainly developed by VMware and later by Pivotal. As you can see in the following diagram, the basic design is easy to understand. There are different layers of services that are split to scale and communicate between them: Cloud Foundry provides the ability to run containers independent of the underlying programming language or framework. It provides so-called service brokers that provide a defined connection to existing PaaS solutions from cloud vendors (for example, MySQL from Azure, DB2 from AWS, and so on). It is completely integrated into the CI/CD pipeline of development, and therefore, it has a lot of users from the DevOps parties. Docker Docker is a software solution that works based on containers. A container is defined as an isolated environment containing applications, tools, configuration files, and libraries. All of them run on a single operating system Kernel without guest operating systems, and we know from virtual machines. If you run container in scale, you will need to define an orchestration solution. In today's Docker environments provided by public cloud vendors, you will find Kubernetes as the management, orchestration, and scaling solution shown as follows: As you can see in the preceding diagram, there are different layers that ensure that the Docker Engine can communicate to the rest of the services and provide a defined API, internally and externally. Each of these layers and design parts is responsible for an important part of the product. OpenShift OpenShift is a container solution, with Kubernetes as the orchestrator, that runs on the RedHat operating system. It is owned by IBM: OpenShift is comparable to Docker itself, but it has some modifications that were designed by RedHat. They have been explicitly implemented into the OS itself. Microservices The technical basis for working with containers is a microservice application architecture. This means that each application should be sliced into the smallest possible (but scalable) services. These services are then moved to containers. To scale a specific part of an application, another container is switched on, and a load balancer, sitting before the microservices container, is responsible for integrating the new container into the application life cycle. The concept of microservices is illustrated in the following diagram: As you can see in the preceding diagram, there is an application frontend, API services, and a background database in this sample design. So we have services talking to the user and acting as the frontend. There is a layer in between for communication and translation and we can find a third layer, which is the database service itself. Docker Services as a Service Depending on the cloud vendor, you will find some, or even all, of the following services as PaaS offerings: Container registries for hosting the container images Container hosts/instances for running the container images Container orchestrators (generally based on Kubernetes) to orchestrate the images Regardless of the overall design of your cloud solution, you will be able to integrate these from different cloud vendors. This means that you can spread the same containers to different clouds, hosted by different vendors, and decide where you would like to place your registry, and where your Kubernetes orchestrator should have its home address. Best practices Regarding best practices for your PaaS container design, you should make sure that you find all the required parts for a solution at one or many public cloud vendors. If we set the networking connectivity with good performance and low latency, for example, there will be no need to, place our container hosts in different environments to provide better availability for customers consuming the cloud services. For example, if a customer is consuming his cloud services from Azure, it could connect to your service within the Azure backbone. If it comes from Amazon Web Services, this may be its connectivity target. Bear in mind that redundancy requirements cannot be solved with more complexity. Even here, Terraforms can help to design a descriptive language that is cloud independent. A real-world design scenario Now let's take a look at a real-world scenario that involves designing a multi-cloud architecture for a mid-size European company with locations in the United States, Japan, and Moscow, in addition to their worldwide headquarters in Munich. They are working in the medical and health care area and decided to completely move every IT service to the cloud, except their data center in Moscow, as their IT team is located there. But even for this location, their goal is to minimize the on-premises servers and even work in the data center environment using public cloud technology, as this would give them the chance to move the last on-premise servers to a public cloud, if somehow, in the future, their data center needs to move. As of today, the company is running the following services: Active Directory on Windows Server 2012 R2 with four domain controllers' in the IT headquarters in Moscow. Each location has two domain controllers' on-premises. 480 member servers, running Windows Server 2008 and higher. 280 Ubuntu 17 servers. VMware, as a virtualization layer. Each server is a virtual machine; even their firewalls (checkpoint) are virtual machines. The company's network is MPLS-based, operated by AT&T. They have a central internet breakout in Moscow. There are about 500 SQL server instances running on Windows and Linux. Each of them is already in the most recent release. About 50 servers are running custom installations developed by the company's development team. They are using Visual Studio for managing their code. About 4200 client devices are being used. Each of them is running Windows 7 and Office 2010. For mobile devices, they are using Apple iPhones. The central solution for business services is SAP, which is currently hosted in their local data center in Germany, and in Moscow as a redundant hot standby environment. They are currently managing their environment by using system center 2012 R2 tools. Their ITSM solution is already running as a SaaS solution in the ServiceNow cloud. This is the only solution that will survive the redesign and even in 2021 and later will run in the ServiceNow cloud. The CEO has decided to have a cloud-first strategy, and all of the cloud migration must be done by the end of 2021, as all the existing data center contracts would be cancelled by then. In addition, they have already decided to implement a two cloud vendor strategy using Microsoft Azure and Amazon Web Services. AWS should mainly be used by their development team, as the chief of development is a former employee of AWS. The head of IT has decided to move all of the infrastructure services mainly to Microsoft Azure. Suppose that you are the responsible, external consultant, helping to design the new IT environment. Solution design This section will describe the project and the final design of the new company's IT environment, based on Azure and AWS technology. Preparations One of the most important steps, when starting to use cloud services is to define cloud governance. Regarding which cloud vendor you decide to use, basically, they are all the same. With Amazon Web Services, the Cloud Adoption Framework looks as follows: With AWS, as the customer, have to work through each of the points, in order to be happy with your cloud governance. With Microsoft Azure Services, there is the Azure Governance Scaffold, as follows: These are the main points that you will need to define your governance and work with Azure properly. Looking at Azure in more detail, we will need to decide on a concept for the following components: As you can see in the preceding diagram, there are different portals on the business side (the Enterprise Portal and the Account Portal), and then a third one to manage the technologies themselves (the Management Portal). If you would like to work with code (JSON), the APIS, and CLI, Visual Studio will be your ideal solution to work with. We will need to merge both of the cloud vendors. For this example, the governance has already been set and we can have a look at the technology itself. Networking Looking at the networking portion of the multi-cloud design, the company decided to work with a partner that supports multi-cloud connections. This means that they, themselves, do not have to manage connectivity. This is being done via remote peering with their networking partner. Our company decided to go with Equinix and Interxion. The following diagram shows the Equinix Cloud Exchange Framework: As you can see in the preceding diagram, the customer has connectivity to Equinix and Equinix will provide the customer with connectivity to the cloud vendors of your choice. Let's take a look at Interxion: Interxion works the same way that Equinix does, and it is another vendor to help you solve your multi-cloud networking configurations. The result will be redundancy and independency and even a connection to the local data center in Moscow without any issues to both cloud vendors. Identity management The company has decided to have a single identity management solution based on the technology, they already run on both public clouds, which is Azure Active Directory: As you can see in the preceding diagram, each cloud service (either public or hybrid, using Azure Stack or a similar service) is using Azure AD as a single IDM solution. Based on their security policies, the company has decided to go with Azure AD Connect, using pass through authentication (PTA): The PTA agent is monitoring the IDM queues in the cloud and authenticating the requests locallys transferring back the authentication token. As Azure AD works with AWS, too, there is single identity management solution in place, as follows: For their on-premises cloud environment, the company has decided to go with Azure Stack in a connected mode, in order to leverage Azure AD, too. The design is illustrated in the following diagram: As you can see in the preceding diagram, Azure Stack and Azure behave the same way technologically, and can therefore be integrated into the express route configuration as another Azure cloud. Modern workplace services With the basic cloud environment in place, the company has decided to go with Office 365 for all workplace services, on all client devices. They will be migrated to Office applications on the client computers, using Office servers as SaaS directly from the cloud. This will be a seamless migration for the user accounts, and everything will work as usual, even when the migration is taking place: As the Office 365 license can also be used on iPhones, all of the employees will be fine. Regarding the company's on-premises exchange server, Skype, and SharePoint, they will move these to Office 365 completely and will get rid of the virtual machines that are present today. Infrastructure services Regarding the existing infrastructure services, you have decided to move most of them to Microsoft Azure and to prepare the migration by first identifying which solution can exist as a PaaS, and what needs to reside on a VM in IaaS. To automatically collect all of the required information, you decide to perform an analysis using Azure Migrate, as follows: The vCenter Service will be connected to Azure and will host the migration service itself. I will be responsible for synchronizing, and later managing, the switch of each VM from on-premises to the cloud. For all of the SQL services, there is a solution called Azure SQL Migrate, as follows: As a result of these cloud services, you will be able to figure out if there are still any virtual machines running SQL. In general, about 80-90% of the SQL servers in the existing design can be moved to PaaS solutions. Using the results of the migration service, you can get an idea of what the cloud SQL solution will look like. It will also help you to work through each migrating step in an informed way. For the SAP environment that is currently running on-premises, you decide to migrate to SAP HEC on Azure, based on the existing blueprint design of Microsoft, as follows: About 68% of the VMs can be migrated to Azure seamlessly, without requiring running them as virtual machines anymore. Another 25% of the VMs need to be migrated to Azure using the lift and shift model. The service that you will need to migrate them to Azure is Azure Site Recovery. This service works as follows: For all of the VMs that need to run in the on-premises data centers that will be left after the move to the cloud, you decide to use Azure Stack. As Azure Stack is the on-premises solution of Azure, the process to migrate the VM is nearly the same. As the company's CEO has decided that a temporary placement of a virtual machine in a public Azure cloud for migration tasks is fine, you three-step migration: Migrate the VM from the on-premises VMware to Azure, using Azure Site Recovery Move the VM from Azure to Azure Stack, using Azure Storage Explorer Bring the VM online again, on Azure Stack From a sizing perspective, you decide to use an eight-node Azure Stack environment from the company's favorite hardware vendor. The sizing has been created using the Azure Stack Capacity Planner. As you can see in the following screenshot, it is an Excel sheet with input and output parameters: Setting up new cloud services For each new cloud service that will be deployed, the CIO has decided to go with Terraforms, in order to have a unique description language for all of the resources, regardless of the cloud flavor of a specific vendor. Terraforms provides an easy way to automate the deployment, and to be flexible when moving resources, even between the clouds. Development environment As the CDO is a former employee of Amazon Web Services, and as all of the existing development code is in AWS, there is no need for him to change this: As Jenkins is supported in Azure, too, the development is flexible. The main task is to design the build pipeline using stage environments. If DevOps decides to implement virtual machines for their services, these may also reside on AWS, but due to the underlying single identity and networking design, this really does not matter at all. The only requirement from the CIO is that if the VM is a Windows server and not Linux, it must be placed on Azure, as in Azure, there is an option to save license costs by using Azure Hybrid Benefits. As you can see in the preceding diagram, there are 41% savings using the Hybrid Benefits and reusing the Windows server licenses in the cloud. So, the plan is to demote a server on-premises and enable it in Azure. With this switch of each VM, you will be able to transfer the license itself. Summary In this tutorial, we learned how to use solutions from different cloud providers and frameworks and create a design to fit into the requirements that will be as transparent as possible to the end customer. If you are looking at completely adapting to any Cloud and its services, Multi-Cloud for Architects will be your go-to guide to find perfect solutions irrespective the size of your infrastructure.  Microsoft Cloud services’ DNS outage results in deleting several Microsoft Azure database records VMware Essential PKS: Use upstream Kubernetes to build a flexible, cost-effective cloud-native platform MariaDB CEO says big proprietary cloud vendors “strip-mining open-source technologies and companies”
Read more
  • 0
  • 0
  • 18728
Modal Close icon
Modal Close icon